[
https://issues.apache.org/jira/browse/BIGTOP-1323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14055731#comment-14055731
]
Roman Shaposhnik commented on BIGTOP-1323:
------------------------------------------
[~jeid] would love to join forces with you on this one. So far, I've been
blocked by BIGTOP-1359 but now that's been resolved and committed here's the
pipeline I'd like to suggest to you (which seems remarkably similar to what
you've been doing).
# Bigtop shall maintain all of its docker containers (regardless of whether
they are used for build or not) under bigtop account on Docker index:
https://hub.docker.com/u/bigtop/ That will be Apache Bigtop Docker Registry in
Docker's speak.
# Under that common registry we will maintain a bunch of repositories one per
function, with the following functions being currently proposed:
#* seed
#* puppet
#* build
# Within each repository different tags will correspond to different OS
environments that we have to support (e.g. bigtop/build:centos-6.4 will be the
full name of a build environment for centos-6.4 OS)
# tags will be codified to designate an OS and the current proposed set of tags
is:
#* centos-6.4
#* fedora-20
#* opensuse-12.3
#* ubuntu-12.04
#* ubuntu-14.04
Every tag must be presented in every repo which will give us a unified matrix
of docker containers bigtop/<function>:<os>
That's the nomenclature, now for the semantics.
# seed repository contains bare images of OSes that we use for everything else.
I really wish we can just re-use officially maintained images in all the cases
(and Ubuntu is really good about maintaining their stuff on Docker), but
opensuse for example comes from somebody else's repo today. Great for testing,
but I wouldn't trust it for official builds. This is our first issue to figure
out (see the list of issues bellow). Regardless of that, though, the
functionality that it enables is that if you just want a clean OS environment
for quickly testing *anything* you can just run: $ docker run -t -i
bigtop/seed:<os name> bash and get and instant access to it. That alone already
is saving me tons of headache.
# puppet repository is seed plus puppet installed. This is produced based by
the "meta Docker file" I'm attaching to this JIRA. The way to
use is this: $ ./Docker.sh <os tag> | docker build -
# build repository will be produced by applying our puppet toolchain code to
the containers from puppet repo. Once we have that, the official recommended
way to build Bigtop packages will be:
{noformat}
$ docker run -v `pwd`:/ws bigtop/slave:centos-6.4 bash -c '.
/etc/profile.d/bigtop.sh; cd /ws ; make bigtop-jsvc-rpm'
{noformat}
this is regardless of whether you run it on your laptop or our Jenkins runs it
on EC2.
Thus the whole progression of our containers look like this: seed -> puppet ->
build. This progression will be managed by our Jenkins build job but we still
have to figure out a few things to fully automate it.
I'm currently stuck at the end of the process of populating puppet repo (I've
got it populated but I don't quite like the result). And also actually
producing build repository. Any help with this (or anything else on the list
bellow) will be greatly appreciated.
List of issues to figure out (perhaps we need to create subjiras from this
list):
# how do we import new OSes into the seed repo? What's the process there?
# how do we make a jenkins job safely publish to our registry on
hub.docker.com? I really don't want to leave credentials in plain sight on
Jenkins
# the way I currently generate build containers is this:
{noformat}
$ docker run -v `pwd`:/etc/puppet/modules:ro bigtop/puppet:ubuntu-$i.04 puppet
apply --debug -e "include bigtop_toolchain::installer"
{noformat}
what's the best way to automate this? (NOTE: Docker files don't support
external folders -v)?
# currently the seed containers are small (all ~100MBs or so), the puppet
containers are larger (~250MB) but the slave containers are huge (2GB). This is
unacceptable and we need to figure out where the bloat is coming from.
> provide an option to containerize Bigtop package build
> -------------------------------------------------------
>
> Key: BIGTOP-1323
> URL: https://issues.apache.org/jira/browse/BIGTOP-1323
> Project: Bigtop
> Issue Type: Improvement
> Components: Build
> Affects Versions: 0.8.0
> Reporter: Roman Shaposhnik
> Assignee: Roman Shaposhnik
> Fix For: 0.8.0
>
>
> Looking at the state of our Jenkins made me scream "I'm mad as Hell and I'm
> not going to take this anymore". The basic problem is simple: our build
> slaves are not actually managed by Puppet (even though we do have Puppet code
> to do that) and they seem to bitrot quite a bit (e.g. git clone from ASF
> repos on SLES and CentOS5 is totally borked).
> There's another, somewhat subtler, problem: currently we have to keep as many
> slaves as we've got platforms to support. This leads to a pretty poor
> utilization and if a slave goes down requires manual intervention.
> What I'd like to introduce in this JIRA is a notion of decoupling OS
> environment dependency from the base OS. The technology I'm proposing here is
> Docker (which really is just a prettier UI on top of Linux Containers). IOW,
> regardless of what OS you'd be running on your host -- as long as you've got
> docker you can have an exact replica of a target os running in a container.
> Here's my plan for simplifying the management aspect of what's going on.
> Note, that this is a complimentary plan. Whatever is happening (or not
> happening!) today on our Jenkins can remain there for as long as we need it.
> # make our build slaves be 100% uniform and fungible instances of CoreOS
> VMs https://coreos.com/ The benefit of CoreOS is pretty much 0 administration
> cost, quick boot up time, full integration with Docker (see bellow) and full
> integration with things like VirtualBox, etc (so that you can replicate
> exactly the kind of process that we will setup on the Jenkins).
> # make the actual builds happen in a container using CoreOS's integration
> with Docker. IOW, instead of running: make foo on the slave the actual
> command will look like: docker run -i -t -v `pwd`:/workspace:rw BUILD-ENV
> bash -c 'cd /workspace ; make foo'
> # add jobs to the jenkins that would maintain images of BUILD-ENV
> containers. Each image will be a base-line OS (RHEL, Ubuntu, etc) plus all
> the build dependencies of a given source package (but nothing else!).
> All of these 3 steps are pretty easy to automate and they do have an added
> benefit of being 100% replicatable on your workstation. IOW, as long as you
> have docker running on your host OS you can build packages for any OS with a
> simple command.
> Thoughts?
--
This message was sent by Atlassian JIRA
(v6.2#6252)
