Julien Eid created BIGTOP-1368:
----------------------------------
Summary: Have Jenkins use Mesos-Jenkins plugin to create
containerized build slaves
Key: BIGTOP-1368
URL: https://issues.apache.org/jira/browse/BIGTOP-1368
Project: Bigtop
Issue Type: Improvement
Components: Build
Reporter: Julien Eid
This ticket is about changing our Jenkins setup to do builds inside of
containers so that we can auto-configure slaves and cleanup our environment as
well as match containers that developers can use to build Bigtop.
I recommend using the Jenkins Mesos plugin
(https://github.com/jenkinsci/mesos-plugin) to start Jenkins slaves,
orchestrate and run jobs in containers, and handle resource allocation to
different slave containers that start up. Apache Mesos handles the launching of
containers and limiting containers resources so that slaves won’t starve each
other out for resources while the Jenkins Mesos plugin sits on top of Mesos and
starts the slave.jar inside of the container which will then run the Jenkins
building the BigTop package you want or doing whatever tests you need in the
isolated environment. It would also not require much changing of our current
Jenkins setup to get it integrated and build steps for most projects would be
unchanged. You would just install mesos-master somewhere and then install
mesos-slave on all of the servers that currently do job builds, and Mesos will
handle using them all and spinning up the Jenkins slaves.
While the above allows you to do BigTop builds in containers using Jenkins, we
should also do some more things to make life easier. We can use the Jenkins
Mesos plugin to run Docker in Docker (dind) so that inside of our Jenkins slave
containers we can launch a Docker container inside of that to build Docker
images for deployment. For example, you could have a Jenkins job that is kicked
off inside of a container that then starts and builds a base Docker image that
will later build a specific package on another job. This would allow our
Jenkins setup to build and test our Puppet toolchain code and then produce
images that would be used by other jobs for builds. This is documented and
talked a lot about here:
http://www.ebaytechblog.com/2014/04/04/delivering-ebays-ci-solution-with-apache-mesos-part-i/#.U7R4t3VdVhF
As for matching builds on peoples computers, that gets pretty easy with Docker.
You can just take the Jenkins Slave docker image and then run jobs against it
without starting the slave.jar. So instead of docker run supervisord which you
would do to get the Jenkins slave running, you can do docker run make hive-deb
or whatever package you wanted. This would make peoples environments match our
Jenkins environment exactly.
I have an experimental git repo with some Dockerfiles that are a pretty good
prototype of this future setup located at
https://github.com/jeid64/bigtop-dockerfiles jenkins-docker/ has an example
Ubuntu 12.04 Dockerfile that once built uses the Puppet manifests in
bigtop_toolchain to setup the build environment. When run by Jenkins, Jenkins
will start a slave inside of the container and the slave would handle all the
build steps for that build. That image can also just be used on your desktop
without needing a Jenkins master or using anything from Jenkins.
Also in that repo is jenkins-dind/ which is based off of
https://github.com/ahunnargikar/jenkins-dind multi-docker setup. This will be
the image for dind image builds in Jenkins for build environments.
I would love to head up this ticket and work on getting everything setup.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
