Happy to collaborate but also given how busy we all are I think it's lightweight enough that anyone of us can implement it in stages- first let's accomplish 1) getting romans dockerfiler images into bigtop and building nightly on the build server.
On Mar 25, 2015 10:42 PM, "Evans Ye" <[email protected]> wrote: > > Agree Jay's approach. At this moment I think we should first conquer "at > least work" problem with the mindset that we'll eventually make it "fully > automated". This can make the release not being blocked by the CI > infrastructure progress. > I'd like to help setting up "jenkins job 3)" or co-work with Jay if we > both interested in it:) > > > > 2015-03-26 8:34 GMT+08:00 jay vyas <[email protected]>: > > > to kick start this, one thing we can do is a minimal viable product, which > > uses romans dockerfiles and has potentially to evolve into romans > > suggestion.... > > > > im 4 jobs, all on one single build server, you can pretty much do 80% of > > the main things we want in bigtop. > > > > jenkins job 1) build docker images nightly based on roman's Dockerfile that > > builds the toolchain > > > > jenkins job 2) dump a shell script into jenkins which runs a job that > > builds bigtop packages night for the main components (hadoop, spark, pig, > > hbase, zookeeper) > > > > jenkins job 3) use vagrant to deploy those in docker containers, and runs > > our existing smoke-tests + bigpetstore-spark (we already have that > > automated in deploy/vagrant-docker/ > > > > jenkins job 4) If (3) passes, copy bigtop output/ do a public dir people > > can add as a yum repo ? > > > > That is a minimal, but really easy and simple and fast way to implement a > > CI. > > And even better anyone can run the exact build from their laptop. > > > > Cos' gradle wrappers combined with our vagrant-docker recipes make this a > > pretty painless task.... > > > > > > On Wed, Mar 25, 2015 at 3:17 PM, Roman Shaposhnik <[email protected]> > > wrote: > > > > > First of all, thanks a million to all of you guys for pitching in! > > > I so wish I can close these loops myself, but I'm completely > > > out of cycles at least till the end of Apache CON. With that > > > disclaimer, here's my braindump on the subject. Note that > > > this is not a plan of action, but rather a list of things that > > > need to be done. Perhaps there's a subset of things in > > > there somewhere that would get us a functional CI without > > > doing all the work that I'm suggesting. This would be up > > > to you guys to figure out: > > > > > > 0. Whoever would be helping with the project would need to > > > get the creds for the AWS account that we're using. I don't think > > > we ever figured out a reliable way to share those credentials, > > > but a few of us have them. Please contact Cos and myself > > > offline. > > > > > > 1. The big idea behind the new CI pipeline was to use Docker > > > containers as the environments running on generic OSes. So > > > far I have been experimenting with CoreOS hoping that its > > > focus on Docker would make it a more reliable environment > > > to run these containers on. Quite the contrary, it seems that the > > > CoreOS slave is down way more than it is up: > > > http://bigtop01.cloudera.org:8080/computer/docker/ > > > We need to figure out a reliable way of spinning these slaves. One > > > option would be to use a Jenkins EC2 plugin (which we have deployed > > > on our Jenkins) to spin new slaves when they are needed. Or we can > > > keep the static slaves around. Either way one thing that needs to be > > > figured out is this: Docker is pretty disk hungry and most of the > > > default > > > AMIs come with a tiny disk images. > > > > > > At any rate, once we have a way to have reliable slaves on which > > > we can run Docker containers we can proceed. > > > > > > 2. Another big idea was to move all 'state' from Jenkins configuration > > > files into a Jenkins DSL that would be checked into our repo. The > > > prototype > > > exist here: > > > bigtop-ci/jenkins/jobsCreator.groovy > > > and hooked up to this job: > > > http://bigtop01.cloudera.org:8080/view/Gradle/job/Gradle-seed/ > > > If click on that link you will see the list of Jobs that was > > generated > > > via > > > running the seed job. This is how it is expected to work -- but there > > > are > > > a few bugs still. Somebody needs to make sure that all the generated > > > jobs are actually bug free. The easiest way is to compare and > > contrast > > > the differences between generated jobs and the static ones we have > > over > > > here: http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/ > > > > > > Once the seed job can generate the jobs that actually produce the > > > right result > > > utilizing Docker builds on the fixed Docker slave from step #1 we can > > > take > > > care of one last detail > > > > > > 3. Currently the Docker containers we're using came from me manually > > > building these containers and pushing them to: > > > https://registry.hub.docker.com/u/bigtop/slaves/ > > > We need to automate two aspects of this: > > > 3.1. we need to have a job on Jenkins that would at least build > > > these > > > containers based on the latest state of our puppet code in > > > bigtop_toolchain > > > > > > 3.2. we need to figure out a way to publish the containers > > > without endangering > > > Bigtop's Docker HUB account creds. > > > > > > Ideally these containers need to be regenrated every time there's a > > > change > > > in bigtop_toolchain puppet code. The prototype job that I was > > > playing with for > > > this purpose is over here: > > > > > > http://bigtop01.cloudera.org:8080/view/Docker/job/Docker-Toolchain/ > > > > > > > > > Hope this helps. I'd be more than happy to answer questions and review > > > JIRAs. > > > > > > Thanks, > > > Roman. > > > > > > > > > On Tue, Mar 24, 2015 at 2:50 PM, Konstantin Boudnik <[email protected]> > > > wrote: > > > > Guys, > > > > > > > > thanks a lot for your commitment - I have chatted with Nate offline and > > > he > > > > might be able to help a bit as well. Let's wait for Roman's input - I > > > will > > > > ping him tonight if we don't hear from him by then, so we have a better > > > > picture of the 1st question. > > > > > > > > Cos > > > > > > > > On Tue, Mar 24, 2015 at 06:06AM, Konstantin Boudnik wrote: > > > >> Guys, > > > >> > > > >> I want to start a separate thread to track the CI preparations for the > > > release > > > >> next month (fingers crossed). Clearly, we can make a release without > > > CI, but > > > >> it'd way easier to test and create binary artifacts if we have a > > working > > > >> environment for official validation. Roman has done a lot in this > > > direction > > > >> (many thanks!), but there are still a few rough edges, which might be > > > easy to > > > >> finish of. > > > >> > > > >> I want to figure out a couple of things: > > > >> - what's the state of CI and how much still needs to be done (Rvs? > > > Could you > > > >> share any first hand feedback?) > > > >> - who would be able to help with the CI completion? I can commit some > > > of my > > > >> cycles, but it'd be great to have few more hands on that. Clearly, > > > some > > > >> Jenkins-foo and prior CI skills won't hurt ;) > > > >> > > > >> Please chime in if you can help. Thanks a lot! > > > >> Cos > > > >> > > > > > > > > > > > -- > > jay vyas > >
