First of all, thanks a million to all of you guys for pitching in!
I so wish I can close these loops myself, but I'm completely
out of cycles at least till the end of Apache CON. With that
disclaimer, here's my braindump on the subject. Note that
this is not a plan of action, but rather a list of things that
need to be done. Perhaps there's a subset of things in
there somewhere that would get us a functional CI without
doing all the work that I'm suggesting. This would be up
to you guys to figure out:
0. Whoever would be helping with the project would need to
get the creds for the AWS account that we're using. I don't think
we ever figured out a reliable way to share those credentials,
but a few of us have them. Please contact Cos and myself
offline.
1. The big idea behind the new CI pipeline was to use Docker
containers as the environments running on generic OSes. So
far I have been experimenting with CoreOS hoping that its
focus on Docker would make it a more reliable environment
to run these containers on. Quite the contrary, it seems that the
CoreOS slave is down way more than it is up:
http://bigtop01.cloudera.org:8080/computer/docker/
We need to figure out a reliable way of spinning these slaves. One
option would be to use a Jenkins EC2 plugin (which we have deployed
on our Jenkins) to spin new slaves when they are needed. Or we can
keep the static slaves around. Either way one thing that needs to be
figured out is this: Docker is pretty disk hungry and most of the default
AMIs come with a tiny disk images.
At any rate, once we have a way to have reliable slaves on which
we can run Docker containers we can proceed.
2. Another big idea was to move all 'state' from Jenkins configuration
files into a Jenkins DSL that would be checked into our repo. The prototype
exist here:
bigtop-ci/jenkins/jobsCreator.groovy
and hooked up to this job:
http://bigtop01.cloudera.org:8080/view/Gradle/job/Gradle-seed/
If click on that link you will see the list of Jobs that was generated via
running the seed job. This is how it is expected to work -- but there are
a few bugs still. Somebody needs to make sure that all the generated
jobs are actually bug free. The easiest way is to compare and contrast
the differences between generated jobs and the static ones we have over
here: http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/
Once the seed job can generate the jobs that actually produce the
right result
utilizing Docker builds on the fixed Docker slave from step #1 we can take
care of one last detail
3. Currently the Docker containers we're using came from me manually
building these containers and pushing them to:
https://registry.hub.docker.com/u/bigtop/slaves/
We need to automate two aspects of this:
3.1. we need to have a job on Jenkins that would at least build these
containers based on the latest state of our puppet code in
bigtop_toolchain
3.2. we need to figure out a way to publish the containers
without endangering
Bigtop's Docker HUB account creds.
Ideally these containers need to be regenrated every time there's a change
in bigtop_toolchain puppet code. The prototype job that I was
playing with for
this purpose is over here:
http://bigtop01.cloudera.org:8080/view/Docker/job/Docker-Toolchain/
Hope this helps. I'd be more than happy to answer questions and review JIRAs.
Thanks,
Roman.
On Tue, Mar 24, 2015 at 2:50 PM, Konstantin Boudnik <[email protected]> wrote:
> Guys,
>
> thanks a lot for your commitment - I have chatted with Nate offline and he
> might be able to help a bit as well. Let's wait for Roman's input - I will
> ping him tonight if we don't hear from him by then, so we have a better
> picture of the 1st question.
>
> Cos
>
> On Tue, Mar 24, 2015 at 06:06AM, Konstantin Boudnik wrote:
>> Guys,
>>
>> I want to start a separate thread to track the CI preparations for the
>> release
>> next month (fingers crossed). Clearly, we can make a release without CI, but
>> it'd way easier to test and create binary artifacts if we have a working
>> environment for official validation. Roman has done a lot in this direction
>> (many thanks!), but there are still a few rough edges, which might be easy to
>> finish of.
>>
>> I want to figure out a couple of things:
>> - what's the state of CI and how much still needs to be done (Rvs? Could you
>> share any first hand feedback?)
>> - who would be able to help with the CI completion? I can commit some of my
>> cycles, but it'd be great to have few more hands on that. Clearly, some
>> Jenkins-foo and prior CI skills won't hurt ;)
>>
>> Please chime in if you can help. Thanks a lot!
>> Cos
>>
