Hi all. Just a quick update about the status of this work: all projects now build within docker containers. It has been a bit painful to set up but it's all good now.
Also, the concept I mentioned in my previous email works like a charm. However, I discovered yesterday a limitation: Brooklyn GitHub repos are owned by Apache, therefore we don't have the permissions to add webhooks. This means that the feature to re-trigger a build from a PR comment by writing "retest this please" won't work. On top of that, it seems that even the DSL command to set this up is missing from the Apache Jenkins installation. Knowing this, are we happy to move anyway, or is it a deal breaker? WDYT? Best. On Tue, 1 May 2018 at 14:33 Thomas Bouron <[email protected]> wrote: > Hi all. > > Continuing my work on this topic, I just pushed a PR [1] in > brooklyn-library that adds a Jenkinsfile. If you are not familiar with what > it is, a Jenkinfile describes a Jenkins job by using the pipeline DSL. That > allows you to capture the configuration / best practices under version > control so it's easier to iterate over and is repeatable. This particular > Jenkinsfile implements the current configuration we have + a fix for > permission issues we recently had (see INFRA ticket [2]) > > To make this work, we still need to manually create the multibranch > pipeline job, and pointing it toward the right GitHub repo. I'll do that > once the PR is merge. But the bonus point of this approach is that it > automatically handles PR builds, with very nice views so that is 2 birds > from one stone. > > Why not using a seed instead that would avoid doing any kind of manual > config? That is a good question and the answer is: we are running on the > apache infra, and I don't think we have the power of using Jenkins seed. > Now if I'm wrong, it's fine as we can just add the seed on top of the > pipelines. > > Anyway, I tested my Jenkinsfile locally with brooklyn-library so it should > work nicely. I was planning to test with this project first, validate the > concept, then do the same for the remaining git submodules. All I need now, > is a review on [1] + merge :) > > Best. > > [1] https://github.com/apache/brooklyn-library/pull/153 > [2] > https://issues.apache.org/jira/browse/INFRA-16417?page=com.atlassian.jira.plugin.system.issuetabpanels%3Aall-tabpanel > > On Wed, 15 Nov 2017 at 16:42 Thomas Bouron < > [email protected]> wrote: > >> Looks like we start to reach a consensus :) >> >> I duplicated all our jobs and configured them to use docker instead: >> - brooklyn-server => >> https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-server-master-docker/ >> - brooklyn-library => >> https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-library-master-docker/ >> - brooklyn-client => >> https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-client-master-docker/ >> - brooklyn-ui => >> https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-ui-master-docker/ >> - brooklyn-dist => >> https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-dist-master-docker/ >> - brooklyn => >> https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-master-build-docker/ >> >> I only "disabled" the other jobs by removing all triggers, just in case >> what I did breaks everything then we can go back. >> >> While I was on the jenkins, I noticed some `incubator-brooklyn-*` jobs, I >> guess those can be deleted isn't it? Also, anyone know the purpose of >> brooklyn-master-windows[1] ? >> >> --- >> In terms of next steps, I'll monitor to see if the new jobs behave as >> expected. If yes, I'll then move on to: >> - migrate the PR's jobs to use docker >> - add the `Dockerfile` within each git submodule >> - leverage the Jenkins' pipeline feature by adding the config directly in >> each git submodule >> - create new job to build and publish docs/website automatically >> >> Best >> >> [1] >> https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-master-windows/ >> >> On Wed, 15 Nov 2017 at 11:18 John McCabe <[email protected]> wrote: >> >>> Huge +1 on consistency/portability. >>> >>> On Tue, 14 Nov 2017 at 16:40 Mark McKenna <[email protected]> >>> wrote: >>> >>> > +1 I echo what Richard says about consistency ... also makes debugging >>> > build failures easier >>> > >>> > >>> > >>> > On 14 November 2017 at 15:48, Richard Downer <[email protected]> >>> wrote: >>> > >>> > > +1 >>> > > >>> > > Having seen some of the conversations on the [email protected] >>> list, it >>> > > does seem like this is the way to go as far as getting the right >>> build >>> > > environment on the Apache Jenkins cluster. This is the best solution >>> for >>> > > getting the `rpm` and `deb` built regularly. >>> > > >>> > > Build consistency is also a great driver. I can see this being >>> useful for >>> > > the release process too, as a release build would have an identical >>> > > environment to that used to build the snapshots. >>> > > >>> > > Richard. >>> > > >>> > > >>> > > On 14 November 2017 at 15:43, Thomas Bouron >>> > <[email protected] >>> > > om >>> > > > wrote: >>> > > >>> > > > Hi Brooklyners! >>> > > > >>> > > > Few months ago, I pushed a PR[1] to simplify `brooklyn-vagrant` by >>> > using >>> > > > the RPM package. This is still on hold because we don't build RPM >>> > package >>> > > > of SNAPSHOT versions and therefore, our vagrant install script >>> won't >>> > work >>> > > > for bleeding edge versions. >>> > > > For the record, RPM are not built on SNAPSHOT because INFRA does >>> not >>> > > offer >>> > > > centos/RHEL machines as Jenkins slaves. >>> > > > >>> > > > But that is coming to an end. INFRA did install (recently-ish) >>> docker >>> > on >>> > > > all Jenkins slaves which means that a new world is opened to us! >>> In the >>> > > > past weeks, I have created a clone[2] of the >>> `brooklyn-dist-master`[3] >>> > > job >>> > > > and configured it to use a custom docker image I created[4]. It is >>> > based >>> > > on >>> > > > `maven:alpine` and adds `rpm` and `dpkg` binaries (for the `dist` >>> tag). >>> > > > As my test was successful, I decided to go a little further by >>> > creating 2 >>> > > > other tags for this image: >>> > > > - `client, based on `maven:alpine` with the `go` binary >>> > > > - `latest` which is the combination of the last 2. >>> > > > >>> > > > My plan is to migrate all jenkins jobs to use docker. I'll donate >>> the >>> > > > `Dockerfile` to each git repo so that jenkins will be able to >>> build and >>> > > > publish them on docker hub instead of using my account/image. Once >>> we >>> > > have >>> > > > everything dockerised, we can look at improving current builds, >>> like >>> > > using >>> > > > jenkins pipelines rather than relying on upstream<->downstream >>> > projects' >>> > > > relationships. >>> > > > >>> > > > It also makes the integration/live tests rise from the ashes. This >>> is >>> > the >>> > > > perfect opportunity to fix and make then part of our CI. >>> > > > >>> > > > WDYT? Even if it's obvious, the main advantage here is the >>> portability >>> > of >>> > > > the build environment. It also means that we can use any jenkins >>> > slaves, >>> > > > potentially saving us a lot of time when waiting for a build >>> executor. >>> > > > >>> > > > Best. >>> > > > >>> > > > [1] https://github.com/apache/brooklyn-dist/pull/105 >>> > > > [2] >>> > > > https://builds.apache.org/view/B/view/Brooklyn/job/ >>> > > > brooklyn-dist-master-docker/ >>> > > > [3] https://builds.apache.org/view/B/view/Brooklyn/job/ >>> > > > brooklyn-dist-master/ >>> > > > [4] https://hub.docker.com/r/tbouron/brooklyn-build/ >>> > > > -- >>> > > > >>> > > > Thomas Bouron • Senior Software Engineer @ Cloudsoft Corporation • >>> > > > https://cloudsoft.io/ >>> > > > Github: https://github.com/tbouron >>> > > > Twitter: https://twitter.com/eltibouron >>> > > > >>> > > >>> > >>> >> -- >> >> Thomas Bouron • Senior Software Engineer @ Cloudsoft Corporation • >> https://cloudsoft.io/ >> Github: https://github.com/tbouron >> Twitter: https://twitter.com/eltibouron >> > -- > > Thomas Bouron • Senior Software Engineer @ Cloudsoft Corporation • > https://cloudsoft.io/ > Github: https://github.com/tbouron > Twitter: https://twitter.com/eltibouron > -- Thomas Bouron Senior Software Engineer *Cloudsoft <https://cloudsoft.io/> *| Bringing Business to the Cloud GitHub: https://github.com/tbouron Twitter: https://twitter.com/eltibouron Need a hand with AWS? Get a Free Consultation.
