Hi all. Continuing my work on this topic, I just pushed a PR [1] in brooklyn-library that adds a Jenkinsfile. If you are not familiar with what it is, a Jenkinfile describes a Jenkins job by using the pipeline DSL. That allows you to capture the configuration / best practices under version control so it's easier to iterate over and is repeatable. This particular Jenkinsfile implements the current configuration we have + a fix for permission issues we recently had (see INFRA ticket [2])
To make this work, we still need to manually create the multibranch pipeline job, and pointing it toward the right GitHub repo. I'll do that once the PR is merge. But the bonus point of this approach is that it automatically handles PR builds, with very nice views so that is 2 birds from one stone. Why not using a seed instead that would avoid doing any kind of manual config? That is a good question and the answer is: we are running on the apache infra, and I don't think we have the power of using Jenkins seed. Now if I'm wrong, it's fine as we can just add the seed on top of the pipelines. Anyway, I tested my Jenkinsfile locally with brooklyn-library so it should work nicely. I was planning to test with this project first, validate the concept, then do the same for the remaining git submodules. All I need now, is a review on [1] + merge :) Best. [1] https://github.com/apache/brooklyn-library/pull/153 [2] https://issues.apache.org/jira/browse/INFRA-16417?page=com.atlassian.jira.plugin.system.issuetabpanels%3Aall-tabpanel On Wed, 15 Nov 2017 at 16:42 Thomas Bouron <thomas.bou...@cloudsoftcorp.com> wrote: > Looks like we start to reach a consensus :) > > I duplicated all our jobs and configured them to use docker instead: > - brooklyn-server => > https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-server-master-docker/ > - brooklyn-library => > https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-library-master-docker/ > - brooklyn-client => > https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-client-master-docker/ > - brooklyn-ui => > https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-ui-master-docker/ > - brooklyn-dist => > https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-dist-master-docker/ > - brooklyn => > https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-master-build-docker/ > > I only "disabled" the other jobs by removing all triggers, just in case > what I did breaks everything then we can go back. > > While I was on the jenkins, I noticed some `incubator-brooklyn-*` jobs, I > guess those can be deleted isn't it? Also, anyone know the purpose of > brooklyn-master-windows[1] ? > > --- > In terms of next steps, I'll monitor to see if the new jobs behave as > expected. If yes, I'll then move on to: > - migrate the PR's jobs to use docker > - add the `Dockerfile` within each git submodule > - leverage the Jenkins' pipeline feature by adding the config directly in > each git submodule > - create new job to build and publish docs/website automatically > > Best > > [1] > https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-master-windows/ > > On Wed, 15 Nov 2017 at 11:18 John McCabe <j...@johnmccabe.net> wrote: > >> Huge +1 on consistency/portability. >> >> On Tue, 14 Nov 2017 at 16:40 Mark McKenna <m4rkmcke...@apache.org> wrote: >> >> > +1 I echo what Richard says about consistency ... also makes debugging >> > build failures easier >> > >> > >> > >> > On 14 November 2017 at 15:48, Richard Downer <rich...@apache.org> >> wrote: >> > >> > > +1 >> > > >> > > Having seen some of the conversations on the bui...@apache.org list, >> it >> > > does seem like this is the way to go as far as getting the right build >> > > environment on the Apache Jenkins cluster. This is the best solution >> for >> > > getting the `rpm` and `deb` built regularly. >> > > >> > > Build consistency is also a great driver. I can see this being useful >> for >> > > the release process too, as a release build would have an identical >> > > environment to that used to build the snapshots. >> > > >> > > Richard. >> > > >> > > >> > > On 14 November 2017 at 15:43, Thomas Bouron >> > <thomas.bouron@cloudsoftcorp.c >> > > om >> > > > wrote: >> > > >> > > > Hi Brooklyners! >> > > > >> > > > Few months ago, I pushed a PR[1] to simplify `brooklyn-vagrant` by >> > using >> > > > the RPM package. This is still on hold because we don't build RPM >> > package >> > > > of SNAPSHOT versions and therefore, our vagrant install script won't >> > work >> > > > for bleeding edge versions. >> > > > For the record, RPM are not built on SNAPSHOT because INFRA does not >> > > offer >> > > > centos/RHEL machines as Jenkins slaves. >> > > > >> > > > But that is coming to an end. INFRA did install (recently-ish) >> docker >> > on >> > > > all Jenkins slaves which means that a new world is opened to us! In >> the >> > > > past weeks, I have created a clone[2] of the >> `brooklyn-dist-master`[3] >> > > job >> > > > and configured it to use a custom docker image I created[4]. It is >> > based >> > > on >> > > > `maven:alpine` and adds `rpm` and `dpkg` binaries (for the `dist` >> tag). >> > > > As my test was successful, I decided to go a little further by >> > creating 2 >> > > > other tags for this image: >> > > > - `client, based on `maven:alpine` with the `go` binary >> > > > - `latest` which is the combination of the last 2. >> > > > >> > > > My plan is to migrate all jenkins jobs to use docker. I'll donate >> the >> > > > `Dockerfile` to each git repo so that jenkins will be able to build >> and >> > > > publish them on docker hub instead of using my account/image. Once >> we >> > > have >> > > > everything dockerised, we can look at improving current builds, like >> > > using >> > > > jenkins pipelines rather than relying on upstream<->downstream >> > projects' >> > > > relationships. >> > > > >> > > > It also makes the integration/live tests rise from the ashes. This >> is >> > the >> > > > perfect opportunity to fix and make then part of our CI. >> > > > >> > > > WDYT? Even if it's obvious, the main advantage here is the >> portability >> > of >> > > > the build environment. It also means that we can use any jenkins >> > slaves, >> > > > potentially saving us a lot of time when waiting for a build >> executor. >> > > > >> > > > Best. >> > > > >> > > > [1] https://github.com/apache/brooklyn-dist/pull/105 >> > > > [2] >> > > > https://builds.apache.org/view/B/view/Brooklyn/job/ >> > > > brooklyn-dist-master-docker/ >> > > > [3] https://builds.apache.org/view/B/view/Brooklyn/job/ >> > > > brooklyn-dist-master/ >> > > > [4] https://hub.docker.com/r/tbouron/brooklyn-build/ >> > > > -- >> > > > >> > > > Thomas Bouron • Senior Software Engineer @ Cloudsoft Corporation • >> > > > https://cloudsoft.io/ >> > > > Github: https://github.com/tbouron >> > > > Twitter: https://twitter.com/eltibouron >> > > > >> > > >> > >> > -- > > Thomas Bouron • Senior Software Engineer @ Cloudsoft Corporation • > https://cloudsoft.io/ > Github: https://github.com/tbouron > Twitter: https://twitter.com/eltibouron > -- Thomas Bouron • Senior Software Engineer @ Cloudsoft Corporation • https://cloudsoft.io/ Github: https://github.com/tbouron Twitter: https://twitter.com/eltibouron
