Hi Richard. Indeed, this will work but I guess we cannot guarantee SLAs ;)
I think there is a bit of confusion because I mixed things, let me clarify. There are two aspect to what I'm doing: 1. Build all Brooklyn projects into Docker 2. Replace the 2 jobs we have per project (to build master + PRs) by a single pipeline per project that take care of all. This solution is really elegant because everything is nicely organised + the configuration lives within git (I don't think I need to explain the benefits of that) "retest this please" is currently available for all Brooklyn PRs and its loss is induced by #2, but has no link with #1. In fact #2 sits on top of #1. But you're right, I should ask infra to add this feature. Will open a JIRA for it. Best. On Tue, 24 Jul 2018 at 10:12 Richard Downer <[email protected]> wrote: > Loss of "retest this please" isn't a deal breaker IMO. Replace it with > "@rdowner retest this please" and it will have the same effect, if somewhat > slower ;-) > > Did we have "retest this please" available before? If so then I'd hope > there's a way to get it working with the Docker-based build, especially > since the Docker path is the one recommended by Infra. If there's a loss of > functionality there then hopefully we could persuade Infra to get this > feature working again. > > Richard. > > > On Tue, 24 Jul 2018 at 09:34, Thomas Bouron < > [email protected]> > wrote: > > > Hi all. > > > > Just a quick update about the status of this work: all projects now build > > within docker containers. It has been a bit painful to set up but it's > all > > good now. > > > > Also, the concept I mentioned in my previous email works like a charm. > > However, I discovered yesterday a limitation: Brooklyn GitHub repos are > > owned by Apache, therefore we don't have the permissions to add webhooks. > > This means that the feature to re-trigger a build from a PR comment by > > writing "retest this please" won't work. On top of that, it seems that > even > > the DSL command to set this up is missing from the Apache > > Jenkins installation. > > > > Knowing this, are we happy to move anyway, or is it a deal breaker? WDYT? > > > > Best. > > > > On Tue, 1 May 2018 at 14:33 Thomas Bouron < > [email protected] > > > > > wrote: > > > > > Hi all. > > > > > > Continuing my work on this topic, I just pushed a PR [1] in > > > brooklyn-library that adds a Jenkinsfile. If you are not familiar with > > what > > > it is, a Jenkinfile describes a Jenkins job by using the pipeline DSL. > > That > > > allows you to capture the configuration / best practices under version > > > control so it's easier to iterate over and is repeatable. This > particular > > > Jenkinsfile implements the current configuration we have + a fix for > > > permission issues we recently had (see INFRA ticket [2]) > > > > > > To make this work, we still need to manually create the multibranch > > > pipeline job, and pointing it toward the right GitHub repo. I'll do > that > > > once the PR is merge. But the bonus point of this approach is that it > > > automatically handles PR builds, with very nice views so that is 2 > birds > > > from one stone. > > > > > > Why not using a seed instead that would avoid doing any kind of manual > > > config? That is a good question and the answer is: we are running on > the > > > apache infra, and I don't think we have the power of using Jenkins > seed. > > > Now if I'm wrong, it's fine as we can just add the seed on top of the > > > pipelines. > > > > > > Anyway, I tested my Jenkinsfile locally with brooklyn-library so it > > should > > > work nicely. I was planning to test with this project first, validate > the > > > concept, then do the same for the remaining git submodules. All I need > > now, > > > is a review on [1] + merge :) > > > > > > Best. > > > > > > [1] https://github.com/apache/brooklyn-library/pull/153 > > > [2] > > > > > > https://issues.apache.org/jira/browse/INFRA-16417?page=com.atlassian.jira.plugin.system.issuetabpanels%3Aall-tabpanel > > > > > > On Wed, 15 Nov 2017 at 16:42 Thomas Bouron < > > > [email protected]> wrote: > > > > > >> Looks like we start to reach a consensus :) > > >> > > >> I duplicated all our jobs and configured them to use docker instead: > > >> - brooklyn-server => > > >> > > > https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-server-master-docker/ > > >> - brooklyn-library => > > >> > > > https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-library-master-docker/ > > >> - brooklyn-client => > > >> > > > https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-client-master-docker/ > > >> - brooklyn-ui => > > >> > > > https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-ui-master-docker/ > > >> - brooklyn-dist => > > >> > > > https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-dist-master-docker/ > > >> - brooklyn => > > >> > > > https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-master-build-docker/ > > >> > > >> I only "disabled" the other jobs by removing all triggers, just in > case > > >> what I did breaks everything then we can go back. > > >> > > >> While I was on the jenkins, I noticed some `incubator-brooklyn-*` > jobs, > > I > > >> guess those can be deleted isn't it? Also, anyone know the purpose of > > >> brooklyn-master-windows[1] ? > > >> > > >> --- > > >> In terms of next steps, I'll monitor to see if the new jobs behave as > > >> expected. If yes, I'll then move on to: > > >> - migrate the PR's jobs to use docker > > >> - add the `Dockerfile` within each git submodule > > >> - leverage the Jenkins' pipeline feature by adding the config directly > > in > > >> each git submodule > > >> - create new job to build and publish docs/website automatically > > >> > > >> Best > > >> > > >> [1] > > >> > > > https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-master-windows/ > > >> > > >> On Wed, 15 Nov 2017 at 11:18 John McCabe <[email protected]> wrote: > > >> > > >>> Huge +1 on consistency/portability. > > >>> > > >>> On Tue, 14 Nov 2017 at 16:40 Mark McKenna <[email protected]> > > >>> wrote: > > >>> > > >>> > +1 I echo what Richard says about consistency ... also makes > > debugging > > >>> > build failures easier > > >>> > > > >>> > > > >>> > > > >>> > On 14 November 2017 at 15:48, Richard Downer <[email protected]> > > >>> wrote: > > >>> > > > >>> > > +1 > > >>> > > > > >>> > > Having seen some of the conversations on the [email protected] > > >>> list, it > > >>> > > does seem like this is the way to go as far as getting the right > > >>> build > > >>> > > environment on the Apache Jenkins cluster. This is the best > > solution > > >>> for > > >>> > > getting the `rpm` and `deb` built regularly. > > >>> > > > > >>> > > Build consistency is also a great driver. I can see this being > > >>> useful for > > >>> > > the release process too, as a release build would have an > identical > > >>> > > environment to that used to build the snapshots. > > >>> > > > > >>> > > Richard. > > >>> > > > > >>> > > > > >>> > > On 14 November 2017 at 15:43, Thomas Bouron > > >>> > <[email protected] > > >>> > > om > > >>> > > > wrote: > > >>> > > > > >>> > > > Hi Brooklyners! > > >>> > > > > > >>> > > > Few months ago, I pushed a PR[1] to simplify `brooklyn-vagrant` > > by > > >>> > using > > >>> > > > the RPM package. This is still on hold because we don't build > RPM > > >>> > package > > >>> > > > of SNAPSHOT versions and therefore, our vagrant install script > > >>> won't > > >>> > work > > >>> > > > for bleeding edge versions. > > >>> > > > For the record, RPM are not built on SNAPSHOT because INFRA > does > > >>> not > > >>> > > offer > > >>> > > > centos/RHEL machines as Jenkins slaves. > > >>> > > > > > >>> > > > But that is coming to an end. INFRA did install (recently-ish) > > >>> docker > > >>> > on > > >>> > > > all Jenkins slaves which means that a new world is opened to > us! > > >>> In the > > >>> > > > past weeks, I have created a clone[2] of the > > >>> `brooklyn-dist-master`[3] > > >>> > > job > > >>> > > > and configured it to use a custom docker image I created[4]. It > > is > > >>> > based > > >>> > > on > > >>> > > > `maven:alpine` and adds `rpm` and `dpkg` binaries (for the > `dist` > > >>> tag). > > >>> > > > As my test was successful, I decided to go a little further by > > >>> > creating 2 > > >>> > > > other tags for this image: > > >>> > > > - `client, based on `maven:alpine` with the `go` binary > > >>> > > > - `latest` which is the combination of the last 2. > > >>> > > > > > >>> > > > My plan is to migrate all jenkins jobs to use docker. I'll > donate > > >>> the > > >>> > > > `Dockerfile` to each git repo so that jenkins will be able to > > >>> build and > > >>> > > > publish them on docker hub instead of using my account/image. > > Once > > >>> we > > >>> > > have > > >>> > > > everything dockerised, we can look at improving current builds, > > >>> like > > >>> > > using > > >>> > > > jenkins pipelines rather than relying on upstream<->downstream > > >>> > projects' > > >>> > > > relationships. > > >>> > > > > > >>> > > > It also makes the integration/live tests rise from the ashes. > > This > > >>> is > > >>> > the > > >>> > > > perfect opportunity to fix and make then part of our CI. > > >>> > > > > > >>> > > > WDYT? Even if it's obvious, the main advantage here is the > > >>> portability > > >>> > of > > >>> > > > the build environment. It also means that we can use any > jenkins > > >>> > slaves, > > >>> > > > potentially saving us a lot of time when waiting for a build > > >>> executor. > > >>> > > > > > >>> > > > Best. > > >>> > > > > > >>> > > > [1] https://github.com/apache/brooklyn-dist/pull/105 > > >>> > > > [2] > > >>> > > > https://builds.apache.org/view/B/view/Brooklyn/job/ > > >>> > > > brooklyn-dist-master-docker/ > > >>> > > > [3] https://builds.apache.org/view/B/view/Brooklyn/job/ > > >>> > > > brooklyn-dist-master/ > > >>> > > > [4] https://hub.docker.com/r/tbouron/brooklyn-build/ > > >>> > > > -- > > >>> > > > > > >>> > > > Thomas Bouron • Senior Software Engineer @ Cloudsoft > Corporation > > • > > >>> > > > https://cloudsoft.io/ > > >>> > > > Github: https://github.com/tbouron > > >>> > > > Twitter: https://twitter.com/eltibouron > > >>> > > > > > >>> > > > > >>> > > > >>> > > >> -- > > >> > > >> Thomas Bouron • Senior Software Engineer @ Cloudsoft Corporation • > > >> https://cloudsoft.io/ > > >> Github: https://github.com/tbouron > > >> Twitter: https://twitter.com/eltibouron > > >> > > > -- > > > > > > Thomas Bouron • Senior Software Engineer @ Cloudsoft Corporation • > > > https://cloudsoft.io/ > > > Github: https://github.com/tbouron > > > Twitter: https://twitter.com/eltibouron > > > > > -- > > > > Thomas Bouron > > Senior Software Engineer > > > > *Cloudsoft <https://cloudsoft.io/> *| Bringing Business to the Cloud > > > > GitHub: https://github.com/tbouron > > Twitter: https://twitter.com/eltibouron > > > > Need a hand with AWS? Get a Free Consultation. > > > -- Thomas Bouron Senior Software Engineer *Cloudsoft <https://cloudsoft.io/> *| Bringing Business to the Cloud GitHub: https://github.com/tbouron Twitter: https://twitter.com/eltibouron Need a hand with AWS? Get a Free Consultation.
