Re: sling ci builds per module
Build feedback is now lot more faster and something one can rely on Thanks Robert!. Chetan Mehrotra On Wed, Oct 12, 2016 at 3:01 AM, Stefan Seifert <sseif...@pro-vision.de> wrote: > that's great, thanks! > > stefan > >>-Original Message- >>From: Robert Munteanu [mailto:romb...@apache.org] >>Sent: Tuesday, October 11, 2016 11:18 PM >>To: dev@sling.apache.org >>Subject: Re: sling ci builds per module >> >>For the record, this is now done. >> >>Robert
RE: sling ci builds per module
that's great, thanks! stefan >-Original Message- >From: Robert Munteanu [mailto:romb...@apache.org] >Sent: Tuesday, October 11, 2016 11:18 PM >To: dev@sling.apache.org >Subject: Re: sling ci builds per module > >For the record, this is now done. > >Robert
Re: sling ci builds per module
For the record, this is now done. Robert On Wed, 2016-09-28 at 22:52 +, Stefan Seifert wrote: > discussed at the Sling Committer Round Table @ adaptTo() 2016 > > we all agreed that we need CI build individually per module/bundle, > not a big build for all bundles at once. > > benefits: > - much faster feedback in case of a broken build, and much more > specific > - if one module is broken only it's own CI build is affected, all > other are still green > - CI build for one module is much faster, leading to much shorter > debugging cycles in case of problems > - it's much easier to use tools like travis without having to work > around build time and log output size limitations > > alternatives discussed: > - we were very unhappy with the apache Jenkins lately, but the reason > may be that the single full CI build was just too big and fragile > - travis would be a simple option to use esp. when sling is migrated > to git and we have one single git repo per module. bertrand pointed > out there is an official support from ASF to use travis. > - we could ask infra to support us with our own jenkins instance we > maintain ourselves. but we do not want to go this way due to its > maintance efforts unless there is no other way. > - we decided to stick with the ASF Jenkins for now, and got the way > to CI builds for each module which robert has already started in > SLING-6061 > > robert currently uses the jenkins job DSL plugin to auto-generate new > jobs and update existing jobs to the current job definition. the > source script and job definition template is maintained in svn, as > soon as this is changed all auto-generated jobs get updated. and > alternative would be to use the Jenkins pipeline plugin, but > currently we think creating individual jobs is easier for us to > understand and maintain. > > some requirements for the ci build ans integration: > > - it would be nice to have a CI build notification within the github > mirrors of the git modules showing red/green status on each commit > (which is available e.g. for travis - is it possible with the ASF > Jenkins as well?) > > - if new branches are created or PRs are submitted a CI build should > run for them as well, reporting it's status in the github views > > - the CI build should run for the supported JDKs for each module, > e.g. java7+java8 or java8-only > > - from the git project it the Jenkins CI job should be easy found. if > a common naming scheme for the Jenkins jobs is used which e.g. > include the git repo name this should be easy. > > - each CI job has to deploy the current module's snapshot to the > apache maven snapshot repo, to make it available to other builds. > > > stefan > > >
Re: sling ci builds per module
On Thu, 2016-09-29 at 15:16 +0200, Oliver Lietz wrote: > > Do you want me to set per-module build jobs for the Karaf build as > > well? > > Yes, please. Didn't look into build job before holidays because of > used > bundles under vote (in features). Done. Now remember, if it fails you can't disable it since it's all automated so you have to fix it :-) . Robert
Re: sling ci builds per module
On Thursday 29 September 2016 11:56:13 Robert Munteanu wrote: > Hi, > > (@Stefan - thanks for summing up what we discussed and following up on > the dev list) +1 indeed, much appreciated (also all the work done for and at adaptTo())! > This is mostly done for the 'main' build, I'm still ironing out some > dependency issues between jobs(1) but should be stable. > > I'll disable the sling-trunk-1.8 job and see how well this setup runs > for a couple of days and then move on the contrib and sample trees. > > Olli, AFAICT the karaf part does not have a Jenkins build set up ATM. Right. I moved it out of contrib next to launchpad. > Do you want me to set per-module build jobs for the Karaf build as > well? Yes, please. Didn't look into build job before holidays because of used bundles under vote (in features). Thanks, Robert! O. > On Wed, 2016-09-28 at 22:52 +, Stefan Seifert wrote: > > discussed at the Sling Committer Round Table @ adaptTo() 2016 > > > > we all agreed that we need CI build individually per module/bundle, > > not a big build for all bundles at once. > > > > benefits: > > - much faster feedback in case of a broken build, and much more > > specific > > - if one module is broken only it's own CI build is affected, all > > other are still green > > - CI build for one module is much faster, leading to much shorter > > debugging cycles in case of problems > > - it's much easier to use tools like travis without having to work > > around build time and log output size limitations > > > > alternatives discussed: > > - we were very unhappy with the apache Jenkins lately, but the reason > > may be that the single full CI build was just too big and fragile > > - travis would be a simple option to use esp. when sling is migrated > > to git and we have one single git repo per module. bertrand pointed > > out there is an official support from ASF to use travis. > > - we could ask infra to support us with our own jenkins instance we > > maintain ourselves. but we do not want to go this way due to its > > maintance efforts unless there is no other way. > > - we decided to stick with the ASF Jenkins for now, and got the way > > to CI builds for each module which robert has already started in > > SLING-6061 > > > > robert currently uses the jenkins job DSL plugin to auto-generate new > > jobs and update existing jobs to the current job definition. the > > source script and job definition template is maintained in svn, as > > soon as this is changed all auto-generated jobs get updated. and > > alternative would be to use the Jenkins pipeline plugin, but > > currently we think creating individual jobs is easier for us to > > understand and maintain. > > > > some requirements for the ci build ans integration: > > > > - it would be nice to have a CI build notification within the github > > mirrors of the git modules showing red/green status on each commit > > (which is available e.g. for travis - is it possible with the ASF > > Jenkins as well?) > > This is a bit linked to the Github migration, but the build status > plugin is enabled for the ASF jenkins instance, so we can reference it. > > We would still need to auto-generate a README file for each modules > referencing the active build jobs. > > > - if new branches are created or PRs are submitted a CI build should > > run for them as well, reporting it's status in the github views > > This is doable once we move to Github with the current setup. > > Once thing I haven't fully figured out yet is show to test such a setup > including the Sling Launchpad tests ( which would be nice ). > > The pipeline approach might be better suited here, but let's see once > we move to git. The benefit of using the dsl plugin is that I can > change the job type (losing job history though ). > > > - the CI build should run for the supported JDKs for each module, > > e.g. java7+java8 or java8-only > > This is already solved in the current setup. > > > - from the git project it the Jenkins CI job should be easy found. if > > a common naming scheme for the Jenkins jobs is used which e.g. > > include the git repo name this should be easy. > > Yes, definitely doable. > > > - each CI job has to deploy the current module's snapshot to the > > apache maven snapshot repo, to make it available to other builds. > > Solved in the current setup, and if we have multiple jobs for various > JDK versions only the one with the lowest JDK version is deployed, for > maximum compatibility. > > Thanks, > > Robert > > 1 - the dependencies are mostly inferred automatically by the maven > plugin supplied by Jenkins, but at the moment it's not able to detect > when a module is referenced in a launchpad via the provisioning module. > I will establish module → launchpad dependencies manually for now, with > a wide approach of 'modules in bundles/ and installer/ will trigger an > org.apache.sling.launchpad build' . We'll see if we need something more > fine-grained.
Re: sling ci builds per module
Hi, (@Stefan - thanks for summing up what we discussed and following up on the dev list) This is mostly done for the 'main' build, I'm still ironing out some dependency issues between jobs(1) but should be stable. I'll disable the sling-trunk-1.8 job and see how well this setup runs for a couple of days and then move on the contrib and sample trees. Olli, AFAICT the karaf part does not have a Jenkins build set up ATM. Do you want me to set per-module build jobs for the Karaf build as well? On Wed, 2016-09-28 at 22:52 +, Stefan Seifert wrote: > discussed at the Sling Committer Round Table @ adaptTo() 2016 > > we all agreed that we need CI build individually per module/bundle, > not a big build for all bundles at once. > > benefits: > - much faster feedback in case of a broken build, and much more > specific > - if one module is broken only it's own CI build is affected, all > other are still green > - CI build for one module is much faster, leading to much shorter > debugging cycles in case of problems > - it's much easier to use tools like travis without having to work > around build time and log output size limitations > > alternatives discussed: > - we were very unhappy with the apache Jenkins lately, but the reason > may be that the single full CI build was just too big and fragile > - travis would be a simple option to use esp. when sling is migrated > to git and we have one single git repo per module. bertrand pointed > out there is an official support from ASF to use travis. > - we could ask infra to support us with our own jenkins instance we > maintain ourselves. but we do not want to go this way due to its > maintance efforts unless there is no other way. > - we decided to stick with the ASF Jenkins for now, and got the way > to CI builds for each module which robert has already started in > SLING-6061 > > robert currently uses the jenkins job DSL plugin to auto-generate new > jobs and update existing jobs to the current job definition. the > source script and job definition template is maintained in svn, as > soon as this is changed all auto-generated jobs get updated. and > alternative would be to use the Jenkins pipeline plugin, but > currently we think creating individual jobs is easier for us to > understand and maintain. > > some requirements for the ci build ans integration: > > - it would be nice to have a CI build notification within the github > mirrors of the git modules showing red/green status on each commit > (which is available e.g. for travis - is it possible with the ASF > Jenkins as well?) This is a bit linked to the Github migration, but the build status plugin is enabled for the ASF jenkins instance, so we can reference it. We would still need to auto-generate a README file for each modules referencing the active build jobs. > > - if new branches are created or PRs are submitted a CI build should > run for them as well, reporting it's status in the github views This is doable once we move to Github with the current setup. Once thing I haven't fully figured out yet is show to test such a setup including the Sling Launchpad tests ( which would be nice ). The pipeline approach might be better suited here, but let's see once we move to git. The benefit of using the dsl plugin is that I can change the job type (losing job history though ). > > - the CI build should run for the supported JDKs for each module, > e.g. java7+java8 or java8-only This is already solved in the current setup. > > - from the git project it the Jenkins CI job should be easy found. if > a common naming scheme for the Jenkins jobs is used which e.g. > include the git repo name this should be easy. Yes, definitely doable. > > - each CI job has to deploy the current module's snapshot to the > apache maven snapshot repo, to make it available to other builds. Solved in the current setup, and if we have multiple jobs for various JDK versions only the one with the lowest JDK version is deployed, for maximum compatibility. Thanks, Robert 1 - the dependencies are mostly inferred automatically by the maven plugin supplied by Jenkins, but at the moment it's not able to detect when a module is referenced in a launchpad via the provisioning module. I will establish module → launchpad dependencies manually for now, with a wide approach of 'modules in bundles/ and installer/ will trigger an org.apache.sling.launchpad build' . We'll see if we need something more fine-grained.
sling ci builds per module
discussed at the Sling Committer Round Table @ adaptTo() 2016 we all agreed that we need CI build individually per module/bundle, not a big build for all bundles at once. benefits: - much faster feedback in case of a broken build, and much more specific - if one module is broken only it's own CI build is affected, all other are still green - CI build for one module is much faster, leading to much shorter debugging cycles in case of problems - it's much easier to use tools like travis without having to work around build time and log output size limitations alternatives discussed: - we were very unhappy with the apache Jenkins lately, but the reason may be that the single full CI build was just too big and fragile - travis would be a simple option to use esp. when sling is migrated to git and we have one single git repo per module. bertrand pointed out there is an official support from ASF to use travis. - we could ask infra to support us with our own jenkins instance we maintain ourselves. but we do not want to go this way due to its maintance efforts unless there is no other way. - we decided to stick with the ASF Jenkins for now, and got the way to CI builds for each module which robert has already started in SLING-6061 robert currently uses the jenkins job DSL plugin to auto-generate new jobs and update existing jobs to the current job definition. the source script and job definition template is maintained in svn, as soon as this is changed all auto-generated jobs get updated. and alternative would be to use the Jenkins pipeline plugin, but currently we think creating individual jobs is easier for us to understand and maintain. some requirements for the ci build ans integration: - it would be nice to have a CI build notification within the github mirrors of the git modules showing red/green status on each commit (which is available e.g. for travis - is it possible with the ASF Jenkins as well?) - if new branches are created or PRs are submitted a CI build should run for them as well, reporting it's status in the github views - the CI build should run for the supported JDKs for each module, e.g. java7+java8 or java8-only - from the git project it the Jenkins CI job should be easy found. if a common naming scheme for the Jenkins jobs is used which e.g. include the git repo name this should be easy. - each CI job has to deploy the current module's snapshot to the apache maven snapshot repo, to make it available to other builds. stefan
