Aha! This is where https://github.com/jenkinsci/workflow-plugin/tree/master/multibranch and/or https://github.com/jenkinsci/literate-plugin/tree/master come into play... and shortly too there will be the multi-repo job type (which (wearing cloudbees hat) we are in the process of open sourcing (removes cloudbees hat)) which will scan all the repositories in your github/bitbucket organization and populate workflow-multibranch / literate projects for all repositories with the marker files.
In short, you just point a mult-repo job type at GitHub, give it an API key and presto... jobs for all your projects and all the branches of all your project and all the PRs of all your projects with no maintenance at all and the full power of workflow to edit the job configuration using the Jenkinsfile at the root of each repo/branch On 25 September 2015 at 04:33, Ben Podgursky <bpodgur...@gmail.com> wrote: > Shouldn't require any additional CI servers -- we have one master with > hundreds of builds. Depending on build frequency, you likely want a couple > slaves -- we have 7 but easily get by with 4. > > Setting up all the builds is some work but Jenkins will help you a lot. We > have a template project which we copy when setting up a new project, so > plugging in a new git repo and name takes less than a minute of work > (actually we have scripts which hit the jenkins API to create new projects, > but it's not critical). And if you set up projects as maven builds, they > will detect downstream builds and trigger necessary rebuilds > automatically. So Jenkins is usually doing less work overall, since one > monolithic project has to rebuild and re-rest * everything *, which would > take many hours. > > On Thu, Sep 24, 2015 at 8:09 PM, Kevin Burton <bur...@spinn3r.com> wrote: > >> Pretty impressive. I think another issue I hadn't thought of is the >> overhead of having a CI server/project for each project. >> >> If anything, you have to enumerate each project so you can see inside of it >> and what's happening. >> >> Having one macro one means you can see everything in one place but of >> course that has drawbacks too. >> >> On Tue, Sep 22, 2015 at 3:28 PM, Ben Podgursky <bpodgur...@gmail.com> >> wrote: >> >> > +1 for aggressive SNAPSHOT use >> > >> > Our setup. YMMV: >> > >> > - We have ~75 independent projects and each of them is versioned and >> > deployed independently. Few million lines of code. >> > - We have CI jenkins builds + unit test suites for every project, and we >> > deploy SNAPSHOT artifacts as soon as a build succeeds. >> > - Our OSS projects follow the same pattern (https://github.com/liveramp/ >> ). >> > We do releases every so often for external users, but internally we use >> > snapshots which build for every new commit. >> > - Every internal dependency is on a SNAPSHOT version. >> > - All developers have all code checked out in intellij, source linked >> > - If you push changes to a library, you update all downstream usages. No >> > exceptions. Since you have all source in intellij, this usually isn't >> > hard. >> > >> > Our use-case is a bit different in that we don't really have "application >> > versions", since everything is back-end data pipeliens. But when we >> > deploy, we build an artifact with locked SNAPSHOT versions which passes >> > integration tests and deploy that artifact. We deploy many projects >> > several times a day. >> > >> > It takes a bit of diligence, but the payoff of not having to manage >> > internal versioning and have formal releases is huge. >> > >> > On Tue, Sep 22, 2015 at 1:14 PM, Ron Wheeler < >> > rwhee...@artifact-software.com >> > > wrote: >> > >> > > +1 (probably better and more complete than my description) >> > > >> > > Has anyone else looked at using an installer like izPack for assembling >> > > test setups? >> > > It integrates with Maven and will pick up all the right versions of >> jars >> > > and configuration files and build an installer that will drop the whole >> > set >> > > where ever you want to test. >> > > You can build an installer that will install on different OSs so you >> can >> > > take the same test jar and run it on Windows, Linux or a Mac and get >> the >> > > "right" configuration files for the target OS. >> > > >> > > Uses maven dependency management to get the jars from your repo and >> uses >> > a >> > > command language in XML to assemble the rest of your supporting data >> and >> > > configuration files. >> > > >> > > A bit smarter than Ant about building application run-time structures >> but >> > > that is all it does. >> > > >> > > Ron >> > > >> > > >> > > On 22/09/2015 3:43 PM, Curtis Rueden wrote: >> > > >> > >> Hi Kevin, >> > >> >> > >> My projects opt for independent versioning of modules to facilitate >> > >> "release early, release often." To do this for large sets of >> components >> > >> like yours requires a Bill of Materials -- i.e., common parent POM >> with >> > >> dependencyManagement section. >> > >> >> > >> FWIW, the docs we have about our projects that work this way are at: >> > >> * http://imagej.net/Architecture >> > >> >> > >> And in particular: >> > >> * http://imagej.net/Architecture#Bill_of_Materials >> > >> * >> > http://imagej.net/Architecture#How_SciJava_achieves_reproducible_builds >> > >> * http://imagej.net/Philosophy#Release_early.2C_release_often >> > >> >> > >> And the BOM stuff is at: >> > >> * >> > >> >> > >> >> > >> https://github.com/scijava/pom-scijava/blob/pom-scijava-8.3.0/pom.xml#L103-L819 >> > >> >> > >> The downside, as you point out, of all components being release >> version >> > >> coupled is that it is annoying to have to do a "release cascade" to >> > >> propagate a bug fix from the lowest level components to the highest >> > level >> > >> ones. We have some tooling to make that easier (I personally live in >> the >> > >> "releases should be as easy/automated/fast as possible" camp), but the >> > >> modularity does cost time sometimes. Hopefully a lot less time than >> > >> building your huge multi-module project from scratch every time, >> though! >> > >> >> > >> I also recently wrote a "melting pot" script to do end-to-end testing >> of >> > >> large component collections: >> > >> >> > >> >> > >> >> > >> https://github.com/scijava/scijava-scripts/blob/d892adc0092c220ee1e597b9fb5a1fb067e4509b/melting-pot.sh >> > >> >> > >> This script builds and runs unit tests for all components of a large >> > >> collection at their respective versions, all in the same Java runtime, >> > to >> > >> ensure that everything _really does_ work together at the versions you >> > are >> > >> currently deploying to end users. >> > >> >> > >> I would be happy to know about other tooling people have created to >> help >> > >> with this sort of project structure. >> > >> >> > >> Regards, >> > >> Curtis >> > >> >> > >> On Tue, Sep 22, 2015 at 12:47 PM, Kevin Burton <bur...@spinn3r.com> >> > >> wrote: >> > >> >> > >> We have a multi-module setup whereby we have about 150 independent >> > >>> modules. >> > >>> >> > >>> Our build takes a long time and actually slows down development as we >> > >>> have >> > >>> to do a compile of a LOT of source code to rebuild the project. >> > >>> >> > >>> Additionally, we have a lot of code that we want to Open Source. >> > >>> >> > >>> This has meant git submodules the IMO git submodules really don’t >> work >> > >>> when >> > >>> using branches. They break and require a whole bunch of custom works >> > and >> > >>> hack and when they DO break it’s confusing how to resolve them. >> > >>> >> > >>> This has meant that we’ve not really done a good job of OSSing our >> code >> > >>> base as its just too hard. >> > >>> >> > >>> What we’ve done to date is just have one major version number across >> > all >> > >>> our projects. So upgrading them and fixing their dependencies means >> > >>> that I >> > >>> just have to change a version number everywhere and I’m done. >> > >>> >> > >>> What I was thinking of is changing this strategy to use the maven >> > >>> "versions:use-latest-versions” plugin. >> > >>> >> > >>> What i would do is have a parent directory named ‘spinn3r’ which just >> > >>> has a >> > >>> bunch of git submodules. We NEVER branch in this directory. >> > >>> >> > >>> It also means that any of our developers can check it out so that >> they >> > >>> have >> > >>> all of our source code. >> > >>> >> > >>> At this point I can use a normal development strategy for each >> project. >> > >>> They don’t use submodules which enables us to branch/merge easily. >> > >>> >> > >>> I can also have a dedicated IntelliJ or Eclipse project for each one >> > and >> > >>> switch between them. >> > >>> >> > >>> Now the main issue I have is how do I bump releases easily and make >> > sure >> > >>> all my code is using the latest version of its sibling projects. >> > >>> >> > >>> In the parent directory I can just run versions:use-latest-versions … >> > on >> > >>> each one of the projects so that it automatically pulled in the >> latest >> > >>> version after a release. >> > >>> >> > >>> The only problem here is that there’s a dependency graph that needs >> to >> > be >> > >>> considered. >> > >>> >> > >>> for example, if project A depends on project B, then we have to bump >> > the >> > >>> version and push project B into maven before we upgrade dependencies >> on >> > >>> project A. >> > >>> >> > >>> This is a frustrating issue… >> > >>> >> > >>> -- >> > >>> >> > >>> We’re hiring if you know of any awesome Java Devops or Linux >> Operations >> > >>> Engineers! >> > >>> >> > >>> Founder/CEO Spinn3r.com >> > >>> Location: *San Francisco, CA* >> > >>> blog: http://burtonator.wordpress.com >> > >>> … or check out my Google+ profile >> > >>> <https://plus.google.com/102718274791889610666/posts> >> > >>> >> > >>> >> > > >> > > -- >> > > Ron Wheeler >> > > President >> > > Artifact Software Inc >> > > email: rwhee...@artifact-software.com >> > > skype: ronaldmwheeler >> > > phone: 866-970-2435, ext 102 >> > > >> > > >> > > --------------------------------------------------------------------- >> > > To unsubscribe, e-mail: users-unsubscr...@maven.apache.org >> > > For additional commands, e-mail: users-h...@maven.apache.org >> > > >> > > >> > >> >> >> >> -- >> >> We’re hiring if you know of any awesome Java Devops or Linux Operations >> Engineers! >> >> Founder/CEO Spinn3r.com >> Location: *San Francisco, CA* >> blog: http://burtonator.wordpress.com >> … or check out my Google+ profile >> <https://plus.google.com/102718274791889610666/posts> >> --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@maven.apache.org For additional commands, e-mail: users-h...@maven.apache.org
