Scanning a github org to generate builds would be pretty slick. Definitely interested in taking a look at that when it's released.
On Fri, Sep 25, 2015 at 5:20 AM, Stephen Connolly < [email protected]> wrote: > Aha! This is where > https://github.com/jenkinsci/workflow-plugin/tree/master/multibranch > and/or https://github.com/jenkinsci/literate-plugin/tree/master come > into play... and shortly too there will be the multi-repo job type > (which (wearing cloudbees hat) we are in the process of open sourcing > (removes cloudbees hat)) which will scan all the repositories in your > github/bitbucket organization and populate workflow-multibranch / > literate projects for all repositories with the marker files. > > In short, you just point a mult-repo job type at GitHub, give it an > API key and presto... jobs for all your projects and all the branches > of all your project and all the PRs of all your projects with no > maintenance at all and the full power of workflow to edit the job > configuration using the Jenkinsfile at the root of each repo/branch > > On 25 September 2015 at 04:33, Ben Podgursky <[email protected]> wrote: > > Shouldn't require any additional CI servers -- we have one master with > > hundreds of builds. Depending on build frequency, you likely want a > couple > > slaves -- we have 7 but easily get by with 4. > > > > Setting up all the builds is some work but Jenkins will help you a lot. > We > > have a template project which we copy when setting up a new project, so > > plugging in a new git repo and name takes less than a minute of work > > (actually we have scripts which hit the jenkins API to create new > projects, > > but it's not critical). And if you set up projects as maven builds, they > > will detect downstream builds and trigger necessary rebuilds > > automatically. So Jenkins is usually doing less work overall, since one > > monolithic project has to rebuild and re-rest * everything *, which would > > take many hours. > > > > On Thu, Sep 24, 2015 at 8:09 PM, Kevin Burton <[email protected]> > wrote: > > > >> Pretty impressive. I think another issue I hadn't thought of is the > >> overhead of having a CI server/project for each project. > >> > >> If anything, you have to enumerate each project so you can see inside > of it > >> and what's happening. > >> > >> Having one macro one means you can see everything in one place but of > >> course that has drawbacks too. > >> > >> On Tue, Sep 22, 2015 at 3:28 PM, Ben Podgursky <[email protected]> > >> wrote: > >> > >> > +1 for aggressive SNAPSHOT use > >> > > >> > Our setup. YMMV: > >> > > >> > - We have ~75 independent projects and each of them is versioned and > >> > deployed independently. Few million lines of code. > >> > - We have CI jenkins builds + unit test suites for every project, and > we > >> > deploy SNAPSHOT artifacts as soon as a build succeeds. > >> > - Our OSS projects follow the same pattern ( > https://github.com/liveramp/ > >> ). > >> > We do releases every so often for external users, but internally we > use > >> > snapshots which build for every new commit. > >> > - Every internal dependency is on a SNAPSHOT version. > >> > - All developers have all code checked out in intellij, source linked > >> > - If you push changes to a library, you update all downstream > usages. No > >> > exceptions. Since you have all source in intellij, this usually isn't > >> > hard. > >> > > >> > Our use-case is a bit different in that we don't really have > "application > >> > versions", since everything is back-end data pipeliens. But when we > >> > deploy, we build an artifact with locked SNAPSHOT versions which > passes > >> > integration tests and deploy that artifact. We deploy many projects > >> > several times a day. > >> > > >> > It takes a bit of diligence, but the payoff of not having to manage > >> > internal versioning and have formal releases is huge. > >> > > >> > On Tue, Sep 22, 2015 at 1:14 PM, Ron Wheeler < > >> > [email protected] > >> > > wrote: > >> > > >> > > +1 (probably better and more complete than my description) > >> > > > >> > > Has anyone else looked at using an installer like izPack for > assembling > >> > > test setups? > >> > > It integrates with Maven and will pick up all the right versions of > >> jars > >> > > and configuration files and build an installer that will drop the > whole > >> > set > >> > > where ever you want to test. > >> > > You can build an installer that will install on different OSs so you > >> can > >> > > take the same test jar and run it on Windows, Linux or a Mac and get > >> the > >> > > "right" configuration files for the target OS. > >> > > > >> > > Uses maven dependency management to get the jars from your repo and > >> uses > >> > a > >> > > command language in XML to assemble the rest of your supporting data > >> and > >> > > configuration files. > >> > > > >> > > A bit smarter than Ant about building application run-time > structures > >> but > >> > > that is all it does. > >> > > > >> > > Ron > >> > > > >> > > > >> > > On 22/09/2015 3:43 PM, Curtis Rueden wrote: > >> > > > >> > >> Hi Kevin, > >> > >> > >> > >> My projects opt for independent versioning of modules to facilitate > >> > >> "release early, release often." To do this for large sets of > >> components > >> > >> like yours requires a Bill of Materials -- i.e., common parent POM > >> with > >> > >> dependencyManagement section. > >> > >> > >> > >> FWIW, the docs we have about our projects that work this way are > at: > >> > >> * http://imagej.net/Architecture > >> > >> > >> > >> And in particular: > >> > >> * http://imagej.net/Architecture#Bill_of_Materials > >> > >> * > >> > > http://imagej.net/Architecture#How_SciJava_achieves_reproducible_builds > >> > >> * http://imagej.net/Philosophy#Release_early.2C_release_often > >> > >> > >> > >> And the BOM stuff is at: > >> > >> * > >> > >> > >> > >> > >> > > >> > https://github.com/scijava/pom-scijava/blob/pom-scijava-8.3.0/pom.xml#L103-L819 > >> > >> > >> > >> The downside, as you point out, of all components being release > >> version > >> > >> coupled is that it is annoying to have to do a "release cascade" to > >> > >> propagate a bug fix from the lowest level components to the highest > >> > level > >> > >> ones. We have some tooling to make that easier (I personally live > in > >> the > >> > >> "releases should be as easy/automated/fast as possible" camp), but > the > >> > >> modularity does cost time sometimes. Hopefully a lot less time than > >> > >> building your huge multi-module project from scratch every time, > >> though! > >> > >> > >> > >> I also recently wrote a "melting pot" script to do end-to-end > testing > >> of > >> > >> large component collections: > >> > >> > >> > >> > >> > >> > >> > > >> > https://github.com/scijava/scijava-scripts/blob/d892adc0092c220ee1e597b9fb5a1fb067e4509b/melting-pot.sh > >> > >> > >> > >> This script builds and runs unit tests for all components of a > large > >> > >> collection at their respective versions, all in the same Java > runtime, > >> > to > >> > >> ensure that everything _really does_ work together at the versions > you > >> > are > >> > >> currently deploying to end users. > >> > >> > >> > >> I would be happy to know about other tooling people have created to > >> help > >> > >> with this sort of project structure. > >> > >> > >> > >> Regards, > >> > >> Curtis > >> > >> > >> > >> On Tue, Sep 22, 2015 at 12:47 PM, Kevin Burton <[email protected] > > > >> > >> wrote: > >> > >> > >> > >> We have a multi-module setup whereby we have about 150 independent > >> > >>> modules. > >> > >>> > >> > >>> Our build takes a long time and actually slows down development > as we > >> > >>> have > >> > >>> to do a compile of a LOT of source code to rebuild the project. > >> > >>> > >> > >>> Additionally, we have a lot of code that we want to Open Source. > >> > >>> > >> > >>> This has meant git submodules the IMO git submodules really don’t > >> work > >> > >>> when > >> > >>> using branches. They break and require a whole bunch of custom > works > >> > and > >> > >>> hack and when they DO break it’s confusing how to resolve them. > >> > >>> > >> > >>> This has meant that we’ve not really done a good job of OSSing our > >> code > >> > >>> base as its just too hard. > >> > >>> > >> > >>> What we’ve done to date is just have one major version number > across > >> > all > >> > >>> our projects. So upgrading them and fixing their dependencies > means > >> > >>> that I > >> > >>> just have to change a version number everywhere and I’m done. > >> > >>> > >> > >>> What I was thinking of is changing this strategy to use the maven > >> > >>> "versions:use-latest-versions” plugin. > >> > >>> > >> > >>> What i would do is have a parent directory named ‘spinn3r’ which > just > >> > >>> has a > >> > >>> bunch of git submodules. We NEVER branch in this directory. > >> > >>> > >> > >>> It also means that any of our developers can check it out so that > >> they > >> > >>> have > >> > >>> all of our source code. > >> > >>> > >> > >>> At this point I can use a normal development strategy for each > >> project. > >> > >>> They don’t use submodules which enables us to branch/merge easily. > >> > >>> > >> > >>> I can also have a dedicated IntelliJ or Eclipse project for each > one > >> > and > >> > >>> switch between them. > >> > >>> > >> > >>> Now the main issue I have is how do I bump releases easily and > make > >> > sure > >> > >>> all my code is using the latest version of its sibling projects. > >> > >>> > >> > >>> In the parent directory I can just run > versions:use-latest-versions … > >> > on > >> > >>> each one of the projects so that it automatically pulled in the > >> latest > >> > >>> version after a release. > >> > >>> > >> > >>> The only problem here is that there’s a dependency graph that > needs > >> to > >> > be > >> > >>> considered. > >> > >>> > >> > >>> for example, if project A depends on project B, then we have to > bump > >> > the > >> > >>> version and push project B into maven before we upgrade > dependencies > >> on > >> > >>> project A. > >> > >>> > >> > >>> This is a frustrating issue… > >> > >>> > >> > >>> -- > >> > >>> > >> > >>> We’re hiring if you know of any awesome Java Devops or Linux > >> Operations > >> > >>> Engineers! > >> > >>> > >> > >>> Founder/CEO Spinn3r.com > >> > >>> Location: *San Francisco, CA* > >> > >>> blog: http://burtonator.wordpress.com > >> > >>> … or check out my Google+ profile > >> > >>> <https://plus.google.com/102718274791889610666/posts> > >> > >>> > >> > >>> > >> > > > >> > > -- > >> > > Ron Wheeler > >> > > President > >> > > Artifact Software Inc > >> > > email: [email protected] > >> > > skype: ronaldmwheeler > >> > > phone: 866-970-2435, ext 102 > >> > > > >> > > > >> > > > --------------------------------------------------------------------- > >> > > To unsubscribe, e-mail: [email protected] > >> > > For additional commands, e-mail: [email protected] > >> > > > >> > > > >> > > >> > >> > >> > >> -- > >> > >> We’re hiring if you know of any awesome Java Devops or Linux Operations > >> Engineers! > >> > >> Founder/CEO Spinn3r.com > >> Location: *San Francisco, CA* > >> blog: http://burtonator.wordpress.com > >> … or check out my Google+ profile > >> <https://plus.google.com/102718274791889610666/posts> > >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
