Some time before November is the plan... But certainly before the end of November if there is any slippage.
The demo I saw this week looked almost done On Friday 25 September 2015, Ben Podgursky <[email protected]> wrote: > Scanning a github org to generate builds would be pretty slick. Definitely > interested in taking a look at that when it's released. > > On Fri, Sep 25, 2015 at 5:20 AM, Stephen Connolly < > [email protected] <javascript:;>> wrote: > > > Aha! This is where > > https://github.com/jenkinsci/workflow-plugin/tree/master/multibranch > > and/or https://github.com/jenkinsci/literate-plugin/tree/master come > > into play... and shortly too there will be the multi-repo job type > > (which (wearing cloudbees hat) we are in the process of open sourcing > > (removes cloudbees hat)) which will scan all the repositories in your > > github/bitbucket organization and populate workflow-multibranch / > > literate projects for all repositories with the marker files. > > > > In short, you just point a mult-repo job type at GitHub, give it an > > API key and presto... jobs for all your projects and all the branches > > of all your project and all the PRs of all your projects with no > > maintenance at all and the full power of workflow to edit the job > > configuration using the Jenkinsfile at the root of each repo/branch > > > > On 25 September 2015 at 04:33, Ben Podgursky <[email protected] > <javascript:;>> wrote: > > > Shouldn't require any additional CI servers -- we have one master with > > > hundreds of builds. Depending on build frequency, you likely want a > > couple > > > slaves -- we have 7 but easily get by with 4. > > > > > > Setting up all the builds is some work but Jenkins will help you a lot. > > We > > > have a template project which we copy when setting up a new project, so > > > plugging in a new git repo and name takes less than a minute of work > > > (actually we have scripts which hit the jenkins API to create new > > projects, > > > but it's not critical). And if you set up projects as maven builds, > they > > > will detect downstream builds and trigger necessary rebuilds > > > automatically. So Jenkins is usually doing less work overall, since > one > > > monolithic project has to rebuild and re-rest * everything *, which > would > > > take many hours. > > > > > > On Thu, Sep 24, 2015 at 8:09 PM, Kevin Burton <[email protected] > <javascript:;>> > > wrote: > > > > > >> Pretty impressive. I think another issue I hadn't thought of is the > > >> overhead of having a CI server/project for each project. > > >> > > >> If anything, you have to enumerate each project so you can see inside > > of it > > >> and what's happening. > > >> > > >> Having one macro one means you can see everything in one place but of > > >> course that has drawbacks too. > > >> > > >> On Tue, Sep 22, 2015 at 3:28 PM, Ben Podgursky <[email protected] > <javascript:;>> > > >> wrote: > > >> > > >> > +1 for aggressive SNAPSHOT use > > >> > > > >> > Our setup. YMMV: > > >> > > > >> > - We have ~75 independent projects and each of them is versioned and > > >> > deployed independently. Few million lines of code. > > >> > - We have CI jenkins builds + unit test suites for every project, > and > > we > > >> > deploy SNAPSHOT artifacts as soon as a build succeeds. > > >> > - Our OSS projects follow the same pattern ( > > https://github.com/liveramp/ > > >> ). > > >> > We do releases every so often for external users, but internally we > > use > > >> > snapshots which build for every new commit. > > >> > - Every internal dependency is on a SNAPSHOT version. > > >> > - All developers have all code checked out in intellij, source > linked > > >> > - If you push changes to a library, you update all downstream > > usages. No > > >> > exceptions. Since you have all source in intellij, this usually > isn't > > >> > hard. > > >> > > > >> > Our use-case is a bit different in that we don't really have > > "application > > >> > versions", since everything is back-end data pipeliens. But when we > > >> > deploy, we build an artifact with locked SNAPSHOT versions which > > passes > > >> > integration tests and deploy that artifact. We deploy many projects > > >> > several times a day. > > >> > > > >> > It takes a bit of diligence, but the payoff of not having to manage > > >> > internal versioning and have formal releases is huge. > > >> > > > >> > On Tue, Sep 22, 2015 at 1:14 PM, Ron Wheeler < > > >> > [email protected] <javascript:;> > > >> > > wrote: > > >> > > > >> > > +1 (probably better and more complete than my description) > > >> > > > > >> > > Has anyone else looked at using an installer like izPack for > > assembling > > >> > > test setups? > > >> > > It integrates with Maven and will pick up all the right versions > of > > >> jars > > >> > > and configuration files and build an installer that will drop the > > whole > > >> > set > > >> > > where ever you want to test. > > >> > > You can build an installer that will install on different OSs so > you > > >> can > > >> > > take the same test jar and run it on Windows, Linux or a Mac and > get > > >> the > > >> > > "right" configuration files for the target OS. > > >> > > > > >> > > Uses maven dependency management to get the jars from your repo > and > > >> uses > > >> > a > > >> > > command language in XML to assemble the rest of your supporting > data > > >> and > > >> > > configuration files. > > >> > > > > >> > > A bit smarter than Ant about building application run-time > > structures > > >> but > > >> > > that is all it does. > > >> > > > > >> > > Ron > > >> > > > > >> > > > > >> > > On 22/09/2015 3:43 PM, Curtis Rueden wrote: > > >> > > > > >> > >> Hi Kevin, > > >> > >> > > >> > >> My projects opt for independent versioning of modules to > facilitate > > >> > >> "release early, release often." To do this for large sets of > > >> components > > >> > >> like yours requires a Bill of Materials -- i.e., common parent > POM > > >> with > > >> > >> dependencyManagement section. > > >> > >> > > >> > >> FWIW, the docs we have about our projects that work this way are > > at: > > >> > >> * http://imagej.net/Architecture > > >> > >> > > >> > >> And in particular: > > >> > >> * http://imagej.net/Architecture#Bill_of_Materials > > >> > >> * > > >> > > > http://imagej.net/Architecture#How_SciJava_achieves_reproducible_builds > > >> > >> * http://imagej.net/Philosophy#Release_early.2C_release_often > > >> > >> > > >> > >> And the BOM stuff is at: > > >> > >> * > > >> > >> > > >> > >> > > >> > > > >> > > > https://github.com/scijava/pom-scijava/blob/pom-scijava-8.3.0/pom.xml#L103-L819 > > >> > >> > > >> > >> The downside, as you point out, of all components being release > > >> version > > >> > >> coupled is that it is annoying to have to do a "release cascade" > to > > >> > >> propagate a bug fix from the lowest level components to the > highest > > >> > level > > >> > >> ones. We have some tooling to make that easier (I personally live > > in > > >> the > > >> > >> "releases should be as easy/automated/fast as possible" camp), > but > > the > > >> > >> modularity does cost time sometimes. Hopefully a lot less time > than > > >> > >> building your huge multi-module project from scratch every time, > > >> though! > > >> > >> > > >> > >> I also recently wrote a "melting pot" script to do end-to-end > > testing > > >> of > > >> > >> large component collections: > > >> > >> > > >> > >> > > >> > >> > > >> > > > >> > > > https://github.com/scijava/scijava-scripts/blob/d892adc0092c220ee1e597b9fb5a1fb067e4509b/melting-pot.sh > > >> > >> > > >> > >> This script builds and runs unit tests for all components of a > > large > > >> > >> collection at their respective versions, all in the same Java > > runtime, > > >> > to > > >> > >> ensure that everything _really does_ work together at the > versions > > you > > >> > are > > >> > >> currently deploying to end users. > > >> > >> > > >> > >> I would be happy to know about other tooling people have created > to > > >> help > > >> > >> with this sort of project structure. > > >> > >> > > >> > >> Regards, > > >> > >> Curtis > > >> > >> > > >> > >> On Tue, Sep 22, 2015 at 12:47 PM, Kevin Burton < > [email protected] <javascript:;> > > > > > >> > >> wrote: > > >> > >> > > >> > >> We have a multi-module setup whereby we have about 150 > independent > > >> > >>> modules. > > >> > >>> > > >> > >>> Our build takes a long time and actually slows down development > > as we > > >> > >>> have > > >> > >>> to do a compile of a LOT of source code to rebuild the project. > > >> > >>> > > >> > >>> Additionally, we have a lot of code that we want to Open Source. > > >> > >>> > > >> > >>> This has meant git submodules the IMO git submodules really > don’t > > >> work > > >> > >>> when > > >> > >>> using branches. They break and require a whole bunch of custom > > works > > >> > and > > >> > >>> hack and when they DO break it’s confusing how to resolve them. > > >> > >>> > > >> > >>> This has meant that we’ve not really done a good job of OSSing > our > > >> code > > >> > >>> base as its just too hard. > > >> > >>> > > >> > >>> What we’ve done to date is just have one major version number > > across > > >> > all > > >> > >>> our projects. So upgrading them and fixing their dependencies > > means > > >> > >>> that I > > >> > >>> just have to change a version number everywhere and I’m done. > > >> > >>> > > >> > >>> What I was thinking of is changing this strategy to use the > maven > > >> > >>> "versions:use-latest-versions” plugin. > > >> > >>> > > >> > >>> What i would do is have a parent directory named ‘spinn3r’ which > > just > > >> > >>> has a > > >> > >>> bunch of git submodules. We NEVER branch in this directory. > > >> > >>> > > >> > >>> It also means that any of our developers can check it out so > that > > >> they > > >> > >>> have > > >> > >>> all of our source code. > > >> > >>> > > >> > >>> At this point I can use a normal development strategy for each > > >> project. > > >> > >>> They don’t use submodules which enables us to branch/merge > easily. > > >> > >>> > > >> > >>> I can also have a dedicated IntelliJ or Eclipse project for each > > one > > >> > and > > >> > >>> switch between them. > > >> > >>> > > >> > >>> Now the main issue I have is how do I bump releases easily and > > make > > >> > sure > > >> > >>> all my code is using the latest version of its sibling projects. > > >> > >>> > > >> > >>> In the parent directory I can just run > > versions:use-latest-versions … > > >> > on > > >> > >>> each one of the projects so that it automatically pulled in the > > >> latest > > >> > >>> version after a release. > > >> > >>> > > >> > >>> The only problem here is that there’s a dependency graph that > > needs > > >> to > > >> > be > > >> > >>> considered. > > >> > >>> > > >> > >>> for example, if project A depends on project B, then we have to > > bump > > >> > the > > >> > >>> version and push project B into maven before we upgrade > > dependencies > > >> on > > >> > >>> project A. > > >> > >>> > > >> > >>> This is a frustrating issue… > > >> > >>> > > >> > >>> -- > > >> > >>> > > >> > >>> We’re hiring if you know of any awesome Java Devops or Linux > > >> Operations > > >> > >>> Engineers! > > >> > >>> > > >> > >>> Founder/CEO Spinn3r.com > > >> > >>> Location: *San Francisco, CA* > > >> > >>> blog: http://burtonator.wordpress.com > > >> > >>> … or check out my Google+ profile > > >> > >>> <https://plus.google.com/102718274791889610666/posts> > > >> > >>> > > >> > >>> > > >> > > > > >> > > -- > > >> > > Ron Wheeler > > >> > > President > > >> > > Artifact Software Inc > > >> > > email: [email protected] <javascript:;> > > >> > > skype: ronaldmwheeler > > >> > > phone: 866-970-2435, ext 102 > > >> > > > > >> > > > > >> > > > > --------------------------------------------------------------------- > > >> > > To unsubscribe, e-mail: [email protected] > <javascript:;> > > >> > > For additional commands, e-mail: [email protected] > <javascript:;> > > >> > > > > >> > > > > >> > > > >> > > >> > > >> > > >> -- > > >> > > >> We’re hiring if you know of any awesome Java Devops or Linux > Operations > > >> Engineers! > > >> > > >> Founder/CEO Spinn3r.com > > >> Location: *San Francisco, CA* > > >> blog: http://burtonator.wordpress.com > > >> … or check out my Google+ profile > > >> <https://plus.google.com/102718274791889610666/posts> > > >> > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > <javascript:;> > > For additional commands, e-mail: [email protected] > <javascript:;> > > > > > -- Sent from my phone
