Nice work. From my recent experience, gradle 4.3 is needed when building with java 9.0.1 See INFRA-15448 -------- Original message --------From: Lukasz Cwik <[email protected]> Date: 11/7/17 5:59 PM (GMT-08:00) To: [email protected] Subject: Re: [DISCUSS] Move away from Apache Maven as build tool The Gradle POC has made significant advances since last week (shading, Python, Go, Docker builds, ...). I believe the current state is close enough to the Maven build system to warrant a comparison.
The largest build differences I noticed are: * Full build takes about ~22mins using Gradle (parallelizing the three rounds of Python tests would reduce this to ~17mins) compared to ~38mins in Maven * Clean build all Java modules (skipping over Go/Python) takes ~8mins in Gradle which takes ~36mins in Maven * Build output is cached allowing for faster subsequent builds with "gradle buildDependents" allowing for most single module changes taking ~2mins to build and test without needing to rely on "mvn install" I have opened PR 4096 <https://github.com/apache/beam/pull/4096> so that the Gradle build files merged and then follow up with new Jenkins precommits which are powered by Gradle. This will allow the community to continuing contributing to the Gradle build and also allow for a comparison of the precommit times on the Jenkins executor when using Maven/Gradle. I suggest that those who are interested try out the PR. On Fri, Nov 3, 2017 at 10:29 PM, Jean-Baptiste Onofré <[email protected]> wrote: > That makes sense. The point is that we have to compare equivalently. I'm > also curious about Gradle PoC assuming it does the same actions as Maven. > > Regards > JB > > On Nov 3, 2017, 20:41, at 20:41, Kenneth Knowles <[email protected]> > wrote: > >I'm confident that any choice will speed things up dramatically even > >beyond > >a fast profile, even if the new tool runs all the extra stuff. But that > >is > >a question that we can answer empirically anyhow. Let's see how it > >goes! > > > >Incidentally, my experiments with Bazel have led me to the conclusion > >that > >it is not the right choice for us so I'm not going to be proposing any > >completed POC of that right now. I'm interested in the outcome of the > >Gradle POC. > > > >Kenn > > > > > >On Fri, Nov 3, 2017 at 3:30 AM, Jean-Baptiste Onofré <[email protected]> > >wrote: > > > >> Hi > >> > >> It's what I said in a previous e-mail: I don't think that just > >changing > >> the build tool will improve a lot the build time. > >> > >> We already know (and discussed while ago) that plugins like findbugs, > >> checkstyle, etc are taking time. > >> > >> So, I think we can already have a fast profile. > >> > >> Regards > >> JB > >> > >> On Nov 3, 2017, 11:16, at 11:16, Romain Manni-Bucau > ><[email protected]> > >> wrote: > >> >Hi guys, > >> > > >> >when you check the duration of each mojo of the build (almost since > >> >python part of the build just breaks it locally) you see that there > >is > >> >no real link with maven for the perf issues beam can encounter: > >> >https://gist.github.com/rmannibucau/f65fdde28d5dab0fdac50633f84554c9 > >> >(generated from the profiling of tesla-profile and parsed with > >> > >>https://gist.github.com/rmannibucau/e329d54b8af6c009f46fd151d10037ad) > >> > > >> >Before PoC-ing other tools which will end up to either have the same > >> >issues if the other builds do the same things (test, checkstyle, > >> >enforcer, findbugs, ...) or have a less reliable build (trying to > >skip > >> >some parts of the build if "untouched" - note that this is a very > >hard > >> >issue since static code anaylizis doesn't give you any guarantee of > >> >what it does with modern code - then maybe some action can be taken > >on > >> >the current build: > >> > > >> >- testing https://github.com/vackosar/gitflow-incremental-builder or > >> >https://github.com/khmarbaise/incremental-module-builder maybe or do > >> >the same kind of extension including the beam needs (/!\ the > >previous > >> >warning is still accurate and requires a full run at some point to > >> >validate the graph detection algorithm didn't get abused by some > >> >indirect code dependency) > >> >- maybe try to get rid of some shades (it is a bit crazy ATM to have > >> >so much shades no?) > >> >- the CI can have profiles based on a PR convention (name of the > >> >branch?) to select the build profile, for instance > >> >fb/elasticsearch_super-nice-PR would build only the elasticsearch > >> >modules, jenkins/travis have this ability since they support > >scripting > >> >- document how to setup a "fastBuild" profile in its settings.xml > >> >which bypasses checkstyle, enforcer plugin, findbugs, etc... for > >fast > >> >development iterations > >> > > >> > > >> > > >> > > >> >Romain Manni-Bucau > >> >@rmannibucau | Blog | Old Blog | Github | LinkedIn > >> > > >> > > >> >2017-11-01 21:02 GMT+01:00 Kenneth Knowles <[email protected]>: > >> >> I have started one, here: > >> >https://github.com/kennknowles/beam/commits/bazel. > >> >> It is not nearly as far along as Luke's. For the POC I am just > >> >putting > >> >> things in one root BUILD, and learning where we might find the > >> >necessary > >> >> plugins as I go. I am happy to grant push access to this branch. > >It > >> >would > >> >> be superb if you had some time to work through the Python steps. > >> >> > >> >> On Wed, Nov 1, 2017 at 10:09 AM, Ahmet Altay > >> ><[email protected]> > >> >> wrote: > >> >> > >> >>> Has anyone started a POC with Bazel? I would be interested in > >> >helping that > >> >>> effort. > >> >>> > >> >>> On Wed, Nov 1, 2017 at 9:27 AM, Lukasz Cwik > >> ><[email protected]> > >> >>> wrote: > >> >>> > >> >>> > I have started a POC for using Gradle here: > >> >>> > https://github.com/lukecwik/incubator-beam/tree/gradle > >> >>> > > >> >>> > Things that work: > >> >>> > * compiling all Java code (src/main and src/test) > >> >>> > * generating source from protos > >> >>> > * generating source from avro > >> >>> > * running rat, checkstyle > >> >>> > > >> >>> > Partially working: > >> >>> > * generating maven pom (albeit with wrong dependencies for some > >> >>> > subprojects) > >> >>> > * running tests (~80% pass, remainder seem to be dependency > >> >related but > >> >>> are > >> >>> > uninvestigated) > >> >>> > > >> >>> > Things that don't work: > >> >>> > * anything Python/Go/Docker compilation related > >> >>> > * many tests fail because I messed up dependencies > >> >>> > * anything shading related > >> >>> > * minor plugins like eclipse code formatter/... > >> >>> > * running @NeedsRunner/@ValidatesRunner/integration tests > >> >>> > > >> >>> > Feel free to reach out to me on Slack if you would like to try > >to > >> >tackle > >> >>> a > >> >>> > piece of the POC to prevent duplication of effort from anyone > >> >working on > >> >>> > it. > >> >>> > > >> >>> > > >> >>> > > >> >>> > On Tue, Oct 31, 2017 at 10:25 PM, Jean-Baptiste Onofré > >> ><[email protected]> > >> >>> > wrote: > >> >>> > > >> >>> > > Agree to move forward on a PoC. > >> >>> > > > >> >>> > > Thanks Reuven for bringing discussion on the mailing list ! > >> >>> > > > >> >>> > > Regards > >> >>> > > JB > >> >>> > > > >> >>> > > On Nov 1, 2017, 03:20, at 03:20, Reuven Lax > >> ><[email protected]> > >> >>> > > wrote: > >> >>> > > >Some good discussion here, and thanks to JB and Romain for > >> >adding to > >> >>> > > >it! > >> >>> > > > > >> >>> > > >JB makes the good point that we still need to release Maven > >> >artifacts, > >> >>> > > >as > >> >>> > > >many Beam users want to develop using Maven. So none of this > >> >>> discussion > >> >>> > > >will affect our release process, as we still need Maven > >> >"releases." > >> >>> > > > > >> >>> > > >At this point, if people are interested, I see no harm in > >> >prototyping. > >> >>> > > >Having working alternatives will give us a better basis for > >> >comparison > >> >>> > > >to > >> >>> > > >understand whether these other build systems give us > >anything > >> >over > >> >>> what > >> >>> > > >Maven does. > >> >>> > > > > >> >>> > > >Reuven > >> >>> > > > > >> >>> > > >On Tue, Oct 31, 2017 at 11:05 AM, Charles Chen > >> ><[email protected] > >> >>> > > >> >>> > > >wrote: > >> >>> > > > > >> >>> > > >> As a contributor to the Beam Python SDK, I noticed that > >many > >> >of the > >> >>> > > >points > >> >>> > > >> above regarding Maven and Gradle pertain mostly to Java > >SDK > >> >>> > > >development. > >> >>> > > >> For Python development, Maven is much less natural, and we > >> >end up > >> >>> > > >just > >> >>> > > >> shelling out to perform builds and tests. For Python SDK > >> >(and > >> >>> > > >upcoming Go > >> >>> > > >> SDK development), an option to use Bazel would be quite > >> >useful. > >> >>> > > >> > >> >>> > > >> On Tue, Oct 31, 2017 at 10:42 AM Robert Bradshaw > >> >>> > > >> <[email protected]> wrote: > >> >>> > > >> > >> >>> > > >> > +1, Maven is both a build tool and a repository, and the > >> >latter is > >> >>> > > >> > essential to keep. Both Gradel and Bazel can interface > >with > >> >this > >> >>> > > >> > repository. > >> >>> > > >> > > >> >>> > > >> > I am, however, very supportive of moving away from Maven > >to > >> >a tool > >> >>> > > >> > that supports correct incremental, hermetic, > >> >dependency-driven, > >> >>> > > >> > multi-langauge, and hopefully fast builds for our own > >> >development. > >> >>> > > >> > > >> >>> > > >> > On Tue, Oct 31, 2017 at 10:00 AM, Kenneth Knowles > >> >>> > > >> > <[email protected]> wrote: > >> >>> > > >> > > Echoing what JB and Reuven said, we absolutely must > >> >provide > >> >>> maven > >> >>> > > >> central > >> >>> > > >> > > artifacts for Java users, just as we provide pypi > >> >artifacts for > >> >>> > > >Python > >> >>> > > >> > > users. > >> >>> > > >> > > > >> >>> > > >> > > I see Maven as still a viable tool for single-module > >Java > >> >>> builds, > >> >>> > > >> > > especially considering its rich plugin ecosystem. > >> >>> > > >> > > > >> >>> > > >> > > On Mon, Oct 30, 2017 at 11:27 PM, Reuven Lax > >> >>> > > ><[email protected] > >> >>> > > >> > > >> >>> > > >> > > wrote: > >> >>> > > >> > > > >> >>> > > >> > >> I think that's a very good point. No matter what > >build > >> >system > >> >>> we > >> >>> > > >use > >> >>> > > >> for > >> >>> > > >> > >> our own personal development, we still need to > >release > >> >Maven > >> >>> > > >artifacts > >> >>> > > >> > and > >> >>> > > >> > >> releases as we need to support our users using Maven. > >> >>> > > >> > >> > >> >>> > > >> > >> On Mon, Oct 30, 2017 at 11:26 PM, Jean-Baptiste > >Onofré < > >> >>> > > >> [email protected] > >> >>> > > >> > > > >> >>> > > >> > >> wrote: > >> >>> > > >> > >> > >> >>> > > >> > >> > Generally speaking, it's interesting to evaluate > >> >>> alternatives, > >> >>> > > >> > especially > >> >>> > > >> > >> > Gradle. My point is also to keep Maven artifacts > >and > >> >>> > > >"releases" as > >> >>> > > >> > most > >> >>> > > >> > >> of > >> >>> > > >> > >> > our users will use Maven. > >> >>> > > >> > >> > For incremental build, afair, there's some > >> >enhancements on > >> >>> > > >Maven > >> >>> > > >> but I > >> >>> > > >> > >> > have to take a look. > >> >>> > > >> > >> > > >> >>> > > >> > >> > Regards > >> >>> > > >> > >> > JB > >> >>> > > >> > >> > > >> >>> > > >> > >> > On Oct 31, 2017, 07:22, at 07:22, Eugene Kirpichov > >> >>> > > >> > >> > <[email protected]> wrote: > >> >>> > > >> > >> > >Hi! > >> >>> > > >> > >> > > > >> >>> > > >> > >> > >Many of these points sound valid, but AFAICT Maven > >> >doesn't > >> >>> > > >really > >> >>> > > >> do > >> >>> > > >> > >> > >incremental builds [1]. The best it can do is, it > >> >seems, > >> >>> > > >recompile > >> >>> > > >> > only > >> >>> > > >> > >> > >changed files, but Java compilation is a tiny part > >of > >> >the > >> >>> > > >overall > >> >>> > > >> > >> > >build. > >> >>> > > >> > >> > > > >> >>> > > >> > >> > >Almost all time is taken by other plugins, such as > >> >unit > >> >>> > > >testing or > >> >>> > > >> > >> > >findbugs > >> >>> > > >> > >> > >- and Maven does not seem to currently support > >> >features such > >> >>> > > >as "do > >> >>> > > >> > not > >> >>> > > >> > >> > >rerun unit tests of a module if the code didn't > >> >change". > >> >>> > > >> > >> > > > >> >>> > > >> > >> > >The fact that the surefire plugin has existed for > >>11 > >> >years > >> >>> > > >> (version > >> >>> > > >> > >> > >2.0 > >> >>> > > >> > >> > >was released in 2006) and still doesn't have this > >> >feature > >> >>> > > >makes me > >> >>> > > >> > >> > >think > >> >>> > > >> > >> > >that it's unlikely to be supported in the next few > >> >years > >> >>> > > >either. > >> >>> > > >> > >> > > > >> >>> > > >> > >> > >I suspect most PRs affect a very small number of > >> >modules, so > >> >>> > > >I > >> >>> > > >> think > >> >>> > > >> > >> > >the > >> >>> > > >> > >> > >performance advantage of a build system truly > >> >supporting > >> >>> > > >> incremental > >> >>> > > >> > >> > >builds > >> >>> > > >> > >> > >may be so overwhelming as to trump many other > >> >factors. Of > >> >>> > > >course, > >> >>> > > >> > we'd > >> >>> > > >> > >> > >need > >> >>> > > >> > >> > >to prototype and have hard numbers in hand to > >discuss > >> >this > >> >>> > > >with > >> >>> > > >> more > >> >>> > > >> > >> > >substance. > >> >>> > > >> > >> > > > >> >>> > > >> > >> > >[1] > >> >>> > > >> > >> > > >> >>https://stackoverflow.com/questions/8918165/does-maven- > >> >>> > > >> > >> > support-incremental-builds > >> >>> > > >> > >> > > > >> >>> > > >> > >> > >On Mon, Oct 30, 2017 at 10:57 PM Romain > >Manni-Bucau > >> >>> > > >> > >> > ><[email protected]> > >> >>> > > >> > >> > >wrote: > >> >>> > > >> > >> > > > >> >>> > > >> > >> > >> Hi > >> >>> > > >> > >> > >> > >> >>> > > >> > >> > >> Even if not a commiter or even PMC, I'd like to > >> >mention a > >> >>> > > >few > >> >>> > > >> > points > >> >>> > > >> > >> > >from > >> >>> > > >> > >> > >> an external eye: > >> >>> > > >> > >> > >> > >> >>> > > >> > >> > >> - Maven stays the most common build tool and > >easier > >> >one > >> >>> for > >> >>> > > >any > >> >>> > > >> > user. > >> >>> > > >> > >> > >It > >> >>> > > >> > >> > >> means it is the best one to hope contributions > >> >IMHO. > >> >>> > > >> > >> > >> - Maven has incremental support but if there is > >any > >> >>> blocker > >> >>> > > >the > >> >>> > > >> > >> > >community > >> >>> > > >> > >> > >> is probably ready to enhance it (has been done > >for > >> >>> compiler > >> >>> > > >> plugin > >> >>> > > >> > >> > >for > >> >>> > > >> > >> > >> instance) > >> >>> > > >> > >> > >> - Gradle hides issues easily with its daemon so > >a > >> >build > >> >>> > > >without > >> >>> > > >> > >> > >daemon is > >> >>> > > >> > >> > >> needed > >> >>> > > >> > >> > >> - Gradle doesnt isolate plugins well enough so > >> >ensure your > >> >>> > > >> planned > >> >>> > > >> > >> > >plugins > >> >>> > > >> > >> > >> doesnt conflict > >> >>> > > >> > >> > >> - Only Maven is correctly supported in > >mainstream > >> >and > >> >>> > > >OS/free IDE > >> >>> > > >> > >> > >> > >> >>> > > >> > >> > >> This is the reasons why I think Maven is better > >- > >> >not even > >> >>> > > >> entering > >> >>> > > >> > >> > >into > >> >>> > > >> > >> > >> the ASF points. > >> >>> > > >> > >> > >> > >> >>> > > >> > >> > >> Now Maven is not perfect but some quick > >> >enhancements can > >> >>> be > >> >>> > > >done: > >> >>> > > >> > >> > >> > >> >>> > > >> > >> > >> - A fast build profile can be created > >> >>> > > >> > >> > >> - Takari scheduler can be used yo enhance the > >> >parallel > >> >>> > > >build > >> >>> > > >> > >> > >> - Scripts can be provided to build a subpart of > >the > >> >>> project > >> >>> > > >> > >> > >> - A beam extension can surely be done to > >optimize > >> >or > >> >>> > > >compute the > >> >>> > > >> > >> > >reactors > >> >>> > > >> > >> > >> more easily based on module names > >> >>> > > >> > >> > >> > >> >>> > > >> > >> > >> Romain > >> >>> > > >> > >> > >> > >> >>> > > >> > >> > >> Le 31 oct. 2017 06:42, "Jean-Baptiste Onofré" > >> >>> > > ><[email protected]> > >> >>> > > >> a > >> >>> > > >> > >> > >écrit : > >> >>> > > >> > >> > >> > >> >>> > > >> > >> > >> -0 > >> >>> > > >> > >> > >> > >> >>> > > >> > >> > >> For the following reasons reasons: > >> >>> > > >> > >> > >> - maven is a Apache project and we can have > >> >>> > > >support/improvement > >> >>> > > >> > >> > >> - I don't see how another build tool would speed > >up > >> >the > >> >>> > > >build by > >> >>> > > >> > >> > >itself > >> >>> > > >> > >> > >> - Apache default release process is based on > >Maven > >> >>> > > >> > >> > >> > >> >>> > > >> > >> > >> On the other hand, Gradle could be interesting. > >> >Anyway > >> >>> it's > >> >>> > > >> > something > >> >>> > > >> > >> > >to > >> >>> > > >> > >> > >> evaluate. > >> >>> > > >> > >> > >> > >> >>> > > >> > >> > >> Regards > >> >>> > > >> > >> > >> JB > >> >>> > > >> > >> > >> > >> >>> > > >> > >> > >> > >> >>> > > >> > >> > >> On Oct 30, 2017, 18:46, at 18:46, Ted Yu > >> >>> > > ><[email protected]> > >> >>> > > >> > wrote: > >> >>> > > >> > >> > >> >I agree with Ben's comment. > >> >>> > > >> > >> > >> > > >> >>> > > >> > >> > >> >Recently I have been using gradle in another > >> >Apache > >> >>> > > >project and > >> >>> > > >> > >> > >found > >> >>> > > >> > >> > >> >it > >> >>> > > >> > >> > >> >interesting. > >> >>> > > >> > >> > >> > > >> >>> > > >> > >> > >> >Cheers > >> >>> > > >> > >> > >> > >> >>> > > >> > >> > > >> >>> > > >> > >> > >> >>> > > >> > > >> >>> > > >> > >> >>> > > > >> >>> > > >> >>> > >> >
