Please see https://dzone.com/articles/java-command-line-interfaces-part-30-finale-observations
On Tue, Nov 7, 2017 at 6:31 PM, Eugene Kirpichov < [email protected]> wrote: > Wow, these are impressive differences! Do you know why is that so? In > particular I'm surprised at the difference in performance of the Java clean > build: feels like the performed work ought to be the same. Does it all come > from better parallelization? (in Maven I believe we don't specify "-T 1C" > on Jenkins, I'm not sure why - but I typically specify it locally when > verifying PRs, and the build takes ~15 minutes, which is still 2x slower > than 8 minutes you got with Gradle) > > On Tue, Nov 7, 2017 at 6:20 PM Lukasz Cwik <[email protected]> > wrote: > > > The Gradle POC has made significant advances since last week (shading, > > Python, Go, Docker builds, ...). I believe the current state is close > > enough to the Maven build system to warrant a comparison. > > > > The largest build differences I noticed are: > > * Full build takes about ~22mins using Gradle (parallelizing the three > > rounds of Python tests would reduce this to ~17mins) compared to ~38mins > in > > Maven > > * Clean build all Java modules (skipping over Go/Python > > <https://goto.google.com/Python>) takes ~8mins in > > Gradle which takes ~36mins in Maven > > * Build output is cached allowing for faster subsequent builds with > "gradle > > buildDependents" allowing for most single module changes taking ~2mins to > > build and test without needing to rely on "mvn install" > > > > I have opened PR 4096 <https://github.com/apache/beam/pull/4096> so that > > the Gradle build files merged and then follow up with new Jenkins > > precommits which are powered by Gradle. This will allow the community to > > continuing contributing to the Gradle build and also allow for a > comparison > > of the precommit times on the Jenkins executor when using Maven/Gradle. I > > suggest that those who are interested try out the PR. > > > > On Fri, Nov 3, 2017 at 10:29 PM, Jean-Baptiste Onofré <[email protected]> > > wrote: > > > > > That makes sense. The point is that we have to compare equivalently. > I'm > > > also curious about Gradle PoC assuming it does the same actions as > Maven. > > > > > > Regards > > > JB > > > > > > On Nov 3, 2017, 20:41, at 20:41, Kenneth Knowles > <[email protected] > > > > > > wrote: > > > >I'm confident that any choice will speed things up dramatically even > > > >beyond > > > >a fast profile, even if the new tool runs all the extra stuff. But > that > > > >is > > > >a question that we can answer empirically anyhow. Let's see how it > > > >goes! > > > > > > > >Incidentally, my experiments with Bazel have led me to the conclusion > > > >that > > > >it is not the right choice for us so I'm not going to be proposing any > > > >completed POC of that right now. I'm interested in the outcome of the > > > >Gradle POC. > > > > > > > >Kenn > > > > > > > > > > > >On Fri, Nov 3, 2017 at 3:30 AM, Jean-Baptiste Onofré <[email protected] > > > > > >wrote: > > > > > > > >> Hi > > > >> > > > >> It's what I said in a previous e-mail: I don't think that just > > > >changing > > > >> the build tool will improve a lot the build time. > > > >> > > > >> We already know (and discussed while ago) that plugins like > findbugs, > > > >> checkstyle, etc are taking time. > > > >> > > > >> So, I think we can already have a fast profile. > > > >> > > > >> Regards > > > >> JB > > > >> > > > >> On Nov 3, 2017, 11:16, at 11:16, Romain Manni-Bucau > > > ><[email protected]> > > > >> wrote: > > > >> >Hi guys, > > > >> > > > > >> >when you check the duration of each mojo of the build (almost since > > > >> >python part of the build just breaks it locally) you see that there > > > >is > > > >> >no real link with maven for the perf issues beam can encounter: > > > >> >https://gist.github.com/rmannibucau/f65fdde28d5dab0fdac50633f84554 > c9 > > > >> >(generated from the profiling of tesla-profile and parsed with > > > >> > > > >>https://gist.github.com/rmannibucau/e329d54b8af6c009f46fd151d10037ad > ) > > > >> > > > > >> >Before PoC-ing other tools which will end up to either have the > same > > > >> >issues if the other builds do the same things (test, checkstyle, > > > >> >enforcer, findbugs, ...) or have a less reliable build (trying to > > > >skip > > > >> >some parts of the build if "untouched" - note that this is a very > > > >hard > > > >> >issue since static code anaylizis doesn't give you any guarantee of > > > >> >what it does with modern code - then maybe some action can be taken > > > >on > > > >> >the current build: > > > >> > > > > >> >- testing https://github.com/vackosar/gitflow-incremental-builder > or > > > >> >https://github.com/khmarbaise/incremental-module-builder maybe or > do > > > >> >the same kind of extension including the beam needs (/!\ the > > > >previous > > > >> >warning is still accurate and requires a full run at some point to > > > >> >validate the graph detection algorithm didn't get abused by some > > > >> >indirect code dependency) > > > >> >- maybe try to get rid of some shades (it is a bit crazy ATM to > have > > > >> >so much shades no?) > > > >> >- the CI can have profiles based on a PR convention (name of the > > > >> >branch?) to select the build profile, for instance > > > >> >fb/elasticsearch_super-nice-PR would build only the elasticsearch > > > >> >modules, jenkins/travis have this ability since they support > > > >scripting > > > >> >- document how to setup a "fastBuild" profile in its settings.xml > > > >> >which bypasses checkstyle, enforcer plugin, findbugs, etc... for > > > >fast > > > >> >development iterations > > > >> > > > > >> > > > > >> > > > > >> > > > > >> >Romain Manni-Bucau > > > >> >@rmannibucau | Blog | Old Blog | Github | LinkedIn > > > >> > > > > >> > > > > >> >2017-11-01 21:02 GMT+01:00 Kenneth Knowles <[email protected] > >: > > > >> >> I have started one, here: > > > >> >https://github.com/kennknowles/beam/commits/bazel. > > > >> >> It is not nearly as far along as Luke's. For the POC I am just > > > >> >putting > > > >> >> things in one root BUILD, and learning where we might find the > > > >> >necessary > > > >> >> plugins as I go. I am happy to grant push access to this branch. > > > >It > > > >> >would > > > >> >> be superb if you had some time to work through the Python steps. > > > >> >> > > > >> >> On Wed, Nov 1, 2017 at 10:09 AM, Ahmet Altay > > > >> ><[email protected]> > > > >> >> wrote: > > > >> >> > > > >> >>> Has anyone started a POC with Bazel? I would be interested in > > > >> >helping that > > > >> >>> effort. > > > >> >>> > > > >> >>> On Wed, Nov 1, 2017 at 9:27 AM, Lukasz Cwik > > > >> ><[email protected]> > > > >> >>> wrote: > > > >> >>> > > > >> >>> > I have started a POC for using Gradle here: > > > >> >>> > https://github.com/lukecwik/incubator-beam/tree/gradle > > > >> >>> > > > > >> >>> > Things that work: > > > >> >>> > * compiling all Java code (src/main and src/test) > > > >> >>> > * generating source from protos > > > >> >>> > * generating source from avro > > > >> >>> > * running rat, checkstyle > > > >> >>> > > > > >> >>> > Partially working: > > > >> >>> > * generating maven pom (albeit with wrong dependencies for > some > > > >> >>> > subprojects) > > > >> >>> > * running tests (~80% pass, remainder seem to be dependency > > > >> >related but > > > >> >>> are > > > >> >>> > uninvestigated) > > > >> >>> > > > > >> >>> > Things that don't work: > > > >> >>> > * anything Python/Go/Docker compilation related > > > >> >>> > * many tests fail because I messed up dependencies > > > >> >>> > * anything shading related > > > >> >>> > * minor plugins like eclipse code formatter/... > > > >> >>> > * running @NeedsRunner/@ValidatesRunner/integration tests > > > >> >>> > > > > >> >>> > Feel free to reach out to me on Slack if you would like to try > > > >to > > > >> >tackle > > > >> >>> a > > > >> >>> > piece of the POC to prevent duplication of effort from anyone > > > >> >working on > > > >> >>> > it. > > > >> >>> > > > > >> >>> > > > > >> >>> > > > > >> >>> > On Tue, Oct 31, 2017 at 10:25 PM, Jean-Baptiste Onofré > > > >> ><[email protected]> > > > >> >>> > wrote: > > > >> >>> > > > > >> >>> > > Agree to move forward on a PoC. > > > >> >>> > > > > > >> >>> > > Thanks Reuven for bringing discussion on the mailing list ! > > > >> >>> > > > > > >> >>> > > Regards > > > >> >>> > > JB > > > >> >>> > > > > > >> >>> > > On Nov 1, 2017, 03:20, at 03:20, Reuven Lax > > > >> ><[email protected]> > > > >> >>> > > wrote: > > > >> >>> > > >Some good discussion here, and thanks to JB and Romain for > > > >> >adding to > > > >> >>> > > >it! > > > >> >>> > > > > > > >> >>> > > >JB makes the good point that we still need to release Maven > > > >> >artifacts, > > > >> >>> > > >as > > > >> >>> > > >many Beam users want to develop using Maven. So none of > this > > > >> >>> discussion > > > >> >>> > > >will affect our release process, as we still need Maven > > > >> >"releases." > > > >> >>> > > > > > > >> >>> > > >At this point, if people are interested, I see no harm in > > > >> >prototyping. > > > >> >>> > > >Having working alternatives will give us a better basis for > > > >> >comparison > > > >> >>> > > >to > > > >> >>> > > >understand whether these other build systems give us > > > >anything > > > >> >over > > > >> >>> what > > > >> >>> > > >Maven does. > > > >> >>> > > > > > > >> >>> > > >Reuven > > > >> >>> > > > > > > >> >>> > > >On Tue, Oct 31, 2017 at 11:05 AM, Charles Chen > > > >> ><[email protected] > > > >> >>> > > > > >> >>> > > >wrote: > > > >> >>> > > > > > > >> >>> > > >> As a contributor to the Beam Python SDK, I noticed that > > > >many > > > >> >of the > > > >> >>> > > >points > > > >> >>> > > >> above regarding Maven and Gradle pertain mostly to Java > > > >SDK > > > >> >>> > > >development. > > > >> >>> > > >> For Python development, Maven is much less natural, and > we > > > >> >end up > > > >> >>> > > >just > > > >> >>> > > >> shelling out to perform builds and tests. For Python SDK > > > >> >(and > > > >> >>> > > >upcoming Go > > > >> >>> > > >> SDK development), an option to use Bazel would be quite > > > >> >useful. > > > >> >>> > > >> > > > >> >>> > > >> On Tue, Oct 31, 2017 at 10:42 AM Robert Bradshaw > > > >> >>> > > >> <[email protected]> wrote: > > > >> >>> > > >> > > > >> >>> > > >> > +1, Maven is both a build tool and a repository, and > the > > > >> >latter is > > > >> >>> > > >> > essential to keep. Both Gradel and Bazel can interface > > > >with > > > >> >this > > > >> >>> > > >> > repository. > > > >> >>> > > >> > > > > >> >>> > > >> > I am, however, very supportive of moving away from > Maven > > > >to > > > >> >a tool > > > >> >>> > > >> > that supports correct incremental, hermetic, > > > >> >dependency-driven, > > > >> >>> > > >> > multi-langauge, and hopefully fast builds for our own > > > >> >development. > > > >> >>> > > >> > > > > >> >>> > > >> > On Tue, Oct 31, 2017 at 10:00 AM, Kenneth Knowles > > > >> >>> > > >> > <[email protected]> wrote: > > > >> >>> > > >> > > Echoing what JB and Reuven said, we absolutely must > > > >> >provide > > > >> >>> maven > > > >> >>> > > >> central > > > >> >>> > > >> > > artifacts for Java users, just as we provide pypi > > > >> >artifacts for > > > >> >>> > > >Python > > > >> >>> > > >> > > users. > > > >> >>> > > >> > > > > > >> >>> > > >> > > I see Maven as still a viable tool for single-module > > > >Java > > > >> >>> builds, > > > >> >>> > > >> > > especially considering its rich plugin ecosystem. > > > >> >>> > > >> > > > > > >> >>> > > >> > > On Mon, Oct 30, 2017 at 11:27 PM, Reuven Lax > > > >> >>> > > ><[email protected] > > > >> >>> > > >> > > > > >> >>> > > >> > > wrote: > > > >> >>> > > >> > > > > > >> >>> > > >> > >> I think that's a very good point. No matter what > > > >build > > > >> >system > > > >> >>> we > > > >> >>> > > >use > > > >> >>> > > >> for > > > >> >>> > > >> > >> our own personal development, we still need to > > > >release > > > >> >Maven > > > >> >>> > > >artifacts > > > >> >>> > > >> > and > > > >> >>> > > >> > >> releases as we need to support our users using > Maven. > > > >> >>> > > >> > >> > > > >> >>> > > >> > >> On Mon, Oct 30, 2017 at 11:26 PM, Jean-Baptiste > > > >Onofré < > > > >> >>> > > >> [email protected] > > > >> >>> > > >> > > > > > >> >>> > > >> > >> wrote: > > > >> >>> > > >> > >> > > > >> >>> > > >> > >> > Generally speaking, it's interesting to evaluate > > > >> >>> alternatives, > > > >> >>> > > >> > especially > > > >> >>> > > >> > >> > Gradle. My point is also to keep Maven artifacts > > > >and > > > >> >>> > > >"releases" as > > > >> >>> > > >> > most > > > >> >>> > > >> > >> of > > > >> >>> > > >> > >> > our users will use Maven. > > > >> >>> > > >> > >> > For incremental build, afair, there's some > > > >> >enhancements on > > > >> >>> > > >Maven > > > >> >>> > > >> but I > > > >> >>> > > >> > >> > have to take a look. > > > >> >>> > > >> > >> > > > > >> >>> > > >> > >> > Regards > > > >> >>> > > >> > >> > JB > > > >> >>> > > >> > >> > > > > >> >>> > > >> > >> > On Oct 31, 2017, 07:22, at 07:22, Eugene Kirpichov > > > >> >>> > > >> > >> > <[email protected]> wrote: > > > >> >>> > > >> > >> > >Hi! > > > >> >>> > > >> > >> > > > > > >> >>> > > >> > >> > >Many of these points sound valid, but AFAICT > Maven > > > >> >doesn't > > > >> >>> > > >really > > > >> >>> > > >> do > > > >> >>> > > >> > >> > >incremental builds [1]. The best it can do is, it > > > >> >seems, > > > >> >>> > > >recompile > > > >> >>> > > >> > only > > > >> >>> > > >> > >> > >changed files, but Java compilation is a tiny > part > > > >of > > > >> >the > > > >> >>> > > >overall > > > >> >>> > > >> > >> > >build. > > > >> >>> > > >> > >> > > > > > >> >>> > > >> > >> > >Almost all time is taken by other plugins, such > as > > > >> >unit > > > >> >>> > > >testing or > > > >> >>> > > >> > >> > >findbugs > > > >> >>> > > >> > >> > >- and Maven does not seem to currently support > > > >> >features such > > > >> >>> > > >as "do > > > >> >>> > > >> > not > > > >> >>> > > >> > >> > >rerun unit tests of a module if the code didn't > > > >> >change". > > > >> >>> > > >> > >> > > > > > >> >>> > > >> > >> > >The fact that the surefire plugin has existed for > > > >>11 > > > >> >years > > > >> >>> > > >> (version > > > >> >>> > > >> > >> > >2.0 > > > >> >>> > > >> > >> > >was released in 2006) and still doesn't have this > > > >> >feature > > > >> >>> > > >makes me > > > >> >>> > > >> > >> > >think > > > >> >>> > > >> > >> > >that it's unlikely to be supported in the next > few > > > >> >years > > > >> >>> > > >either. > > > >> >>> > > >> > >> > > > > > >> >>> > > >> > >> > >I suspect most PRs affect a very small number of > > > >> >modules, so > > > >> >>> > > >I > > > >> >>> > > >> think > > > >> >>> > > >> > >> > >the > > > >> >>> > > >> > >> > >performance advantage of a build system truly > > > >> >supporting > > > >> >>> > > >> incremental > > > >> >>> > > >> > >> > >builds > > > >> >>> > > >> > >> > >may be so overwhelming as to trump many other > > > >> >factors. Of > > > >> >>> > > >course, > > > >> >>> > > >> > we'd > > > >> >>> > > >> > >> > >need > > > >> >>> > > >> > >> > >to prototype and have hard numbers in hand to > > > >discuss > > > >> >this > > > >> >>> > > >with > > > >> >>> > > >> more > > > >> >>> > > >> > >> > >substance. > > > >> >>> > > >> > >> > > > > > >> >>> > > >> > >> > >[1] > > > >> >>> > > >> > >> > > > > >> >>https://stackoverflow.com/questions/8918165/does-maven- > > > >> >>> > > >> > >> > support-incremental-builds > > > >> >>> > > >> > >> > > > > > >> >>> > > >> > >> > >On Mon, Oct 30, 2017 at 10:57 PM Romain > > > >Manni-Bucau > > > >> >>> > > >> > >> > ><[email protected]> > > > >> >>> > > >> > >> > >wrote: > > > >> >>> > > >> > >> > > > > > >> >>> > > >> > >> > >> Hi > > > >> >>> > > >> > >> > >> > > > >> >>> > > >> > >> > >> Even if not a commiter or even PMC, I'd like to > > > >> >mention a > > > >> >>> > > >few > > > >> >>> > > >> > points > > > >> >>> > > >> > >> > >from > > > >> >>> > > >> > >> > >> an external eye: > > > >> >>> > > >> > >> > >> > > > >> >>> > > >> > >> > >> - Maven stays the most common build tool and > > > >easier > > > >> >one > > > >> >>> for > > > >> >>> > > >any > > > >> >>> > > >> > user. > > > >> >>> > > >> > >> > >It > > > >> >>> > > >> > >> > >> means it is the best one to hope contributions > > > >> >IMHO. > > > >> >>> > > >> > >> > >> - Maven has incremental support but if there is > > > >any > > > >> >>> blocker > > > >> >>> > > >the > > > >> >>> > > >> > >> > >community > > > >> >>> > > >> > >> > >> is probably ready to enhance it (has been done > > > >for > > > >> >>> compiler > > > >> >>> > > >> plugin > > > >> >>> > > >> > >> > >for > > > >> >>> > > >> > >> > >> instance) > > > >> >>> > > >> > >> > >> - Gradle hides issues easily with its daemon so > > > >a > > > >> >build > > > >> >>> > > >without > > > >> >>> > > >> > >> > >daemon is > > > >> >>> > > >> > >> > >> needed > > > >> >>> > > >> > >> > >> - Gradle doesnt isolate plugins well enough so > > > >> >ensure your > > > >> >>> > > >> planned > > > >> >>> > > >> > >> > >plugins > > > >> >>> > > >> > >> > >> doesnt conflict > > > >> >>> > > >> > >> > >> - Only Maven is correctly supported in > > > >mainstream > > > >> >and > > > >> >>> > > >OS/free IDE > > > >> >>> > > >> > >> > >> > > > >> >>> > > >> > >> > >> This is the reasons why I think Maven is better > > > >- > > > >> >not even > > > >> >>> > > >> entering > > > >> >>> > > >> > >> > >into > > > >> >>> > > >> > >> > >> the ASF points. > > > >> >>> > > >> > >> > >> > > > >> >>> > > >> > >> > >> Now Maven is not perfect but some quick > > > >> >enhancements can > > > >> >>> be > > > >> >>> > > >done: > > > >> >>> > > >> > >> > >> > > > >> >>> > > >> > >> > >> - A fast build profile can be created > > > >> >>> > > >> > >> > >> - Takari scheduler can be used yo enhance the > > > >> >parallel > > > >> >>> > > >build > > > >> >>> > > >> > >> > >> - Scripts can be provided to build a subpart of > > > >the > > > >> >>> project > > > >> >>> > > >> > >> > >> - A beam extension can surely be done to > > > >optimize > > > >> >or > > > >> >>> > > >compute the > > > >> >>> > > >> > >> > >reactors > > > >> >>> > > >> > >> > >> more easily based on module names > > > >> >>> > > >> > >> > >> > > > >> >>> > > >> > >> > >> Romain > > > >> >>> > > >> > >> > >> > > > >> >>> > > >> > >> > >> Le 31 oct. 2017 06:42, "Jean-Baptiste Onofré" > > > >> >>> > > ><[email protected]> > > > >> >>> > > >> a > > > >> >>> > > >> > >> > >écrit : > > > >> >>> > > >> > >> > >> > > > >> >>> > > >> > >> > >> -0 > > > >> >>> > > >> > >> > >> > > > >> >>> > > >> > >> > >> For the following reasons reasons: > > > >> >>> > > >> > >> > >> - maven is a Apache project and we can have > > > >> >>> > > >support/improvement > > > >> >>> > > >> > >> > >> - I don't see how another build tool would > speed > > > >up > > > >> >the > > > >> >>> > > >build by > > > >> >>> > > >> > >> > >itself > > > >> >>> > > >> > >> > >> - Apache default release process is based on > > > >Maven > > > >> >>> > > >> > >> > >> > > > >> >>> > > >> > >> > >> On the other hand, Gradle could be interesting. > > > >> >Anyway > > > >> >>> it's > > > >> >>> > > >> > something > > > >> >>> > > >> > >> > >to > > > >> >>> > > >> > >> > >> evaluate. > > > >> >>> > > >> > >> > >> > > > >> >>> > > >> > >> > >> Regards > > > >> >>> > > >> > >> > >> JB > > > >> >>> > > >> > >> > >> > > > >> >>> > > >> > >> > >> > > > >> >>> > > >> > >> > >> On Oct 30, 2017, 18:46, at 18:46, Ted Yu > > > >> >>> > > ><[email protected]> > > > >> >>> > > >> > wrote: > > > >> >>> > > >> > >> > >> >I agree with Ben's comment. > > > >> >>> > > >> > >> > >> > > > > >> >>> > > >> > >> > >> >Recently I have been using gradle in another > > > >> >Apache > > > >> >>> > > >project and > > > >> >>> > > >> > >> > >found > > > >> >>> > > >> > >> > >> >it > > > >> >>> > > >> > >> > >> >interesting. > > > >> >>> > > >> > >> > >> > > > > >> >>> > > >> > >> > >> >Cheers > > > >> >>> > > >> > >> > >> > > > >> >>> > > >> > >> > > > > >> >>> > > >> > >> > > > >> >>> > > >> > > > > >> >>> > > >> > > > >> >>> > > > > > >> >>> > > > > >> >>> > > > >> > > > > > >
