Hi guys, when you check the duration of each mojo of the build (almost since python part of the build just breaks it locally) you see that there is no real link with maven for the perf issues beam can encounter: https://gist.github.com/rmannibucau/f65fdde28d5dab0fdac50633f84554c9 (generated from the profiling of tesla-profile and parsed with https://gist.github.com/rmannibucau/e329d54b8af6c009f46fd151d10037ad)
Before PoC-ing other tools which will end up to either have the same issues if the other builds do the same things (test, checkstyle, enforcer, findbugs, ...) or have a less reliable build (trying to skip some parts of the build if "untouched" - note that this is a very hard issue since static code anaylizis doesn't give you any guarantee of what it does with modern code - then maybe some action can be taken on the current build: - testing https://github.com/vackosar/gitflow-incremental-builder or https://github.com/khmarbaise/incremental-module-builder maybe or do the same kind of extension including the beam needs (/!\ the previous warning is still accurate and requires a full run at some point to validate the graph detection algorithm didn't get abused by some indirect code dependency) - maybe try to get rid of some shades (it is a bit crazy ATM to have so much shades no?) - the CI can have profiles based on a PR convention (name of the branch?) to select the build profile, for instance fb/elasticsearch_super-nice-PR would build only the elasticsearch modules, jenkins/travis have this ability since they support scripting - document how to setup a "fastBuild" profile in its settings.xml which bypasses checkstyle, enforcer plugin, findbugs, etc... for fast development iterations Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github | LinkedIn 2017-11-01 21:02 GMT+01:00 Kenneth Knowles <[email protected]>: > I have started one, here: https://github.com/kennknowles/beam/commits/bazel. > It is not nearly as far along as Luke's. For the POC I am just putting > things in one root BUILD, and learning where we might find the necessary > plugins as I go. I am happy to grant push access to this branch. It would > be superb if you had some time to work through the Python steps. > > On Wed, Nov 1, 2017 at 10:09 AM, Ahmet Altay <[email protected]> > wrote: > >> Has anyone started a POC with Bazel? I would be interested in helping that >> effort. >> >> On Wed, Nov 1, 2017 at 9:27 AM, Lukasz Cwik <[email protected]> >> wrote: >> >> > I have started a POC for using Gradle here: >> > https://github.com/lukecwik/incubator-beam/tree/gradle >> > >> > Things that work: >> > * compiling all Java code (src/main and src/test) >> > * generating source from protos >> > * generating source from avro >> > * running rat, checkstyle >> > >> > Partially working: >> > * generating maven pom (albeit with wrong dependencies for some >> > subprojects) >> > * running tests (~80% pass, remainder seem to be dependency related but >> are >> > uninvestigated) >> > >> > Things that don't work: >> > * anything Python/Go/Docker compilation related >> > * many tests fail because I messed up dependencies >> > * anything shading related >> > * minor plugins like eclipse code formatter/... >> > * running @NeedsRunner/@ValidatesRunner/integration tests >> > >> > Feel free to reach out to me on Slack if you would like to try to tackle >> a >> > piece of the POC to prevent duplication of effort from anyone working on >> > it. >> > >> > >> > >> > On Tue, Oct 31, 2017 at 10:25 PM, Jean-Baptiste Onofré <[email protected]> >> > wrote: >> > >> > > Agree to move forward on a PoC. >> > > >> > > Thanks Reuven for bringing discussion on the mailing list ! >> > > >> > > Regards >> > > JB >> > > >> > > On Nov 1, 2017, 03:20, at 03:20, Reuven Lax <[email protected]> >> > > wrote: >> > > >Some good discussion here, and thanks to JB and Romain for adding to >> > > >it! >> > > > >> > > >JB makes the good point that we still need to release Maven artifacts, >> > > >as >> > > >many Beam users want to develop using Maven. So none of this >> discussion >> > > >will affect our release process, as we still need Maven "releases." >> > > > >> > > >At this point, if people are interested, I see no harm in prototyping. >> > > >Having working alternatives will give us a better basis for comparison >> > > >to >> > > >understand whether these other build systems give us anything over >> what >> > > >Maven does. >> > > > >> > > >Reuven >> > > > >> > > >On Tue, Oct 31, 2017 at 11:05 AM, Charles Chen <[email protected] >> > >> > > >wrote: >> > > > >> > > >> As a contributor to the Beam Python SDK, I noticed that many of the >> > > >points >> > > >> above regarding Maven and Gradle pertain mostly to Java SDK >> > > >development. >> > > >> For Python development, Maven is much less natural, and we end up >> > > >just >> > > >> shelling out to perform builds and tests. For Python SDK (and >> > > >upcoming Go >> > > >> SDK development), an option to use Bazel would be quite useful. >> > > >> >> > > >> On Tue, Oct 31, 2017 at 10:42 AM Robert Bradshaw >> > > >> <[email protected]> wrote: >> > > >> >> > > >> > +1, Maven is both a build tool and a repository, and the latter is >> > > >> > essential to keep. Both Gradel and Bazel can interface with this >> > > >> > repository. >> > > >> > >> > > >> > I am, however, very supportive of moving away from Maven to a tool >> > > >> > that supports correct incremental, hermetic, dependency-driven, >> > > >> > multi-langauge, and hopefully fast builds for our own development. >> > > >> > >> > > >> > On Tue, Oct 31, 2017 at 10:00 AM, Kenneth Knowles >> > > >> > <[email protected]> wrote: >> > > >> > > Echoing what JB and Reuven said, we absolutely must provide >> maven >> > > >> central >> > > >> > > artifacts for Java users, just as we provide pypi artifacts for >> > > >Python >> > > >> > > users. >> > > >> > > >> > > >> > > I see Maven as still a viable tool for single-module Java >> builds, >> > > >> > > especially considering its rich plugin ecosystem. >> > > >> > > >> > > >> > > On Mon, Oct 30, 2017 at 11:27 PM, Reuven Lax >> > > ><[email protected] >> > > >> > >> > > >> > > wrote: >> > > >> > > >> > > >> > >> I think that's a very good point. No matter what build system >> we >> > > >use >> > > >> for >> > > >> > >> our own personal development, we still need to release Maven >> > > >artifacts >> > > >> > and >> > > >> > >> releases as we need to support our users using Maven. >> > > >> > >> >> > > >> > >> On Mon, Oct 30, 2017 at 11:26 PM, Jean-Baptiste Onofré < >> > > >> [email protected] >> > > >> > > >> > > >> > >> wrote: >> > > >> > >> >> > > >> > >> > Generally speaking, it's interesting to evaluate >> alternatives, >> > > >> > especially >> > > >> > >> > Gradle. My point is also to keep Maven artifacts and >> > > >"releases" as >> > > >> > most >> > > >> > >> of >> > > >> > >> > our users will use Maven. >> > > >> > >> > For incremental build, afair, there's some enhancements on >> > > >Maven >> > > >> but I >> > > >> > >> > have to take a look. >> > > >> > >> > >> > > >> > >> > Regards >> > > >> > >> > JB >> > > >> > >> > >> > > >> > >> > On Oct 31, 2017, 07:22, at 07:22, Eugene Kirpichov >> > > >> > >> > <[email protected]> wrote: >> > > >> > >> > >Hi! >> > > >> > >> > > >> > > >> > >> > >Many of these points sound valid, but AFAICT Maven doesn't >> > > >really >> > > >> do >> > > >> > >> > >incremental builds [1]. The best it can do is, it seems, >> > > >recompile >> > > >> > only >> > > >> > >> > >changed files, but Java compilation is a tiny part of the >> > > >overall >> > > >> > >> > >build. >> > > >> > >> > > >> > > >> > >> > >Almost all time is taken by other plugins, such as unit >> > > >testing or >> > > >> > >> > >findbugs >> > > >> > >> > >- and Maven does not seem to currently support features such >> > > >as "do >> > > >> > not >> > > >> > >> > >rerun unit tests of a module if the code didn't change". >> > > >> > >> > > >> > > >> > >> > >The fact that the surefire plugin has existed for >11 years >> > > >> (version >> > > >> > >> > >2.0 >> > > >> > >> > >was released in 2006) and still doesn't have this feature >> > > >makes me >> > > >> > >> > >think >> > > >> > >> > >that it's unlikely to be supported in the next few years >> > > >either. >> > > >> > >> > > >> > > >> > >> > >I suspect most PRs affect a very small number of modules, so >> > > >I >> > > >> think >> > > >> > >> > >the >> > > >> > >> > >performance advantage of a build system truly supporting >> > > >> incremental >> > > >> > >> > >builds >> > > >> > >> > >may be so overwhelming as to trump many other factors. Of >> > > >course, >> > > >> > we'd >> > > >> > >> > >need >> > > >> > >> > >to prototype and have hard numbers in hand to discuss this >> > > >with >> > > >> more >> > > >> > >> > >substance. >> > > >> > >> > > >> > > >> > >> > >[1] >> > > >> > >> > >https://stackoverflow.com/questions/8918165/does-maven- >> > > >> > >> > support-incremental-builds >> > > >> > >> > > >> > > >> > >> > >On Mon, Oct 30, 2017 at 10:57 PM Romain Manni-Bucau >> > > >> > >> > ><[email protected]> >> > > >> > >> > >wrote: >> > > >> > >> > > >> > > >> > >> > >> Hi >> > > >> > >> > >> >> > > >> > >> > >> Even if not a commiter or even PMC, I'd like to mention a >> > > >few >> > > >> > points >> > > >> > >> > >from >> > > >> > >> > >> an external eye: >> > > >> > >> > >> >> > > >> > >> > >> - Maven stays the most common build tool and easier one >> for >> > > >any >> > > >> > user. >> > > >> > >> > >It >> > > >> > >> > >> means it is the best one to hope contributions IMHO. >> > > >> > >> > >> - Maven has incremental support but if there is any >> blocker >> > > >the >> > > >> > >> > >community >> > > >> > >> > >> is probably ready to enhance it (has been done for >> compiler >> > > >> plugin >> > > >> > >> > >for >> > > >> > >> > >> instance) >> > > >> > >> > >> - Gradle hides issues easily with its daemon so a build >> > > >without >> > > >> > >> > >daemon is >> > > >> > >> > >> needed >> > > >> > >> > >> - Gradle doesnt isolate plugins well enough so ensure your >> > > >> planned >> > > >> > >> > >plugins >> > > >> > >> > >> doesnt conflict >> > > >> > >> > >> - Only Maven is correctly supported in mainstream and >> > > >OS/free IDE >> > > >> > >> > >> >> > > >> > >> > >> This is the reasons why I think Maven is better - not even >> > > >> entering >> > > >> > >> > >into >> > > >> > >> > >> the ASF points. >> > > >> > >> > >> >> > > >> > >> > >> Now Maven is not perfect but some quick enhancements can >> be >> > > >done: >> > > >> > >> > >> >> > > >> > >> > >> - A fast build profile can be created >> > > >> > >> > >> - Takari scheduler can be used yo enhance the parallel >> > > >build >> > > >> > >> > >> - Scripts can be provided to build a subpart of the >> project >> > > >> > >> > >> - A beam extension can surely be done to optimize or >> > > >compute the >> > > >> > >> > >reactors >> > > >> > >> > >> more easily based on module names >> > > >> > >> > >> >> > > >> > >> > >> Romain >> > > >> > >> > >> >> > > >> > >> > >> Le 31 oct. 2017 06:42, "Jean-Baptiste Onofré" >> > > ><[email protected]> >> > > >> a >> > > >> > >> > >écrit : >> > > >> > >> > >> >> > > >> > >> > >> -0 >> > > >> > >> > >> >> > > >> > >> > >> For the following reasons reasons: >> > > >> > >> > >> - maven is a Apache project and we can have >> > > >support/improvement >> > > >> > >> > >> - I don't see how another build tool would speed up the >> > > >build by >> > > >> > >> > >itself >> > > >> > >> > >> - Apache default release process is based on Maven >> > > >> > >> > >> >> > > >> > >> > >> On the other hand, Gradle could be interesting. Anyway >> it's >> > > >> > something >> > > >> > >> > >to >> > > >> > >> > >> evaluate. >> > > >> > >> > >> >> > > >> > >> > >> Regards >> > > >> > >> > >> JB >> > > >> > >> > >> >> > > >> > >> > >> >> > > >> > >> > >> On Oct 30, 2017, 18:46, at 18:46, Ted Yu >> > > ><[email protected]> >> > > >> > wrote: >> > > >> > >> > >> >I agree with Ben's comment. >> > > >> > >> > >> > >> > > >> > >> > >> >Recently I have been using gradle in another Apache >> > > >project and >> > > >> > >> > >found >> > > >> > >> > >> >it >> > > >> > >> > >> >interesting. >> > > >> > >> > >> > >> > > >> > >> > >> >Cheers >> > > >> > >> > >> >> > > >> > >> > >> > > >> > >> >> > > >> > >> > > >> >> > > >> > >>
