Hi It's what I said in a previous e-mail: I don't think that just changing the build tool will improve a lot the build time.
We already know (and discussed while ago) that plugins like findbugs, checkstyle, etc are taking time. So, I think we can already have a fast profile. Regards JB On Nov 3, 2017, 11:16, at 11:16, Romain Manni-Bucau <rmannibu...@gmail.com> wrote: >Hi guys, > >when you check the duration of each mojo of the build (almost since >python part of the build just breaks it locally) you see that there is >no real link with maven for the perf issues beam can encounter: >https://gist.github.com/rmannibucau/f65fdde28d5dab0fdac50633f84554c9 >(generated from the profiling of tesla-profile and parsed with >https://gist.github.com/rmannibucau/e329d54b8af6c009f46fd151d10037ad) > >Before PoC-ing other tools which will end up to either have the same >issues if the other builds do the same things (test, checkstyle, >enforcer, findbugs, ...) or have a less reliable build (trying to skip >some parts of the build if "untouched" - note that this is a very hard >issue since static code anaylizis doesn't give you any guarantee of >what it does with modern code - then maybe some action can be taken on >the current build: > >- testing https://github.com/vackosar/gitflow-incremental-builder or >https://github.com/khmarbaise/incremental-module-builder maybe or do >the same kind of extension including the beam needs (/!\ the previous >warning is still accurate and requires a full run at some point to >validate the graph detection algorithm didn't get abused by some >indirect code dependency) >- maybe try to get rid of some shades (it is a bit crazy ATM to have >so much shades no?) >- the CI can have profiles based on a PR convention (name of the >branch?) to select the build profile, for instance >fb/elasticsearch_super-nice-PR would build only the elasticsearch >modules, jenkins/travis have this ability since they support scripting >- document how to setup a "fastBuild" profile in its settings.xml >which bypasses checkstyle, enforcer plugin, findbugs, etc... for fast >development iterations > > > > >Romain Manni-Bucau >@rmannibucau | Blog | Old Blog | Github | LinkedIn > > >2017-11-01 21:02 GMT+01:00 Kenneth Knowles <k...@google.com.invalid>: >> I have started one, here: >https://github.com/kennknowles/beam/commits/bazel. >> It is not nearly as far along as Luke's. For the POC I am just >putting >> things in one root BUILD, and learning where we might find the >necessary >> plugins as I go. I am happy to grant push access to this branch. It >would >> be superb if you had some time to work through the Python steps. >> >> On Wed, Nov 1, 2017 at 10:09 AM, Ahmet Altay ><al...@google.com.invalid> >> wrote: >> >>> Has anyone started a POC with Bazel? I would be interested in >helping that >>> effort. >>> >>> On Wed, Nov 1, 2017 at 9:27 AM, Lukasz Cwik ><lc...@google.com.invalid> >>> wrote: >>> >>> > I have started a POC for using Gradle here: >>> > https://github.com/lukecwik/incubator-beam/tree/gradle >>> > >>> > Things that work: >>> > * compiling all Java code (src/main and src/test) >>> > * generating source from protos >>> > * generating source from avro >>> > * running rat, checkstyle >>> > >>> > Partially working: >>> > * generating maven pom (albeit with wrong dependencies for some >>> > subprojects) >>> > * running tests (~80% pass, remainder seem to be dependency >related but >>> are >>> > uninvestigated) >>> > >>> > Things that don't work: >>> > * anything Python/Go/Docker compilation related >>> > * many tests fail because I messed up dependencies >>> > * anything shading related >>> > * minor plugins like eclipse code formatter/... >>> > * running @NeedsRunner/@ValidatesRunner/integration tests >>> > >>> > Feel free to reach out to me on Slack if you would like to try to >tackle >>> a >>> > piece of the POC to prevent duplication of effort from anyone >working on >>> > it. >>> > >>> > >>> > >>> > On Tue, Oct 31, 2017 at 10:25 PM, Jean-Baptiste Onofré ><j...@nanthrax.net> >>> > wrote: >>> > >>> > > Agree to move forward on a PoC. >>> > > >>> > > Thanks Reuven for bringing discussion on the mailing list ! >>> > > >>> > > Regards >>> > > JB >>> > > >>> > > On Nov 1, 2017, 03:20, at 03:20, Reuven Lax ><re...@google.com.INVALID> >>> > > wrote: >>> > > >Some good discussion here, and thanks to JB and Romain for >adding to >>> > > >it! >>> > > > >>> > > >JB makes the good point that we still need to release Maven >artifacts, >>> > > >as >>> > > >many Beam users want to develop using Maven. So none of this >>> discussion >>> > > >will affect our release process, as we still need Maven >"releases." >>> > > > >>> > > >At this point, if people are interested, I see no harm in >prototyping. >>> > > >Having working alternatives will give us a better basis for >comparison >>> > > >to >>> > > >understand whether these other build systems give us anything >over >>> what >>> > > >Maven does. >>> > > > >>> > > >Reuven >>> > > > >>> > > >On Tue, Oct 31, 2017 at 11:05 AM, Charles Chen ><c...@google.com.invalid >>> > >>> > > >wrote: >>> > > > >>> > > >> As a contributor to the Beam Python SDK, I noticed that many >of the >>> > > >points >>> > > >> above regarding Maven and Gradle pertain mostly to Java SDK >>> > > >development. >>> > > >> For Python development, Maven is much less natural, and we >end up >>> > > >just >>> > > >> shelling out to perform builds and tests. For Python SDK >(and >>> > > >upcoming Go >>> > > >> SDK development), an option to use Bazel would be quite >useful. >>> > > >> >>> > > >> On Tue, Oct 31, 2017 at 10:42 AM Robert Bradshaw >>> > > >> <rober...@google.com.invalid> wrote: >>> > > >> >>> > > >> > +1, Maven is both a build tool and a repository, and the >latter is >>> > > >> > essential to keep. Both Gradel and Bazel can interface with >this >>> > > >> > repository. >>> > > >> > >>> > > >> > I am, however, very supportive of moving away from Maven to >a tool >>> > > >> > that supports correct incremental, hermetic, >dependency-driven, >>> > > >> > multi-langauge, and hopefully fast builds for our own >development. >>> > > >> > >>> > > >> > On Tue, Oct 31, 2017 at 10:00 AM, Kenneth Knowles >>> > > >> > <k...@google.com.invalid> wrote: >>> > > >> > > Echoing what JB and Reuven said, we absolutely must >provide >>> maven >>> > > >> central >>> > > >> > > artifacts for Java users, just as we provide pypi >artifacts for >>> > > >Python >>> > > >> > > users. >>> > > >> > > >>> > > >> > > I see Maven as still a viable tool for single-module Java >>> builds, >>> > > >> > > especially considering its rich plugin ecosystem. >>> > > >> > > >>> > > >> > > On Mon, Oct 30, 2017 at 11:27 PM, Reuven Lax >>> > > ><re...@google.com.invalid >>> > > >> > >>> > > >> > > wrote: >>> > > >> > > >>> > > >> > >> I think that's a very good point. No matter what build >system >>> we >>> > > >use >>> > > >> for >>> > > >> > >> our own personal development, we still need to release >Maven >>> > > >artifacts >>> > > >> > and >>> > > >> > >> releases as we need to support our users using Maven. >>> > > >> > >> >>> > > >> > >> On Mon, Oct 30, 2017 at 11:26 PM, Jean-Baptiste Onofré < >>> > > >> j...@nanthrax.net >>> > > >> > > >>> > > >> > >> wrote: >>> > > >> > >> >>> > > >> > >> > Generally speaking, it's interesting to evaluate >>> alternatives, >>> > > >> > especially >>> > > >> > >> > Gradle. My point is also to keep Maven artifacts and >>> > > >"releases" as >>> > > >> > most >>> > > >> > >> of >>> > > >> > >> > our users will use Maven. >>> > > >> > >> > For incremental build, afair, there's some >enhancements on >>> > > >Maven >>> > > >> but I >>> > > >> > >> > have to take a look. >>> > > >> > >> > >>> > > >> > >> > Regards >>> > > >> > >> > JB >>> > > >> > >> > >>> > > >> > >> > On Oct 31, 2017, 07:22, at 07:22, Eugene Kirpichov >>> > > >> > >> > <kirpic...@google.com.INVALID> wrote: >>> > > >> > >> > >Hi! >>> > > >> > >> > > >>> > > >> > >> > >Many of these points sound valid, but AFAICT Maven >doesn't >>> > > >really >>> > > >> do >>> > > >> > >> > >incremental builds [1]. The best it can do is, it >seems, >>> > > >recompile >>> > > >> > only >>> > > >> > >> > >changed files, but Java compilation is a tiny part of >the >>> > > >overall >>> > > >> > >> > >build. >>> > > >> > >> > > >>> > > >> > >> > >Almost all time is taken by other plugins, such as >unit >>> > > >testing or >>> > > >> > >> > >findbugs >>> > > >> > >> > >- and Maven does not seem to currently support >features such >>> > > >as "do >>> > > >> > not >>> > > >> > >> > >rerun unit tests of a module if the code didn't >change". >>> > > >> > >> > > >>> > > >> > >> > >The fact that the surefire plugin has existed for >11 >years >>> > > >> (version >>> > > >> > >> > >2.0 >>> > > >> > >> > >was released in 2006) and still doesn't have this >feature >>> > > >makes me >>> > > >> > >> > >think >>> > > >> > >> > >that it's unlikely to be supported in the next few >years >>> > > >either. >>> > > >> > >> > > >>> > > >> > >> > >I suspect most PRs affect a very small number of >modules, so >>> > > >I >>> > > >> think >>> > > >> > >> > >the >>> > > >> > >> > >performance advantage of a build system truly >supporting >>> > > >> incremental >>> > > >> > >> > >builds >>> > > >> > >> > >may be so overwhelming as to trump many other >factors. Of >>> > > >course, >>> > > >> > we'd >>> > > >> > >> > >need >>> > > >> > >> > >to prototype and have hard numbers in hand to discuss >this >>> > > >with >>> > > >> more >>> > > >> > >> > >substance. >>> > > >> > >> > > >>> > > >> > >> > >[1] >>> > > >> > >> > >>https://stackoverflow.com/questions/8918165/does-maven- >>> > > >> > >> > support-incremental-builds >>> > > >> > >> > > >>> > > >> > >> > >On Mon, Oct 30, 2017 at 10:57 PM Romain Manni-Bucau >>> > > >> > >> > ><rmannibu...@gmail.com> >>> > > >> > >> > >wrote: >>> > > >> > >> > > >>> > > >> > >> > >> Hi >>> > > >> > >> > >> >>> > > >> > >> > >> Even if not a commiter or even PMC, I'd like to >mention a >>> > > >few >>> > > >> > points >>> > > >> > >> > >from >>> > > >> > >> > >> an external eye: >>> > > >> > >> > >> >>> > > >> > >> > >> - Maven stays the most common build tool and easier >one >>> for >>> > > >any >>> > > >> > user. >>> > > >> > >> > >It >>> > > >> > >> > >> means it is the best one to hope contributions >IMHO. >>> > > >> > >> > >> - Maven has incremental support but if there is any >>> blocker >>> > > >the >>> > > >> > >> > >community >>> > > >> > >> > >> is probably ready to enhance it (has been done for >>> compiler >>> > > >> plugin >>> > > >> > >> > >for >>> > > >> > >> > >> instance) >>> > > >> > >> > >> - Gradle hides issues easily with its daemon so a >build >>> > > >without >>> > > >> > >> > >daemon is >>> > > >> > >> > >> needed >>> > > >> > >> > >> - Gradle doesnt isolate plugins well enough so >ensure your >>> > > >> planned >>> > > >> > >> > >plugins >>> > > >> > >> > >> doesnt conflict >>> > > >> > >> > >> - Only Maven is correctly supported in mainstream >and >>> > > >OS/free IDE >>> > > >> > >> > >> >>> > > >> > >> > >> This is the reasons why I think Maven is better - >not even >>> > > >> entering >>> > > >> > >> > >into >>> > > >> > >> > >> the ASF points. >>> > > >> > >> > >> >>> > > >> > >> > >> Now Maven is not perfect but some quick >enhancements can >>> be >>> > > >done: >>> > > >> > >> > >> >>> > > >> > >> > >> - A fast build profile can be created >>> > > >> > >> > >> - Takari scheduler can be used yo enhance the >parallel >>> > > >build >>> > > >> > >> > >> - Scripts can be provided to build a subpart of the >>> project >>> > > >> > >> > >> - A beam extension can surely be done to optimize >or >>> > > >compute the >>> > > >> > >> > >reactors >>> > > >> > >> > >> more easily based on module names >>> > > >> > >> > >> >>> > > >> > >> > >> Romain >>> > > >> > >> > >> >>> > > >> > >> > >> Le 31 oct. 2017 06:42, "Jean-Baptiste Onofré" >>> > > ><j...@nanthrax.net> >>> > > >> a >>> > > >> > >> > >écrit : >>> > > >> > >> > >> >>> > > >> > >> > >> -0 >>> > > >> > >> > >> >>> > > >> > >> > >> For the following reasons reasons: >>> > > >> > >> > >> - maven is a Apache project and we can have >>> > > >support/improvement >>> > > >> > >> > >> - I don't see how another build tool would speed up >the >>> > > >build by >>> > > >> > >> > >itself >>> > > >> > >> > >> - Apache default release process is based on Maven >>> > > >> > >> > >> >>> > > >> > >> > >> On the other hand, Gradle could be interesting. >Anyway >>> it's >>> > > >> > something >>> > > >> > >> > >to >>> > > >> > >> > >> evaluate. >>> > > >> > >> > >> >>> > > >> > >> > >> Regards >>> > > >> > >> > >> JB >>> > > >> > >> > >> >>> > > >> > >> > >> >>> > > >> > >> > >> On Oct 30, 2017, 18:46, at 18:46, Ted Yu >>> > > ><yuzhih...@gmail.com> >>> > > >> > wrote: >>> > > >> > >> > >> >I agree with Ben's comment. >>> > > >> > >> > >> > >>> > > >> > >> > >> >Recently I have been using gradle in another >Apache >>> > > >project and >>> > > >> > >> > >found >>> > > >> > >> > >> >it >>> > > >> > >> > >> >interesting. >>> > > >> > >> > >> > >>> > > >> > >> > >> >Cheers >>> > > >> > >> > >> >>> > > >> > >> > >>> > > >> > >> >>> > > >> > >>> > > >> >>> > > >>> > >>>
