I have started one, here: https://github.com/kennknowles/beam/commits/bazel. It is not nearly as far along as Luke's. For the POC I am just putting things in one root BUILD, and learning where we might find the necessary plugins as I go. I am happy to grant push access to this branch. It would be superb if you had some time to work through the Python steps.
On Wed, Nov 1, 2017 at 10:09 AM, Ahmet Altay <[email protected]> wrote: > Has anyone started a POC with Bazel? I would be interested in helping that > effort. > > On Wed, Nov 1, 2017 at 9:27 AM, Lukasz Cwik <[email protected]> > wrote: > > > I have started a POC for using Gradle here: > > https://github.com/lukecwik/incubator-beam/tree/gradle > > > > Things that work: > > * compiling all Java code (src/main and src/test) > > * generating source from protos > > * generating source from avro > > * running rat, checkstyle > > > > Partially working: > > * generating maven pom (albeit with wrong dependencies for some > > subprojects) > > * running tests (~80% pass, remainder seem to be dependency related but > are > > uninvestigated) > > > > Things that don't work: > > * anything Python/Go/Docker compilation related > > * many tests fail because I messed up dependencies > > * anything shading related > > * minor plugins like eclipse code formatter/... > > * running @NeedsRunner/@ValidatesRunner/integration tests > > > > Feel free to reach out to me on Slack if you would like to try to tackle > a > > piece of the POC to prevent duplication of effort from anyone working on > > it. > > > > > > > > On Tue, Oct 31, 2017 at 10:25 PM, Jean-Baptiste Onofré <[email protected]> > > wrote: > > > > > Agree to move forward on a PoC. > > > > > > Thanks Reuven for bringing discussion on the mailing list ! > > > > > > Regards > > > JB > > > > > > On Nov 1, 2017, 03:20, at 03:20, Reuven Lax <[email protected]> > > > wrote: > > > >Some good discussion here, and thanks to JB and Romain for adding to > > > >it! > > > > > > > >JB makes the good point that we still need to release Maven artifacts, > > > >as > > > >many Beam users want to develop using Maven. So none of this > discussion > > > >will affect our release process, as we still need Maven "releases." > > > > > > > >At this point, if people are interested, I see no harm in prototyping. > > > >Having working alternatives will give us a better basis for comparison > > > >to > > > >understand whether these other build systems give us anything over > what > > > >Maven does. > > > > > > > >Reuven > > > > > > > >On Tue, Oct 31, 2017 at 11:05 AM, Charles Chen <[email protected] > > > > > >wrote: > > > > > > > >> As a contributor to the Beam Python SDK, I noticed that many of the > > > >points > > > >> above regarding Maven and Gradle pertain mostly to Java SDK > > > >development. > > > >> For Python development, Maven is much less natural, and we end up > > > >just > > > >> shelling out to perform builds and tests. For Python SDK (and > > > >upcoming Go > > > >> SDK development), an option to use Bazel would be quite useful. > > > >> > > > >> On Tue, Oct 31, 2017 at 10:42 AM Robert Bradshaw > > > >> <[email protected]> wrote: > > > >> > > > >> > +1, Maven is both a build tool and a repository, and the latter is > > > >> > essential to keep. Both Gradel and Bazel can interface with this > > > >> > repository. > > > >> > > > > >> > I am, however, very supportive of moving away from Maven to a tool > > > >> > that supports correct incremental, hermetic, dependency-driven, > > > >> > multi-langauge, and hopefully fast builds for our own development. > > > >> > > > > >> > On Tue, Oct 31, 2017 at 10:00 AM, Kenneth Knowles > > > >> > <[email protected]> wrote: > > > >> > > Echoing what JB and Reuven said, we absolutely must provide > maven > > > >> central > > > >> > > artifacts for Java users, just as we provide pypi artifacts for > > > >Python > > > >> > > users. > > > >> > > > > > >> > > I see Maven as still a viable tool for single-module Java > builds, > > > >> > > especially considering its rich plugin ecosystem. > > > >> > > > > > >> > > On Mon, Oct 30, 2017 at 11:27 PM, Reuven Lax > > > ><[email protected] > > > >> > > > > >> > > wrote: > > > >> > > > > > >> > >> I think that's a very good point. No matter what build system > we > > > >use > > > >> for > > > >> > >> our own personal development, we still need to release Maven > > > >artifacts > > > >> > and > > > >> > >> releases as we need to support our users using Maven. > > > >> > >> > > > >> > >> On Mon, Oct 30, 2017 at 11:26 PM, Jean-Baptiste Onofré < > > > >> [email protected] > > > >> > > > > > >> > >> wrote: > > > >> > >> > > > >> > >> > Generally speaking, it's interesting to evaluate > alternatives, > > > >> > especially > > > >> > >> > Gradle. My point is also to keep Maven artifacts and > > > >"releases" as > > > >> > most > > > >> > >> of > > > >> > >> > our users will use Maven. > > > >> > >> > For incremental build, afair, there's some enhancements on > > > >Maven > > > >> but I > > > >> > >> > have to take a look. > > > >> > >> > > > > >> > >> > Regards > > > >> > >> > JB > > > >> > >> > > > > >> > >> > On Oct 31, 2017, 07:22, at 07:22, Eugene Kirpichov > > > >> > >> > <[email protected]> wrote: > > > >> > >> > >Hi! > > > >> > >> > > > > > >> > >> > >Many of these points sound valid, but AFAICT Maven doesn't > > > >really > > > >> do > > > >> > >> > >incremental builds [1]. The best it can do is, it seems, > > > >recompile > > > >> > only > > > >> > >> > >changed files, but Java compilation is a tiny part of the > > > >overall > > > >> > >> > >build. > > > >> > >> > > > > > >> > >> > >Almost all time is taken by other plugins, such as unit > > > >testing or > > > >> > >> > >findbugs > > > >> > >> > >- and Maven does not seem to currently support features such > > > >as "do > > > >> > not > > > >> > >> > >rerun unit tests of a module if the code didn't change". > > > >> > >> > > > > > >> > >> > >The fact that the surefire plugin has existed for >11 years > > > >> (version > > > >> > >> > >2.0 > > > >> > >> > >was released in 2006) and still doesn't have this feature > > > >makes me > > > >> > >> > >think > > > >> > >> > >that it's unlikely to be supported in the next few years > > > >either. > > > >> > >> > > > > > >> > >> > >I suspect most PRs affect a very small number of modules, so > > > >I > > > >> think > > > >> > >> > >the > > > >> > >> > >performance advantage of a build system truly supporting > > > >> incremental > > > >> > >> > >builds > > > >> > >> > >may be so overwhelming as to trump many other factors. Of > > > >course, > > > >> > we'd > > > >> > >> > >need > > > >> > >> > >to prototype and have hard numbers in hand to discuss this > > > >with > > > >> more > > > >> > >> > >substance. > > > >> > >> > > > > > >> > >> > >[1] > > > >> > >> > >https://stackoverflow.com/questions/8918165/does-maven- > > > >> > >> > support-incremental-builds > > > >> > >> > > > > > >> > >> > >On Mon, Oct 30, 2017 at 10:57 PM Romain Manni-Bucau > > > >> > >> > ><[email protected]> > > > >> > >> > >wrote: > > > >> > >> > > > > > >> > >> > >> Hi > > > >> > >> > >> > > > >> > >> > >> Even if not a commiter or even PMC, I'd like to mention a > > > >few > > > >> > points > > > >> > >> > >from > > > >> > >> > >> an external eye: > > > >> > >> > >> > > > >> > >> > >> - Maven stays the most common build tool and easier one > for > > > >any > > > >> > user. > > > >> > >> > >It > > > >> > >> > >> means it is the best one to hope contributions IMHO. > > > >> > >> > >> - Maven has incremental support but if there is any > blocker > > > >the > > > >> > >> > >community > > > >> > >> > >> is probably ready to enhance it (has been done for > compiler > > > >> plugin > > > >> > >> > >for > > > >> > >> > >> instance) > > > >> > >> > >> - Gradle hides issues easily with its daemon so a build > > > >without > > > >> > >> > >daemon is > > > >> > >> > >> needed > > > >> > >> > >> - Gradle doesnt isolate plugins well enough so ensure your > > > >> planned > > > >> > >> > >plugins > > > >> > >> > >> doesnt conflict > > > >> > >> > >> - Only Maven is correctly supported in mainstream and > > > >OS/free IDE > > > >> > >> > >> > > > >> > >> > >> This is the reasons why I think Maven is better - not even > > > >> entering > > > >> > >> > >into > > > >> > >> > >> the ASF points. > > > >> > >> > >> > > > >> > >> > >> Now Maven is not perfect but some quick enhancements can > be > > > >done: > > > >> > >> > >> > > > >> > >> > >> - A fast build profile can be created > > > >> > >> > >> - Takari scheduler can be used yo enhance the parallel > > > >build > > > >> > >> > >> - Scripts can be provided to build a subpart of the > project > > > >> > >> > >> - A beam extension can surely be done to optimize or > > > >compute the > > > >> > >> > >reactors > > > >> > >> > >> more easily based on module names > > > >> > >> > >> > > > >> > >> > >> Romain > > > >> > >> > >> > > > >> > >> > >> Le 31 oct. 2017 06:42, "Jean-Baptiste Onofré" > > > ><[email protected]> > > > >> a > > > >> > >> > >écrit : > > > >> > >> > >> > > > >> > >> > >> -0 > > > >> > >> > >> > > > >> > >> > >> For the following reasons reasons: > > > >> > >> > >> - maven is a Apache project and we can have > > > >support/improvement > > > >> > >> > >> - I don't see how another build tool would speed up the > > > >build by > > > >> > >> > >itself > > > >> > >> > >> - Apache default release process is based on Maven > > > >> > >> > >> > > > >> > >> > >> On the other hand, Gradle could be interesting. Anyway > it's > > > >> > something > > > >> > >> > >to > > > >> > >> > >> evaluate. > > > >> > >> > >> > > > >> > >> > >> Regards > > > >> > >> > >> JB > > > >> > >> > >> > > > >> > >> > >> > > > >> > >> > >> On Oct 30, 2017, 18:46, at 18:46, Ted Yu > > > ><[email protected]> > > > >> > wrote: > > > >> > >> > >> >I agree with Ben's comment. > > > >> > >> > >> > > > > >> > >> > >> >Recently I have been using gradle in another Apache > > > >project and > > > >> > >> > >found > > > >> > >> > >> >it > > > >> > >> > >> >interesting. > > > >> > >> > >> > > > > >> > >> > >> >Cheers > > > >> > >> > >> > > > >> > >> > > > > >> > >> > > > >> > > > > >> > > > > > >
