I wonder if we could prototype both Bazel and Gradle to do a better comparison (and also to compare the results with our current Maven build).
On Mon, Oct 30, 2017 at 10:32 AM, Ben Chambers <[email protected] > wrote: > I think both Gradle and Bazel are worth exploring. Gradle is definitely > more common in the wild, but Bazel may be a better fit for the large > mixture of languages being developed in one codebase within Beam. It might > be helpful for us to list what functionality we want from such a tool, and > then have someone who is interested in this dig in a bit and let us know > more details how well Gradle or Bazel would fit our needs. > > On Mon, Oct 30, 2017 at 10:27 AM Kenneth Knowles <[email protected]> > wrote: > > > I also support exploring a move away from Apache Maven for orchestrating > > our build. > > > > For a single-module project, I still think it can be a good build tool, > and > > we could still use it for this sort of thing, but I think we are > reaching a > > multi-module scale where it does not work well. Almost all of our jobs > > build things that are not needed and run tests that are redundant, and it > > is not easy to do better, even with a sequence of maven commands. > > > > I'd like to lay out what we hope for from a tool. Here's a start: > > > > General: > > > > - Dependency-driven build so devs working on one thing build & test only > > what is needed > > - Supports orchestration across Protobuf, Java, Python, Go, Docker > images > > - Meets devs where they are, letting folks in one language use familiar > > tools > > - Caching across builds as much as possible > > - Easily extensible for when it doesn't have the feature we need > (writing > > a maven plugin is too much, using maven-exec-plugin is too crufty) > > - Preferably a declarative configuration language > > > > Java needs beyond the basics, which could be executed by the orchestrator > > or my module-local mvn builds, etc. > > > > - Pulling deps from maven central and alternate repos > > - Findbugs > > - RAT > > - Dependency rule enforcement > > - IWYU (Include What You Use) > > - Multiple Java versions in same project > > - ASF release workflow > > > > I probably missed some must-haves or nice-to-haves. I'd love to compile > > thoughts on other languages' needs. > > > > Based on these, another project I would consider is Bazel. We could very > > easily use it to orchestrate, but use Maven (or Gradle!) on the leaves. I > > also think that Gradle is also more focused on the JVM ecosystem, so it > is > > not quite as neutral as Bazel, and uses Groovy which is a bit more > esoteric > > than Python for writing Bazel rules. > > > > Kenn > > > > On Mon, Oct 30, 2017 at 9:37 AM, Lukasz Cwik <[email protected]> > > wrote: > > > > > I wanted to make this thread more visible. This discussion stems from > > Ken's > > > thread about Jenkins pre/post commit issues[1]. > > > > > > I did some investigation as for ways to improve the quality of the > signal > > > from Jenkins by trying to modify the Jenkins jobs spawned from Groovy. > I > > > had limited success but everything I felt like I was doing was just > > > patching symptoms of the problem which is that our build is just too > > slow. > > > For example, we keep adding all these profiles to Maven or tuning how a > > > plugin runs to eek out a small decrease in build time. I believe > swapping > > > away from Apache Maven to a build tool which only builds the things > which > > > have changed in a PR would be the best approach. > > > > > > I would suggest that we migrate to Gradle as our build tool. I am > > > suggesting Gradle because: > > > * It is used in lots of open source projects and has a very large > > community > > > behind it. > > > * It has better support for building languages other then Java > > > (PyGradle[2], GoGradle[3], ...) > > > * Its incremental build support works and only builds things that > changed > > > through the use of a build cache. Even without the build cache (or for > > > clean builds), it is much faster. > > > * Apache Infra already has Gradle v4.x installed on the Jenkins > machines. > > > > > > Any alternatives that should be considered or should we stick with > Apache > > > Maven? > > > > > > 1: > > > https://lists.apache.org/thread.html/25311e0e95be5c49afb168d9b4b4d3 > > > 57984c10c39c7b01da8ff3baaf@%3Cdev.beam.apache.org%3E > > > 2: https://github.com/linkedin/pygradle > > > 3: https://github.com/gogradle/gogradle > > > > > >
