Great !
What explain these difference ? I'm curious especially for the clean build all
Java modules: is it a question of parallel execution ?
Regards
JB
On 11/08/2017 02:59 AM, Lukasz Cwik wrote:
The Gradle POC has made significant advances since last week (shading,
Python, Go, Docker builds, ...). I believe the current state is close
enough to the Maven build system to warrant a comparison.
The largest build differences I noticed are:
* Full build takes about ~22mins using Gradle (parallelizing the three
rounds of Python tests would reduce this to ~17mins) compared to ~38mins in
Maven
* Clean build all Java modules (skipping over Go/Python) takes ~8mins in
Gradle which takes ~36mins in Maven
* Build output is cached allowing for faster subsequent builds with "gradle
buildDependents" allowing for most single module changes taking ~2mins to
build and test without needing to rely on "mvn install"
I have opened PR 4096 <https://github.com/apache/beam/pull/4096> so that
the Gradle build files merged and then follow up with new Jenkins
precommits which are powered by Gradle. This will allow the community to
continuing contributing to the Gradle build and also allow for a comparison
of the precommit times on the Jenkins executor when using Maven/Gradle. I
suggest that those who are interested try out the PR.
On Fri, Nov 3, 2017 at 10:29 PM, Jean-Baptiste Onofré <[email protected]>
wrote:
That makes sense. The point is that we have to compare equivalently. I'm
also curious about Gradle PoC assuming it does the same actions as Maven.
Regards
JB
On Nov 3, 2017, 20:41, at 20:41, Kenneth Knowles <[email protected]>
wrote:
I'm confident that any choice will speed things up dramatically even
beyond
a fast profile, even if the new tool runs all the extra stuff. But that
is
a question that we can answer empirically anyhow. Let's see how it
goes!
Incidentally, my experiments with Bazel have led me to the conclusion
that
it is not the right choice for us so I'm not going to be proposing any
completed POC of that right now. I'm interested in the outcome of the
Gradle POC.
Kenn
On Fri, Nov 3, 2017 at 3:30 AM, Jean-Baptiste Onofré <[email protected]>
wrote:
Hi
It's what I said in a previous e-mail: I don't think that just
changing
the build tool will improve a lot the build time.
We already know (and discussed while ago) that plugins like findbugs,
checkstyle, etc are taking time.
So, I think we can already have a fast profile.
Regards
JB
On Nov 3, 2017, 11:16, at 11:16, Romain Manni-Bucau
<[email protected]>
wrote:
Hi guys,
when you check the duration of each mojo of the build (almost since
python part of the build just breaks it locally) you see that there
is
no real link with maven for the perf issues beam can encounter:
https://gist.github.com/rmannibucau/f65fdde28d5dab0fdac50633f84554c9
(generated from the profiling of tesla-profile and parsed with
https://gist.github.com/rmannibucau/e329d54b8af6c009f46fd151d10037ad)
Before PoC-ing other tools which will end up to either have the same
issues if the other builds do the same things (test, checkstyle,
enforcer, findbugs, ...) or have a less reliable build (trying to
skip
some parts of the build if "untouched" - note that this is a very
hard
issue since static code anaylizis doesn't give you any guarantee of
what it does with modern code - then maybe some action can be taken
on
the current build:
- testing https://github.com/vackosar/gitflow-incremental-builder or
https://github.com/khmarbaise/incremental-module-builder maybe or do
the same kind of extension including the beam needs (/!\ the
previous
warning is still accurate and requires a full run at some point to
validate the graph detection algorithm didn't get abused by some
indirect code dependency)
- maybe try to get rid of some shades (it is a bit crazy ATM to have
so much shades no?)
- the CI can have profiles based on a PR convention (name of the
branch?) to select the build profile, for instance
fb/elasticsearch_super-nice-PR would build only the elasticsearch
modules, jenkins/travis have this ability since they support
scripting
- document how to setup a "fastBuild" profile in its settings.xml
which bypasses checkstyle, enforcer plugin, findbugs, etc... for
fast
development iterations
Romain Manni-Bucau
@rmannibucau | Blog | Old Blog | Github | LinkedIn
2017-11-01 21:02 GMT+01:00 Kenneth Knowles <[email protected]>:
I have started one, here:
https://github.com/kennknowles/beam/commits/bazel.
It is not nearly as far along as Luke's. For the POC I am just
putting
things in one root BUILD, and learning where we might find the
necessary
plugins as I go. I am happy to grant push access to this branch.
It
would
be superb if you had some time to work through the Python steps.
On Wed, Nov 1, 2017 at 10:09 AM, Ahmet Altay
<[email protected]>
wrote:
Has anyone started a POC with Bazel? I would be interested in
helping that
effort.
On Wed, Nov 1, 2017 at 9:27 AM, Lukasz Cwik
<[email protected]>
wrote:
I have started a POC for using Gradle here:
https://github.com/lukecwik/incubator-beam/tree/gradle
Things that work:
* compiling all Java code (src/main and src/test)
* generating source from protos
* generating source from avro
* running rat, checkstyle
Partially working:
* generating maven pom (albeit with wrong dependencies for some
subprojects)
* running tests (~80% pass, remainder seem to be dependency
related but
are
uninvestigated)
Things that don't work:
* anything Python/Go/Docker compilation related
* many tests fail because I messed up dependencies
* anything shading related
* minor plugins like eclipse code formatter/...
* running @NeedsRunner/@ValidatesRunner/integration tests
Feel free to reach out to me on Slack if you would like to try
to
tackle
a
piece of the POC to prevent duplication of effort from anyone
working on
it.
On Tue, Oct 31, 2017 at 10:25 PM, Jean-Baptiste Onofré
<[email protected]>
wrote:
Agree to move forward on a PoC.
Thanks Reuven for bringing discussion on the mailing list !
Regards
JB
On Nov 1, 2017, 03:20, at 03:20, Reuven Lax
<[email protected]>
wrote:
Some good discussion here, and thanks to JB and Romain for
adding to
it!
JB makes the good point that we still need to release Maven
artifacts,
as
many Beam users want to develop using Maven. So none of this
discussion
will affect our release process, as we still need Maven
"releases."
At this point, if people are interested, I see no harm in
prototyping.
Having working alternatives will give us a better basis for
comparison
to
understand whether these other build systems give us
anything
over
what
Maven does.
Reuven
On Tue, Oct 31, 2017 at 11:05 AM, Charles Chen
<[email protected]
wrote:
As a contributor to the Beam Python SDK, I noticed that
many
of the
points
above regarding Maven and Gradle pertain mostly to Java
SDK
development.
For Python development, Maven is much less natural, and we
end up
just
shelling out to perform builds and tests. For Python SDK
(and
upcoming Go
SDK development), an option to use Bazel would be quite
useful.
On Tue, Oct 31, 2017 at 10:42 AM Robert Bradshaw
<[email protected]> wrote:
+1, Maven is both a build tool and a repository, and the
latter is
essential to keep. Both Gradel and Bazel can interface
with
this
repository.
I am, however, very supportive of moving away from Maven
to
a tool
that supports correct incremental, hermetic,
dependency-driven,
multi-langauge, and hopefully fast builds for our own
development.
On Tue, Oct 31, 2017 at 10:00 AM, Kenneth Knowles
<[email protected]> wrote:
Echoing what JB and Reuven said, we absolutely must
provide
maven
central
artifacts for Java users, just as we provide pypi
artifacts for
Python
users.
I see Maven as still a viable tool for single-module
Java
builds,
especially considering its rich plugin ecosystem.
On Mon, Oct 30, 2017 at 11:27 PM, Reuven Lax
<[email protected]
wrote:
I think that's a very good point. No matter what
build
system
we
use
for
our own personal development, we still need to
release
Maven
artifacts
and
releases as we need to support our users using Maven.
On Mon, Oct 30, 2017 at 11:26 PM, Jean-Baptiste
Onofré <
[email protected]
wrote:
Generally speaking, it's interesting to evaluate
alternatives,
especially
Gradle. My point is also to keep Maven artifacts
and
"releases" as
most
of
our users will use Maven.
For incremental build, afair, there's some
enhancements on
Maven
but I
have to take a look.
Regards
JB
On Oct 31, 2017, 07:22, at 07:22, Eugene Kirpichov
<[email protected]> wrote:
Hi!
Many of these points sound valid, but AFAICT Maven
doesn't
really
do
incremental builds [1]. The best it can do is, it
seems,
recompile
only
changed files, but Java compilation is a tiny part
of
the
overall
build.
Almost all time is taken by other plugins, such as
unit
testing or
findbugs
- and Maven does not seem to currently support
features such
as "do
not
rerun unit tests of a module if the code didn't
change".
The fact that the surefire plugin has existed for
11
years
(version
2.0
was released in 2006) and still doesn't have this
feature
makes me
think
that it's unlikely to be supported in the next few
years
either.
I suspect most PRs affect a very small number of
modules, so
I
think
the
performance advantage of a build system truly
supporting
incremental
builds
may be so overwhelming as to trump many other
factors. Of
course,
we'd
need
to prototype and have hard numbers in hand to
discuss
this
with
more
substance.
[1]
https://stackoverflow.com/questions/8918165/does-maven-
support-incremental-builds
On Mon, Oct 30, 2017 at 10:57 PM Romain
Manni-Bucau
<[email protected]>
wrote:
Hi
Even if not a commiter or even PMC, I'd like to
mention a
few
points
from
an external eye:
- Maven stays the most common build tool and
easier
one
for
any
user.
It
means it is the best one to hope contributions
IMHO.
- Maven has incremental support but if there is
any
blocker
the
community
is probably ready to enhance it (has been done
for
compiler
plugin
for
instance)
- Gradle hides issues easily with its daemon so
a
build
without
daemon is
needed
- Gradle doesnt isolate plugins well enough so
ensure your
planned
plugins
doesnt conflict
- Only Maven is correctly supported in
mainstream
and
OS/free IDE
This is the reasons why I think Maven is better
-
not even
entering
into
the ASF points.
Now Maven is not perfect but some quick
enhancements can
be
done:
- A fast build profile can be created
- Takari scheduler can be used yo enhance the
parallel
build
- Scripts can be provided to build a subpart of
the
project
- A beam extension can surely be done to
optimize
or
compute the
reactors
more easily based on module names
Romain
Le 31 oct. 2017 06:42, "Jean-Baptiste Onofré"
<[email protected]>
a
écrit :
-0
For the following reasons reasons:
- maven is a Apache project and we can have
support/improvement
- I don't see how another build tool would speed
up
the
build by
itself
- Apache default release process is based on
Maven
On the other hand, Gradle could be interesting.
Anyway
it's
something
to
evaluate.
Regards
JB
On Oct 30, 2017, 18:46, at 18:46, Ted Yu
<[email protected]>
wrote:
I agree with Ben's comment.
Recently I have been using gradle in another
Apache
project and
found
it
interesting.
Cheers
--
Jean-Baptiste Onofré
[email protected]
http://blog.nanthrax.net
Talend - http://www.talend.com
