Re: [DISCUSS] Move away from Apache Maven as build tool

[Romain Manni-Bucau]

@Lukasz: was it tested in current setup? I know groovy does it but never
checked it myself. If not the vote must be conditional to that IMHO.

Le 28 nov. 2017 19:19, "Robert Bradshaw"  a écrit :

> I also did an apache github query
>
> select count(*) as apache_projects, sum(uses_maven=true) as
> uses_maven, sum(uses_gradle=true) as uses_gradle from (
> select
> repo_name,
> max(path contains 'pom.xml') as uses_maven,
> max(path contains 'gradle') as uses_gradle
> from [bigquery-public-data:github_repos.files]
> where instr(repo_name, 'apache/') == 1
> group by repo_name);
>
> Of 425 total apache projects on gitub, just over half (249) use maven,
> and only 25 use gradle. So we'd be in the minority, but certainly not
> alone.
>
> I don't think we need to use the most common tool, rather we should
> use what fits the project well, and the popularity criteria is simply
> that we don't want to choose a tool where obscurity would be a
> hinderance. Both gradle and maven seem to clear this bar (as do a host
> of others that are even more popular, but would be unsuitable for
> other reasons, e.g. plain old make).
>
> We would certainly not switch over to gradle if we couldn't do a
> release. IIRC, there's still some work to be done to push this
> through, but at this point it doesn't seem like there's any reason to
> expect it couldn't be done.
>
> Is there any more data that should be gathered before a vote? (Or
> should the vote perhaps have a "+/-0, need more information [please
> provide details]" option.)
>
>
> On Tue, Nov 28, 2017 at 9:45 AM, Scott Wegner  wrote:
> > To add one more data point measuring general adoption of gradle vs.
> maven,
> > we can look at Stackoverflow trends comparing the two tags [1]. This
> shows
> > the percentage of new SO questions in a given month by tag. 'gradle'
> > represents ~0.25% of questions, while maven is ~0.45%. So, maven is more
> > dominant in the Stackoverflow community, but they are at least similar
> > orders of magnitude. Also, the data is a bit noisy to draw a trendline,
> but
> > it seems that maven's growth has flattened while gradle is still
> increasing.
> >
> > [1] https://insights.stackoverflow.com/trends?tags=maven%2Cgradle
> >
> > On Tue, Nov 28, 2017 at 9:14 AM Kenneth Knowles  wrote:
> >>
> >> Yea, I think voting is the next step. Luke - I think you are obviously
> the
> >> right person to set up the email of what exactly we are voting on, since
> >> you've driven this improvement.
> >>
> >> On Tue, Nov 28, 2017 at 12:08 AM, Robert Bradshaw 
> >> wrote:
> >>>
> >>> It's great to see all the discussion going on here.
> >>>
> >>> I think it's important to point out that merging a parallel set of
> >>> gradle build scripts is a separate (and much less disruptive) step
> >>> than, say, switching over the default (or even recommended)
> >>> build/release process to use them, let alone removing the maven build
> >>> files entirely. The latter two should definitely be gated by a formal
> >>> vote (each, probably), with the current state the gradle files can
> >>> mostly be ignored by most people. In particular, this is the kind of
> >>> change that needs to be in master to be evaluated--if it's on a branch
> >>> we can't very well see how it impacts presubmits, and most importantly
> >>> people can't try it out for real development.
> >>>
> >>> I agree that the choice of build tool may attract some contributors
> >>> and discourage others. Having builds that are fast, correct, and
> >>> reproducible will probably matter more to potential contributors than
> >>> the particular command to run. While maven can surely be improved, I
> >>> doubt a 2x improvement (and many more times that for incremental
> >>> builds) is low-hanging fruit, and many of the issues seem quite
> >>> fundamental (e.g. all the special treatment we need for NeedsRunner
> >>> tests, and having to do a (global-by-default) mvn install to skip
> >>> tests of dependencies when testing a leaf module).
> >>>
> >>> Getting data on what other apache projects use could be interesting,
> >>> but unless we gather why such choices were made I don't know that it'd
> >>> be that influential once we've established that both tools are widely
> >>> supported generally.
> >>>
> >>> To re-emphasize, we'll continue to produce and publish maven
> >>> artifacts, so our choice of build system won't matter for those only
> >>> using Beam as a dependency.
> >>>
> >>>
> >>>
> >>> On Mon, Nov 27, 2017 at 10:48 PM, Jean-Baptiste Onofré <
> j...@nanthrax.net>
> >>> wrote:
> >>> > Yeah, especially, I think it would have been great to have a vote
> >>> > before
> >>> > merging on master.
> >>> >
> >>> > Not a big deal, however, I'm really community focus ;)
> >>> >
> >>> > Regards
> >>> > JB
> >>> >
> >>> > On 11/28/2017 07:36 AM, Reuven Lax wrote:
> >>> >>
> >>> >> Agreed. I thinking having a formal vote before Luke had 

Re: [DISCUSS] Move away from Apache Maven as build tool

[Robert Bradshaw]

I also did an apache github query

select count(*) as apache_projects, sum(uses_maven=true) as
uses_maven, sum(uses_gradle=true) as uses_gradle from (
select
repo_name,
max(path contains 'pom.xml') as uses_maven,
max(path contains 'gradle') as uses_gradle
from [bigquery-public-data:github_repos.files]
where instr(repo_name, 'apache/') == 1
group by repo_name);

Of 425 total apache projects on gitub, just over half (249) use maven,
and only 25 use gradle. So we'd be in the minority, but certainly not
alone.

I don't think we need to use the most common tool, rather we should
use what fits the project well, and the popularity criteria is simply
that we don't want to choose a tool where obscurity would be a
hinderance. Both gradle and maven seem to clear this bar (as do a host
of others that are even more popular, but would be unsuitable for
other reasons, e.g. plain old make).

We would certainly not switch over to gradle if we couldn't do a
release. IIRC, there's still some work to be done to push this
through, but at this point it doesn't seem like there's any reason to
expect it couldn't be done.

Is there any more data that should be gathered before a vote? (Or
should the vote perhaps have a "+/-0, need more information [please
provide details]" option.)


On Tue, Nov 28, 2017 at 9:45 AM, Scott Wegner  wrote:
> To add one more data point measuring general adoption of gradle vs. maven,
> we can look at Stackoverflow trends comparing the two tags [1]. This shows
> the percentage of new SO questions in a given month by tag. 'gradle'
> represents ~0.25% of questions, while maven is ~0.45%. So, maven is more
> dominant in the Stackoverflow community, but they are at least similar
> orders of magnitude. Also, the data is a bit noisy to draw a trendline, but
> it seems that maven's growth has flattened while gradle is still increasing.
>
> [1] https://insights.stackoverflow.com/trends?tags=maven%2Cgradle
>
> On Tue, Nov 28, 2017 at 9:14 AM Kenneth Knowles  wrote:
>>
>> Yea, I think voting is the next step. Luke - I think you are obviously the
>> right person to set up the email of what exactly we are voting on, since
>> you've driven this improvement.
>>
>> On Tue, Nov 28, 2017 at 12:08 AM, Robert Bradshaw 
>> wrote:
>>>
>>> It's great to see all the discussion going on here.
>>>
>>> I think it's important to point out that merging a parallel set of
>>> gradle build scripts is a separate (and much less disruptive) step
>>> than, say, switching over the default (or even recommended)
>>> build/release process to use them, let alone removing the maven build
>>> files entirely. The latter two should definitely be gated by a formal
>>> vote (each, probably), with the current state the gradle files can
>>> mostly be ignored by most people. In particular, this is the kind of
>>> change that needs to be in master to be evaluated--if it's on a branch
>>> we can't very well see how it impacts presubmits, and most importantly
>>> people can't try it out for real development.
>>>
>>> I agree that the choice of build tool may attract some contributors
>>> and discourage others. Having builds that are fast, correct, and
>>> reproducible will probably matter more to potential contributors than
>>> the particular command to run. While maven can surely be improved, I
>>> doubt a 2x improvement (and many more times that for incremental
>>> builds) is low-hanging fruit, and many of the issues seem quite
>>> fundamental (e.g. all the special treatment we need for NeedsRunner
>>> tests, and having to do a (global-by-default) mvn install to skip
>>> tests of dependencies when testing a leaf module).
>>>
>>> Getting data on what other apache projects use could be interesting,
>>> but unless we gather why such choices were made I don't know that it'd
>>> be that influential once we've established that both tools are widely
>>> supported generally.
>>>
>>> To re-emphasize, we'll continue to produce and publish maven
>>> artifacts, so our choice of build system won't matter for those only
>>> using Beam as a dependency.
>>>
>>>
>>>
>>> On Mon, Nov 27, 2017 at 10:48 PM, Jean-Baptiste Onofré 
>>> wrote:
>>> > Yeah, especially, I think it would have been great to have a vote
>>> > before
>>> > merging on master.
>>> >
>>> > Not a big deal, however, I'm really community focus ;)
>>> >
>>> > Regards
>>> > JB
>>> >
>>> > On 11/28/2017 07:36 AM, Reuven Lax wrote:
>>> >>
>>> >> Agreed. I thinking having a formal vote before Luke had numbers and
>>> >> results would have been too early. However now that we have such
>>> >> numbers, we
>>> >> should think about having a vote.
>>> >>
>>> >> Also, while I disagree with Romain that Gradle is not "enterprise
>>> >> ready"
>>> >> (it's heavily used by Netflix, LinkedIn, Siemens, and is the default
>>> >> build
>>> >> framework for Android apps), it would be interesting to see if any
>>> >> other ASF
>>> >> 

Re: [DISCUSS] Move away from Apache Maven as build tool

[Lukasz Cwik]

Romain, Gradle has a Nexus plugin[1] which can sign and publish artifacts.
Gradle also has excellent support to run Ant tasks since Ant can perform
the entire release process for an ASF project.

1: https://github.com/bmuschko/gradle-nexus-plugin

On Tue, Nov 28, 2017 at 9:45 AM, Scott Wegner  wrote:

> To add one more data point measuring general adoption of gradle vs. maven,
> we can look at Stackoverflow trends comparing the two tags [1]. This shows
> the percentage of new SO questions in a given month by tag. 'gradle'
> represents ~0.25% of questions, while maven is ~0.45%. So, maven is more
> dominant in the Stackoverflow community, but they are at least similar
> orders of magnitude. Also, the data is a bit noisy to draw a trendline, but
> it seems that maven's growth has flattened while gradle is still increasing.
>
> [1] https://insights.stackoverflow.com/trends?tags=maven%2Cgradle
>
> On Tue, Nov 28, 2017 at 9:14 AM Kenneth Knowles  wrote:
>
>> Yea, I think voting is the next step. Luke - I think you are obviously
>> the right person to set up the email of what exactly we are voting on,
>> since you've driven this improvement.
>>
>> On Tue, Nov 28, 2017 at 12:08 AM, Robert Bradshaw 
>> wrote:
>>
>>> It's great to see all the discussion going on here.
>>>
>>> I think it's important to point out that merging a parallel set of
>>> gradle build scripts is a separate (and much less disruptive) step
>>> than, say, switching over the default (or even recommended)
>>> build/release process to use them, let alone removing the maven build
>>> files entirely. The latter two should definitely be gated by a formal
>>> vote (each, probably), with the current state the gradle files can
>>> mostly be ignored by most people. In particular, this is the kind of
>>> change that needs to be in master to be evaluated--if it's on a branch
>>> we can't very well see how it impacts presubmits, and most importantly
>>> people can't try it out for real development.
>>>
>>> I agree that the choice of build tool may attract some contributors
>>> and discourage others. Having builds that are fast, correct, and
>>> reproducible will probably matter more to potential contributors than
>>> the particular command to run. While maven can surely be improved, I
>>> doubt a 2x improvement (and many more times that for incremental
>>> builds) is low-hanging fruit, and many of the issues seem quite
>>> fundamental (e.g. all the special treatment we need for NeedsRunner
>>> tests, and having to do a (global-by-default) mvn install to skip
>>> tests of dependencies when testing a leaf module).
>>>
>>> Getting data on what other apache projects use could be interesting,
>>> but unless we gather why such choices were made I don't know that it'd
>>> be that influential once we've established that both tools are widely
>>> supported generally.
>>>
>>> To re-emphasize, we'll continue to produce and publish maven
>>> artifacts, so our choice of build system won't matter for those only
>>> using Beam as a dependency.
>>>
>>>
>>>
>>> On Mon, Nov 27, 2017 at 10:48 PM, Jean-Baptiste Onofré 
>>> wrote:
>>> > Yeah, especially, I think it would have been great to have a vote
>>> before
>>> > merging on master.
>>> >
>>> > Not a big deal, however, I'm really community focus ;)
>>> >
>>> > Regards
>>> > JB
>>> >
>>> > On 11/28/2017 07:36 AM, Reuven Lax wrote:
>>> >>
>>> >> Agreed. I thinking having a formal vote before Luke had numbers and
>>> >> results would have been too early. However now that we have such
>>> numbers, we
>>> >> should think about having a vote.
>>> >>
>>> >> Also, while I disagree with Romain that Gradle is not "enterprise
>>> ready"
>>> >> (it's heavily used by Netflix, LinkedIn, Siemens, and is the default
>>> build
>>> >> framework for Android apps), it would be interesting to see if any
>>> other ASF
>>> >> projects are using it. I don't think that should not make or break the
>>> >> decision - we should do what's best for the Beam project, and
>>> "everyone else
>>> >> is doing something" is rarely a good argument - it will provide good
>>> data
>>> >> points for us to evaluate.
>>> >>
>>> >> Reuven
>>> >>
>>> >> On Mon, Nov 27, 2017 at 10:23 PM, Jean-Baptiste Onofré <
>>> j...@nanthrax.net
>>> >> > wrote:
>>> >>
>>> >> Hi Luke,
>>> >>
>>> >> just curious (and maybe I missed it): did we do a formal vote to
>>> merge
>>> >> the
>>> >> gradle build ?
>>> >> Gradle is now on master, we have some Jira to update the release
>>> guide
>>> >> with
>>> >> gradle. It's fine, but I remember only a discussion, not a vote.
>>> >>
>>> >> In order to embrace the community and avoid to have some
>>> contributors
>>> >> "frustrated" (meaning that "this project doesn't care about
>>> >> contributor,
>>> >> they just do whatever they want"), I would have love to see a
>>> formal
>>> >> vote
>>> >> about 

Re: [DISCUSS] Move away from Apache Maven as build tool

[Scott Wegner]

To add one more data point measuring general adoption of gradle vs. maven,
we can look at Stackoverflow trends comparing the two tags [1]. This shows
the percentage of new SO questions in a given month by tag. 'gradle'
represents ~0.25% of questions, while maven is ~0.45%. So, maven is more
dominant in the Stackoverflow community, but they are at least similar
orders of magnitude. Also, the data is a bit noisy to draw a trendline, but
it seems that maven's growth has flattened while gradle is still increasing.

[1] https://insights.stackoverflow.com/trends?tags=maven%2Cgradle

On Tue, Nov 28, 2017 at 9:14 AM Kenneth Knowles  wrote:

> Yea, I think voting is the next step. Luke - I think you are obviously the
> right person to set up the email of what exactly we are voting on, since
> you've driven this improvement.
>
> On Tue, Nov 28, 2017 at 12:08 AM, Robert Bradshaw 
> wrote:
>
>> It's great to see all the discussion going on here.
>>
>> I think it's important to point out that merging a parallel set of
>> gradle build scripts is a separate (and much less disruptive) step
>> than, say, switching over the default (or even recommended)
>> build/release process to use them, let alone removing the maven build
>> files entirely. The latter two should definitely be gated by a formal
>> vote (each, probably), with the current state the gradle files can
>> mostly be ignored by most people. In particular, this is the kind of
>> change that needs to be in master to be evaluated--if it's on a branch
>> we can't very well see how it impacts presubmits, and most importantly
>> people can't try it out for real development.
>>
>> I agree that the choice of build tool may attract some contributors
>> and discourage others. Having builds that are fast, correct, and
>> reproducible will probably matter more to potential contributors than
>> the particular command to run. While maven can surely be improved, I
>> doubt a 2x improvement (and many more times that for incremental
>> builds) is low-hanging fruit, and many of the issues seem quite
>> fundamental (e.g. all the special treatment we need for NeedsRunner
>> tests, and having to do a (global-by-default) mvn install to skip
>> tests of dependencies when testing a leaf module).
>>
>> Getting data on what other apache projects use could be interesting,
>> but unless we gather why such choices were made I don't know that it'd
>> be that influential once we've established that both tools are widely
>> supported generally.
>>
>> To re-emphasize, we'll continue to produce and publish maven
>> artifacts, so our choice of build system won't matter for those only
>> using Beam as a dependency.
>>
>>
>>
>> On Mon, Nov 27, 2017 at 10:48 PM, Jean-Baptiste Onofré 
>> wrote:
>> > Yeah, especially, I think it would have been great to have a vote before
>> > merging on master.
>> >
>> > Not a big deal, however, I'm really community focus ;)
>> >
>> > Regards
>> > JB
>> >
>> > On 11/28/2017 07:36 AM, Reuven Lax wrote:
>> >>
>> >> Agreed. I thinking having a formal vote before Luke had numbers and
>> >> results would have been too early. However now that we have such
>> numbers, we
>> >> should think about having a vote.
>> >>
>> >> Also, while I disagree with Romain that Gradle is not "enterprise
>> ready"
>> >> (it's heavily used by Netflix, LinkedIn, Siemens, and is the default
>> build
>> >> framework for Android apps), it would be interesting to see if any
>> other ASF
>> >> projects are using it. I don't think that should not make or break the
>> >> decision - we should do what's best for the Beam project, and
>> "everyone else
>> >> is doing something" is rarely a good argument - it will provide good
>> data
>> >> points for us to evaluate.
>> >>
>> >> Reuven
>> >>
>> >> On Mon, Nov 27, 2017 at 10:23 PM, Jean-Baptiste Onofré <
>> j...@nanthrax.net
>> >> > wrote:
>> >>
>> >> Hi Luke,
>> >>
>> >> just curious (and maybe I missed it): did we do a formal vote to
>> merge
>> >> the
>> >> gradle build ?
>> >> Gradle is now on master, we have some Jira to update the release
>> guide
>> >> with
>> >> gradle. It's fine, but I remember only a discussion, not a vote.
>> >>
>> >> In order to embrace the community and avoid to have some
>> contributors
>> >> "frustrated" (meaning that "this project doesn't care about
>> >> contributor,
>> >> they just do whatever they want"), I would have love to see a
>> formal
>> >> vote
>> >> about Gradle more than just a discussion.
>> >>
>> >> My $0.01
>> >>
>> >> Regards
>> >> JB
>> >>
>> >> On 11/27/2017 07:46 PM, Lukasz Cwik wrote:
>> >>
>> >> I have collected data by running several builds against master
>> >> using Gradle
>> >> and Maven without using Gradle's support for incremental
>> builds.
>> >>
>> >> Gradle (mins)
>> >> min: 25.04
>> >> max: 160.14
>> >> median: 

Re: [DISCUSS] Move away from Apache Maven as build tool

[Romain Manni-Bucau]

Did you try a release (you can create a temporary staging repo on ASF
nexus if it helps) before starting a vote? Cause you migrate and the
project is no more able to release it can be a rude blocker - which
never happens when needed ;). Release has a few more plugins I didn't
find in gradle (can have missed them) like the signing of artifacts
etc.

Romain Manni-Bucau
@rmannibucau |  Blog | Old Blog | Github | LinkedIn


2017-11-28 18:14 GMT+01:00 Kenneth Knowles :
> Yea, I think voting is the next step. Luke - I think you are obviously the
> right person to set up the email of what exactly we are voting on, since
> you've driven this improvement.
>
> On Tue, Nov 28, 2017 at 12:08 AM, Robert Bradshaw 
> wrote:
>>
>> It's great to see all the discussion going on here.
>>
>> I think it's important to point out that merging a parallel set of
>> gradle build scripts is a separate (and much less disruptive) step
>> than, say, switching over the default (or even recommended)
>> build/release process to use them, let alone removing the maven build
>> files entirely. The latter two should definitely be gated by a formal
>> vote (each, probably), with the current state the gradle files can
>> mostly be ignored by most people. In particular, this is the kind of
>> change that needs to be in master to be evaluated--if it's on a branch
>> we can't very well see how it impacts presubmits, and most importantly
>> people can't try it out for real development.
>>
>> I agree that the choice of build tool may attract some contributors
>> and discourage others. Having builds that are fast, correct, and
>> reproducible will probably matter more to potential contributors than
>> the particular command to run. While maven can surely be improved, I
>> doubt a 2x improvement (and many more times that for incremental
>> builds) is low-hanging fruit, and many of the issues seem quite
>> fundamental (e.g. all the special treatment we need for NeedsRunner
>> tests, and having to do a (global-by-default) mvn install to skip
>> tests of dependencies when testing a leaf module).
>>
>> Getting data on what other apache projects use could be interesting,
>> but unless we gather why such choices were made I don't know that it'd
>> be that influential once we've established that both tools are widely
>> supported generally.
>>
>> To re-emphasize, we'll continue to produce and publish maven
>> artifacts, so our choice of build system won't matter for those only
>> using Beam as a dependency.
>>
>>
>>
>> On Mon, Nov 27, 2017 at 10:48 PM, Jean-Baptiste Onofré 
>> wrote:
>> > Yeah, especially, I think it would have been great to have a vote before
>> > merging on master.
>> >
>> > Not a big deal, however, I'm really community focus ;)
>> >
>> > Regards
>> > JB
>> >
>> > On 11/28/2017 07:36 AM, Reuven Lax wrote:
>> >>
>> >> Agreed. I thinking having a formal vote before Luke had numbers and
>> >> results would have been too early. However now that we have such
>> >> numbers, we
>> >> should think about having a vote.
>> >>
>> >> Also, while I disagree with Romain that Gradle is not "enterprise
>> >> ready"
>> >> (it's heavily used by Netflix, LinkedIn, Siemens, and is the default
>> >> build
>> >> framework for Android apps), it would be interesting to see if any
>> >> other ASF
>> >> projects are using it. I don't think that should not make or break the
>> >> decision - we should do what's best for the Beam project, and "everyone
>> >> else
>> >> is doing something" is rarely a good argument - it will provide good
>> >> data
>> >> points for us to evaluate.
>> >>
>> >> Reuven
>> >>
>> >> On Mon, Nov 27, 2017 at 10:23 PM, Jean-Baptiste Onofré > >> > wrote:
>> >>
>> >> Hi Luke,
>> >>
>> >> just curious (and maybe I missed it): did we do a formal vote to
>> >> merge
>> >> the
>> >> gradle build ?
>> >> Gradle is now on master, we have some Jira to update the release
>> >> guide
>> >> with
>> >> gradle. It's fine, but I remember only a discussion, not a vote.
>> >>
>> >> In order to embrace the community and avoid to have some
>> >> contributors
>> >> "frustrated" (meaning that "this project doesn't care about
>> >> contributor,
>> >> they just do whatever they want"), I would have love to see a
>> >> formal
>> >> vote
>> >> about Gradle more than just a discussion.
>> >>
>> >> My $0.01
>> >>
>> >> Regards
>> >> JB
>> >>
>> >> On 11/27/2017 07:46 PM, Lukasz Cwik wrote:
>> >>
>> >> I have collected data by running several builds against master
>> >> using Gradle
>> >> and Maven without using Gradle's support for incremental
>> >> builds.
>> >>
>> >> Gradle (mins)
>> >> min: 25.04
>> >> max: 160.14
>> >> median: 45.78
>> >> average: 52.19
>> >> stdev: 30.80
>> >>
>> >> Maven (mins)
>> >> min: 56.86
>> >> max: 

Re: [DISCUSS] Move away from Apache Maven as build tool

[Kenneth Knowles]

Yea, I think voting is the next step. Luke - I think you are obviously the
right person to set up the email of what exactly we are voting on, since
you've driven this improvement.

On Tue, Nov 28, 2017 at 12:08 AM, Robert Bradshaw 
wrote:

> It's great to see all the discussion going on here.
>
> I think it's important to point out that merging a parallel set of
> gradle build scripts is a separate (and much less disruptive) step
> than, say, switching over the default (or even recommended)
> build/release process to use them, let alone removing the maven build
> files entirely. The latter two should definitely be gated by a formal
> vote (each, probably), with the current state the gradle files can
> mostly be ignored by most people. In particular, this is the kind of
> change that needs to be in master to be evaluated--if it's on a branch
> we can't very well see how it impacts presubmits, and most importantly
> people can't try it out for real development.
>
> I agree that the choice of build tool may attract some contributors
> and discourage others. Having builds that are fast, correct, and
> reproducible will probably matter more to potential contributors than
> the particular command to run. While maven can surely be improved, I
> doubt a 2x improvement (and many more times that for incremental
> builds) is low-hanging fruit, and many of the issues seem quite
> fundamental (e.g. all the special treatment we need for NeedsRunner
> tests, and having to do a (global-by-default) mvn install to skip
> tests of dependencies when testing a leaf module).
>
> Getting data on what other apache projects use could be interesting,
> but unless we gather why such choices were made I don't know that it'd
> be that influential once we've established that both tools are widely
> supported generally.
>
> To re-emphasize, we'll continue to produce and publish maven
> artifacts, so our choice of build system won't matter for those only
> using Beam as a dependency.
>
>
>
> On Mon, Nov 27, 2017 at 10:48 PM, Jean-Baptiste Onofré 
> wrote:
> > Yeah, especially, I think it would have been great to have a vote before
> > merging on master.
> >
> > Not a big deal, however, I'm really community focus ;)
> >
> > Regards
> > JB
> >
> > On 11/28/2017 07:36 AM, Reuven Lax wrote:
> >>
> >> Agreed. I thinking having a formal vote before Luke had numbers and
> >> results would have been too early. However now that we have such
> numbers, we
> >> should think about having a vote.
> >>
> >> Also, while I disagree with Romain that Gradle is not "enterprise ready"
> >> (it's heavily used by Netflix, LinkedIn, Siemens, and is the default
> build
> >> framework for Android apps), it would be interesting to see if any
> other ASF
> >> projects are using it. I don't think that should not make or break the
> >> decision - we should do what's best for the Beam project, and "everyone
> else
> >> is doing something" is rarely a good argument - it will provide good
> data
> >> points for us to evaluate.
> >>
> >> Reuven
> >>
> >> On Mon, Nov 27, 2017 at 10:23 PM, Jean-Baptiste Onofré  >> > wrote:
> >>
> >> Hi Luke,
> >>
> >> just curious (and maybe I missed it): did we do a formal vote to
> merge
> >> the
> >> gradle build ?
> >> Gradle is now on master, we have some Jira to update the release
> guide
> >> with
> >> gradle. It's fine, but I remember only a discussion, not a vote.
> >>
> >> In order to embrace the community and avoid to have some
> contributors
> >> "frustrated" (meaning that "this project doesn't care about
> >> contributor,
> >> they just do whatever they want"), I would have love to see a formal
> >> vote
> >> about Gradle more than just a discussion.
> >>
> >> My $0.01
> >>
> >> Regards
> >> JB
> >>
> >> On 11/27/2017 07:46 PM, Lukasz Cwik wrote:
> >>
> >> I have collected data by running several builds against master
> >> using Gradle
> >> and Maven without using Gradle's support for incremental builds.
> >>
> >> Gradle (mins)
> >> min: 25.04
> >> max: 160.14
> >> median: 45.78
> >> average: 52.19
> >> stdev: 30.80
> >>
> >> Maven (mins)
> >> min: 56.86
> >> max: 216.55 (actually > 240 mins because this data does not
> >> include
> >> timeouts)
> >> median: 87.93
> >> average: 109.10
> >> stdev: 48.01
> >>
> >> I excluded a few timeouts (240 mins) that happened during the
> >> Maven build
> >> from its numbers but we can see conclusively that Gradle is
> twice
> >> as fast
> >> for the build when compared to Maven when run using Jenkins.
> >> On my desktop, I have enabled incremental builds and have seen a
> >> major
> >> improvement on the above numbers but it doesn't yet work
> correctly
> >> because
> >> of incorrectly 

Re: [DISCUSS] Move away from Apache Maven as build tool

[Romain Manni-Bucau]

Guys,

just realized a lot of modules have threadCount=4 or so in the
surefire/failsafe config. It makes it impossible to adapt the
parallelism to the machine and therefore makes the parallelism
inadapted and useless. Can it be a variable at least? -T1C (or -T2C)
should allow to be smoother and aligned with the config machine.

Romain Manni-Bucau
@rmannibucau |  Blog | Old Blog | Github | LinkedIn


2017-11-28 9:53 GMT+01:00 Romain Manni-Bucau :
> Just to answer a previous question:
>
> An ASF github search gives me these stats:
>
> - mvn (org:apache apache filename:pom.xml path:/): 731
> - gradle (org:apache apache filename:build.gradle path:/): 31
>
> Which is consistent with what I saw in enterprises and private repos.
> So way different from the whole github stats which is not done on a
> representative sample if not pre-filtered.
>
> Romain Manni-Bucau
> @rmannibucau |  Blog | Old Blog | Github | LinkedIn
>
>
> 2017-11-28 9:08 GMT+01:00 Robert Bradshaw :
>> It's great to see all the discussion going on here.
>>
>> I think it's important to point out that merging a parallel set of
>> gradle build scripts is a separate (and much less disruptive) step
>> than, say, switching over the default (or even recommended)
>> build/release process to use them, let alone removing the maven build
>> files entirely. The latter two should definitely be gated by a formal
>> vote (each, probably), with the current state the gradle files can
>> mostly be ignored by most people. In particular, this is the kind of
>> change that needs to be in master to be evaluated--if it's on a branch
>> we can't very well see how it impacts presubmits, and most importantly
>> people can't try it out for real development.
>>
>> I agree that the choice of build tool may attract some contributors
>> and discourage others. Having builds that are fast, correct, and
>> reproducible will probably matter more to potential contributors than
>> the particular command to run. While maven can surely be improved, I
>> doubt a 2x improvement (and many more times that for incremental
>> builds) is low-hanging fruit, and many of the issues seem quite
>> fundamental (e.g. all the special treatment we need for NeedsRunner
>> tests, and having to do a (global-by-default) mvn install to skip
>> tests of dependencies when testing a leaf module).
>>
>> Getting data on what other apache projects use could be interesting,
>> but unless we gather why such choices were made I don't know that it'd
>> be that influential once we've established that both tools are widely
>> supported generally.
>>
>> To re-emphasize, we'll continue to produce and publish maven
>> artifacts, so our choice of build system won't matter for those only
>> using Beam as a dependency.
>>
>>
>>
>> On Mon, Nov 27, 2017 at 10:48 PM, Jean-Baptiste Onofré  
>> wrote:
>>> Yeah, especially, I think it would have been great to have a vote before
>>> merging on master.
>>>
>>> Not a big deal, however, I'm really community focus ;)
>>>
>>> Regards
>>> JB
>>>
>>> On 11/28/2017 07:36 AM, Reuven Lax wrote:

 Agreed. I thinking having a formal vote before Luke had numbers and
 results would have been too early. However now that we have such numbers, 
 we
 should think about having a vote.

 Also, while I disagree with Romain that Gradle is not "enterprise ready"
 (it's heavily used by Netflix, LinkedIn, Siemens, and is the default build
 framework for Android apps), it would be interesting to see if any other 
 ASF
 projects are using it. I don't think that should not make or break the
 decision - we should do what's best for the Beam project, and "everyone 
 else
 is doing something" is rarely a good argument - it will provide good data
 points for us to evaluate.

 Reuven

 On Mon, Nov 27, 2017 at 10:23 PM, Jean-Baptiste Onofré > wrote:

 Hi Luke,

 just curious (and maybe I missed it): did we do a formal vote to merge
 the
 gradle build ?
 Gradle is now on master, we have some Jira to update the release guide
 with
 gradle. It's fine, but I remember only a discussion, not a vote.

 In order to embrace the community and avoid to have some contributors
 "frustrated" (meaning that "this project doesn't care about
 contributor,
 they just do whatever they want"), I would have love to see a formal
 vote
 about Gradle more than just a discussion.

 My $0.01

 Regards
 JB

 On 11/27/2017 07:46 PM, Lukasz Cwik wrote:

 I have collected data by running several builds against master
 using Gradle
 and Maven without using Gradle's support for incremental builds.

 Gradle (mins)
 min: 25.04
 max: 160.14
  

Re: [DISCUSS] Move away from Apache Maven as build tool

[Robert Bradshaw]

It's great to see all the discussion going on here.

I think it's important to point out that merging a parallel set of
gradle build scripts is a separate (and much less disruptive) step
than, say, switching over the default (or even recommended)
build/release process to use them, let alone removing the maven build
files entirely. The latter two should definitely be gated by a formal
vote (each, probably), with the current state the gradle files can
mostly be ignored by most people. In particular, this is the kind of
change that needs to be in master to be evaluated--if it's on a branch
we can't very well see how it impacts presubmits, and most importantly
people can't try it out for real development.

I agree that the choice of build tool may attract some contributors
and discourage others. Having builds that are fast, correct, and
reproducible will probably matter more to potential contributors than
the particular command to run. While maven can surely be improved, I
doubt a 2x improvement (and many more times that for incremental
builds) is low-hanging fruit, and many of the issues seem quite
fundamental (e.g. all the special treatment we need for NeedsRunner
tests, and having to do a (global-by-default) mvn install to skip
tests of dependencies when testing a leaf module).

Getting data on what other apache projects use could be interesting,
but unless we gather why such choices were made I don't know that it'd
be that influential once we've established that both tools are widely
supported generally.

To re-emphasize, we'll continue to produce and publish maven
artifacts, so our choice of build system won't matter for those only
using Beam as a dependency.



On Mon, Nov 27, 2017 at 10:48 PM, Jean-Baptiste Onofré  
wrote:
> Yeah, especially, I think it would have been great to have a vote before
> merging on master.
>
> Not a big deal, however, I'm really community focus ;)
>
> Regards
> JB
>
> On 11/28/2017 07:36 AM, Reuven Lax wrote:
>>
>> Agreed. I thinking having a formal vote before Luke had numbers and
>> results would have been too early. However now that we have such numbers, we
>> should think about having a vote.
>>
>> Also, while I disagree with Romain that Gradle is not "enterprise ready"
>> (it's heavily used by Netflix, LinkedIn, Siemens, and is the default build
>> framework for Android apps), it would be interesting to see if any other ASF
>> projects are using it. I don't think that should not make or break the
>> decision - we should do what's best for the Beam project, and "everyone else
>> is doing something" is rarely a good argument - it will provide good data
>> points for us to evaluate.
>>
>> Reuven
>>
>> On Mon, Nov 27, 2017 at 10:23 PM, Jean-Baptiste Onofré > > wrote:
>>
>> Hi Luke,
>>
>> just curious (and maybe I missed it): did we do a formal vote to merge
>> the
>> gradle build ?
>> Gradle is now on master, we have some Jira to update the release guide
>> with
>> gradle. It's fine, but I remember only a discussion, not a vote.
>>
>> In order to embrace the community and avoid to have some contributors
>> "frustrated" (meaning that "this project doesn't care about
>> contributor,
>> they just do whatever they want"), I would have love to see a formal
>> vote
>> about Gradle more than just a discussion.
>>
>> My $0.01
>>
>> Regards
>> JB
>>
>> On 11/27/2017 07:46 PM, Lukasz Cwik wrote:
>>
>> I have collected data by running several builds against master
>> using Gradle
>> and Maven without using Gradle's support for incremental builds.
>>
>> Gradle (mins)
>> min: 25.04
>> max: 160.14
>> median: 45.78
>> average: 52.19
>> stdev: 30.80
>>
>> Maven (mins)
>> min: 56.86
>> max: 216.55 (actually > 240 mins because this data does not
>> include
>> timeouts)
>> median: 87.93
>> average: 109.10
>> stdev: 48.01
>>
>> I excluded a few timeouts (240 mins) that happened during the
>> Maven build
>> from its numbers but we can see conclusively that Gradle is twice
>> as fast
>> for the build when compared to Maven when run using Jenkins.
>> On my desktop, I have enabled incremental builds and have seen a
>> major
>> improvement on the above numbers but it doesn't yet work correctly
>> because
>> of incorrectly specified inputs/outputs for certain tasks.
>>
>> The data is available here
>>
>> https://docs.google.com/spreadsheets/d/1MHVjF-xoI49_NJqEQakUgnNIQ7Qbjzu8Y1q_h3dbF1M/edit?usp=sharing
>>
>> 
>>
>> With this data, I feel confident that we should swap and have
>> opened the
>> following issue https://issues.apache.org/jira/browse/BEAM-3249
>> 

Re: [DISCUSS] Move away from Apache Maven as build tool

[Jean-Baptiste Onofré]

Yeah, especially, I think it would have been great to have a vote before merging 
on master.


Not a big deal, however, I'm really community focus ;)

Regards
JB

On 11/28/2017 07:36 AM, Reuven Lax wrote:
Agreed. I thinking having a formal vote before Luke had numbers and results 
would have been too early. However now that we have such numbers, we should 
think about having a vote.


Also, while I disagree with Romain that Gradle is not "enterprise ready" (it's 
heavily used by Netflix, LinkedIn, Siemens, and is the default build framework 
for Android apps), it would be interesting to see if any other ASF projects are 
using it. I don't think that should not make or break the decision - we should 
do what's best for the Beam project, and "everyone else is doing something" is 
rarely a good argument - it will provide good data points for us to evaluate.


Reuven

On Mon, Nov 27, 2017 at 10:23 PM, Jean-Baptiste Onofré > wrote:


Hi Luke,

just curious (and maybe I missed it): did we do a formal vote to merge the
gradle build ?
Gradle is now on master, we have some Jira to update the release guide with
gradle. It's fine, but I remember only a discussion, not a vote.

In order to embrace the community and avoid to have some contributors
"frustrated" (meaning that "this project doesn't care about contributor,
they just do whatever they want"), I would have love to see a formal vote
about Gradle more than just a discussion.

My $0.01

Regards
JB

On 11/27/2017 07:46 PM, Lukasz Cwik wrote:

I have collected data by running several builds against master using 
Gradle
and Maven without using Gradle's support for incremental builds.

Gradle (mins)
min: 25.04
max: 160.14
median: 45.78
average: 52.19
stdev: 30.80

Maven (mins)
min: 56.86
max: 216.55 (actually > 240 mins because this data does not include
timeouts)
median: 87.93
average: 109.10
stdev: 48.01

I excluded a few timeouts (240 mins) that happened during the Maven 
build
from its numbers but we can see conclusively that Gradle is twice as 
fast
for the build when compared to Maven when run using Jenkins.
On my desktop, I have enabled incremental builds and have seen a major
improvement on the above numbers but it doesn't yet work correctly 
because
of incorrectly specified inputs/outputs for certain tasks.

The data is available here

https://docs.google.com/spreadsheets/d/1MHVjF-xoI49_NJqEQakUgnNIQ7Qbjzu8Y1q_h3dbF1M/edit?usp=sharing



With this data, I feel confident that we should swap and have opened the
following issue https://issues.apache.org/jira/browse/BEAM-3249
 and related
sub-tasks.

On Sun, Nov 19, 2017 at 11:23 AM, Jean-Baptiste Onofré 
>
wrote:

Thanks for the update Luke.

I'm updating my local working copy to do new tests.

Regards
JB

On 11/19/2017 08:21 PM, Lukasz Cwik wrote:

The gradle build rules have been merged, I'm adding a
precommit[1] to
start
collecting data about the build times. It currently only mirrors
the Java
mvn install precommit. I'll gather data over the next two weeks 
and
provide
a summary here.

You can rerun the precommit by issuing "Run Java Gradle 
PreCommit"

1: https://github.com/apache/beam/pull/4146



On Mon, Nov 13, 2017 at 9:08 AM, Lukasz Cwik > wrote:

There has been plenty of time for comments on the PR and the
approach.


So far Ken Knowles has provided the most feedback on the PR,
Ken would
you
like to finish the review?



On Fri, Nov 10, 2017 at 1:22 PM, Romain Manni-Bucau <
rmannibu...@gmail.com 

wrote:


This is only a setup thing and better to not break the
master history for

poc/tests, in particular when no very localized.
Alternative can be to
ask
another temp repo to infra and have a synchro between
both but dont
think
  

Re: [DISCUSS] Move away from Apache Maven as build tool

[Reuven Lax]

Agreed. I thinking having a formal vote before Luke had numbers and results
would have been too early. However now that we have such numbers, we should
think about having a vote.

Also, while I disagree with Romain that Gradle is not "enterprise ready"
(it's heavily used by Netflix, LinkedIn, Siemens, and is the default build
framework for Android apps), it would be interesting to see if any other
ASF projects are using it. I don't think that should not make or break the
decision - we should do what's best for the Beam project, and "everyone
else is doing something" is rarely a good argument - it will provide good
data points for us to evaluate.

Reuven

On Mon, Nov 27, 2017 at 10:23 PM, Jean-Baptiste Onofré 
wrote:

> Hi Luke,
>
> just curious (and maybe I missed it): did we do a formal vote to merge the
> gradle build ?
> Gradle is now on master, we have some Jira to update the release guide
> with gradle. It's fine, but I remember only a discussion, not a vote.
>
> In order to embrace the community and avoid to have some contributors
> "frustrated" (meaning that "this project doesn't care about contributor,
> they just do whatever they want"), I would have love to see a formal vote
> about Gradle more than just a discussion.
>
> My $0.01
>
> Regards
> JB
>
> On 11/27/2017 07:46 PM, Lukasz Cwik wrote:
>
>> I have collected data by running several builds against master using
>> Gradle
>> and Maven without using Gradle's support for incremental builds.
>>
>> Gradle (mins)
>> min: 25.04
>> max: 160.14
>> median: 45.78
>> average: 52.19
>> stdev: 30.80
>>
>> Maven (mins)
>> min: 56.86
>> max: 216.55 (actually > 240 mins because this data does not include
>> timeouts)
>> median: 87.93
>> average: 109.10
>> stdev: 48.01
>>
>> I excluded a few timeouts (240 mins) that happened during the Maven build
>> from its numbers but we can see conclusively that Gradle is twice as fast
>> for the build when compared to Maven when run using Jenkins.
>> On my desktop, I have enabled incremental builds and have seen a major
>> improvement on the above numbers but it doesn't yet work correctly because
>> of incorrectly specified inputs/outputs for certain tasks.
>>
>> The data is available here
>> https://docs.google.com/spreadsheets/d/1MHVjF-xoI49_NJqEQakU
>> gnNIQ7Qbjzu8Y1q_h3dbF1M/edit?usp=sharing
>>
>> With this data, I feel confident that we should swap and have opened the
>> following issue https://issues.apache.org/jira/browse/BEAM-3249 and
>> related
>> sub-tasks.
>>
>> On Sun, Nov 19, 2017 at 11:23 AM, Jean-Baptiste Onofré 
>> wrote:
>>
>> Thanks for the update Luke.
>>>
>>> I'm updating my local working copy to do new tests.
>>>
>>> Regards
>>> JB
>>>
>>> On 11/19/2017 08:21 PM, Lukasz Cwik wrote:
>>>
>>> The gradle build rules have been merged, I'm adding a precommit[1] to
 start
 collecting data about the build times. It currently only mirrors the
 Java
 mvn install precommit. I'll gather data over the next two weeks and
 provide
 a summary here.

 You can rerun the precommit by issuing "Run Java Gradle PreCommit"

 1: https://github.com/apache/beam/pull/4146


 On Mon, Nov 13, 2017 at 9:08 AM, Lukasz Cwik  wrote:

 There has been plenty of time for comments on the PR and the approach.

>
> So far Ken Knowles has provided the most feedback on the PR, Ken would
> you
> like to finish the review?
>
>
>
> On Fri, Nov 10, 2017 at 1:22 PM, Romain Manni-Bucau <
> rmannibu...@gmail.com
>
> wrote:
>>
>>
> This is only a setup thing and better to not break the master history
> for
>
>> poc/tests, in particular when no very localized. Alternative can be to
>> ask
>> another temp repo to infra and have a synchro between both but dont
>> think
>> it does worth it personally.
>>
>>
>>
>> Le 10 nov. 2017 18:57, "Lukasz Cwik"  a
>> écrit :
>>
>> The reason to get it on master is because that is where all the PRs
>>
>>>
>>> are. An
>>
>> upstream branch without any development means no data.
>>> Also, our Jenkins setup via job-dsl doesn't honor using the Jenkins
>>> configuration on the branch because the seed job always runs against
>>> master.
>>>
>>> On Thu, Nov 9, 2017 at 9:59 PM, Romain Manni-Bucau <
>>>
>>> rmannibu...@gmail.com>
>>
>> wrote:
>>>
>>> What about pushing it on a "upstream" branch and testing it for 1
>>>

 week in
>>>
>>
>> parallel of the maven reference build? If gradle is always 50% faster
>>>

 on
>>>
>>
>> jenkins then it could become master setup without much discussion I
>>>

 guess.
>>>
>>> We can even have 2 jenkins jobs: one with the daemon etc and one

 without.
>>>

Re: [DISCUSS] Move away from Apache Maven as build tool

[Jean-Baptiste Onofré]

Hi Luke,

just curious (and maybe I missed it): did we do a formal vote to merge the 
gradle build ?
Gradle is now on master, we have some Jira to update the release guide with 
gradle. It's fine, but I remember only a discussion, not a vote.


In order to embrace the community and avoid to have some contributors 
"frustrated" (meaning that "this project doesn't care about contributor, they 
just do whatever they want"), I would have love to see a formal vote about 
Gradle more than just a discussion.


My $0.01

Regards
JB

On 11/27/2017 07:46 PM, Lukasz Cwik wrote:

I have collected data by running several builds against master using Gradle
and Maven without using Gradle's support for incremental builds.

Gradle (mins)
min: 25.04
max: 160.14
median: 45.78
average: 52.19
stdev: 30.80

Maven (mins)
min: 56.86
max: 216.55 (actually > 240 mins because this data does not include
timeouts)
median: 87.93
average: 109.10
stdev: 48.01

I excluded a few timeouts (240 mins) that happened during the Maven build
from its numbers but we can see conclusively that Gradle is twice as fast
for the build when compared to Maven when run using Jenkins.
On my desktop, I have enabled incremental builds and have seen a major
improvement on the above numbers but it doesn't yet work correctly because
of incorrectly specified inputs/outputs for certain tasks.

The data is available here
https://docs.google.com/spreadsheets/d/1MHVjF-xoI49_NJqEQakUgnNIQ7Qbjzu8Y1q_h3dbF1M/edit?usp=sharing

With this data, I feel confident that we should swap and have opened the
following issue https://issues.apache.org/jira/browse/BEAM-3249 and related
sub-tasks.

On Sun, Nov 19, 2017 at 11:23 AM, Jean-Baptiste Onofré 
wrote:


Thanks for the update Luke.

I'm updating my local working copy to do new tests.

Regards
JB

On 11/19/2017 08:21 PM, Lukasz Cwik wrote:


The gradle build rules have been merged, I'm adding a precommit[1] to
start
collecting data about the build times. It currently only mirrors the Java
mvn install precommit. I'll gather data over the next two weeks and
provide
a summary here.

You can rerun the precommit by issuing "Run Java Gradle PreCommit"

1: https://github.com/apache/beam/pull/4146


On Mon, Nov 13, 2017 at 9:08 AM, Lukasz Cwik  wrote:

There has been plenty of time for comments on the PR and the approach.


So far Ken Knowles has provided the most feedback on the PR, Ken would
you
like to finish the review?



On Fri, Nov 10, 2017 at 1:22 PM, Romain Manni-Bucau <
rmannibu...@gmail.com


wrote:



This is only a setup thing and better to not break the master history for

poc/tests, in particular when no very localized. Alternative can be to
ask
another temp repo to infra and have a synchro between both but dont
think
it does worth it personally.



Le 10 nov. 2017 18:57, "Lukasz Cwik"  a
écrit :

The reason to get it on master is because that is where all the PRs



are. An


upstream branch without any development means no data.
Also, our Jenkins setup via job-dsl doesn't honor using the Jenkins
configuration on the branch because the seed job always runs against
master.

On Thu, Nov 9, 2017 at 9:59 PM, Romain Manni-Bucau <


rmannibu...@gmail.com>


wrote:

What about pushing it on a "upstream" branch and testing it for 1



week in



parallel of the maven reference build? If gradle is always 50% faster



on



jenkins then it could become master setup without much discussion I



guess.


We can even have 2 jenkins jobs: one with the daemon etc and one


without.





Also noticed yesterday that gradle build is killing my machine (all 8


cores


are 100%) during the first minutes vs maven build which let me do


something


else. Then all the consumed time which makes gradle not that fast is


about


python. Will try to send figures later today.

Le 10 nov. 2017 00:10, "Lukasz Cwik"  a


écrit



:



I wouldn't mind merging this change in so I could setup those Gradle

Jenkins precommits.

As per our contribution guidelines, any committer willing to sign


off



on


the PR?


On Thu, Nov 9, 2017 at 2:12 PM, Romain Manni-Bucau <


rmannibu...@gmail.com>


wrote:

Le 9 nov. 2017 21:31, "Kenneth Knowles" 



a



écrit :




Keep in mind that a clean build is unusual during development (it


is



common



for mvn use and that is a bug) and also not necessary for


precommits



if


the



build tool is correct enough that caching is safe. So while this


number



matters, it is not the most important.



Not sure, in dev you bypass the build tool most of the time


anyway -



thanks



to IDE or other shortcuts - but not on PR and CI. Keep in mind


that



not


doing a clean and killing gradle daemon makes the build not



reproducible



and therefore useful :(. Starting to build from a subpart of the



reactor



-


with the mentionned mvn plugin for instance - can be nice on some


CI

Re: [DISCUSS] Move away from Apache Maven as build tool

[Romain Manni-Bucau]

Le 27 nov. 2017 23:07, "Eugene Kirpichov"  a
écrit :

On Mon, Nov 27, 2017 at 11:52 AM Romain Manni-Bucau 
wrote:

> 2017-11-27 20:26 GMT+01:00 Lukasz Cwik :
> > Romain, as mentioned earlier, I identified that Maven was slower because
> it
> > needed to finish building the entire module before dependent modules
> could
> > start which included running tests, performing checkstyle, etc...
> > Gradle is able to increase the parallelism of the build process since it
> > has task driven parallelism so as long as the files are compiled, the
> > dependent projects can start.
>
> This means we can implement a maven graph builder which is better than
> the default one - surely with a thread safe local repo - and
> contribute it back to solve it durably.
>

That would be indeed a wonderful contribution to the Java ecosystem as a
whole, but I'd like to reiterate that Maven has existed for very long, I'm
sure a lot of people have noticed its poor parallelization in the
meanwhile, and still nobody has even tried doing this AFAIK.

So it seems unlikely that this is doable in a reasonable timeframe, and I
don't think that the possibility of this is a strong argument against
switching to Gradle.


Agree. Just to clarify: my strong argument against gradle is that it is not
enterprise mainstream IMHO - dont forget github projects are a lot about
personal tests, small projects, so hard to use all the repos as a valid
stat - which leads to enhancing mvn rather than switching.

Again, as an ASF project, not even contacting our colleagues is not very
sane so I would go and check with them rather than just doing like a github
project maintained by a single company.

Side note on previous mail: mvn extensibility is "mature". Mojo let you do
what you want and you can customize the lifecycle as well. The main
difference is a standardized build (descriptive) vs a fully custom and "by
developer" styled script.



>
> >
> > Maven and Gradle are both heavily used since there are ~146k Maven
> projects
> > on Github while there are ~122k Gradle project on Github. Do you have
> data
> > which shows that Maven is significantly more "mainstream"?
>
> Yep, project i worked on in companies using gradle: 0, all were based
> on maven and maven was "tool-ed" versus gradle was "best effort" in
> term of plugins.
> Now  - with my EE background - I can guarantee you gradle is not able
> to handle properly its build since it flatten the classpath and
> plugins conflicts very quickly (their plugin dependency feature never
> worked and almost no plugin impl it correctly).
>
> Wonder if it is easy to have the ASF stats, anyone knows?
>
> >
> > I believe we want a rich multi-language SDK and community and feel as
> > though it would be unwise to treat non JDK based languages as second
> class.
>
> Hmm, not sure how it is related to the build tool since Maven and
> Gradle have the same level of support - actually surprsingly maven is
> better for js and surely as bad as gradle for others - or here again
> we can create plugin like the frontend-maven-plugin if needed for
> other languages.
>
> That said it can be an interesting other thread since people consuming
> these languages will probably want their mainstream build tool and a
> "standard" repository layout rather than a java one. But this is
> harder to measure.
>
> >
> > On Mon, Nov 27, 2017 at 11:00 AM, Romain Manni-Bucau <
> rmannibu...@gmail.com>
> > wrote:
> >
> >> Hi Lukasz,
> >>
> >> Did you manage to identify how maven was slower and test tesla stuff
> >> and potentially a few other fixes?
> >>
> >> Side note: figures without python can be interesting cause locally - =
> >> for me - python tends to flatten the figures whereas I get something
> >> close to your conclusions without python part.
> >>
> >> My point is mainly that switching now on gradle and being back on
> >> maven in a few months cause gradle ecosystem is far to support java 9
> >> - or any other volatile reason like this one - is probably not a good
> >> choice for a community. Maven is way more mainstream than gradle so
> >> helps to encourage people to contribute - vs gradle will increase the
> >> step to do it.
> >>
> >> I'd like to be sure before a switch that it is a one way decision and
> >> that the build tool was not just challenged by itself and its current
> >> state but also in the way it could be improved (= its community and
> >> potentially some local hacks).
> >>
> >> Romain Manni-Bucau
> >> @rmannibucau |  Blog | Old Blog | Github | LinkedIn
> >>
> >>
> >> 2017-11-27 19:46 GMT+01:00 Lukasz Cwik :
> >> > I have collected data by running several builds against master using
> >> Gradle
> >> > and Maven without using Gradle's support for incremental builds.
> >> >
> >> > Gradle (mins)
> >> > min: 25.04
> >> > max: 160.14
> >> > median: 45.78
> >> > average: 52.19
> >> > stdev: 30.80
> >> >
> >> > Maven 

Re: [DISCUSS] Move away from Apache Maven as build tool

[Lukasz Cwik]

I don't believe that such a switch could happen immediately as the scope of
the change is to replace 15k lines of pom files so anything which isn't
gradual is unlikely to work.

Below is a more thorough list of points which compare the two build systems
beyond speed.

Maven
Java Support: Mature
Python Support: None (via mvn exec plugin)
Go Support: Rudimentary (via mvn plugin)
Protobuf Support: Rudimentary (via mvn plugin)
Docker Support: Rudimentary (via mvn plugin)
ASF Release Automation: Mature
Jenkins Support: Mature
Configuration Language: XML
Multiple Java Versions: Yes
Static Analysis Tools: Some
ASF Release Audit Tool (RAT): Rudimentary (plugin complete and longstanding
but poor)
IntelliJ Integration: Mature
Eclipse Integration: Mature
Extensibility: Rudimentary
Number of GitHub Projects Using It: 146k
Continuous build daemon: None
Incremental build support: None (note that this is not the same as
incremental compile support offered by the compiler plugin)
Intra-module dependencies: Rudimentary (requires the use of many profiles
to get per runner dependencies)

Gradle
Java Support: Mature
Python Support: Rudimentary (pygradle, lacks pypi support)
Go Support: Rudimentary (gogradle plugin)
Protobuf Support: Rudimentary (via protobuf plugin)
Docker Support: Rudimentary (via docker plugin)
ASF Release Automation: ?
Jenkins Support: Mature
Configuration Language: Groovy
Multiple Java Versions: Yes
Static Analysis Tools: Some
ASF Release Audit Tool (RAT): Rudimentary (plugin just calls Apache Maven
ANT plugin)
IntelliJ Integration: Mature
Eclipse Integration: Mature
Extensibility: Mature
Number of GitHub Projects Using It: 122k
Continuous build daemon: Mature
Incremental build support: Mature
Intra-module dependencies: Mature (via configurations)

Most of the above points show that they are very similar except for the
faster builds, better intra-module dependencies, and incremental build
support.


On Mon, Nov 27, 2017 at 1:06 PM, Ismaël Mejía  wrote:

> I have been a little bit out of the discussion on maven vs gradle
> because I was expecting the technical proof of concepts to evaluate
> the best approach. I deeply appreciate all the effort that Lukasz has
> put into the gradle version, and I also think that during the
> discussion Romain and others have bring some serious and important
> points that make the decision less simple than I expected (in the end
> sadly is not as simple as the fastest wins). In any case I don’t think
> that it is wise just to switch immediately to gradle, at least if
> switching means removing the maven files, we have to consider that the
> ‘full’ build/tests were introduced in the CI around one week ago, and
> I am not sure that this is sufficient time to evaluate any possible
> regression. Also I am particularly curious to know if the artifacts
> are correct and complete. Has somebody already simulated a release
> with the gradle build for example, this for me is a prerrequisite
> before we even start discussing about the switch.
>
>
> On Mon, Nov 27, 2017 at 9:34 PM, Lukasz Cwik 
> wrote:
> > On Mon, Nov 27, 2017 at 11:51 AM, Romain Manni-Bucau <
> rmannibu...@gmail.com>
> > wrote:
> >
> >> 2017-11-27 20:26 GMT+01:00 Lukasz Cwik :
> >> > Romain, as mentioned earlier, I identified that Maven was slower
> because
> >> it
> >> > needed to finish building the entire module before dependent modules
> >> could
> >> > start which included running tests, performing checkstyle, etc...
> >> > Gradle is able to increase the parallelism of the build process since
> it
> >> > has task driven parallelism so as long as the files are compiled, the
> >> > dependent projects can start.
> >>
> >> This means we can implement a maven graph builder which is better than
> >> the default one - surely with a thread safe local repo - and
> >> contribute it back to solve it durably.
> >>
> >> If speed for a clean build was the only problem then maybe but lack of
> > incremental builds across tasks is a goal we can actually achieve using
> > Gradle and won't require rewriting almost all of the Maven plugins to
> > support incremental builds.
> >
> >
> >> >
> >> > Maven and Gradle are both heavily used since there are ~146k Maven
> >> projects
> >> > on Github while there are ~122k Gradle project on Github. Do you have
> >> data
> >> > which shows that Maven is significantly more "mainstream"?
> >>
> >> Yep, project i worked on in companies using gradle: 0, all were based
> >> on maven and maven was "tool-ed" versus gradle was "best effort" in
> >> term of plugins.
> >> Now  - with my EE background - I can guarantee you gradle is not able
> >> to handle properly its build since it flatten the classpath and
> >> plugins conflicts very quickly (their plugin dependency feature never
> >> worked and almost no plugin impl it correctly).
> >>
> >> Wonder if it is easy to have the ASF stats, anyone knows?
> >
> >>
> >> > I believe we 

Re: [DISCUSS] Move away from Apache Maven as build tool

[Ismaël Mejía]

I have been a little bit out of the discussion on maven vs gradle
because I was expecting the technical proof of concepts to evaluate
the best approach. I deeply appreciate all the effort that Lukasz has
put into the gradle version, and I also think that during the
discussion Romain and others have bring some serious and important
points that make the decision less simple than I expected (in the end
sadly is not as simple as the fastest wins). In any case I don’t think
that it is wise just to switch immediately to gradle, at least if
switching means removing the maven files, we have to consider that the
‘full’ build/tests were introduced in the CI around one week ago, and
I am not sure that this is sufficient time to evaluate any possible
regression. Also I am particularly curious to know if the artifacts
are correct and complete. Has somebody already simulated a release
with the gradle build for example, this for me is a prerrequisite
before we even start discussing about the switch.


On Mon, Nov 27, 2017 at 9:34 PM, Lukasz Cwik  wrote:
> On Mon, Nov 27, 2017 at 11:51 AM, Romain Manni-Bucau 
> wrote:
>
>> 2017-11-27 20:26 GMT+01:00 Lukasz Cwik :
>> > Romain, as mentioned earlier, I identified that Maven was slower because
>> it
>> > needed to finish building the entire module before dependent modules
>> could
>> > start which included running tests, performing checkstyle, etc...
>> > Gradle is able to increase the parallelism of the build process since it
>> > has task driven parallelism so as long as the files are compiled, the
>> > dependent projects can start.
>>
>> This means we can implement a maven graph builder which is better than
>> the default one - surely with a thread safe local repo - and
>> contribute it back to solve it durably.
>>
>> If speed for a clean build was the only problem then maybe but lack of
> incremental builds across tasks is a goal we can actually achieve using
> Gradle and won't require rewriting almost all of the Maven plugins to
> support incremental builds.
>
>
>> >
>> > Maven and Gradle are both heavily used since there are ~146k Maven
>> projects
>> > on Github while there are ~122k Gradle project on Github. Do you have
>> data
>> > which shows that Maven is significantly more "mainstream"?
>>
>> Yep, project i worked on in companies using gradle: 0, all were based
>> on maven and maven was "tool-ed" versus gradle was "best effort" in
>> term of plugins.
>> Now  - with my EE background - I can guarantee you gradle is not able
>> to handle properly its build since it flatten the classpath and
>> plugins conflicts very quickly (their plugin dependency feature never
>> worked and almost no plugin impl it correctly).
>>
>> Wonder if it is easy to have the ASF stats, anyone knows?
>
>>
>> > I believe we want a rich multi-language SDK and community and feel as
>> > though it would be unwise to treat non JDK based languages as second
>> class.
>>
>> Hmm, not sure how it is related to the build tool since Maven and
>> Gradle have the same level of support - actually surprsingly maven is
>> better for js and surely as bad as gradle for others - or here again
>> we can create plugin like the frontend-maven-plugin if needed for
>> other languages.
>>
>> That said it can be an interesting other thread since people consuming
>> these languages will probably want their mainstream build tool and a
>> "standard" repository layout rather than a java one. But this is
>> harder to measure.
>>
>>
>>
>> > On Mon, Nov 27, 2017 at 11:00 AM, Romain Manni-Bucau <
>> rmannibu...@gmail.com>
>> > wrote:
>> >
>> >> Hi Lukasz,
>> >>
>> >> Did you manage to identify how maven was slower and test tesla stuff
>> >> and potentially a few other fixes?
>> >>
>> >> Side note: figures without python can be interesting cause locally - =
>> >> for me - python tends to flatten the figures whereas I get something
>> >> close to your conclusions without python part.
>> >>
>> >> My point is mainly that switching now on gradle and being back on
>> >> maven in a few months cause gradle ecosystem is far to support java 9
>> >> - or any other volatile reason like this one - is probably not a good
>> >> choice for a community. Maven is way more mainstream than gradle so
>> >> helps to encourage people to contribute - vs gradle will increase the
>> >> step to do it.
>> >>
>> >> I'd like to be sure before a switch that it is a one way decision and
>> >> that the build tool was not just challenged by itself and its current
>> >> state but also in the way it could be improved (= its community and
>> >> potentially some local hacks).
>> >>
>> >> Romain Manni-Bucau
>> >> @rmannibucau |  Blog | Old Blog | Github | LinkedIn
>> >>
>> >>
>> >> 2017-11-27 19:46 GMT+01:00 Lukasz Cwik :
>> >> > I have collected data by running several builds against master using
>> >> Gradle
>> >> > and Maven without using Gradle's support for 

Re: [DISCUSS] Move away from Apache Maven as build tool

[Lukasz Cwik]

On Mon, Nov 27, 2017 at 11:51 AM, Romain Manni-Bucau 
wrote:

> 2017-11-27 20:26 GMT+01:00 Lukasz Cwik :
> > Romain, as mentioned earlier, I identified that Maven was slower because
> it
> > needed to finish building the entire module before dependent modules
> could
> > start which included running tests, performing checkstyle, etc...
> > Gradle is able to increase the parallelism of the build process since it
> > has task driven parallelism so as long as the files are compiled, the
> > dependent projects can start.
>
> This means we can implement a maven graph builder which is better than
> the default one - surely with a thread safe local repo - and
> contribute it back to solve it durably.
>
> If speed for a clean build was the only problem then maybe but lack of
incremental builds across tasks is a goal we can actually achieve using
Gradle and won't require rewriting almost all of the Maven plugins to
support incremental builds.


> >
> > Maven and Gradle are both heavily used since there are ~146k Maven
> projects
> > on Github while there are ~122k Gradle project on Github. Do you have
> data
> > which shows that Maven is significantly more "mainstream"?
>
> Yep, project i worked on in companies using gradle: 0, all were based
> on maven and maven was "tool-ed" versus gradle was "best effort" in
> term of plugins.
> Now  - with my EE background - I can guarantee you gradle is not able
> to handle properly its build since it flatten the classpath and
> plugins conflicts very quickly (their plugin dependency feature never
> worked and almost no plugin impl it correctly).
>
> Wonder if it is easy to have the ASF stats, anyone knows?

>
> > I believe we want a rich multi-language SDK and community and feel as
> > though it would be unwise to treat non JDK based languages as second
> class.
>
> Hmm, not sure how it is related to the build tool since Maven and
> Gradle have the same level of support - actually surprsingly maven is
> better for js and surely as bad as gradle for others - or here again
> we can create plugin like the frontend-maven-plugin if needed for
> other languages.
>
> That said it can be an interesting other thread since people consuming
> these languages will probably want their mainstream build tool and a
> "standard" repository layout rather than a java one. But this is
> harder to measure.
>
>
>
> > On Mon, Nov 27, 2017 at 11:00 AM, Romain Manni-Bucau <
> rmannibu...@gmail.com>
> > wrote:
> >
> >> Hi Lukasz,
> >>
> >> Did you manage to identify how maven was slower and test tesla stuff
> >> and potentially a few other fixes?
> >>
> >> Side note: figures without python can be interesting cause locally - =
> >> for me - python tends to flatten the figures whereas I get something
> >> close to your conclusions without python part.
> >>
> >> My point is mainly that switching now on gradle and being back on
> >> maven in a few months cause gradle ecosystem is far to support java 9
> >> - or any other volatile reason like this one - is probably not a good
> >> choice for a community. Maven is way more mainstream than gradle so
> >> helps to encourage people to contribute - vs gradle will increase the
> >> step to do it.
> >>
> >> I'd like to be sure before a switch that it is a one way decision and
> >> that the build tool was not just challenged by itself and its current
> >> state but also in the way it could be improved (= its community and
> >> potentially some local hacks).
> >>
> >> Romain Manni-Bucau
> >> @rmannibucau |  Blog | Old Blog | Github | LinkedIn
> >>
> >>
> >> 2017-11-27 19:46 GMT+01:00 Lukasz Cwik :
> >> > I have collected data by running several builds against master using
> >> Gradle
> >> > and Maven without using Gradle's support for incremental builds.
> >> >
> >> > Gradle (mins)
> >> > min: 25.04
> >> > max: 160.14
> >> > median: 45.78
> >> > average: 52.19
> >> > stdev: 30.80
> >> >
> >> > Maven (mins)
> >> > min: 56.86
> >> > max: 216.55 (actually > 240 mins because this data does not include
> >> > timeouts)
> >> > median: 87.93
> >> > average: 109.10
> >> > stdev: 48.01
> >> >
> >> > I excluded a few timeouts (240 mins) that happened during the Maven
> build
> >> > from its numbers but we can see conclusively that Gradle is twice as
> fast
> >> > for the build when compared to Maven when run using Jenkins.
> >> > On my desktop, I have enabled incremental builds and have seen a major
> >> > improvement on the above numbers but it doesn't yet work correctly
> >> because
> >> > of incorrectly specified inputs/outputs for certain tasks.
> >> >
> >> > The data is available here
> >> > https://docs.google.com/spreadsheets/d/1MHVjF-xoI49_
> >> NJqEQakUgnNIQ7Qbjzu8Y1q_h3dbF1M/edit?usp=sharing
> >> >
> >> > With this data, I feel confident that we should swap and have opened
> the
> >> > following issue https://issues.apache.org/jira/browse/BEAM-3249 and
> >> related
> >> > sub-tasks.
> >> >

Re: [DISCUSS] Move away from Apache Maven as build tool

[Romain Manni-Bucau]

2017-11-27 20:26 GMT+01:00 Lukasz Cwik :
> Romain, as mentioned earlier, I identified that Maven was slower because it
> needed to finish building the entire module before dependent modules could
> start which included running tests, performing checkstyle, etc...
> Gradle is able to increase the parallelism of the build process since it
> has task driven parallelism so as long as the files are compiled, the
> dependent projects can start.

This means we can implement a maven graph builder which is better than
the default one - surely with a thread safe local repo - and
contribute it back to solve it durably.

>
> Maven and Gradle are both heavily used since there are ~146k Maven projects
> on Github while there are ~122k Gradle project on Github. Do you have data
> which shows that Maven is significantly more "mainstream"?

Yep, project i worked on in companies using gradle: 0, all were based
on maven and maven was "tool-ed" versus gradle was "best effort" in
term of plugins.
Now  - with my EE background - I can guarantee you gradle is not able
to handle properly its build since it flatten the classpath and
plugins conflicts very quickly (their plugin dependency feature never
worked and almost no plugin impl it correctly).

Wonder if it is easy to have the ASF stats, anyone knows?

>
> I believe we want a rich multi-language SDK and community and feel as
> though it would be unwise to treat non JDK based languages as second class.

Hmm, not sure how it is related to the build tool since Maven and
Gradle have the same level of support - actually surprsingly maven is
better for js and surely as bad as gradle for others - or here again
we can create plugin like the frontend-maven-plugin if needed for
other languages.

That said it can be an interesting other thread since people consuming
these languages will probably want their mainstream build tool and a
"standard" repository layout rather than a java one. But this is
harder to measure.

>
> On Mon, Nov 27, 2017 at 11:00 AM, Romain Manni-Bucau 
> wrote:
>
>> Hi Lukasz,
>>
>> Did you manage to identify how maven was slower and test tesla stuff
>> and potentially a few other fixes?
>>
>> Side note: figures without python can be interesting cause locally - =
>> for me - python tends to flatten the figures whereas I get something
>> close to your conclusions without python part.
>>
>> My point is mainly that switching now on gradle and being back on
>> maven in a few months cause gradle ecosystem is far to support java 9
>> - or any other volatile reason like this one - is probably not a good
>> choice for a community. Maven is way more mainstream than gradle so
>> helps to encourage people to contribute - vs gradle will increase the
>> step to do it.
>>
>> I'd like to be sure before a switch that it is a one way decision and
>> that the build tool was not just challenged by itself and its current
>> state but also in the way it could be improved (= its community and
>> potentially some local hacks).
>>
>> Romain Manni-Bucau
>> @rmannibucau |  Blog | Old Blog | Github | LinkedIn
>>
>>
>> 2017-11-27 19:46 GMT+01:00 Lukasz Cwik :
>> > I have collected data by running several builds against master using
>> Gradle
>> > and Maven without using Gradle's support for incremental builds.
>> >
>> > Gradle (mins)
>> > min: 25.04
>> > max: 160.14
>> > median: 45.78
>> > average: 52.19
>> > stdev: 30.80
>> >
>> > Maven (mins)
>> > min: 56.86
>> > max: 216.55 (actually > 240 mins because this data does not include
>> > timeouts)
>> > median: 87.93
>> > average: 109.10
>> > stdev: 48.01
>> >
>> > I excluded a few timeouts (240 mins) that happened during the Maven build
>> > from its numbers but we can see conclusively that Gradle is twice as fast
>> > for the build when compared to Maven when run using Jenkins.
>> > On my desktop, I have enabled incremental builds and have seen a major
>> > improvement on the above numbers but it doesn't yet work correctly
>> because
>> > of incorrectly specified inputs/outputs for certain tasks.
>> >
>> > The data is available here
>> > https://docs.google.com/spreadsheets/d/1MHVjF-xoI49_
>> NJqEQakUgnNIQ7Qbjzu8Y1q_h3dbF1M/edit?usp=sharing
>> >
>> > With this data, I feel confident that we should swap and have opened the
>> > following issue https://issues.apache.org/jira/browse/BEAM-3249 and
>> related
>> > sub-tasks.
>> >
>> > On Sun, Nov 19, 2017 at 11:23 AM, Jean-Baptiste Onofré 
>> > wrote:
>> >
>> >> Thanks for the update Luke.
>> >>
>> >> I'm updating my local working copy to do new tests.
>> >>
>> >> Regards
>> >> JB
>> >>
>> >> On 11/19/2017 08:21 PM, Lukasz Cwik wrote:
>> >>
>> >>> The gradle build rules have been merged, I'm adding a precommit[1] to
>> >>> start
>> >>> collecting data about the build times. It currently only mirrors the
>> Java
>> >>> mvn install precommit. I'll gather data over the next two weeks and
>> >>> provide
>> 

Re: [DISCUSS] Move away from Apache Maven as build tool

[Lukasz Cwik]

Romain, as mentioned earlier, I identified that Maven was slower because it
needed to finish building the entire module before dependent modules could
start which included running tests, performing checkstyle, etc...
Gradle is able to increase the parallelism of the build process since it
has task driven parallelism so as long as the files are compiled, the
dependent projects can start.

Maven and Gradle are both heavily used since there are ~146k Maven projects
on Github while there are ~122k Gradle project on Github. Do you have data
which shows that Maven is significantly more "mainstream"?

I believe we want a rich multi-language SDK and community and feel as
though it would be unwise to treat non JDK based languages as second class.

On Mon, Nov 27, 2017 at 11:00 AM, Romain Manni-Bucau 
wrote:

> Hi Lukasz,
>
> Did you manage to identify how maven was slower and test tesla stuff
> and potentially a few other fixes?
>
> Side note: figures without python can be interesting cause locally - =
> for me - python tends to flatten the figures whereas I get something
> close to your conclusions without python part.
>
> My point is mainly that switching now on gradle and being back on
> maven in a few months cause gradle ecosystem is far to support java 9
> - or any other volatile reason like this one - is probably not a good
> choice for a community. Maven is way more mainstream than gradle so
> helps to encourage people to contribute - vs gradle will increase the
> step to do it.
>
> I'd like to be sure before a switch that it is a one way decision and
> that the build tool was not just challenged by itself and its current
> state but also in the way it could be improved (= its community and
> potentially some local hacks).
>
> Romain Manni-Bucau
> @rmannibucau |  Blog | Old Blog | Github | LinkedIn
>
>
> 2017-11-27 19:46 GMT+01:00 Lukasz Cwik :
> > I have collected data by running several builds against master using
> Gradle
> > and Maven without using Gradle's support for incremental builds.
> >
> > Gradle (mins)
> > min: 25.04
> > max: 160.14
> > median: 45.78
> > average: 52.19
> > stdev: 30.80
> >
> > Maven (mins)
> > min: 56.86
> > max: 216.55 (actually > 240 mins because this data does not include
> > timeouts)
> > median: 87.93
> > average: 109.10
> > stdev: 48.01
> >
> > I excluded a few timeouts (240 mins) that happened during the Maven build
> > from its numbers but we can see conclusively that Gradle is twice as fast
> > for the build when compared to Maven when run using Jenkins.
> > On my desktop, I have enabled incremental builds and have seen a major
> > improvement on the above numbers but it doesn't yet work correctly
> because
> > of incorrectly specified inputs/outputs for certain tasks.
> >
> > The data is available here
> > https://docs.google.com/spreadsheets/d/1MHVjF-xoI49_
> NJqEQakUgnNIQ7Qbjzu8Y1q_h3dbF1M/edit?usp=sharing
> >
> > With this data, I feel confident that we should swap and have opened the
> > following issue https://issues.apache.org/jira/browse/BEAM-3249 and
> related
> > sub-tasks.
> >
> > On Sun, Nov 19, 2017 at 11:23 AM, Jean-Baptiste Onofré 
> > wrote:
> >
> >> Thanks for the update Luke.
> >>
> >> I'm updating my local working copy to do new tests.
> >>
> >> Regards
> >> JB
> >>
> >> On 11/19/2017 08:21 PM, Lukasz Cwik wrote:
> >>
> >>> The gradle build rules have been merged, I'm adding a precommit[1] to
> >>> start
> >>> collecting data about the build times. It currently only mirrors the
> Java
> >>> mvn install precommit. I'll gather data over the next two weeks and
> >>> provide
> >>> a summary here.
> >>>
> >>> You can rerun the precommit by issuing "Run Java Gradle PreCommit"
> >>>
> >>> 1: https://github.com/apache/beam/pull/4146
> >>>
> >>>
> >>> On Mon, Nov 13, 2017 at 9:08 AM, Lukasz Cwik  wrote:
> >>>
> >>> There has been plenty of time for comments on the PR and the approach.
> 
>  So far Ken Knowles has provided the most feedback on the PR, Ken would
>  you
>  like to finish the review?
> 
> 
> 
>  On Fri, Nov 10, 2017 at 1:22 PM, Romain Manni-Bucau <
>  rmannibu...@gmail.com
> 
> > wrote:
> >
> 
>  This is only a setup thing and better to not break the master history
> for
> > poc/tests, in particular when no very localized. Alternative can be
> to
> > ask
> > another temp repo to infra and have a synchro between both but dont
> > think
> > it does worth it personally.
> >
> >
> >
> > Le 10 nov. 2017 18:57, "Lukasz Cwik"  a
> > écrit :
> >
> > The reason to get it on master is because that is where all the PRs
> >>
> > are. An
> >
> >> upstream branch without any development means no data.
> >> Also, our Jenkins setup via job-dsl doesn't honor using the Jenkins
> >> configuration on the branch because the seed job 

Re: [DISCUSS] Move away from Apache Maven as build tool

[Romain Manni-Bucau]

Hi Lukasz,

Did you manage to identify how maven was slower and test tesla stuff
and potentially a few other fixes?

Side note: figures without python can be interesting cause locally - =
for me - python tends to flatten the figures whereas I get something
close to your conclusions without python part.

My point is mainly that switching now on gradle and being back on
maven in a few months cause gradle ecosystem is far to support java 9
- or any other volatile reason like this one - is probably not a good
choice for a community. Maven is way more mainstream than gradle so
helps to encourage people to contribute - vs gradle will increase the
step to do it.

I'd like to be sure before a switch that it is a one way decision and
that the build tool was not just challenged by itself and its current
state but also in the way it could be improved (= its community and
potentially some local hacks).

Romain Manni-Bucau
@rmannibucau |  Blog | Old Blog | Github | LinkedIn


2017-11-27 19:46 GMT+01:00 Lukasz Cwik :
> I have collected data by running several builds against master using Gradle
> and Maven without using Gradle's support for incremental builds.
>
> Gradle (mins)
> min: 25.04
> max: 160.14
> median: 45.78
> average: 52.19
> stdev: 30.80
>
> Maven (mins)
> min: 56.86
> max: 216.55 (actually > 240 mins because this data does not include
> timeouts)
> median: 87.93
> average: 109.10
> stdev: 48.01
>
> I excluded a few timeouts (240 mins) that happened during the Maven build
> from its numbers but we can see conclusively that Gradle is twice as fast
> for the build when compared to Maven when run using Jenkins.
> On my desktop, I have enabled incremental builds and have seen a major
> improvement on the above numbers but it doesn't yet work correctly because
> of incorrectly specified inputs/outputs for certain tasks.
>
> The data is available here
> https://docs.google.com/spreadsheets/d/1MHVjF-xoI49_NJqEQakUgnNIQ7Qbjzu8Y1q_h3dbF1M/edit?usp=sharing
>
> With this data, I feel confident that we should swap and have opened the
> following issue https://issues.apache.org/jira/browse/BEAM-3249 and related
> sub-tasks.
>
> On Sun, Nov 19, 2017 at 11:23 AM, Jean-Baptiste Onofré 
> wrote:
>
>> Thanks for the update Luke.
>>
>> I'm updating my local working copy to do new tests.
>>
>> Regards
>> JB
>>
>> On 11/19/2017 08:21 PM, Lukasz Cwik wrote:
>>
>>> The gradle build rules have been merged, I'm adding a precommit[1] to
>>> start
>>> collecting data about the build times. It currently only mirrors the Java
>>> mvn install precommit. I'll gather data over the next two weeks and
>>> provide
>>> a summary here.
>>>
>>> You can rerun the precommit by issuing "Run Java Gradle PreCommit"
>>>
>>> 1: https://github.com/apache/beam/pull/4146
>>>
>>>
>>> On Mon, Nov 13, 2017 at 9:08 AM, Lukasz Cwik  wrote:
>>>
>>> There has been plenty of time for comments on the PR and the approach.

 So far Ken Knowles has provided the most feedback on the PR, Ken would
 you
 like to finish the review?



 On Fri, Nov 10, 2017 at 1:22 PM, Romain Manni-Bucau <
 rmannibu...@gmail.com

> wrote:
>

 This is only a setup thing and better to not break the master history for
> poc/tests, in particular when no very localized. Alternative can be to
> ask
> another temp repo to infra and have a synchro between both but dont
> think
> it does worth it personally.
>
>
>
> Le 10 nov. 2017 18:57, "Lukasz Cwik"  a
> écrit :
>
> The reason to get it on master is because that is where all the PRs
>>
> are. An
>
>> upstream branch without any development means no data.
>> Also, our Jenkins setup via job-dsl doesn't honor using the Jenkins
>> configuration on the branch because the seed job always runs against
>> master.
>>
>> On Thu, Nov 9, 2017 at 9:59 PM, Romain Manni-Bucau <
>>
> rmannibu...@gmail.com>
>
>> wrote:
>>
>> What about pushing it on a "upstream" branch and testing it for 1
>>>
>> week in
>
>> parallel of the maven reference build? If gradle is always 50% faster
>>>
>> on
>
>> jenkins then it could become master setup without much discussion I
>>>
>> guess.
>>
>>> We can even have 2 jenkins jobs: one with the daemon etc and one
>>>
>> without.
>
>>
>>> Also noticed yesterday that gradle build is killing my machine (all 8
>>>
>> cores
>>
>>> are 100%) during the first minutes vs maven build which let me do
>>>
>> something
>>
>>> else. Then all the consumed time which makes gradle not that fast is
>>>
>> about
>>
>>> python. Will try to send figures later today.
>>>
>>> Le 10 nov. 2017 00:10, "Lukasz Cwik"  a
>>>
>> écrit
>
>> :

Re: [DISCUSS] Move away from Apache Maven as build tool

[Lukasz Cwik]

I have collected data by running several builds against master using Gradle
and Maven without using Gradle's support for incremental builds.

Gradle (mins)
min: 25.04
max: 160.14
median: 45.78
average: 52.19
stdev: 30.80

Maven (mins)
min: 56.86
max: 216.55 (actually > 240 mins because this data does not include
timeouts)
median: 87.93
average: 109.10
stdev: 48.01

I excluded a few timeouts (240 mins) that happened during the Maven build
from its numbers but we can see conclusively that Gradle is twice as fast
for the build when compared to Maven when run using Jenkins.
On my desktop, I have enabled incremental builds and have seen a major
improvement on the above numbers but it doesn't yet work correctly because
of incorrectly specified inputs/outputs for certain tasks.

The data is available here
https://docs.google.com/spreadsheets/d/1MHVjF-xoI49_NJqEQakUgnNIQ7Qbjzu8Y1q_h3dbF1M/edit?usp=sharing

With this data, I feel confident that we should swap and have opened the
following issue https://issues.apache.org/jira/browse/BEAM-3249 and related
sub-tasks.

On Sun, Nov 19, 2017 at 11:23 AM, Jean-Baptiste Onofré 
wrote:

> Thanks for the update Luke.
>
> I'm updating my local working copy to do new tests.
>
> Regards
> JB
>
> On 11/19/2017 08:21 PM, Lukasz Cwik wrote:
>
>> The gradle build rules have been merged, I'm adding a precommit[1] to
>> start
>> collecting data about the build times. It currently only mirrors the Java
>> mvn install precommit. I'll gather data over the next two weeks and
>> provide
>> a summary here.
>>
>> You can rerun the precommit by issuing "Run Java Gradle PreCommit"
>>
>> 1: https://github.com/apache/beam/pull/4146
>>
>>
>> On Mon, Nov 13, 2017 at 9:08 AM, Lukasz Cwik  wrote:
>>
>> There has been plenty of time for comments on the PR and the approach.
>>>
>>> So far Ken Knowles has provided the most feedback on the PR, Ken would
>>> you
>>> like to finish the review?
>>>
>>>
>>>
>>> On Fri, Nov 10, 2017 at 1:22 PM, Romain Manni-Bucau <
>>> rmannibu...@gmail.com
>>>
 wrote:

>>>
>>> This is only a setup thing and better to not break the master history for
 poc/tests, in particular when no very localized. Alternative can be to
 ask
 another temp repo to infra and have a synchro between both but dont
 think
 it does worth it personally.



 Le 10 nov. 2017 18:57, "Lukasz Cwik"  a
 écrit :

 The reason to get it on master is because that is where all the PRs
>
 are. An

> upstream branch without any development means no data.
> Also, our Jenkins setup via job-dsl doesn't honor using the Jenkins
> configuration on the branch because the seed job always runs against
> master.
>
> On Thu, Nov 9, 2017 at 9:59 PM, Romain Manni-Bucau <
>
 rmannibu...@gmail.com>

> wrote:
>
> What about pushing it on a "upstream" branch and testing it for 1
>>
> week in

> parallel of the maven reference build? If gradle is always 50% faster
>>
> on

> jenkins then it could become master setup without much discussion I
>>
> guess.
>
>> We can even have 2 jenkins jobs: one with the daemon etc and one
>>
> without.

>
>> Also noticed yesterday that gradle build is killing my machine (all 8
>>
> cores
>
>> are 100%) during the first minutes vs maven build which let me do
>>
> something
>
>> else. Then all the consumed time which makes gradle not that fast is
>>
> about
>
>> python. Will try to send figures later today.
>>
>> Le 10 nov. 2017 00:10, "Lukasz Cwik"  a
>>
> écrit

> :
>
>>
>> I wouldn't mind merging this change in so I could setup those Gradle
>>> Jenkins precommits.
>>>
>>> As per our contribution guidelines, any committer willing to sign
>>>
>> off

> on
>
>> the PR?
>>>
>>> On Thu, Nov 9, 2017 at 2:12 PM, Romain Manni-Bucau <
>>>
>> rmannibu...@gmail.com>
>>
>>> wrote:
>>>
>>> Le 9 nov. 2017 21:31, "Kenneth Knowles" 

>>> a

> écrit :
>>>

 Keep in mind that a clean build is unusual during development (it

>>> is

> common
>>>
 for mvn use and that is a bug) and also not necessary for

>>> precommits

> if
>
>> the
>>>
 build tool is correct enough that caching is safe. So while this

>>> number
>
>> matters, it is not the most important.


 Not sure, in dev you bypass the build tool most of the time

>>> anyway -

> thanks
>>>
 to IDE or other shortcuts - but not on PR and CI. Keep in mind

>>> that

> not
>
>> doing a clean and killing 

Re: [DISCUSS] Move away from Apache Maven as build tool

[Jean-Baptiste Onofré]

Thanks for the update Luke.

I'm updating my local working copy to do new tests.

Regards
JB

On 11/19/2017 08:21 PM, Lukasz Cwik wrote:

The gradle build rules have been merged, I'm adding a precommit[1] to start
collecting data about the build times. It currently only mirrors the Java
mvn install precommit. I'll gather data over the next two weeks and provide
a summary here.

You can rerun the precommit by issuing "Run Java Gradle PreCommit"

1: https://github.com/apache/beam/pull/4146


On Mon, Nov 13, 2017 at 9:08 AM, Lukasz Cwik  wrote:


There has been plenty of time for comments on the PR and the approach.

So far Ken Knowles has provided the most feedback on the PR, Ken would you
like to finish the review?



On Fri, Nov 10, 2017 at 1:22 PM, Romain Manni-Bucau  a écrit :


The reason to get it on master is because that is where all the PRs

are. An

upstream branch without any development means no data.
Also, our Jenkins setup via job-dsl doesn't honor using the Jenkins
configuration on the branch because the seed job always runs against
master.

On Thu, Nov 9, 2017 at 9:59 PM, Romain Manni-Bucau <

rmannibu...@gmail.com>

wrote:


What about pushing it on a "upstream" branch and testing it for 1

week in

parallel of the maven reference build? If gradle is always 50% faster

on

jenkins then it could become master setup without much discussion I

guess.

We can even have 2 jenkins jobs: one with the daemon etc and one

without.


Also noticed yesterday that gradle build is killing my machine (all 8

cores

are 100%) during the first minutes vs maven build which let me do

something

else. Then all the consumed time which makes gradle not that fast is

about

python. Will try to send figures later today.

Le 10 nov. 2017 00:10, "Lukasz Cwik"  a

écrit

:



I wouldn't mind merging this change in so I could setup those Gradle
Jenkins precommits.

As per our contribution guidelines, any committer willing to sign

off

on

the PR?

On Thu, Nov 9, 2017 at 2:12 PM, Romain Manni-Bucau <

rmannibu...@gmail.com>

wrote:


Le 9 nov. 2017 21:31, "Kenneth Knowles" 

a

écrit :


Keep in mind that a clean build is unusual during development (it

is

common

for mvn use and that is a bug) and also not necessary for

precommits

if

the

build tool is correct enough that caching is safe. So while this

number

matters, it is not the most important.


Not sure, in dev you bypass the build tool most of the time

anyway -

thanks

to IDE or other shortcuts - but not on PR and CI. Keep in mind

that

not

doing a clean and killing gradle daemon makes the build not

reproducible

and therefore useful :(. Starting to build from a subpart of the

reactor

-

with the mentionned mvn plugin for instance - can be nice on some

CI

like

travis if the caching is well configured but still not a guarantee

the

build is "green".

My trade off is to ensure an easy build and relevant result over

the

time

criteria. Do you share it as well or prefer time over other

criteria

-

which leads to other conclusions and options indeed and can make

us

not

understanding each other?


On Thu, Nov 9, 2017 at 11:30 AM, Romain Manni-Bucau <

rmannibu...@gmail.com



wrote:


I will try next week yes but the 2 runs i did were 28mn vs 32mn

from

memory

- after having downloaded all deps once.

Le 9 nov. 2017 19:45, "Lukasz Cwik" 

a

écrit :



If Gradle was slow, do you mind running the build with

--profile

and

sharing that and also sharing the Maven build log?

On Thu, Nov 9, 2017 at 10:43 AM, Lukasz Cwik <

lc...@google.com>

wrote:



Romain, I don't understand your last comment, were you

trying

to

say

that

you had the same Gradle build times like I did and it was an

improvement

over Maven or that you did not and you experienced build

times

that

were

equivalent to Maven?

On Thu, Nov 9, 2017 at 9:51 AM, Romain Manni-Bucau <

rmannibu...@gmail.com>

wrote:


2017-11-09 18:38 GMT+01:00 Kenneth Knowles



wrote:


(this is another topic so we can maybe open another

thread)

issue

is

not much about python but more about the fact the build

is

not

self

contained. it is a maven build and maven should be

sufficient

without

having to install python + dependencies.



Let's leave out the topic of whether our build should

install

things

like

JDKs, Python, Golang, Docker, protoc, findbugs, RAT, etc.

That

issue

is

somewhat independent of build tool, and the new build

Re: [DISCUSS] Move away from Apache Maven as build tool

[Lukasz Cwik]

The gradle build rules have been merged, I'm adding a precommit[1] to start
collecting data about the build times. It currently only mirrors the Java
mvn install precommit. I'll gather data over the next two weeks and provide
a summary here.

You can rerun the precommit by issuing "Run Java Gradle PreCommit"

1: https://github.com/apache/beam/pull/4146


On Mon, Nov 13, 2017 at 9:08 AM, Lukasz Cwik  wrote:

> There has been plenty of time for comments on the PR and the approach.
>
> So far Ken Knowles has provided the most feedback on the PR, Ken would you
> like to finish the review?
>
>
>
> On Fri, Nov 10, 2017 at 1:22 PM, Romain Manni-Bucau  > wrote:
>
>> This is only a setup thing and better to not break the master history for
>> poc/tests, in particular when no very localized. Alternative can be to ask
>> another temp repo to infra and have a synchro between both but dont think
>> it does worth it personally.
>>
>>
>>
>> Le 10 nov. 2017 18:57, "Lukasz Cwik"  a écrit :
>>
>> > The reason to get it on master is because that is where all the PRs
>> are. An
>> > upstream branch without any development means no data.
>> > Also, our Jenkins setup via job-dsl doesn't honor using the Jenkins
>> > configuration on the branch because the seed job always runs against
>> > master.
>> >
>> > On Thu, Nov 9, 2017 at 9:59 PM, Romain Manni-Bucau <
>> rmannibu...@gmail.com>
>> > wrote:
>> >
>> > > What about pushing it on a "upstream" branch and testing it for 1
>> week in
>> > > parallel of the maven reference build? If gradle is always 50% faster
>> on
>> > > jenkins then it could become master setup without much discussion I
>> > guess.
>> > > We can even have 2 jenkins jobs: one with the daemon etc and one
>> without.
>> > >
>> > > Also noticed yesterday that gradle build is killing my machine (all 8
>> > cores
>> > > are 100%) during the first minutes vs maven build which let me do
>> > something
>> > > else. Then all the consumed time which makes gradle not that fast is
>> > about
>> > > python. Will try to send figures later today.
>> > >
>> > > Le 10 nov. 2017 00:10, "Lukasz Cwik"  a
>> écrit
>> > :
>> > >
>> > > > I wouldn't mind merging this change in so I could setup those Gradle
>> > > > Jenkins precommits.
>> > > >
>> > > > As per our contribution guidelines, any committer willing to sign
>> off
>> > on
>> > > > the PR?
>> > > >
>> > > > On Thu, Nov 9, 2017 at 2:12 PM, Romain Manni-Bucau <
>> > > rmannibu...@gmail.com>
>> > > > wrote:
>> > > >
>> > > > > Le 9 nov. 2017 21:31, "Kenneth Knowles" 
>> a
>> > > > écrit :
>> > > > >
>> > > > > Keep in mind that a clean build is unusual during development (it
>> is
>> > > > common
>> > > > > for mvn use and that is a bug) and also not necessary for
>> precommits
>> > if
>> > > > the
>> > > > > build tool is correct enough that caching is safe. So while this
>> > number
>> > > > > matters, it is not the most important.
>> > > > >
>> > > > >
>> > > > > Not sure, in dev you bypass the build tool most of the time
>> anyway -
>> > > > thanks
>> > > > > to IDE or other shortcuts - but not on PR and CI. Keep in mind
>> that
>> > not
>> > > > > doing a clean and killing gradle daemon makes the build not
>> > > reproducible
>> > > > > and therefore useful :(. Starting to build from a subpart of the
>> > > reactor
>> > > > -
>> > > > > with the mentionned mvn plugin for instance - can be nice on some
>> CI
>> > > like
>> > > > > travis if the caching is well configured but still not a guarantee
>> > the
>> > > > > build is "green".
>> > > > >
>> > > > > My trade off is to ensure an easy build and relevant result over
>> the
>> > > time
>> > > > > criteria. Do you share it as well or prefer time over other
>> criteria
>> > -
>> > > > > which leads to other conclusions and options indeed and can make
>> us
>> > not
>> > > > > understanding each other?
>> > > > >
>> > > > >
>> > > > > On Thu, Nov 9, 2017 at 11:30 AM, Romain Manni-Bucau <
>> > > > rmannibu...@gmail.com
>> > > > > >
>> > > > > wrote:
>> > > > >
>> > > > > > I will try next week yes but the 2 runs i did were 28mn vs 32mn
>> > from
>> > > > > memory
>> > > > > > - after having downloaded all deps once.
>> > > > > >
>> > > > > > Le 9 nov. 2017 19:45, "Lukasz Cwik" 
>> a
>> > > > écrit :
>> > > > > >
>> > > > > > > If Gradle was slow, do you mind running the build with
>> --profile
>> > > and
>> > > > > > > sharing that and also sharing the Maven build log?
>> > > > > > >
>> > > > > > > On Thu, Nov 9, 2017 at 10:43 AM, Lukasz Cwik <
>> lc...@google.com>
>> > > > wrote:
>> > > > > > >
>> > > > > > > > Romain, I don't understand your last comment, were you
>> trying
>> > to
>> > > > say
>> > > > > > that
>> > > > > > > > you had the same Gradle build times like I did and it was an
>> > > > > > improvement
>> > > > > > > > over Maven or that you did not and you experienced 

Re: [DISCUSS] Move away from Apache Maven as build tool

[Lukasz Cwik]

The reason to get it on master is because that is where all the PRs are. An
upstream branch without any development means no data.
Also, our Jenkins setup via job-dsl doesn't honor using the Jenkins
configuration on the branch because the seed job always runs against master.

On Thu, Nov 9, 2017 at 9:59 PM, Romain Manni-Bucau 
wrote:

> What about pushing it on a "upstream" branch and testing it for 1 week in
> parallel of the maven reference build? If gradle is always 50% faster on
> jenkins then it could become master setup without much discussion I guess.
> We can even have 2 jenkins jobs: one with the daemon etc and one without.
>
> Also noticed yesterday that gradle build is killing my machine (all 8 cores
> are 100%) during the first minutes vs maven build which let me do something
> else. Then all the consumed time which makes gradle not that fast is about
> python. Will try to send figures later today.
>
> Le 10 nov. 2017 00:10, "Lukasz Cwik"  a écrit :
>
> > I wouldn't mind merging this change in so I could setup those Gradle
> > Jenkins precommits.
> >
> > As per our contribution guidelines, any committer willing to sign off on
> > the PR?
> >
> > On Thu, Nov 9, 2017 at 2:12 PM, Romain Manni-Bucau <
> rmannibu...@gmail.com>
> > wrote:
> >
> > > Le 9 nov. 2017 21:31, "Kenneth Knowles"  a
> > écrit :
> > >
> > > Keep in mind that a clean build is unusual during development (it is
> > common
> > > for mvn use and that is a bug) and also not necessary for precommits if
> > the
> > > build tool is correct enough that caching is safe. So while this number
> > > matters, it is not the most important.
> > >
> > >
> > > Not sure, in dev you bypass the build tool most of the time anyway -
> > thanks
> > > to IDE or other shortcuts - but not on PR and CI. Keep in mind that not
> > > doing a clean and killing gradle daemon makes the build not
> reproducible
> > > and therefore useful :(. Starting to build from a subpart of the
> reactor
> > -
> > > with the mentionned mvn plugin for instance - can be nice on some CI
> like
> > > travis if the caching is well configured but still not a guarantee the
> > > build is "green".
> > >
> > > My trade off is to ensure an easy build and relevant result over the
> time
> > > criteria. Do you share it as well or prefer time over other criteria -
> > > which leads to other conclusions and options indeed and can make us not
> > > understanding each other?
> > >
> > >
> > > On Thu, Nov 9, 2017 at 11:30 AM, Romain Manni-Bucau <
> > rmannibu...@gmail.com
> > > >
> > > wrote:
> > >
> > > > I will try next week yes but the 2 runs i did were 28mn vs 32mn from
> > > memory
> > > > - after having downloaded all deps once.
> > > >
> > > > Le 9 nov. 2017 19:45, "Lukasz Cwik"  a
> > écrit :
> > > >
> > > > > If Gradle was slow, do you mind running the build with --profile
> and
> > > > > sharing that and also sharing the Maven build log?
> > > > >
> > > > > On Thu, Nov 9, 2017 at 10:43 AM, Lukasz Cwik 
> > wrote:
> > > > >
> > > > > > Romain, I don't understand your last comment, were you trying to
> > say
> > > > that
> > > > > > you had the same Gradle build times like I did and it was an
> > > > improvement
> > > > > > over Maven or that you did not and you experienced build times
> that
> > > > were
> > > > > > equivalent to Maven?
> > > > > >
> > > > > > On Thu, Nov 9, 2017 at 9:51 AM, Romain Manni-Bucau <
> > > > > rmannibu...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > >> 2017-11-09 18:38 GMT+01:00 Kenneth Knowles
>  > > >:
> > > > > >> > On Thu, Nov 9, 2017 at 9:11 AM, Romain Manni-Bucau <
> > > > > >> rmannibu...@gmail.com>
> > > > > >> > wrote:
> > > > > >> >
> > > > > >> >> (this is another topic so we can maybe open another thread)
> > issue
> > > > is
> > > > > >> >> not much about python but more about the fact the build is
> not
> > > self
> > > > > >> >> contained. it is a maven build and maven should be sufficient
> > > > without
> > > > > >> >> having to install python + dependencies.
> > > > > >> >
> > > > > >> >
> > > > > >> > Let's leave out the topic of whether our build should install
> > > things
> > > > > >> like
> > > > > >> > JDKs, Python, Golang, Docker, protoc, findbugs, RAT, etc. That
> > > issue
> > > > > is
> > > > > >> > somewhat independent of build tool, and the new build isn't
> > worse
> > > > than
> > > > > >> the
> > > > > >> > old one as far as it goes.
> > > > > >>
> > > > > >>
> > > > > >> Yep, globally the same time with clean and killing the daemon.
> > > > > >>
> > > > > >> >
> > > > > >> > Kenn
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> >> I don't see any technical
> > > > > >> >> blockers to do it (except time ;)) but it is always a bit
> > > annoying
> > > > to
> > > > > >> >> git clone then not be able to build.
> > > > > >> >>
> > > > > >> >> Romain Manni-Bucau
> > > > 

Re: [DISCUSS] Move away from Apache Maven as build tool

[Romain Manni-Bucau]

What about pushing it on a "upstream" branch and testing it for 1 week in
parallel of the maven reference build? If gradle is always 50% faster on
jenkins then it could become master setup without much discussion I guess.
We can even have 2 jenkins jobs: one with the daemon etc and one without.

Also noticed yesterday that gradle build is killing my machine (all 8 cores
are 100%) during the first minutes vs maven build which let me do something
else. Then all the consumed time which makes gradle not that fast is about
python. Will try to send figures later today.

Le 10 nov. 2017 00:10, "Lukasz Cwik"  a écrit :

> I wouldn't mind merging this change in so I could setup those Gradle
> Jenkins precommits.
>
> As per our contribution guidelines, any committer willing to sign off on
> the PR?
>
> On Thu, Nov 9, 2017 at 2:12 PM, Romain Manni-Bucau 
> wrote:
>
> > Le 9 nov. 2017 21:31, "Kenneth Knowles"  a
> écrit :
> >
> > Keep in mind that a clean build is unusual during development (it is
> common
> > for mvn use and that is a bug) and also not necessary for precommits if
> the
> > build tool is correct enough that caching is safe. So while this number
> > matters, it is not the most important.
> >
> >
> > Not sure, in dev you bypass the build tool most of the time anyway -
> thanks
> > to IDE or other shortcuts - but not on PR and CI. Keep in mind that not
> > doing a clean and killing gradle daemon makes the build not reproducible
> > and therefore useful :(. Starting to build from a subpart of the reactor
> -
> > with the mentionned mvn plugin for instance - can be nice on some CI like
> > travis if the caching is well configured but still not a guarantee the
> > build is "green".
> >
> > My trade off is to ensure an easy build and relevant result over the time
> > criteria. Do you share it as well or prefer time over other criteria -
> > which leads to other conclusions and options indeed and can make us not
> > understanding each other?
> >
> >
> > On Thu, Nov 9, 2017 at 11:30 AM, Romain Manni-Bucau <
> rmannibu...@gmail.com
> > >
> > wrote:
> >
> > > I will try next week yes but the 2 runs i did were 28mn vs 32mn from
> > memory
> > > - after having downloaded all deps once.
> > >
> > > Le 9 nov. 2017 19:45, "Lukasz Cwik"  a
> écrit :
> > >
> > > > If Gradle was slow, do you mind running the build with --profile and
> > > > sharing that and also sharing the Maven build log?
> > > >
> > > > On Thu, Nov 9, 2017 at 10:43 AM, Lukasz Cwik 
> wrote:
> > > >
> > > > > Romain, I don't understand your last comment, were you trying to
> say
> > > that
> > > > > you had the same Gradle build times like I did and it was an
> > > improvement
> > > > > over Maven or that you did not and you experienced build times that
> > > were
> > > > > equivalent to Maven?
> > > > >
> > > > > On Thu, Nov 9, 2017 at 9:51 AM, Romain Manni-Bucau <
> > > > rmannibu...@gmail.com>
> > > > > wrote:
> > > > >
> > > > >> 2017-11-09 18:38 GMT+01:00 Kenneth Knowles  > >:
> > > > >> > On Thu, Nov 9, 2017 at 9:11 AM, Romain Manni-Bucau <
> > > > >> rmannibu...@gmail.com>
> > > > >> > wrote:
> > > > >> >
> > > > >> >> (this is another topic so we can maybe open another thread)
> issue
> > > is
> > > > >> >> not much about python but more about the fact the build is not
> > self
> > > > >> >> contained. it is a maven build and maven should be sufficient
> > > without
> > > > >> >> having to install python + dependencies.
> > > > >> >
> > > > >> >
> > > > >> > Let's leave out the topic of whether our build should install
> > things
> > > > >> like
> > > > >> > JDKs, Python, Golang, Docker, protoc, findbugs, RAT, etc. That
> > issue
> > > > is
> > > > >> > somewhat independent of build tool, and the new build isn't
> worse
> > > than
> > > > >> the
> > > > >> > old one as far as it goes.
> > > > >>
> > > > >>
> > > > >> Yep, globally the same time with clean and killing the daemon.
> > > > >>
> > > > >> >
> > > > >> > Kenn
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> >> I don't see any technical
> > > > >> >> blockers to do it (except time ;)) but it is always a bit
> > annoying
> > > to
> > > > >> >> git clone then not be able to build.
> > > > >> >>
> > > > >> >> Romain Manni-Bucau
> > > > >> >> @rmannibucau |  Blog | Old Blog | Github | LinkedIn
> > > > >> >>
> > > > >> >>
> > > > >> >> 2017-11-09 18:07 GMT+01:00 Lukasz Cwik
>  > >:
> > > > >> >> > Hmm, I have had good luck when following the Python quick
> start
> > > > setup
> > > > >> >> >  on
> > multiple
> > > > >> >> machines
> > > > >> >> > by ensuring the installed version of setuptools, virtualenv
> and
> > > pip
> > > > >> are
> > > > >> >> new
> > > > >> >> > enough versions.
> > > > >> >> >
> > > > >> >> > You can always skip the Python portion of the build 

Re: [DISCUSS] Move away from Apache Maven as build tool

[Lukasz Cwik]

I wouldn't mind merging this change in so I could setup those Gradle
Jenkins precommits.

As per our contribution guidelines, any committer willing to sign off on
the PR?

On Thu, Nov 9, 2017 at 2:12 PM, Romain Manni-Bucau 
wrote:

> Le 9 nov. 2017 21:31, "Kenneth Knowles"  a écrit :
>
> Keep in mind that a clean build is unusual during development (it is common
> for mvn use and that is a bug) and also not necessary for precommits if the
> build tool is correct enough that caching is safe. So while this number
> matters, it is not the most important.
>
>
> Not sure, in dev you bypass the build tool most of the time anyway - thanks
> to IDE or other shortcuts - but not on PR and CI. Keep in mind that not
> doing a clean and killing gradle daemon makes the build not reproducible
> and therefore useful :(. Starting to build from a subpart of the reactor -
> with the mentionned mvn plugin for instance - can be nice on some CI like
> travis if the caching is well configured but still not a guarantee the
> build is "green".
>
> My trade off is to ensure an easy build and relevant result over the time
> criteria. Do you share it as well or prefer time over other criteria -
> which leads to other conclusions and options indeed and can make us not
> understanding each other?
>
>
> On Thu, Nov 9, 2017 at 11:30 AM, Romain Manni-Bucau  >
> wrote:
>
> > I will try next week yes but the 2 runs i did were 28mn vs 32mn from
> memory
> > - after having downloaded all deps once.
> >
> > Le 9 nov. 2017 19:45, "Lukasz Cwik"  a écrit :
> >
> > > If Gradle was slow, do you mind running the build with --profile and
> > > sharing that and also sharing the Maven build log?
> > >
> > > On Thu, Nov 9, 2017 at 10:43 AM, Lukasz Cwik  wrote:
> > >
> > > > Romain, I don't understand your last comment, were you trying to say
> > that
> > > > you had the same Gradle build times like I did and it was an
> > improvement
> > > > over Maven or that you did not and you experienced build times that
> > were
> > > > equivalent to Maven?
> > > >
> > > > On Thu, Nov 9, 2017 at 9:51 AM, Romain Manni-Bucau <
> > > rmannibu...@gmail.com>
> > > > wrote:
> > > >
> > > >> 2017-11-09 18:38 GMT+01:00 Kenneth Knowles  >:
> > > >> > On Thu, Nov 9, 2017 at 9:11 AM, Romain Manni-Bucau <
> > > >> rmannibu...@gmail.com>
> > > >> > wrote:
> > > >> >
> > > >> >> (this is another topic so we can maybe open another thread) issue
> > is
> > > >> >> not much about python but more about the fact the build is not
> self
> > > >> >> contained. it is a maven build and maven should be sufficient
> > without
> > > >> >> having to install python + dependencies.
> > > >> >
> > > >> >
> > > >> > Let's leave out the topic of whether our build should install
> things
> > > >> like
> > > >> > JDKs, Python, Golang, Docker, protoc, findbugs, RAT, etc. That
> issue
> > > is
> > > >> > somewhat independent of build tool, and the new build isn't worse
> > than
> > > >> the
> > > >> > old one as far as it goes.
> > > >>
> > > >>
> > > >> Yep, globally the same time with clean and killing the daemon.
> > > >>
> > > >> >
> > > >> > Kenn
> > > >> >
> > > >> >
> > > >> >
> > > >> >> I don't see any technical
> > > >> >> blockers to do it (except time ;)) but it is always a bit
> annoying
> > to
> > > >> >> git clone then not be able to build.
> > > >> >>
> > > >> >> Romain Manni-Bucau
> > > >> >> @rmannibucau |  Blog | Old Blog | Github | LinkedIn
> > > >> >>
> > > >> >>
> > > >> >> 2017-11-09 18:07 GMT+01:00 Lukasz Cwik  >:
> > > >> >> > Hmm, I have had good luck when following the Python quick start
> > > setup
> > > >> >> >  on
> multiple
> > > >> >> machines
> > > >> >> > by ensuring the installed version of setuptools, virtualenv and
> > pip
> > > >> are
> > > >> >> new
> > > >> >> > enough versions.
> > > >> >> >
> > > >> >> > You can always skip the Python portion of the build by
> excluding
> > > the
> > > >> >> build
> > > >> >> > task as so:
> > > >> >> > ./gradlew build -x ":beam-sdks-parent:beam-sdks-python:build"
> > > >> >> >
> > > >> >> > On Thu, Nov 9, 2017 at 8:58 AM, Romain Manni-Bucau <
> > > >> >> rmannibu...@gmail.com>
> > > >> >> > wrote:
> > > >> >> >
> > > >> >> >> The 1.3.5 file is when i installed the python dependencies
> > > manually
> > > >> >> >> to make the build passing (the pip command never passed on my
> > > >> computer
> > > >> >> >> and therefore the build always has been broken until i
> installed
> > > it
> > > >> >> >> manually - independently from the build tool).
> > > >> >> >>
> > > >> >> >> Romain Manni-Bucau
> > > >> >> >> @rmannibucau |  Blog | Old Blog | Github | LinkedIn
> > > >> >> >>
> > > >> >> >>
> > > >> >> >> 2017-11-09 17:51 GMT+01:00 Lukasz Cwik
>  > > >:
> > > >> >> >> > It turns out 

Re: [DISCUSS] Move away from Apache Maven as build tool

[Romain Manni-Bucau]

Le 9 nov. 2017 21:31, "Kenneth Knowles"  a écrit :

Keep in mind that a clean build is unusual during development (it is common
for mvn use and that is a bug) and also not necessary for precommits if the
build tool is correct enough that caching is safe. So while this number
matters, it is not the most important.


Not sure, in dev you bypass the build tool most of the time anyway - thanks
to IDE or other shortcuts - but not on PR and CI. Keep in mind that not
doing a clean and killing gradle daemon makes the build not reproducible
and therefore useful :(. Starting to build from a subpart of the reactor -
with the mentionned mvn plugin for instance - can be nice on some CI like
travis if the caching is well configured but still not a guarantee the
build is "green".

My trade off is to ensure an easy build and relevant result over the time
criteria. Do you share it as well or prefer time over other criteria -
which leads to other conclusions and options indeed and can make us not
understanding each other?


On Thu, Nov 9, 2017 at 11:30 AM, Romain Manni-Bucau 
wrote:

> I will try next week yes but the 2 runs i did were 28mn vs 32mn from
memory
> - after having downloaded all deps once.
>
> Le 9 nov. 2017 19:45, "Lukasz Cwik"  a écrit :
>
> > If Gradle was slow, do you mind running the build with --profile and
> > sharing that and also sharing the Maven build log?
> >
> > On Thu, Nov 9, 2017 at 10:43 AM, Lukasz Cwik  wrote:
> >
> > > Romain, I don't understand your last comment, were you trying to say
> that
> > > you had the same Gradle build times like I did and it was an
> improvement
> > > over Maven or that you did not and you experienced build times that
> were
> > > equivalent to Maven?
> > >
> > > On Thu, Nov 9, 2017 at 9:51 AM, Romain Manni-Bucau <
> > rmannibu...@gmail.com>
> > > wrote:
> > >
> > >> 2017-11-09 18:38 GMT+01:00 Kenneth Knowles :
> > >> > On Thu, Nov 9, 2017 at 9:11 AM, Romain Manni-Bucau <
> > >> rmannibu...@gmail.com>
> > >> > wrote:
> > >> >
> > >> >> (this is another topic so we can maybe open another thread) issue
> is
> > >> >> not much about python but more about the fact the build is not
self
> > >> >> contained. it is a maven build and maven should be sufficient
> without
> > >> >> having to install python + dependencies.
> > >> >
> > >> >
> > >> > Let's leave out the topic of whether our build should install
things
> > >> like
> > >> > JDKs, Python, Golang, Docker, protoc, findbugs, RAT, etc. That
issue
> > is
> > >> > somewhat independent of build tool, and the new build isn't worse
> than
> > >> the
> > >> > old one as far as it goes.
> > >>
> > >>
> > >> Yep, globally the same time with clean and killing the daemon.
> > >>
> > >> >
> > >> > Kenn
> > >> >
> > >> >
> > >> >
> > >> >> I don't see any technical
> > >> >> blockers to do it (except time ;)) but it is always a bit annoying
> to
> > >> >> git clone then not be able to build.
> > >> >>
> > >> >> Romain Manni-Bucau
> > >> >> @rmannibucau |  Blog | Old Blog | Github | LinkedIn
> > >> >>
> > >> >>
> > >> >> 2017-11-09 18:07 GMT+01:00 Lukasz Cwik :
> > >> >> > Hmm, I have had good luck when following the Python quick start
> > setup
> > >> >> >  on multiple
> > >> >> machines
> > >> >> > by ensuring the installed version of setuptools, virtualenv and
> pip
> > >> are
> > >> >> new
> > >> >> > enough versions.
> > >> >> >
> > >> >> > You can always skip the Python portion of the build by excluding
> > the
> > >> >> build
> > >> >> > task as so:
> > >> >> > ./gradlew build -x ":beam-sdks-parent:beam-sdks-python:build"
> > >> >> >
> > >> >> > On Thu, Nov 9, 2017 at 8:58 AM, Romain Manni-Bucau <
> > >> >> rmannibu...@gmail.com>
> > >> >> > wrote:
> > >> >> >
> > >> >> >> The 1.3.5 file is when i installed the python dependencies
> > manually
> > >> >> >> to make the build passing (the pip command never passed on my
> > >> computer
> > >> >> >> and therefore the build always has been broken until i
installed
> > it
> > >> >> >> manually - independently from the build tool).
> > >> >> >>
> > >> >> >> Romain Manni-Bucau
> > >> >> >> @rmannibucau |  Blog | Old Blog | Github | LinkedIn
> > >> >> >>
> > >> >> >>
> > >> >> >> 2017-11-09 17:51 GMT+01:00 Lukasz Cwik
 > >:
> > >> >> >> > It turns out that the Apache Rat Ant task and the Apache Rat
> > Maven
> > >> >> plugin
> > >> >> >> > differ in that the plugin automatically excludes certain
files
> > by
> > >> >> default
> > >> >> >> > while the Ant task does not.
> > >> >> >> > See:
> > >> >> >> > http://creadur.apache.org/rat/apache-rat-plugin/check-mojo.
> > >> >> >> html#useDefaultExcludes
> > >> >> >> >
> > >> >> >> > I fixed the list to exclude ".idea/" instead of "idea/" since
> > >> there
> > >> >> was a
> > >> >> >> > typo.
> > >> >> >> >
> > >> >> 

Re: [DISCUSS] Move away from Apache Maven as build tool

[Kenneth Knowles]

Keep in mind that a clean build is unusual during development (it is common
for mvn use and that is a bug) and also not necessary for precommits if the
build tool is correct enough that caching is safe. So while this number
matters, it is not the most important.

On Thu, Nov 9, 2017 at 11:30 AM, Romain Manni-Bucau 
wrote:

> I will try next week yes but the 2 runs i did were 28mn vs 32mn from memory
> - after having downloaded all deps once.
>
> Le 9 nov. 2017 19:45, "Lukasz Cwik"  a écrit :
>
> > If Gradle was slow, do you mind running the build with --profile and
> > sharing that and also sharing the Maven build log?
> >
> > On Thu, Nov 9, 2017 at 10:43 AM, Lukasz Cwik  wrote:
> >
> > > Romain, I don't understand your last comment, were you trying to say
> that
> > > you had the same Gradle build times like I did and it was an
> improvement
> > > over Maven or that you did not and you experienced build times that
> were
> > > equivalent to Maven?
> > >
> > > On Thu, Nov 9, 2017 at 9:51 AM, Romain Manni-Bucau <
> > rmannibu...@gmail.com>
> > > wrote:
> > >
> > >> 2017-11-09 18:38 GMT+01:00 Kenneth Knowles :
> > >> > On Thu, Nov 9, 2017 at 9:11 AM, Romain Manni-Bucau <
> > >> rmannibu...@gmail.com>
> > >> > wrote:
> > >> >
> > >> >> (this is another topic so we can maybe open another thread) issue
> is
> > >> >> not much about python but more about the fact the build is not self
> > >> >> contained. it is a maven build and maven should be sufficient
> without
> > >> >> having to install python + dependencies.
> > >> >
> > >> >
> > >> > Let's leave out the topic of whether our build should install things
> > >> like
> > >> > JDKs, Python, Golang, Docker, protoc, findbugs, RAT, etc. That issue
> > is
> > >> > somewhat independent of build tool, and the new build isn't worse
> than
> > >> the
> > >> > old one as far as it goes.
> > >>
> > >>
> > >> Yep, globally the same time with clean and killing the daemon.
> > >>
> > >> >
> > >> > Kenn
> > >> >
> > >> >
> > >> >
> > >> >> I don't see any technical
> > >> >> blockers to do it (except time ;)) but it is always a bit annoying
> to
> > >> >> git clone then not be able to build.
> > >> >>
> > >> >> Romain Manni-Bucau
> > >> >> @rmannibucau |  Blog | Old Blog | Github | LinkedIn
> > >> >>
> > >> >>
> > >> >> 2017-11-09 18:07 GMT+01:00 Lukasz Cwik :
> > >> >> > Hmm, I have had good luck when following the Python quick start
> > setup
> > >> >> >  on multiple
> > >> >> machines
> > >> >> > by ensuring the installed version of setuptools, virtualenv and
> pip
> > >> are
> > >> >> new
> > >> >> > enough versions.
> > >> >> >
> > >> >> > You can always skip the Python portion of the build by excluding
> > the
> > >> >> build
> > >> >> > task as so:
> > >> >> > ./gradlew build -x ":beam-sdks-parent:beam-sdks-python:build"
> > >> >> >
> > >> >> > On Thu, Nov 9, 2017 at 8:58 AM, Romain Manni-Bucau <
> > >> >> rmannibu...@gmail.com>
> > >> >> > wrote:
> > >> >> >
> > >> >> >> The 1.3.5 file is when i installed the python dependencies
> > manually
> > >> >> >> to make the build passing (the pip command never passed on my
> > >> computer
> > >> >> >> and therefore the build always has been broken until i installed
> > it
> > >> >> >> manually - independently from the build tool).
> > >> >> >>
> > >> >> >> Romain Manni-Bucau
> > >> >> >> @rmannibucau |  Blog | Old Blog | Github | LinkedIn
> > >> >> >>
> > >> >> >>
> > >> >> >> 2017-11-09 17:51 GMT+01:00 Lukasz Cwik  > >:
> > >> >> >> > It turns out that the Apache Rat Ant task and the Apache Rat
> > Maven
> > >> >> plugin
> > >> >> >> > differ in that the plugin automatically excludes certain files
> > by
> > >> >> default
> > >> >> >> > while the Ant task does not.
> > >> >> >> > See:
> > >> >> >> > http://creadur.apache.org/rat/apache-rat-plugin/check-mojo.
> > >> >> >> html#useDefaultExcludes
> > >> >> >> >
> > >> >> >> > I fixed the list to exclude ".idea/" instead of "idea/" since
> > >> there
> > >> >> was a
> > >> >> >> > typo.
> > >> >> >> >
> > >> >> >> > I have no idea what the file "=1.3.5" is. Can you take a look
> at
> > >> the
> > >> >> >> > contents?
> > >> >> >> >
> > >> >> >> > On Thu, Nov 9, 2017 at 12:03 AM, Romain Manni-Bucau <
> > >> >> >> rmannibu...@gmail.com>
> > >> >> >> > wrote:
> > >> >> >> >
> > >> >> >> >> Ok, the rat issues I got were:
> > >> >> >> >>
> > >> >> >> >> == File: /home/rmannibucau/1_dev/beam/.idea/*
> > >> >> >> >> == File: /home/rmannibucau/1_dev/beam/sdks/python/=1.3.5
> > >> >> >> >>
> > >> >> >> >> The first one could be in my default exclude - even if
> > >> eclipse/idea
> > >> >> >> >> files should be in the default exclude set of beam rat config
> > >> IMHO,
> > >> >> >> >> the last one is more a "?" can probably be exclude as well if
> > >> created
> > >> >> >> >> by the build 

Re: [DISCUSS] Move away from Apache Maven as build tool

[Romain Manni-Bucau]

I will try next week yes but the 2 runs i did were 28mn vs 32mn from memory
- after having downloaded all deps once.

Le 9 nov. 2017 19:45, "Lukasz Cwik"  a écrit :

> If Gradle was slow, do you mind running the build with --profile and
> sharing that and also sharing the Maven build log?
>
> On Thu, Nov 9, 2017 at 10:43 AM, Lukasz Cwik  wrote:
>
> > Romain, I don't understand your last comment, were you trying to say that
> > you had the same Gradle build times like I did and it was an improvement
> > over Maven or that you did not and you experienced build times that were
> > equivalent to Maven?
> >
> > On Thu, Nov 9, 2017 at 9:51 AM, Romain Manni-Bucau <
> rmannibu...@gmail.com>
> > wrote:
> >
> >> 2017-11-09 18:38 GMT+01:00 Kenneth Knowles :
> >> > On Thu, Nov 9, 2017 at 9:11 AM, Romain Manni-Bucau <
> >> rmannibu...@gmail.com>
> >> > wrote:
> >> >
> >> >> (this is another topic so we can maybe open another thread) issue is
> >> >> not much about python but more about the fact the build is not self
> >> >> contained. it is a maven build and maven should be sufficient without
> >> >> having to install python + dependencies.
> >> >
> >> >
> >> > Let's leave out the topic of whether our build should install things
> >> like
> >> > JDKs, Python, Golang, Docker, protoc, findbugs, RAT, etc. That issue
> is
> >> > somewhat independent of build tool, and the new build isn't worse than
> >> the
> >> > old one as far as it goes.
> >>
> >>
> >> Yep, globally the same time with clean and killing the daemon.
> >>
> >> >
> >> > Kenn
> >> >
> >> >
> >> >
> >> >> I don't see any technical
> >> >> blockers to do it (except time ;)) but it is always a bit annoying to
> >> >> git clone then not be able to build.
> >> >>
> >> >> Romain Manni-Bucau
> >> >> @rmannibucau |  Blog | Old Blog | Github | LinkedIn
> >> >>
> >> >>
> >> >> 2017-11-09 18:07 GMT+01:00 Lukasz Cwik :
> >> >> > Hmm, I have had good luck when following the Python quick start
> setup
> >> >> >  on multiple
> >> >> machines
> >> >> > by ensuring the installed version of setuptools, virtualenv and pip
> >> are
> >> >> new
> >> >> > enough versions.
> >> >> >
> >> >> > You can always skip the Python portion of the build by excluding
> the
> >> >> build
> >> >> > task as so:
> >> >> > ./gradlew build -x ":beam-sdks-parent:beam-sdks-python:build"
> >> >> >
> >> >> > On Thu, Nov 9, 2017 at 8:58 AM, Romain Manni-Bucau <
> >> >> rmannibu...@gmail.com>
> >> >> > wrote:
> >> >> >
> >> >> >> The 1.3.5 file is when i installed the python dependencies
> manually
> >> >> >> to make the build passing (the pip command never passed on my
> >> computer
> >> >> >> and therefore the build always has been broken until i installed
> it
> >> >> >> manually - independently from the build tool).
> >> >> >>
> >> >> >> Romain Manni-Bucau
> >> >> >> @rmannibucau |  Blog | Old Blog | Github | LinkedIn
> >> >> >>
> >> >> >>
> >> >> >> 2017-11-09 17:51 GMT+01:00 Lukasz Cwik  >:
> >> >> >> > It turns out that the Apache Rat Ant task and the Apache Rat
> Maven
> >> >> plugin
> >> >> >> > differ in that the plugin automatically excludes certain files
> by
> >> >> default
> >> >> >> > while the Ant task does not.
> >> >> >> > See:
> >> >> >> > http://creadur.apache.org/rat/apache-rat-plugin/check-mojo.
> >> >> >> html#useDefaultExcludes
> >> >> >> >
> >> >> >> > I fixed the list to exclude ".idea/" instead of "idea/" since
> >> there
> >> >> was a
> >> >> >> > typo.
> >> >> >> >
> >> >> >> > I have no idea what the file "=1.3.5" is. Can you take a look at
> >> the
> >> >> >> > contents?
> >> >> >> >
> >> >> >> > On Thu, Nov 9, 2017 at 12:03 AM, Romain Manni-Bucau <
> >> >> >> rmannibu...@gmail.com>
> >> >> >> > wrote:
> >> >> >> >
> >> >> >> >> Ok, the rat issues I got were:
> >> >> >> >>
> >> >> >> >> == File: /home/rmannibucau/1_dev/beam/.idea/*
> >> >> >> >> == File: /home/rmannibucau/1_dev/beam/sdks/python/=1.3.5
> >> >> >> >>
> >> >> >> >> The first one could be in my default exclude - even if
> >> eclipse/idea
> >> >> >> >> files should be in the default exclude set of beam rat config
> >> IMHO,
> >> >> >> >> the last one is more a "?" can probably be exclude as well if
> >> created
> >> >> >> >> by the build at some point.
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> Romain Manni-Bucau
> >> >> >> >> @rmannibucau |  Blog | Old Blog | Github | LinkedIn
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> 2017-11-08 19:17 GMT+01:00 Jean-Baptiste Onofré <
> j...@nanthrax.net
> >> >:
> >> >> >> >> > Thanks for the update. I was swamped on some meetings. I'm
> >> back to
> >> >> >> test
> >> >> >> >> the latest changes.
> >> >> >> >> >
> >> >> >> >> > Regards
> >> >> >> >> > JB
> >> >> >> >> >
> >> >> >> >> > On Nov 8, 2017, 18:56, at 18:56, Lukasz Cwik
> >> >>  >> >> >> >
> >> >> >> >> wrote:
> >> >> 

Re: [DISCUSS] Move away from Apache Maven as build tool

[Lukasz Cwik]

Romain, I don't understand your last comment, were you trying to say that
you had the same Gradle build times like I did and it was an improvement
over Maven or that you did not and you experienced build times that were
equivalent to Maven?

On Thu, Nov 9, 2017 at 9:51 AM, Romain Manni-Bucau 
wrote:

> 2017-11-09 18:38 GMT+01:00 Kenneth Knowles :
> > On Thu, Nov 9, 2017 at 9:11 AM, Romain Manni-Bucau <
> rmannibu...@gmail.com>
> > wrote:
> >
> >> (this is another topic so we can maybe open another thread) issue is
> >> not much about python but more about the fact the build is not self
> >> contained. it is a maven build and maven should be sufficient without
> >> having to install python + dependencies.
> >
> >
> > Let's leave out the topic of whether our build should install things like
> > JDKs, Python, Golang, Docker, protoc, findbugs, RAT, etc. That issue is
> > somewhat independent of build tool, and the new build isn't worse than
> the
> > old one as far as it goes.
>
>
> Yep, globally the same time with clean and killing the daemon.
>
> >
> > Kenn
> >
> >
> >
> >> I don't see any technical
> >> blockers to do it (except time ;)) but it is always a bit annoying to
> >> git clone then not be able to build.
> >>
> >> Romain Manni-Bucau
> >> @rmannibucau |  Blog | Old Blog | Github | LinkedIn
> >>
> >>
> >> 2017-11-09 18:07 GMT+01:00 Lukasz Cwik :
> >> > Hmm, I have had good luck when following the Python quick start setup
> >> >  on multiple
> >> machines
> >> > by ensuring the installed version of setuptools, virtualenv and pip
> are
> >> new
> >> > enough versions.
> >> >
> >> > You can always skip the Python portion of the build by excluding the
> >> build
> >> > task as so:
> >> > ./gradlew build -x ":beam-sdks-parent:beam-sdks-python:build"
> >> >
> >> > On Thu, Nov 9, 2017 at 8:58 AM, Romain Manni-Bucau <
> >> rmannibu...@gmail.com>
> >> > wrote:
> >> >
> >> >> The 1.3.5 file is when i installed the python dependencies manually
> >> >> to make the build passing (the pip command never passed on my
> computer
> >> >> and therefore the build always has been broken until i installed it
> >> >> manually - independently from the build tool).
> >> >>
> >> >> Romain Manni-Bucau
> >> >> @rmannibucau |  Blog | Old Blog | Github | LinkedIn
> >> >>
> >> >>
> >> >> 2017-11-09 17:51 GMT+01:00 Lukasz Cwik :
> >> >> > It turns out that the Apache Rat Ant task and the Apache Rat Maven
> >> plugin
> >> >> > differ in that the plugin automatically excludes certain files by
> >> default
> >> >> > while the Ant task does not.
> >> >> > See:
> >> >> > http://creadur.apache.org/rat/apache-rat-plugin/check-mojo.
> >> >> html#useDefaultExcludes
> >> >> >
> >> >> > I fixed the list to exclude ".idea/" instead of "idea/" since there
> >> was a
> >> >> > typo.
> >> >> >
> >> >> > I have no idea what the file "=1.3.5" is. Can you take a look at
> the
> >> >> > contents?
> >> >> >
> >> >> > On Thu, Nov 9, 2017 at 12:03 AM, Romain Manni-Bucau <
> >> >> rmannibu...@gmail.com>
> >> >> > wrote:
> >> >> >
> >> >> >> Ok, the rat issues I got were:
> >> >> >>
> >> >> >> == File: /home/rmannibucau/1_dev/beam/.idea/*
> >> >> >> == File: /home/rmannibucau/1_dev/beam/sdks/python/=1.3.5
> >> >> >>
> >> >> >> The first one could be in my default exclude - even if
> eclipse/idea
> >> >> >> files should be in the default exclude set of beam rat config
> IMHO,
> >> >> >> the last one is more a "?" can probably be exclude as well if
> created
> >> >> >> by the build at some point.
> >> >> >>
> >> >> >>
> >> >> >> Romain Manni-Bucau
> >> >> >> @rmannibucau |  Blog | Old Blog | Github | LinkedIn
> >> >> >>
> >> >> >>
> >> >> >> 2017-11-08 19:17 GMT+01:00 Jean-Baptiste Onofré  >:
> >> >> >> > Thanks for the update. I was swamped on some meetings. I'm back
> to
> >> >> test
> >> >> >> the latest changes.
> >> >> >> >
> >> >> >> > Regards
> >> >> >> > JB
> >> >> >> >
> >> >> >> > On Nov 8, 2017, 18:56, at 18:56, Lukasz Cwik
> >>  >> >> >
> >> >> >> wrote:
> >> >> >> >>Thanks everyone for trying this build out in different
> workspaces /
> >> >> >> >>configurations. This will help make sure the build works for
> more
> >> >> >> >>people
> >> >> >> >>and will get rid of any rough edges.
> >> >> >> >>
> >> >> >> >>Performance (All):
> >> >> >> >>Maven performs parallelization at the module level, an entire
> >> module
> >> >> >> >>needs
> >> >> >> >>to complete before any dependent modules can start, this means
> >> running
> >> >> >> >>all
> >> >> >> >>the checks like findbugs, checkstyle, tests need to finish.
> Gradle
> >> has
> >> >> >> >>task
> >> >> >> >>level parallelism between subprojects which means that as soon
> as
> >> the
> >> >> >> >>compile and shade steps are done for a project, and dependent
> >> >> >> >>subprojects
> >> >> >> >>can 

Re: [DISCUSS] Move away from Apache Maven as build tool

[Romain Manni-Bucau]

2017-11-09 18:38 GMT+01:00 Kenneth Knowles :
> On Thu, Nov 9, 2017 at 9:11 AM, Romain Manni-Bucau 
> wrote:
>
>> (this is another topic so we can maybe open another thread) issue is
>> not much about python but more about the fact the build is not self
>> contained. it is a maven build and maven should be sufficient without
>> having to install python + dependencies.
>
>
> Let's leave out the topic of whether our build should install things like
> JDKs, Python, Golang, Docker, protoc, findbugs, RAT, etc. That issue is
> somewhat independent of build tool, and the new build isn't worse than the
> old one as far as it goes.


Yep, globally the same time with clean and killing the daemon.

>
> Kenn
>
>
>
>> I don't see any technical
>> blockers to do it (except time ;)) but it is always a bit annoying to
>> git clone then not be able to build.
>>
>> Romain Manni-Bucau
>> @rmannibucau |  Blog | Old Blog | Github | LinkedIn
>>
>>
>> 2017-11-09 18:07 GMT+01:00 Lukasz Cwik :
>> > Hmm, I have had good luck when following the Python quick start setup
>> >  on multiple
>> machines
>> > by ensuring the installed version of setuptools, virtualenv and pip are
>> new
>> > enough versions.
>> >
>> > You can always skip the Python portion of the build by excluding the
>> build
>> > task as so:
>> > ./gradlew build -x ":beam-sdks-parent:beam-sdks-python:build"
>> >
>> > On Thu, Nov 9, 2017 at 8:58 AM, Romain Manni-Bucau <
>> rmannibu...@gmail.com>
>> > wrote:
>> >
>> >> The 1.3.5 file is when i installed the python dependencies manually
>> >> to make the build passing (the pip command never passed on my computer
>> >> and therefore the build always has been broken until i installed it
>> >> manually - independently from the build tool).
>> >>
>> >> Romain Manni-Bucau
>> >> @rmannibucau |  Blog | Old Blog | Github | LinkedIn
>> >>
>> >>
>> >> 2017-11-09 17:51 GMT+01:00 Lukasz Cwik :
>> >> > It turns out that the Apache Rat Ant task and the Apache Rat Maven
>> plugin
>> >> > differ in that the plugin automatically excludes certain files by
>> default
>> >> > while the Ant task does not.
>> >> > See:
>> >> > http://creadur.apache.org/rat/apache-rat-plugin/check-mojo.
>> >> html#useDefaultExcludes
>> >> >
>> >> > I fixed the list to exclude ".idea/" instead of "idea/" since there
>> was a
>> >> > typo.
>> >> >
>> >> > I have no idea what the file "=1.3.5" is. Can you take a look at the
>> >> > contents?
>> >> >
>> >> > On Thu, Nov 9, 2017 at 12:03 AM, Romain Manni-Bucau <
>> >> rmannibu...@gmail.com>
>> >> > wrote:
>> >> >
>> >> >> Ok, the rat issues I got were:
>> >> >>
>> >> >> == File: /home/rmannibucau/1_dev/beam/.idea/*
>> >> >> == File: /home/rmannibucau/1_dev/beam/sdks/python/=1.3.5
>> >> >>
>> >> >> The first one could be in my default exclude - even if eclipse/idea
>> >> >> files should be in the default exclude set of beam rat config IMHO,
>> >> >> the last one is more a "?" can probably be exclude as well if created
>> >> >> by the build at some point.
>> >> >>
>> >> >>
>> >> >> Romain Manni-Bucau
>> >> >> @rmannibucau |  Blog | Old Blog | Github | LinkedIn
>> >> >>
>> >> >>
>> >> >> 2017-11-08 19:17 GMT+01:00 Jean-Baptiste Onofré :
>> >> >> > Thanks for the update. I was swamped on some meetings. I'm back to
>> >> test
>> >> >> the latest changes.
>> >> >> >
>> >> >> > Regards
>> >> >> > JB
>> >> >> >
>> >> >> > On Nov 8, 2017, 18:56, at 18:56, Lukasz Cwik
>> > >> >
>> >> >> wrote:
>> >> >> >>Thanks everyone for trying this build out in different workspaces /
>> >> >> >>configurations. This will help make sure the build works for more
>> >> >> >>people
>> >> >> >>and will get rid of any rough edges.
>> >> >> >>
>> >> >> >>Performance (All):
>> >> >> >>Maven performs parallelization at the module level, an entire
>> module
>> >> >> >>needs
>> >> >> >>to complete before any dependent modules can start, this means
>> running
>> >> >> >>all
>> >> >> >>the checks like findbugs, checkstyle, tests need to finish. Gradle
>> has
>> >> >> >>task
>> >> >> >>level parallelism between subprojects which means that as soon as
>> the
>> >> >> >>compile and shade steps are done for a project, and dependent
>> >> >> >>subprojects
>> >> >> >>can typically start. This means that we get increased parallelism
>> due
>> >> >> >>to
>> >> >> >>not needing to wait for findbugs, checkstyle, tests to run. I
>> >> typically
>> >> >> >>see
>> >> >> >>~20 tasks (at peak) running on my desktop in parallel.
>> >> >> >>
>> >> >> >>Apache Rat (JB / Romain):
>> >> >> >>What files are in the rat report that fail (its likely that I'm
>> >> missing
>> >> >> >>some exclusion for a build time artifact)? Also, please try the
>> build
>> >> >> >>again
>> >> >> >>after running `git clean -fdx` in your workspace.
>> >> >> >>
>> >> >> >>Python (JB):
>> >> >> >>As 

Re: [DISCUSS] Move away from Apache Maven as build tool

[Kenneth Knowles]

On Thu, Nov 9, 2017 at 9:11 AM, Romain Manni-Bucau 
wrote:

> (this is another topic so we can maybe open another thread) issue is
> not much about python but more about the fact the build is not self
> contained. it is a maven build and maven should be sufficient without
> having to install python + dependencies.


Let's leave out the topic of whether our build should install things like
JDKs, Python, Golang, Docker, protoc, findbugs, RAT, etc. That issue is
somewhat independent of build tool, and the new build isn't worse than the
old one as far as it goes.

Kenn



> I don't see any technical
> blockers to do it (except time ;)) but it is always a bit annoying to
> git clone then not be able to build.
>
> Romain Manni-Bucau
> @rmannibucau |  Blog | Old Blog | Github | LinkedIn
>
>
> 2017-11-09 18:07 GMT+01:00 Lukasz Cwik :
> > Hmm, I have had good luck when following the Python quick start setup
> >  on multiple
> machines
> > by ensuring the installed version of setuptools, virtualenv and pip are
> new
> > enough versions.
> >
> > You can always skip the Python portion of the build by excluding the
> build
> > task as so:
> > ./gradlew build -x ":beam-sdks-parent:beam-sdks-python:build"
> >
> > On Thu, Nov 9, 2017 at 8:58 AM, Romain Manni-Bucau <
> rmannibu...@gmail.com>
> > wrote:
> >
> >> The 1.3.5 file is when i installed the python dependencies manually
> >> to make the build passing (the pip command never passed on my computer
> >> and therefore the build always has been broken until i installed it
> >> manually - independently from the build tool).
> >>
> >> Romain Manni-Bucau
> >> @rmannibucau |  Blog | Old Blog | Github | LinkedIn
> >>
> >>
> >> 2017-11-09 17:51 GMT+01:00 Lukasz Cwik :
> >> > It turns out that the Apache Rat Ant task and the Apache Rat Maven
> plugin
> >> > differ in that the plugin automatically excludes certain files by
> default
> >> > while the Ant task does not.
> >> > See:
> >> > http://creadur.apache.org/rat/apache-rat-plugin/check-mojo.
> >> html#useDefaultExcludes
> >> >
> >> > I fixed the list to exclude ".idea/" instead of "idea/" since there
> was a
> >> > typo.
> >> >
> >> > I have no idea what the file "=1.3.5" is. Can you take a look at the
> >> > contents?
> >> >
> >> > On Thu, Nov 9, 2017 at 12:03 AM, Romain Manni-Bucau <
> >> rmannibu...@gmail.com>
> >> > wrote:
> >> >
> >> >> Ok, the rat issues I got were:
> >> >>
> >> >> == File: /home/rmannibucau/1_dev/beam/.idea/*
> >> >> == File: /home/rmannibucau/1_dev/beam/sdks/python/=1.3.5
> >> >>
> >> >> The first one could be in my default exclude - even if eclipse/idea
> >> >> files should be in the default exclude set of beam rat config IMHO,
> >> >> the last one is more a "?" can probably be exclude as well if created
> >> >> by the build at some point.
> >> >>
> >> >>
> >> >> Romain Manni-Bucau
> >> >> @rmannibucau |  Blog | Old Blog | Github | LinkedIn
> >> >>
> >> >>
> >> >> 2017-11-08 19:17 GMT+01:00 Jean-Baptiste Onofré :
> >> >> > Thanks for the update. I was swamped on some meetings. I'm back to
> >> test
> >> >> the latest changes.
> >> >> >
> >> >> > Regards
> >> >> > JB
> >> >> >
> >> >> > On Nov 8, 2017, 18:56, at 18:56, Lukasz Cwik
>  >> >
> >> >> wrote:
> >> >> >>Thanks everyone for trying this build out in different workspaces /
> >> >> >>configurations. This will help make sure the build works for more
> >> >> >>people
> >> >> >>and will get rid of any rough edges.
> >> >> >>
> >> >> >>Performance (All):
> >> >> >>Maven performs parallelization at the module level, an entire
> module
> >> >> >>needs
> >> >> >>to complete before any dependent modules can start, this means
> running
> >> >> >>all
> >> >> >>the checks like findbugs, checkstyle, tests need to finish. Gradle
> has
> >> >> >>task
> >> >> >>level parallelism between subprojects which means that as soon as
> the
> >> >> >>compile and shade steps are done for a project, and dependent
> >> >> >>subprojects
> >> >> >>can typically start. This means that we get increased parallelism
> due
> >> >> >>to
> >> >> >>not needing to wait for findbugs, checkstyle, tests to run. I
> >> typically
> >> >> >>see
> >> >> >>~20 tasks (at peak) running on my desktop in parallel.
> >> >> >>
> >> >> >>Apache Rat (JB / Romain):
> >> >> >>What files are in the rat report that fail (its likely that I'm
> >> missing
> >> >> >>some exclusion for a build time artifact)? Also, please try the
> build
> >> >> >>again
> >> >> >>after running `git clean -fdx` in your workspace.
> >> >> >>
> >> >> >>Python (JB):
> >> >> >>As for the Python SDK, you'll need to share more details about the
> >> >> >>failure.
> >> >> >>
> >> >> >>Gradle 4.3:
> >> >> >>I would like to defer the swap to Gradle 4.3 until after this PR
> since
> >> >> >>it
> >> >> >>will be a much smaller set of changes.
> >> >> >>
> >> >> 

Re: [DISCUSS] Move away from Apache Maven as build tool

[Romain Manni-Bucau]

(this is another topic so we can maybe open another thread) issue is
not much about python but more about the fact the build is not self
contained. it is a maven build and maven should be sufficient without
having to install python + dependencies. I don't see any technical
blockers to do it (except time ;)) but it is always a bit annoying to
git clone then not be able to build.

Romain Manni-Bucau
@rmannibucau |  Blog | Old Blog | Github | LinkedIn


2017-11-09 18:07 GMT+01:00 Lukasz Cwik :
> Hmm, I have had good luck when following the Python quick start setup
>  on multiple machines
> by ensuring the installed version of setuptools, virtualenv and pip are new
> enough versions.
>
> You can always skip the Python portion of the build by excluding the build
> task as so:
> ./gradlew build -x ":beam-sdks-parent:beam-sdks-python:build"
>
> On Thu, Nov 9, 2017 at 8:58 AM, Romain Manni-Bucau 
> wrote:
>
>> The 1.3.5 file is when i installed the python dependencies manually
>> to make the build passing (the pip command never passed on my computer
>> and therefore the build always has been broken until i installed it
>> manually - independently from the build tool).
>>
>> Romain Manni-Bucau
>> @rmannibucau |  Blog | Old Blog | Github | LinkedIn
>>
>>
>> 2017-11-09 17:51 GMT+01:00 Lukasz Cwik :
>> > It turns out that the Apache Rat Ant task and the Apache Rat Maven plugin
>> > differ in that the plugin automatically excludes certain files by default
>> > while the Ant task does not.
>> > See:
>> > http://creadur.apache.org/rat/apache-rat-plugin/check-mojo.
>> html#useDefaultExcludes
>> >
>> > I fixed the list to exclude ".idea/" instead of "idea/" since there was a
>> > typo.
>> >
>> > I have no idea what the file "=1.3.5" is. Can you take a look at the
>> > contents?
>> >
>> > On Thu, Nov 9, 2017 at 12:03 AM, Romain Manni-Bucau <
>> rmannibu...@gmail.com>
>> > wrote:
>> >
>> >> Ok, the rat issues I got were:
>> >>
>> >> == File: /home/rmannibucau/1_dev/beam/.idea/*
>> >> == File: /home/rmannibucau/1_dev/beam/sdks/python/=1.3.5
>> >>
>> >> The first one could be in my default exclude - even if eclipse/idea
>> >> files should be in the default exclude set of beam rat config IMHO,
>> >> the last one is more a "?" can probably be exclude as well if created
>> >> by the build at some point.
>> >>
>> >>
>> >> Romain Manni-Bucau
>> >> @rmannibucau |  Blog | Old Blog | Github | LinkedIn
>> >>
>> >>
>> >> 2017-11-08 19:17 GMT+01:00 Jean-Baptiste Onofré :
>> >> > Thanks for the update. I was swamped on some meetings. I'm back to
>> test
>> >> the latest changes.
>> >> >
>> >> > Regards
>> >> > JB
>> >> >
>> >> > On Nov 8, 2017, 18:56, at 18:56, Lukasz Cwik > >
>> >> wrote:
>> >> >>Thanks everyone for trying this build out in different workspaces /
>> >> >>configurations. This will help make sure the build works for more
>> >> >>people
>> >> >>and will get rid of any rough edges.
>> >> >>
>> >> >>Performance (All):
>> >> >>Maven performs parallelization at the module level, an entire module
>> >> >>needs
>> >> >>to complete before any dependent modules can start, this means running
>> >> >>all
>> >> >>the checks like findbugs, checkstyle, tests need to finish. Gradle has
>> >> >>task
>> >> >>level parallelism between subprojects which means that as soon as the
>> >> >>compile and shade steps are done for a project, and dependent
>> >> >>subprojects
>> >> >>can typically start. This means that we get increased parallelism due
>> >> >>to
>> >> >>not needing to wait for findbugs, checkstyle, tests to run. I
>> typically
>> >> >>see
>> >> >>~20 tasks (at peak) running on my desktop in parallel.
>> >> >>
>> >> >>Apache Rat (JB / Romain):
>> >> >>What files are in the rat report that fail (its likely that I'm
>> missing
>> >> >>some exclusion for a build time artifact)? Also, please try the build
>> >> >>again
>> >> >>after running `git clean -fdx` in your workspace.
>> >> >>
>> >> >>Python (JB):
>> >> >>As for the Python SDK, you'll need to share more details about the
>> >> >>failure.
>> >> >>
>> >> >>Gradle 4.3:
>> >> >>I would like to defer the swap to Gradle 4.3 until after this PR since
>> >> >>it
>> >> >>will be a much smaller set of changes.
>> >> >>
>> >> >>On Wed, Nov 8, 2017 at 12:54 AM, Jean-Baptiste Onofré <
>> j...@nanthrax.net>
>> >> >>wrote:
>> >> >>
>> >> >>> Same for me for rat and python build too:
>> >> >>>
>> >> >>> FAILURE: Build completed with 2 failures.
>> >> >>>
>> >> >>> 1: Task failed with an exception.
>> >> >>> ---
>> >> >>> * What went wrong:
>> >> >>> Execution failed for task ':rat'.
>> >> >>> > Found 905 files with unapproved/unknown licenses. See
>> >> >>> file:/home/jbonofre/Workspace/beam/build/reports/rat/rat-report.txt
>> >> >>>
>> >> >>> * Try:
>> >> >>> Run with --stacktrace option to get the stack trace. Run 

Re: [DISCUSS] Move away from Apache Maven as build tool

[Lukasz Cwik]

Hmm, I have had good luck when following the Python quick start setup
 on multiple machines
by ensuring the installed version of setuptools, virtualenv and pip are new
enough versions.

You can always skip the Python portion of the build by excluding the build
task as so:
./gradlew build -x ":beam-sdks-parent:beam-sdks-python:build"

On Thu, Nov 9, 2017 at 8:58 AM, Romain Manni-Bucau 
wrote:

> The 1.3.5 file is when i installed the python dependencies manually
> to make the build passing (the pip command never passed on my computer
> and therefore the build always has been broken until i installed it
> manually - independently from the build tool).
>
> Romain Manni-Bucau
> @rmannibucau |  Blog | Old Blog | Github | LinkedIn
>
>
> 2017-11-09 17:51 GMT+01:00 Lukasz Cwik :
> > It turns out that the Apache Rat Ant task and the Apache Rat Maven plugin
> > differ in that the plugin automatically excludes certain files by default
> > while the Ant task does not.
> > See:
> > http://creadur.apache.org/rat/apache-rat-plugin/check-mojo.
> html#useDefaultExcludes
> >
> > I fixed the list to exclude ".idea/" instead of "idea/" since there was a
> > typo.
> >
> > I have no idea what the file "=1.3.5" is. Can you take a look at the
> > contents?
> >
> > On Thu, Nov 9, 2017 at 12:03 AM, Romain Manni-Bucau <
> rmannibu...@gmail.com>
> > wrote:
> >
> >> Ok, the rat issues I got were:
> >>
> >> == File: /home/rmannibucau/1_dev/beam/.idea/*
> >> == File: /home/rmannibucau/1_dev/beam/sdks/python/=1.3.5
> >>
> >> The first one could be in my default exclude - even if eclipse/idea
> >> files should be in the default exclude set of beam rat config IMHO,
> >> the last one is more a "?" can probably be exclude as well if created
> >> by the build at some point.
> >>
> >>
> >> Romain Manni-Bucau
> >> @rmannibucau |  Blog | Old Blog | Github | LinkedIn
> >>
> >>
> >> 2017-11-08 19:17 GMT+01:00 Jean-Baptiste Onofré :
> >> > Thanks for the update. I was swamped on some meetings. I'm back to
> test
> >> the latest changes.
> >> >
> >> > Regards
> >> > JB
> >> >
> >> > On Nov 8, 2017, 18:56, at 18:56, Lukasz Cwik  >
> >> wrote:
> >> >>Thanks everyone for trying this build out in different workspaces /
> >> >>configurations. This will help make sure the build works for more
> >> >>people
> >> >>and will get rid of any rough edges.
> >> >>
> >> >>Performance (All):
> >> >>Maven performs parallelization at the module level, an entire module
> >> >>needs
> >> >>to complete before any dependent modules can start, this means running
> >> >>all
> >> >>the checks like findbugs, checkstyle, tests need to finish. Gradle has
> >> >>task
> >> >>level parallelism between subprojects which means that as soon as the
> >> >>compile and shade steps are done for a project, and dependent
> >> >>subprojects
> >> >>can typically start. This means that we get increased parallelism due
> >> >>to
> >> >>not needing to wait for findbugs, checkstyle, tests to run. I
> typically
> >> >>see
> >> >>~20 tasks (at peak) running on my desktop in parallel.
> >> >>
> >> >>Apache Rat (JB / Romain):
> >> >>What files are in the rat report that fail (its likely that I'm
> missing
> >> >>some exclusion for a build time artifact)? Also, please try the build
> >> >>again
> >> >>after running `git clean -fdx` in your workspace.
> >> >>
> >> >>Python (JB):
> >> >>As for the Python SDK, you'll need to share more details about the
> >> >>failure.
> >> >>
> >> >>Gradle 4.3:
> >> >>I would like to defer the swap to Gradle 4.3 until after this PR since
> >> >>it
> >> >>will be a much smaller set of changes.
> >> >>
> >> >>On Wed, Nov 8, 2017 at 12:54 AM, Jean-Baptiste Onofré <
> j...@nanthrax.net>
> >> >>wrote:
> >> >>
> >> >>> Same for me for rat and python build too:
> >> >>>
> >> >>> FAILURE: Build completed with 2 failures.
> >> >>>
> >> >>> 1: Task failed with an exception.
> >> >>> ---
> >> >>> * What went wrong:
> >> >>> Execution failed for task ':rat'.
> >> >>> > Found 905 files with unapproved/unknown licenses. See
> >> >>> file:/home/jbonofre/Workspace/beam/build/reports/rat/rat-report.txt
> >> >>>
> >> >>> * Try:
> >> >>> Run with --stacktrace option to get the stack trace. Run with --info
> >> >>or
> >> >>> --debug option to get more log output.
> >> >>> 
> >> >>> ==
> >> >>>
> >> >>> 2: Task failed with an exception.
> >> >>> ---
> >> >>> * Where:
> >> >>> Build file '/home/jbonofre/Workspace/beam/sdks/python/build.gradle'
> >> >>line:
> >> >>> 64
> >> >>>
> >> >>> * What went wrong:
> >> >>> Execution failed for task ':beam-sdks-parent:beam-sdks-
> python:lint'.
> >> >>> > Process 'command 'tox'' finished with non-zero exit value 1
> >> >>>
> >> >>>
> >> >>>
> >> >>> On 11/08/2017 09:51 AM, Romain Manni-Bucau wrote:
> >> >>>
> >> 

Re: [DISCUSS] Move away from Apache Maven as build tool

[Romain Manni-Bucau]

The 1.3.5 file is when i installed the python dependencies manually
to make the build passing (the pip command never passed on my computer
and therefore the build always has been broken until i installed it
manually - independently from the build tool).

Romain Manni-Bucau
@rmannibucau |  Blog | Old Blog | Github | LinkedIn


2017-11-09 17:51 GMT+01:00 Lukasz Cwik :
> It turns out that the Apache Rat Ant task and the Apache Rat Maven plugin
> differ in that the plugin automatically excludes certain files by default
> while the Ant task does not.
> See:
> http://creadur.apache.org/rat/apache-rat-plugin/check-mojo.html#useDefaultExcludes
>
> I fixed the list to exclude ".idea/" instead of "idea/" since there was a
> typo.
>
> I have no idea what the file "=1.3.5" is. Can you take a look at the
> contents?
>
> On Thu, Nov 9, 2017 at 12:03 AM, Romain Manni-Bucau 
> wrote:
>
>> Ok, the rat issues I got were:
>>
>> == File: /home/rmannibucau/1_dev/beam/.idea/*
>> == File: /home/rmannibucau/1_dev/beam/sdks/python/=1.3.5
>>
>> The first one could be in my default exclude - even if eclipse/idea
>> files should be in the default exclude set of beam rat config IMHO,
>> the last one is more a "?" can probably be exclude as well if created
>> by the build at some point.
>>
>>
>> Romain Manni-Bucau
>> @rmannibucau |  Blog | Old Blog | Github | LinkedIn
>>
>>
>> 2017-11-08 19:17 GMT+01:00 Jean-Baptiste Onofré :
>> > Thanks for the update. I was swamped on some meetings. I'm back to test
>> the latest changes.
>> >
>> > Regards
>> > JB
>> >
>> > On Nov 8, 2017, 18:56, at 18:56, Lukasz Cwik 
>> wrote:
>> >>Thanks everyone for trying this build out in different workspaces /
>> >>configurations. This will help make sure the build works for more
>> >>people
>> >>and will get rid of any rough edges.
>> >>
>> >>Performance (All):
>> >>Maven performs parallelization at the module level, an entire module
>> >>needs
>> >>to complete before any dependent modules can start, this means running
>> >>all
>> >>the checks like findbugs, checkstyle, tests need to finish. Gradle has
>> >>task
>> >>level parallelism between subprojects which means that as soon as the
>> >>compile and shade steps are done for a project, and dependent
>> >>subprojects
>> >>can typically start. This means that we get increased parallelism due
>> >>to
>> >>not needing to wait for findbugs, checkstyle, tests to run. I typically
>> >>see
>> >>~20 tasks (at peak) running on my desktop in parallel.
>> >>
>> >>Apache Rat (JB / Romain):
>> >>What files are in the rat report that fail (its likely that I'm missing
>> >>some exclusion for a build time artifact)? Also, please try the build
>> >>again
>> >>after running `git clean -fdx` in your workspace.
>> >>
>> >>Python (JB):
>> >>As for the Python SDK, you'll need to share more details about the
>> >>failure.
>> >>
>> >>Gradle 4.3:
>> >>I would like to defer the swap to Gradle 4.3 until after this PR since
>> >>it
>> >>will be a much smaller set of changes.
>> >>
>> >>On Wed, Nov 8, 2017 at 12:54 AM, Jean-Baptiste Onofré 
>> >>wrote:
>> >>
>> >>> Same for me for rat and python build too:
>> >>>
>> >>> FAILURE: Build completed with 2 failures.
>> >>>
>> >>> 1: Task failed with an exception.
>> >>> ---
>> >>> * What went wrong:
>> >>> Execution failed for task ':rat'.
>> >>> > Found 905 files with unapproved/unknown licenses. See
>> >>> file:/home/jbonofre/Workspace/beam/build/reports/rat/rat-report.txt
>> >>>
>> >>> * Try:
>> >>> Run with --stacktrace option to get the stack trace. Run with --info
>> >>or
>> >>> --debug option to get more log output.
>> >>> 
>> >>> ==
>> >>>
>> >>> 2: Task failed with an exception.
>> >>> ---
>> >>> * Where:
>> >>> Build file '/home/jbonofre/Workspace/beam/sdks/python/build.gradle'
>> >>line:
>> >>> 64
>> >>>
>> >>> * What went wrong:
>> >>> Execution failed for task ':beam-sdks-parent:beam-sdks-python:lint'.
>> >>> > Process 'command 'tox'' finished with non-zero exit value 1
>> >>>
>> >>>
>> >>>
>> >>> On 11/08/2017 09:51 AM, Romain Manni-Bucau wrote:
>> >>>
>>  gradle branch doesnt build for me (some rat issues)
>> 
>>  Romain Manni-Bucau
>>  @rmannibucau |  Blog | Old Blog | Github | LinkedIn
>> 
>> 
>>  2017-11-08 5:41 GMT+01:00 Jean-Baptiste Onofré :
>> 
>> > Great !
>> >
>> > What explain these difference ? I'm curious especially for the
>> >>clean
>> > build
>> > all Java modules: is it a question of parallel execution ?
>> >
>> > Regards
>> > JB
>> >
>> >
>> > On 11/08/2017 02:59 AM, Lukasz Cwik wrote:
>> >
>> >>
>> >> The Gradle POC has made significant advances since last week
>> >>(shading,
>> >> Python, Go, Docker builds, ...). I believe the current state is
>> 

Re: [DISCUSS] Move away from Apache Maven as build tool

[Lukasz Cwik]

It turns out that the Apache Rat Ant task and the Apache Rat Maven plugin
differ in that the plugin automatically excludes certain files by default
while the Ant task does not.
See:
http://creadur.apache.org/rat/apache-rat-plugin/check-mojo.html#useDefaultExcludes

I fixed the list to exclude ".idea/" instead of "idea/" since there was a
typo.

I have no idea what the file "=1.3.5" is. Can you take a look at the
contents?

On Thu, Nov 9, 2017 at 12:03 AM, Romain Manni-Bucau 
wrote:

> Ok, the rat issues I got were:
>
> == File: /home/rmannibucau/1_dev/beam/.idea/*
> == File: /home/rmannibucau/1_dev/beam/sdks/python/=1.3.5
>
> The first one could be in my default exclude - even if eclipse/idea
> files should be in the default exclude set of beam rat config IMHO,
> the last one is more a "?" can probably be exclude as well if created
> by the build at some point.
>
>
> Romain Manni-Bucau
> @rmannibucau |  Blog | Old Blog | Github | LinkedIn
>
>
> 2017-11-08 19:17 GMT+01:00 Jean-Baptiste Onofré :
> > Thanks for the update. I was swamped on some meetings. I'm back to test
> the latest changes.
> >
> > Regards
> > JB
> >
> > On Nov 8, 2017, 18:56, at 18:56, Lukasz Cwik 
> wrote:
> >>Thanks everyone for trying this build out in different workspaces /
> >>configurations. This will help make sure the build works for more
> >>people
> >>and will get rid of any rough edges.
> >>
> >>Performance (All):
> >>Maven performs parallelization at the module level, an entire module
> >>needs
> >>to complete before any dependent modules can start, this means running
> >>all
> >>the checks like findbugs, checkstyle, tests need to finish. Gradle has
> >>task
> >>level parallelism between subprojects which means that as soon as the
> >>compile and shade steps are done for a project, and dependent
> >>subprojects
> >>can typically start. This means that we get increased parallelism due
> >>to
> >>not needing to wait for findbugs, checkstyle, tests to run. I typically
> >>see
> >>~20 tasks (at peak) running on my desktop in parallel.
> >>
> >>Apache Rat (JB / Romain):
> >>What files are in the rat report that fail (its likely that I'm missing
> >>some exclusion for a build time artifact)? Also, please try the build
> >>again
> >>after running `git clean -fdx` in your workspace.
> >>
> >>Python (JB):
> >>As for the Python SDK, you'll need to share more details about the
> >>failure.
> >>
> >>Gradle 4.3:
> >>I would like to defer the swap to Gradle 4.3 until after this PR since
> >>it
> >>will be a much smaller set of changes.
> >>
> >>On Wed, Nov 8, 2017 at 12:54 AM, Jean-Baptiste Onofré 
> >>wrote:
> >>
> >>> Same for me for rat and python build too:
> >>>
> >>> FAILURE: Build completed with 2 failures.
> >>>
> >>> 1: Task failed with an exception.
> >>> ---
> >>> * What went wrong:
> >>> Execution failed for task ':rat'.
> >>> > Found 905 files with unapproved/unknown licenses. See
> >>> file:/home/jbonofre/Workspace/beam/build/reports/rat/rat-report.txt
> >>>
> >>> * Try:
> >>> Run with --stacktrace option to get the stack trace. Run with --info
> >>or
> >>> --debug option to get more log output.
> >>> 
> >>> ==
> >>>
> >>> 2: Task failed with an exception.
> >>> ---
> >>> * Where:
> >>> Build file '/home/jbonofre/Workspace/beam/sdks/python/build.gradle'
> >>line:
> >>> 64
> >>>
> >>> * What went wrong:
> >>> Execution failed for task ':beam-sdks-parent:beam-sdks-python:lint'.
> >>> > Process 'command 'tox'' finished with non-zero exit value 1
> >>>
> >>>
> >>>
> >>> On 11/08/2017 09:51 AM, Romain Manni-Bucau wrote:
> >>>
>  gradle branch doesnt build for me (some rat issues)
> 
>  Romain Manni-Bucau
>  @rmannibucau |  Blog | Old Blog | Github | LinkedIn
> 
> 
>  2017-11-08 5:41 GMT+01:00 Jean-Baptiste Onofré :
> 
> > Great !
> >
> > What explain these difference ? I'm curious especially for the
> >>clean
> > build
> > all Java modules: is it a question of parallel execution ?
> >
> > Regards
> > JB
> >
> >
> > On 11/08/2017 02:59 AM, Lukasz Cwik wrote:
> >
> >>
> >> The Gradle POC has made significant advances since last week
> >>(shading,
> >> Python, Go, Docker builds, ...). I believe the current state is
> >>close
> >> enough to the Maven build system to warrant a comparison.
> >>
> >> The largest build differences I noticed are:
> >> * Full build takes about ~22mins using Gradle (parallelizing the
> >>three
> >> rounds of Python tests would reduce this to ~17mins) compared to
> >>~38mins
> >> in
> >> Maven
> >> * Clean build all Java modules (skipping over Go/Python
> >> ) takes ~8mins in
> >> Gradle which takes ~36mins in Maven
> >> * Build output is cached 

Re: [DISCUSS] Move away from Apache Maven as build tool

[Romain Manni-Bucau]

Ok, the rat issues I got were:

== File: /home/rmannibucau/1_dev/beam/.idea/*
== File: /home/rmannibucau/1_dev/beam/sdks/python/=1.3.5

The first one could be in my default exclude - even if eclipse/idea
files should be in the default exclude set of beam rat config IMHO,
the last one is more a "?" can probably be exclude as well if created
by the build at some point.


Romain Manni-Bucau
@rmannibucau |  Blog | Old Blog | Github | LinkedIn


2017-11-08 19:17 GMT+01:00 Jean-Baptiste Onofré :
> Thanks for the update. I was swamped on some meetings. I'm back to test the 
> latest changes.
>
> Regards
> JB
>
> On Nov 8, 2017, 18:56, at 18:56, Lukasz Cwik  wrote:
>>Thanks everyone for trying this build out in different workspaces /
>>configurations. This will help make sure the build works for more
>>people
>>and will get rid of any rough edges.
>>
>>Performance (All):
>>Maven performs parallelization at the module level, an entire module
>>needs
>>to complete before any dependent modules can start, this means running
>>all
>>the checks like findbugs, checkstyle, tests need to finish. Gradle has
>>task
>>level parallelism between subprojects which means that as soon as the
>>compile and shade steps are done for a project, and dependent
>>subprojects
>>can typically start. This means that we get increased parallelism due
>>to
>>not needing to wait for findbugs, checkstyle, tests to run. I typically
>>see
>>~20 tasks (at peak) running on my desktop in parallel.
>>
>>Apache Rat (JB / Romain):
>>What files are in the rat report that fail (its likely that I'm missing
>>some exclusion for a build time artifact)? Also, please try the build
>>again
>>after running `git clean -fdx` in your workspace.
>>
>>Python (JB):
>>As for the Python SDK, you'll need to share more details about the
>>failure.
>>
>>Gradle 4.3:
>>I would like to defer the swap to Gradle 4.3 until after this PR since
>>it
>>will be a much smaller set of changes.
>>
>>On Wed, Nov 8, 2017 at 12:54 AM, Jean-Baptiste Onofré 
>>wrote:
>>
>>> Same for me for rat and python build too:
>>>
>>> FAILURE: Build completed with 2 failures.
>>>
>>> 1: Task failed with an exception.
>>> ---
>>> * What went wrong:
>>> Execution failed for task ':rat'.
>>> > Found 905 files with unapproved/unknown licenses. See
>>> file:/home/jbonofre/Workspace/beam/build/reports/rat/rat-report.txt
>>>
>>> * Try:
>>> Run with --stacktrace option to get the stack trace. Run with --info
>>or
>>> --debug option to get more log output.
>>> 
>>> ==
>>>
>>> 2: Task failed with an exception.
>>> ---
>>> * Where:
>>> Build file '/home/jbonofre/Workspace/beam/sdks/python/build.gradle'
>>line:
>>> 64
>>>
>>> * What went wrong:
>>> Execution failed for task ':beam-sdks-parent:beam-sdks-python:lint'.
>>> > Process 'command 'tox'' finished with non-zero exit value 1
>>>
>>>
>>>
>>> On 11/08/2017 09:51 AM, Romain Manni-Bucau wrote:
>>>
 gradle branch doesnt build for me (some rat issues)

 Romain Manni-Bucau
 @rmannibucau |  Blog | Old Blog | Github | LinkedIn


 2017-11-08 5:41 GMT+01:00 Jean-Baptiste Onofré :

> Great !
>
> What explain these difference ? I'm curious especially for the
>>clean
> build
> all Java modules: is it a question of parallel execution ?
>
> Regards
> JB
>
>
> On 11/08/2017 02:59 AM, Lukasz Cwik wrote:
>
>>
>> The Gradle POC has made significant advances since last week
>>(shading,
>> Python, Go, Docker builds, ...). I believe the current state is
>>close
>> enough to the Maven build system to warrant a comparison.
>>
>> The largest build differences I noticed are:
>> * Full build takes about ~22mins using Gradle (parallelizing the
>>three
>> rounds of Python tests would reduce this to ~17mins) compared to
>>~38mins
>> in
>> Maven
>> * Clean build all Java modules (skipping over Go/Python
>> ) takes ~8mins in
>> Gradle which takes ~36mins in Maven
>> * Build output is cached allowing for faster subsequent builds
>>with
>> "gradle
>> buildDependents" allowing for most single module changes taking
>>~2mins
>> to
>> build and test without needing to rely on "mvn install"
>>
>> I have opened PR 4096 
>>so
>> that
>> the Gradle build files merged and then follow up with new Jenkins
>> precommits which are powered by Gradle. This will allow the
>>community to
>> continuing contributing to the Gradle build and also allow for a
>> comparison
>> of the precommit times on the Jenkins executor when using
>>Maven/Gradle.
>> I
>> suggest that those who are interested try out the PR.
>>
>> On Fri, Nov 3, 2017 at 10:29 PM, Jean-Baptiste Onofré

Re: [DISCUSS] Move away from Apache Maven as build tool

[Jean-Baptiste Onofré]

Thanks for the update. I was swamped on some meetings. I'm back to test the 
latest changes.

Regards
JB

On Nov 8, 2017, 18:56, at 18:56, Lukasz Cwik  wrote:
>Thanks everyone for trying this build out in different workspaces /
>configurations. This will help make sure the build works for more
>people
>and will get rid of any rough edges.
>
>Performance (All):
>Maven performs parallelization at the module level, an entire module
>needs
>to complete before any dependent modules can start, this means running
>all
>the checks like findbugs, checkstyle, tests need to finish. Gradle has
>task
>level parallelism between subprojects which means that as soon as the
>compile and shade steps are done for a project, and dependent
>subprojects
>can typically start. This means that we get increased parallelism due
>to
>not needing to wait for findbugs, checkstyle, tests to run. I typically
>see
>~20 tasks (at peak) running on my desktop in parallel.
>
>Apache Rat (JB / Romain):
>What files are in the rat report that fail (its likely that I'm missing
>some exclusion for a build time artifact)? Also, please try the build
>again
>after running `git clean -fdx` in your workspace.
>
>Python (JB):
>As for the Python SDK, you'll need to share more details about the
>failure.
>
>Gradle 4.3:
>I would like to defer the swap to Gradle 4.3 until after this PR since
>it
>will be a much smaller set of changes.
>
>On Wed, Nov 8, 2017 at 12:54 AM, Jean-Baptiste Onofré 
>wrote:
>
>> Same for me for rat and python build too:
>>
>> FAILURE: Build completed with 2 failures.
>>
>> 1: Task failed with an exception.
>> ---
>> * What went wrong:
>> Execution failed for task ':rat'.
>> > Found 905 files with unapproved/unknown licenses. See
>> file:/home/jbonofre/Workspace/beam/build/reports/rat/rat-report.txt
>>
>> * Try:
>> Run with --stacktrace option to get the stack trace. Run with --info
>or
>> --debug option to get more log output.
>> 
>> ==
>>
>> 2: Task failed with an exception.
>> ---
>> * Where:
>> Build file '/home/jbonofre/Workspace/beam/sdks/python/build.gradle'
>line:
>> 64
>>
>> * What went wrong:
>> Execution failed for task ':beam-sdks-parent:beam-sdks-python:lint'.
>> > Process 'command 'tox'' finished with non-zero exit value 1
>>
>>
>>
>> On 11/08/2017 09:51 AM, Romain Manni-Bucau wrote:
>>
>>> gradle branch doesnt build for me (some rat issues)
>>>
>>> Romain Manni-Bucau
>>> @rmannibucau |  Blog | Old Blog | Github | LinkedIn
>>>
>>>
>>> 2017-11-08 5:41 GMT+01:00 Jean-Baptiste Onofré :
>>>
 Great !

 What explain these difference ? I'm curious especially for the
>clean
 build
 all Java modules: is it a question of parallel execution ?

 Regards
 JB


 On 11/08/2017 02:59 AM, Lukasz Cwik wrote:

>
> The Gradle POC has made significant advances since last week
>(shading,
> Python, Go, Docker builds, ...). I believe the current state is
>close
> enough to the Maven build system to warrant a comparison.
>
> The largest build differences I noticed are:
> * Full build takes about ~22mins using Gradle (parallelizing the
>three
> rounds of Python tests would reduce this to ~17mins) compared to
>~38mins
> in
> Maven
> * Clean build all Java modules (skipping over Go/Python
> ) takes ~8mins in
> Gradle which takes ~36mins in Maven
> * Build output is cached allowing for faster subsequent builds
>with
> "gradle
> buildDependents" allowing for most single module changes taking
>~2mins
> to
> build and test without needing to rely on "mvn install"
>
> I have opened PR 4096 
>so
> that
> the Gradle build files merged and then follow up with new Jenkins
> precommits which are powered by Gradle. This will allow the
>community to
> continuing contributing to the Gradle build and also allow for a
> comparison
> of the precommit times on the Jenkins executor when using
>Maven/Gradle.
> I
> suggest that those who are interested try out the PR.
>
> On Fri, Nov 3, 2017 at 10:29 PM, Jean-Baptiste Onofré
>
> wrote:
>
> That makes sense. The point is that we have to compare
>equivalently. I'm
>> also curious about Gradle PoC assuming it does the same actions
>as
>> Maven.
>>
>> Regards
>> JB
>>
>> On Nov 3, 2017, 20:41, at 20:41, Kenneth Knowles
>> 
>> wrote:
>>
>>>
>>> I'm confident that any choice will speed things up dramatically
>even
>>> beyond
>>> a fast profile, even if the new tool runs all the extra stuff.
>But
>>> that
>>> is
>>> a question that we can answer empirically anyhow. Let's see how
>it
>>> goes!
>>>

Re: [DISCUSS] Move away from Apache Maven as build tool

[Lukasz Cwik]

Thanks everyone for trying this build out in different workspaces /
configurations. This will help make sure the build works for more people
and will get rid of any rough edges.

Performance (All):
Maven performs parallelization at the module level, an entire module needs
to complete before any dependent modules can start, this means running all
the checks like findbugs, checkstyle, tests need to finish. Gradle has task
level parallelism between subprojects which means that as soon as the
compile and shade steps are done for a project, and dependent subprojects
can typically start. This means that we get increased parallelism due to
not needing to wait for findbugs, checkstyle, tests to run. I typically see
~20 tasks (at peak) running on my desktop in parallel.

Apache Rat (JB / Romain):
What files are in the rat report that fail (its likely that I'm missing
some exclusion for a build time artifact)? Also, please try the build again
after running `git clean -fdx` in your workspace.

Python (JB):
As for the Python SDK, you'll need to share more details about the failure.

Gradle 4.3:
I would like to defer the swap to Gradle 4.3 until after this PR since it
will be a much smaller set of changes.

On Wed, Nov 8, 2017 at 12:54 AM, Jean-Baptiste Onofré 
wrote:

> Same for me for rat and python build too:
>
> FAILURE: Build completed with 2 failures.
>
> 1: Task failed with an exception.
> ---
> * What went wrong:
> Execution failed for task ':rat'.
> > Found 905 files with unapproved/unknown licenses. See
> file:/home/jbonofre/Workspace/beam/build/reports/rat/rat-report.txt
>
> * Try:
> Run with --stacktrace option to get the stack trace. Run with --info or
> --debug option to get more log output.
> 
> ==
>
> 2: Task failed with an exception.
> ---
> * Where:
> Build file '/home/jbonofre/Workspace/beam/sdks/python/build.gradle' line:
> 64
>
> * What went wrong:
> Execution failed for task ':beam-sdks-parent:beam-sdks-python:lint'.
> > Process 'command 'tox'' finished with non-zero exit value 1
>
>
>
> On 11/08/2017 09:51 AM, Romain Manni-Bucau wrote:
>
>> gradle branch doesnt build for me (some rat issues)
>>
>> Romain Manni-Bucau
>> @rmannibucau |  Blog | Old Blog | Github | LinkedIn
>>
>>
>> 2017-11-08 5:41 GMT+01:00 Jean-Baptiste Onofré :
>>
>>> Great !
>>>
>>> What explain these difference ? I'm curious especially for the clean
>>> build
>>> all Java modules: is it a question of parallel execution ?
>>>
>>> Regards
>>> JB
>>>
>>>
>>> On 11/08/2017 02:59 AM, Lukasz Cwik wrote:
>>>

 The Gradle POC has made significant advances since last week (shading,
 Python, Go, Docker builds, ...). I believe the current state is close
 enough to the Maven build system to warrant a comparison.

 The largest build differences I noticed are:
 * Full build takes about ~22mins using Gradle (parallelizing the three
 rounds of Python tests would reduce this to ~17mins) compared to ~38mins
 in
 Maven
 * Clean build all Java modules (skipping over Go/Python
 ) takes ~8mins in
 Gradle which takes ~36mins in Maven
 * Build output is cached allowing for faster subsequent builds with
 "gradle
 buildDependents" allowing for most single module changes taking ~2mins
 to
 build and test without needing to rely on "mvn install"

 I have opened PR 4096  so
 that
 the Gradle build files merged and then follow up with new Jenkins
 precommits which are powered by Gradle. This will allow the community to
 continuing contributing to the Gradle build and also allow for a
 comparison
 of the precommit times on the Jenkins executor when using Maven/Gradle.
 I
 suggest that those who are interested try out the PR.

 On Fri, Nov 3, 2017 at 10:29 PM, Jean-Baptiste Onofré 
 wrote:

 That makes sense. The point is that we have to compare equivalently. I'm
> also curious about Gradle PoC assuming it does the same actions as
> Maven.
>
> Regards
> JB
>
> On Nov 3, 2017, 20:41, at 20:41, Kenneth Knowles
> 
> wrote:
>
>>
>> I'm confident that any choice will speed things up dramatically even
>> beyond
>> a fast profile, even if the new tool runs all the extra stuff. But
>> that
>> is
>> a question that we can answer empirically anyhow. Let's see how it
>> goes!
>>
>> Incidentally, my experiments with Bazel have led me to the conclusion
>> that
>> it is not the right choice for us so I'm not going to be proposing any
>> completed POC of that right now. I'm interested in the outcome of the
>> Gradle POC.
>>
>> Kenn
>>
>>
>> On Fri, Nov 3, 2017 at 3:30 AM, Jean-Baptiste 

Re: [DISCUSS] Move away from Apache Maven as build tool

[Jean-Baptiste Onofré]

Same for me for rat and python build too:

FAILURE: Build completed with 2 failures.

1: Task failed with an exception.
---
* What went wrong:
Execution failed for task ':rat'.
> Found 905 files with unapproved/unknown licenses. See 
file:/home/jbonofre/Workspace/beam/build/reports/rat/rat-report.txt


* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output.

==

2: Task failed with an exception.
---
* Where:
Build file '/home/jbonofre/Workspace/beam/sdks/python/build.gradle' line: 64

* What went wrong:
Execution failed for task ':beam-sdks-parent:beam-sdks-python:lint'.
> Process 'command 'tox'' finished with non-zero exit value 1


On 11/08/2017 09:51 AM, Romain Manni-Bucau wrote:

gradle branch doesnt build for me (some rat issues)

Romain Manni-Bucau
@rmannibucau |  Blog | Old Blog | Github | LinkedIn


2017-11-08 5:41 GMT+01:00 Jean-Baptiste Onofré :

Great !

What explain these difference ? I'm curious especially for the clean build
all Java modules: is it a question of parallel execution ?

Regards
JB


On 11/08/2017 02:59 AM, Lukasz Cwik wrote:


The Gradle POC has made significant advances since last week (shading,
Python, Go, Docker builds, ...). I believe the current state is close
enough to the Maven build system to warrant a comparison.

The largest build differences I noticed are:
* Full build takes about ~22mins using Gradle (parallelizing the three
rounds of Python tests would reduce this to ~17mins) compared to ~38mins
in
Maven
* Clean build all Java modules (skipping over Go/Python) takes ~8mins in
Gradle which takes ~36mins in Maven
* Build output is cached allowing for faster subsequent builds with
"gradle
buildDependents" allowing for most single module changes taking ~2mins to
build and test without needing to rely on "mvn install"

I have opened PR 4096  so that
the Gradle build files merged and then follow up with new Jenkins
precommits which are powered by Gradle. This will allow the community to
continuing contributing to the Gradle build and also allow for a
comparison
of the precommit times on the Jenkins executor when using Maven/Gradle. I
suggest that those who are interested try out the PR.

On Fri, Nov 3, 2017 at 10:29 PM, Jean-Baptiste Onofré 
wrote:


That makes sense. The point is that we have to compare equivalently. I'm
also curious about Gradle PoC assuming it does the same actions as Maven.

Regards
JB

On Nov 3, 2017, 20:41, at 20:41, Kenneth Knowles 
wrote:


I'm confident that any choice will speed things up dramatically even
beyond
a fast profile, even if the new tool runs all the extra stuff. But that
is
a question that we can answer empirically anyhow. Let's see how it
goes!

Incidentally, my experiments with Bazel have led me to the conclusion
that
it is not the right choice for us so I'm not going to be proposing any
completed POC of that right now. I'm interested in the outcome of the
Gradle POC.

Kenn


On Fri, Nov 3, 2017 at 3:30 AM, Jean-Baptiste Onofré 
wrote:


Hi

It's what I said in a previous e-mail: I don't think that just


changing


the build tool will improve a lot the build time.

We already know (and discussed while ago) that plugins like findbugs,
checkstyle, etc are taking time.

So, I think we can already have a fast profile.

Regards
JB

On Nov 3, 2017, 11:16, at 11:16, Romain Manni-Bucau





wrote:


Hi guys,

when you check the duration of each mojo of the build (almost since
python part of the build just breaks it locally) you see that there


is


no real link with maven for the perf issues beam can encounter:
https://gist.github.com/rmannibucau/f65fdde28d5dab0fdac50633f84554c9
(generated from the profiling of tesla-profile and parsed with



https://gist.github.com/rmannibucau/e329d54b8af6c009f46fd151d10037ad)



Before PoC-ing other tools which will end up to either have the same
issues if the other builds do the same things (test, checkstyle,
enforcer, findbugs, ...) or have a less reliable build (trying to


skip


some parts of the build if "untouched" - note that this is a very


hard


issue since static code anaylizis doesn't give you any guarantee of
what it does with modern code - then maybe some action can be taken


on


the current build:

- testing https://github.com/vackosar/gitflow-incremental-builder or
https://github.com/khmarbaise/incremental-module-builder maybe or do
the same kind of extension including the beam needs (/!\ the


previous


warning is still accurate and requires a full run at some point to
validate the graph detection algorithm didn't get abused by some
indirect code dependency)
- maybe try to get rid of some shades (it is a bit crazy ATM to have
so much shades no?)
- the CI can have profiles based on 

Re: [DISCUSS] Move away from Apache Maven as build tool

[Romain Manni-Bucau]

gradle branch doesnt build for me (some rat issues)

Romain Manni-Bucau
@rmannibucau |  Blog | Old Blog | Github | LinkedIn


2017-11-08 5:41 GMT+01:00 Jean-Baptiste Onofré :
> Great !
>
> What explain these difference ? I'm curious especially for the clean build
> all Java modules: is it a question of parallel execution ?
>
> Regards
> JB
>
>
> On 11/08/2017 02:59 AM, Lukasz Cwik wrote:
>>
>> The Gradle POC has made significant advances since last week (shading,
>> Python, Go, Docker builds, ...). I believe the current state is close
>> enough to the Maven build system to warrant a comparison.
>>
>> The largest build differences I noticed are:
>> * Full build takes about ~22mins using Gradle (parallelizing the three
>> rounds of Python tests would reduce this to ~17mins) compared to ~38mins
>> in
>> Maven
>> * Clean build all Java modules (skipping over Go/Python) takes ~8mins in
>> Gradle which takes ~36mins in Maven
>> * Build output is cached allowing for faster subsequent builds with
>> "gradle
>> buildDependents" allowing for most single module changes taking ~2mins to
>> build and test without needing to rely on "mvn install"
>>
>> I have opened PR 4096  so that
>> the Gradle build files merged and then follow up with new Jenkins
>> precommits which are powered by Gradle. This will allow the community to
>> continuing contributing to the Gradle build and also allow for a
>> comparison
>> of the precommit times on the Jenkins executor when using Maven/Gradle. I
>> suggest that those who are interested try out the PR.
>>
>> On Fri, Nov 3, 2017 at 10:29 PM, Jean-Baptiste Onofré 
>> wrote:
>>
>>> That makes sense. The point is that we have to compare equivalently. I'm
>>> also curious about Gradle PoC assuming it does the same actions as Maven.
>>>
>>> Regards
>>> JB
>>>
>>> On Nov 3, 2017, 20:41, at 20:41, Kenneth Knowles 
>>> wrote:

 I'm confident that any choice will speed things up dramatically even
 beyond
 a fast profile, even if the new tool runs all the extra stuff. But that
 is
 a question that we can answer empirically anyhow. Let's see how it
 goes!

 Incidentally, my experiments with Bazel have led me to the conclusion
 that
 it is not the right choice for us so I'm not going to be proposing any
 completed POC of that right now. I'm interested in the outcome of the
 Gradle POC.

 Kenn


 On Fri, Nov 3, 2017 at 3:30 AM, Jean-Baptiste Onofré 
 wrote:

> Hi
>
> It's what I said in a previous e-mail: I don't think that just

 changing
>
> the build tool will improve a lot the build time.
>
> We already know (and discussed while ago) that plugins like findbugs,
> checkstyle, etc are taking time.
>
> So, I think we can already have a fast profile.
>
> Regards
> JB
>
> On Nov 3, 2017, 11:16, at 11:16, Romain Manni-Bucau

 
>
> wrote:
>>
>> Hi guys,
>>
>> when you check the duration of each mojo of the build (almost since
>> python part of the build just breaks it locally) you see that there

 is
>>
>> no real link with maven for the perf issues beam can encounter:
>> https://gist.github.com/rmannibucau/f65fdde28d5dab0fdac50633f84554c9
>> (generated from the profiling of tesla-profile and parsed with
>
>
> https://gist.github.com/rmannibucau/e329d54b8af6c009f46fd151d10037ad)
>>
>>
>> Before PoC-ing other tools which will end up to either have the same
>> issues if the other builds do the same things (test, checkstyle,
>> enforcer, findbugs, ...) or have a less reliable build (trying to

 skip
>>
>> some parts of the build if "untouched" - note that this is a very

 hard
>>
>> issue since static code anaylizis doesn't give you any guarantee of
>> what it does with modern code - then maybe some action can be taken

 on
>>
>> the current build:
>>
>> - testing https://github.com/vackosar/gitflow-incremental-builder or
>> https://github.com/khmarbaise/incremental-module-builder maybe or do
>> the same kind of extension including the beam needs (/!\ the

 previous
>>
>> warning is still accurate and requires a full run at some point to
>> validate the graph detection algorithm didn't get abused by some
>> indirect code dependency)
>> - maybe try to get rid of some shades (it is a bit crazy ATM to have
>> so much shades no?)
>> - the CI can have profiles based on a PR convention (name of the
>> branch?) to select the build profile, for instance
>> fb/elasticsearch_super-nice-PR would build only the elasticsearch
>> modules, jenkins/travis have this ability since they support

 scripting
>>
>> - 

Re: [DISCUSS] Move away from Apache Maven as build tool

[Jean-Baptiste Onofré]

Great !

What explain these difference ? I'm curious especially for the clean build all 
Java modules: is it a question of parallel execution ?


Regards
JB

On 11/08/2017 02:59 AM, Lukasz Cwik wrote:

The Gradle POC has made significant advances since last week (shading,
Python, Go, Docker builds, ...). I believe the current state is close
enough to the Maven build system to warrant a comparison.

The largest build differences I noticed are:
* Full build takes about ~22mins using Gradle (parallelizing the three
rounds of Python tests would reduce this to ~17mins) compared to ~38mins in
Maven
* Clean build all Java modules (skipping over Go/Python) takes ~8mins in
Gradle which takes ~36mins in Maven
* Build output is cached allowing for faster subsequent builds with "gradle
buildDependents" allowing for most single module changes taking ~2mins to
build and test without needing to rely on "mvn install"

I have opened PR 4096  so that
the Gradle build files merged and then follow up with new Jenkins
precommits which are powered by Gradle. This will allow the community to
continuing contributing to the Gradle build and also allow for a comparison
of the precommit times on the Jenkins executor when using Maven/Gradle. I
suggest that those who are interested try out the PR.

On Fri, Nov 3, 2017 at 10:29 PM, Jean-Baptiste Onofré 
wrote:


That makes sense. The point is that we have to compare equivalently. I'm
also curious about Gradle PoC assuming it does the same actions as Maven.

Regards
JB

On Nov 3, 2017, 20:41, at 20:41, Kenneth Knowles 
wrote:

I'm confident that any choice will speed things up dramatically even
beyond
a fast profile, even if the new tool runs all the extra stuff. But that
is
a question that we can answer empirically anyhow. Let's see how it
goes!

Incidentally, my experiments with Bazel have led me to the conclusion
that
it is not the right choice for us so I'm not going to be proposing any
completed POC of that right now. I'm interested in the outcome of the
Gradle POC.

Kenn


On Fri, Nov 3, 2017 at 3:30 AM, Jean-Baptiste Onofré 
wrote:


Hi

It's what I said in a previous e-mail: I don't think that just

changing

the build tool will improve a lot the build time.

We already know (and discussed while ago) that plugins like findbugs,
checkstyle, etc are taking time.

So, I think we can already have a fast profile.

Regards
JB

On Nov 3, 2017, 11:16, at 11:16, Romain Manni-Bucau



wrote:

Hi guys,

when you check the duration of each mojo of the build (almost since
python part of the build just breaks it locally) you see that there

is

no real link with maven for the perf issues beam can encounter:
https://gist.github.com/rmannibucau/f65fdde28d5dab0fdac50633f84554c9
(generated from the profiling of tesla-profile and parsed with


https://gist.github.com/rmannibucau/e329d54b8af6c009f46fd151d10037ad)


Before PoC-ing other tools which will end up to either have the same
issues if the other builds do the same things (test, checkstyle,
enforcer, findbugs, ...) or have a less reliable build (trying to

skip

some parts of the build if "untouched" - note that this is a very

hard

issue since static code anaylizis doesn't give you any guarantee of
what it does with modern code - then maybe some action can be taken

on

the current build:

- testing https://github.com/vackosar/gitflow-incremental-builder or
https://github.com/khmarbaise/incremental-module-builder maybe or do
the same kind of extension including the beam needs (/!\ the

previous

warning is still accurate and requires a full run at some point to
validate the graph detection algorithm didn't get abused by some
indirect code dependency)
- maybe try to get rid of some shades (it is a bit crazy ATM to have
so much shades no?)
- the CI can have profiles based on a PR convention (name of the
branch?) to select the build profile, for instance
fb/elasticsearch_super-nice-PR would build only the elasticsearch
modules, jenkins/travis have this ability since they support

scripting

- document how to setup a "fastBuild" profile in its settings.xml
which bypasses checkstyle, enforcer plugin, findbugs, etc... for

fast

development iterations




Romain Manni-Bucau
@rmannibucau |  Blog | Old Blog | Github | LinkedIn


2017-11-01 21:02 GMT+01:00 Kenneth Knowles :

I have started one, here:

https://github.com/kennknowles/beam/commits/bazel.

It is not nearly as far along as Luke's. For the POC I am just

putting

things in one root BUILD, and learning where we might find the

necessary

plugins as I go. I am happy to grant push access to this branch.

It

would

be superb if you had some time to work through the Python steps.

On Wed, Nov 1, 2017 at 10:09 AM, Ahmet Altay



wrote:


Has anyone started a POC with Bazel? I would be interested in

helping that

Re: [DISCUSS] Move away from Apache Maven as build tool

[Ted Yu]

Nice work.
From my recent experience, gradle 4.3 is needed when building with java 9.0.1
See INFRA-15448
 Original message From: Lukasz Cwik <lc...@google.com.INVALID> 
Date: 11/7/17  5:59 PM  (GMT-08:00) To: dev@beam.apache.org Subject: Re: 
[DISCUSS] Move away from Apache Maven as build tool 
The Gradle POC has made significant advances since last week (shading,
Python, Go, Docker builds, ...). I believe the current state is close
enough to the Maven build system to warrant a comparison.

The largest build differences I noticed are:
* Full build takes about ~22mins using Gradle (parallelizing the three
rounds of Python tests would reduce this to ~17mins) compared to ~38mins in
Maven
* Clean build all Java modules (skipping over Go/Python) takes ~8mins in
Gradle which takes ~36mins in Maven
* Build output is cached allowing for faster subsequent builds with "gradle
buildDependents" allowing for most single module changes taking ~2mins to
build and test without needing to rely on "mvn install"

I have opened PR 4096 <https://github.com/apache/beam/pull/4096> so that
the Gradle build files merged and then follow up with new Jenkins
precommits which are powered by Gradle. This will allow the community to
continuing contributing to the Gradle build and also allow for a comparison
of the precommit times on the Jenkins executor when using Maven/Gradle. I
suggest that those who are interested try out the PR.

On Fri, Nov 3, 2017 at 10:29 PM, Jean-Baptiste Onofré <j...@nanthrax.net>
wrote:

> That makes sense. The point is that we have to compare equivalently. I'm
> also curious about Gradle PoC assuming it does the same actions as Maven.
>
> Regards
> JB
>
> On Nov 3, 2017, 20:41, at 20:41, Kenneth Knowles <k...@google.com.INVALID>
> wrote:
> >I'm confident that any choice will speed things up dramatically even
> >beyond
> >a fast profile, even if the new tool runs all the extra stuff. But that
> >is
> >a question that we can answer empirically anyhow. Let's see how it
> >goes!
> >
> >Incidentally, my experiments with Bazel have led me to the conclusion
> >that
> >it is not the right choice for us so I'm not going to be proposing any
> >completed POC of that right now. I'm interested in the outcome of the
> >Gradle POC.
> >
> >Kenn
> >
> >
> >On Fri, Nov 3, 2017 at 3:30 AM, Jean-Baptiste Onofré <j...@nanthrax.net>
> >wrote:
> >
> >> Hi
> >>
> >> It's what I said in a previous e-mail: I don't think that just
> >changing
> >> the build tool will improve a lot the build time.
> >>
> >> We already know (and discussed while ago) that plugins like findbugs,
> >> checkstyle, etc are taking time.
> >>
> >> So, I think we can already have a fast profile.
> >>
> >> Regards
> >> JB
> >>
> >> On Nov 3, 2017, 11:16, at 11:16, Romain Manni-Bucau
> ><rmannibu...@gmail.com>
> >> wrote:
> >> >Hi guys,
> >> >
> >> >when you check the duration of each mojo of the build (almost since
> >> >python part of the build just breaks it locally) you see that there
> >is
> >> >no real link with maven for the perf issues beam can encounter:
> >> >https://gist.github.com/rmannibucau/f65fdde28d5dab0fdac50633f84554c9
> >> >(generated from the profiling of tesla-profile and parsed with
> >>
> >>https://gist.github.com/rmannibucau/e329d54b8af6c009f46fd151d10037ad)
> >> >
> >> >Before PoC-ing other tools which will end up to either have the same
> >> >issues if the other builds do the same things (test, checkstyle,
> >> >enforcer, findbugs, ...) or have a less reliable build (trying to
> >skip
> >> >some parts of the build if "untouched" - note that this is a very
> >hard
> >> >issue since static code anaylizis doesn't give you any guarantee of
> >> >what it does with modern code - then maybe some action can be taken
> >on
> >> >the current build:
> >> >
> >> >- testing https://github.com/vackosar/gitflow-incremental-builder or
> >> >https://github.com/khmarbaise/incremental-module-builder maybe or do
> >> >the same kind of extension including the beam needs (/!\ the
> >previous
> >> >warning is still accurate and requires a full run at some point to
> >> >validate the graph detection algorithm didn't get abused by some
> >> >indirect code dependency)
> >> >- maybe try to get rid of some shades (it is a bit crazy ATM to have
> >> >so much shades no?)
> &

Re: [DISCUSS] Move away from Apache Maven as build tool

[Eugene Kirpichov]

Wow, these are impressive differences! Do you know why is that so? In
particular I'm surprised at the difference in performance of the Java clean
build: feels like the performed work ought to be the same. Does it all come
from better parallelization? (in Maven I believe we don't specify "-T 1C"
on Jenkins, I'm not sure why - but I typically specify it locally when
verifying PRs, and the build takes ~15 minutes, which is still 2x slower
than 8 minutes you got with Gradle)

On Tue, Nov 7, 2017 at 6:20 PM Lukasz Cwik  wrote:

> The Gradle POC has made significant advances since last week (shading,
> Python, Go, Docker builds, ...). I believe the current state is close
> enough to the Maven build system to warrant a comparison.
>
> The largest build differences I noticed are:
> * Full build takes about ~22mins using Gradle (parallelizing the three
> rounds of Python tests would reduce this to ~17mins) compared to ~38mins in
> Maven
> * Clean build all Java modules (skipping over Go/Python
> ) takes ~8mins in
> Gradle which takes ~36mins in Maven
> * Build output is cached allowing for faster subsequent builds with "gradle
> buildDependents" allowing for most single module changes taking ~2mins to
> build and test without needing to rely on "mvn install"
>
> I have opened PR 4096  so that
> the Gradle build files merged and then follow up with new Jenkins
> precommits which are powered by Gradle. This will allow the community to
> continuing contributing to the Gradle build and also allow for a comparison
> of the precommit times on the Jenkins executor when using Maven/Gradle. I
> suggest that those who are interested try out the PR.
>
> On Fri, Nov 3, 2017 at 10:29 PM, Jean-Baptiste Onofré 
> wrote:
>
> > That makes sense. The point is that we have to compare equivalently. I'm
> > also curious about Gradle PoC assuming it does the same actions as Maven.
> >
> > Regards
> > JB
> >
> > On Nov 3, 2017, 20:41, at 20:41, Kenneth Knowles  >
> > wrote:
> > >I'm confident that any choice will speed things up dramatically even
> > >beyond
> > >a fast profile, even if the new tool runs all the extra stuff. But that
> > >is
> > >a question that we can answer empirically anyhow. Let's see how it
> > >goes!
> > >
> > >Incidentally, my experiments with Bazel have led me to the conclusion
> > >that
> > >it is not the right choice for us so I'm not going to be proposing any
> > >completed POC of that right now. I'm interested in the outcome of the
> > >Gradle POC.
> > >
> > >Kenn
> > >
> > >
> > >On Fri, Nov 3, 2017 at 3:30 AM, Jean-Baptiste Onofré 
> > >wrote:
> > >
> > >> Hi
> > >>
> > >> It's what I said in a previous e-mail: I don't think that just
> > >changing
> > >> the build tool will improve a lot the build time.
> > >>
> > >> We already know (and discussed while ago) that plugins like findbugs,
> > >> checkstyle, etc are taking time.
> > >>
> > >> So, I think we can already have a fast profile.
> > >>
> > >> Regards
> > >> JB
> > >>
> > >> On Nov 3, 2017, 11:16, at 11:16, Romain Manni-Bucau
> > >
> > >> wrote:
> > >> >Hi guys,
> > >> >
> > >> >when you check the duration of each mojo of the build (almost since
> > >> >python part of the build just breaks it locally) you see that there
> > >is
> > >> >no real link with maven for the perf issues beam can encounter:
> > >> >https://gist.github.com/rmannibucau/f65fdde28d5dab0fdac50633f84554c9
> > >> >(generated from the profiling of tesla-profile and parsed with
> > >>
> > >>https://gist.github.com/rmannibucau/e329d54b8af6c009f46fd151d10037ad)
> > >> >
> > >> >Before PoC-ing other tools which will end up to either have the same
> > >> >issues if the other builds do the same things (test, checkstyle,
> > >> >enforcer, findbugs, ...) or have a less reliable build (trying to
> > >skip
> > >> >some parts of the build if "untouched" - note that this is a very
> > >hard
> > >> >issue since static code anaylizis doesn't give you any guarantee of
> > >> >what it does with modern code - then maybe some action can be taken
> > >on
> > >> >the current build:
> > >> >
> > >> >- testing https://github.com/vackosar/gitflow-incremental-builder or
> > >> >https://github.com/khmarbaise/incremental-module-builder maybe or do
> > >> >the same kind of extension including the beam needs (/!\ the
> > >previous
> > >> >warning is still accurate and requires a full run at some point to
> > >> >validate the graph detection algorithm didn't get abused by some
> > >> >indirect code dependency)
> > >> >- maybe try to get rid of some shades (it is a bit crazy ATM to have
> > >> >so much shades no?)
> > >> >- the CI can have profiles based on a PR convention (name of the
> > >> >branch?) to select the build profile, for instance
> > >> >fb/elasticsearch_super-nice-PR would build only the elasticsearch
> > >> 

Re: [DISCUSS] Move away from Apache Maven as build tool

[Lukasz Cwik]

The Gradle POC has made significant advances since last week (shading,
Python, Go, Docker builds, ...). I believe the current state is close
enough to the Maven build system to warrant a comparison.

The largest build differences I noticed are:
* Full build takes about ~22mins using Gradle (parallelizing the three
rounds of Python tests would reduce this to ~17mins) compared to ~38mins in
Maven
* Clean build all Java modules (skipping over Go/Python) takes ~8mins in
Gradle which takes ~36mins in Maven
* Build output is cached allowing for faster subsequent builds with "gradle
buildDependents" allowing for most single module changes taking ~2mins to
build and test without needing to rely on "mvn install"

I have opened PR 4096  so that
the Gradle build files merged and then follow up with new Jenkins
precommits which are powered by Gradle. This will allow the community to
continuing contributing to the Gradle build and also allow for a comparison
of the precommit times on the Jenkins executor when using Maven/Gradle. I
suggest that those who are interested try out the PR.

On Fri, Nov 3, 2017 at 10:29 PM, Jean-Baptiste Onofré 
wrote:

> That makes sense. The point is that we have to compare equivalently. I'm
> also curious about Gradle PoC assuming it does the same actions as Maven.
>
> Regards
> JB
>
> On Nov 3, 2017, 20:41, at 20:41, Kenneth Knowles 
> wrote:
> >I'm confident that any choice will speed things up dramatically even
> >beyond
> >a fast profile, even if the new tool runs all the extra stuff. But that
> >is
> >a question that we can answer empirically anyhow. Let's see how it
> >goes!
> >
> >Incidentally, my experiments with Bazel have led me to the conclusion
> >that
> >it is not the right choice for us so I'm not going to be proposing any
> >completed POC of that right now. I'm interested in the outcome of the
> >Gradle POC.
> >
> >Kenn
> >
> >
> >On Fri, Nov 3, 2017 at 3:30 AM, Jean-Baptiste Onofré 
> >wrote:
> >
> >> Hi
> >>
> >> It's what I said in a previous e-mail: I don't think that just
> >changing
> >> the build tool will improve a lot the build time.
> >>
> >> We already know (and discussed while ago) that plugins like findbugs,
> >> checkstyle, etc are taking time.
> >>
> >> So, I think we can already have a fast profile.
> >>
> >> Regards
> >> JB
> >>
> >> On Nov 3, 2017, 11:16, at 11:16, Romain Manni-Bucau
> >
> >> wrote:
> >> >Hi guys,
> >> >
> >> >when you check the duration of each mojo of the build (almost since
> >> >python part of the build just breaks it locally) you see that there
> >is
> >> >no real link with maven for the perf issues beam can encounter:
> >> >https://gist.github.com/rmannibucau/f65fdde28d5dab0fdac50633f84554c9
> >> >(generated from the profiling of tesla-profile and parsed with
> >>
> >>https://gist.github.com/rmannibucau/e329d54b8af6c009f46fd151d10037ad)
> >> >
> >> >Before PoC-ing other tools which will end up to either have the same
> >> >issues if the other builds do the same things (test, checkstyle,
> >> >enforcer, findbugs, ...) or have a less reliable build (trying to
> >skip
> >> >some parts of the build if "untouched" - note that this is a very
> >hard
> >> >issue since static code anaylizis doesn't give you any guarantee of
> >> >what it does with modern code - then maybe some action can be taken
> >on
> >> >the current build:
> >> >
> >> >- testing https://github.com/vackosar/gitflow-incremental-builder or
> >> >https://github.com/khmarbaise/incremental-module-builder maybe or do
> >> >the same kind of extension including the beam needs (/!\ the
> >previous
> >> >warning is still accurate and requires a full run at some point to
> >> >validate the graph detection algorithm didn't get abused by some
> >> >indirect code dependency)
> >> >- maybe try to get rid of some shades (it is a bit crazy ATM to have
> >> >so much shades no?)
> >> >- the CI can have profiles based on a PR convention (name of the
> >> >branch?) to select the build profile, for instance
> >> >fb/elasticsearch_super-nice-PR would build only the elasticsearch
> >> >modules, jenkins/travis have this ability since they support
> >scripting
> >> >- document how to setup a "fastBuild" profile in its settings.xml
> >> >which bypasses checkstyle, enforcer plugin, findbugs, etc... for
> >fast
> >> >development iterations
> >> >
> >> >
> >> >
> >> >
> >> >Romain Manni-Bucau
> >> >@rmannibucau |  Blog | Old Blog | Github | LinkedIn
> >> >
> >> >
> >> >2017-11-01 21:02 GMT+01:00 Kenneth Knowles :
> >> >> I have started one, here:
> >> >https://github.com/kennknowles/beam/commits/bazel.
> >> >> It is not nearly as far along as Luke's. For the POC I am just
> >> >putting
> >> >> things in one root BUILD, and learning where we might find the
> >> >necessary
> >> >> plugins as I go. I am happy to grant push access to this branch.
> >It
> 

Re: [DISCUSS] Move away from Apache Maven as build tool

[Jean-Baptiste Onofré]

That makes sense. The point is that we have to compare equivalently. I'm also 
curious about Gradle PoC assuming it does the same actions as Maven.

Regards
JB

On Nov 3, 2017, 20:41, at 20:41, Kenneth Knowles  
wrote:
>I'm confident that any choice will speed things up dramatically even
>beyond
>a fast profile, even if the new tool runs all the extra stuff. But that
>is
>a question that we can answer empirically anyhow. Let's see how it
>goes!
>
>Incidentally, my experiments with Bazel have led me to the conclusion
>that
>it is not the right choice for us so I'm not going to be proposing any
>completed POC of that right now. I'm interested in the outcome of the
>Gradle POC.
>
>Kenn
>
>
>On Fri, Nov 3, 2017 at 3:30 AM, Jean-Baptiste Onofré 
>wrote:
>
>> Hi
>>
>> It's what I said in a previous e-mail: I don't think that just
>changing
>> the build tool will improve a lot the build time.
>>
>> We already know (and discussed while ago) that plugins like findbugs,
>> checkstyle, etc are taking time.
>>
>> So, I think we can already have a fast profile.
>>
>> Regards
>> JB
>>
>> On Nov 3, 2017, 11:16, at 11:16, Romain Manni-Bucau
>
>> wrote:
>> >Hi guys,
>> >
>> >when you check the duration of each mojo of the build (almost since
>> >python part of the build just breaks it locally) you see that there
>is
>> >no real link with maven for the perf issues beam can encounter:
>> >https://gist.github.com/rmannibucau/f65fdde28d5dab0fdac50633f84554c9
>> >(generated from the profiling of tesla-profile and parsed with
>>
>>https://gist.github.com/rmannibucau/e329d54b8af6c009f46fd151d10037ad)
>> >
>> >Before PoC-ing other tools which will end up to either have the same
>> >issues if the other builds do the same things (test, checkstyle,
>> >enforcer, findbugs, ...) or have a less reliable build (trying to
>skip
>> >some parts of the build if "untouched" - note that this is a very
>hard
>> >issue since static code anaylizis doesn't give you any guarantee of
>> >what it does with modern code - then maybe some action can be taken
>on
>> >the current build:
>> >
>> >- testing https://github.com/vackosar/gitflow-incremental-builder or
>> >https://github.com/khmarbaise/incremental-module-builder maybe or do
>> >the same kind of extension including the beam needs (/!\ the
>previous
>> >warning is still accurate and requires a full run at some point to
>> >validate the graph detection algorithm didn't get abused by some
>> >indirect code dependency)
>> >- maybe try to get rid of some shades (it is a bit crazy ATM to have
>> >so much shades no?)
>> >- the CI can have profiles based on a PR convention (name of the
>> >branch?) to select the build profile, for instance
>> >fb/elasticsearch_super-nice-PR would build only the elasticsearch
>> >modules, jenkins/travis have this ability since they support
>scripting
>> >- document how to setup a "fastBuild" profile in its settings.xml
>> >which bypasses checkstyle, enforcer plugin, findbugs, etc... for
>fast
>> >development iterations
>> >
>> >
>> >
>> >
>> >Romain Manni-Bucau
>> >@rmannibucau |  Blog | Old Blog | Github | LinkedIn
>> >
>> >
>> >2017-11-01 21:02 GMT+01:00 Kenneth Knowles :
>> >> I have started one, here:
>> >https://github.com/kennknowles/beam/commits/bazel.
>> >> It is not nearly as far along as Luke's. For the POC I am just
>> >putting
>> >> things in one root BUILD, and learning where we might find the
>> >necessary
>> >> plugins as I go. I am happy to grant push access to this branch.
>It
>> >would
>> >> be superb if you had some time to work through the Python steps.
>> >>
>> >> On Wed, Nov 1, 2017 at 10:09 AM, Ahmet Altay
>> >
>> >> wrote:
>> >>
>> >>> Has anyone started a POC with Bazel? I would be interested in
>> >helping that
>> >>> effort.
>> >>>
>> >>> On Wed, Nov 1, 2017 at 9:27 AM, Lukasz Cwik
>> >
>> >>> wrote:
>> >>>
>> >>> > I have started a POC for using Gradle here:
>> >>> > https://github.com/lukecwik/incubator-beam/tree/gradle
>> >>> >
>> >>> > Things that work:
>> >>> > * compiling all Java code (src/main and src/test)
>> >>> > * generating source from protos
>> >>> > * generating source from avro
>> >>> > * running rat, checkstyle
>> >>> >
>> >>> > Partially working:
>> >>> > * generating maven pom (albeit with wrong dependencies for some
>> >>> > subprojects)
>> >>> > * running tests (~80% pass, remainder seem to be dependency
>> >related but
>> >>> are
>> >>> > uninvestigated)
>> >>> >
>> >>> > Things that don't work:
>> >>> > * anything Python/Go/Docker compilation related
>> >>> > * many tests fail because I messed up dependencies
>> >>> > * anything shading related
>> >>> > * minor plugins like eclipse code formatter/...
>> >>> > * running @NeedsRunner/@ValidatesRunner/integration tests
>> >>> >
>> >>> > Feel free to reach out to me on Slack if you would like to try
>to
>> >tackle
>> >>> a
>> >>> > 

Re: [DISCUSS] Move away from Apache Maven as build tool

[Kenneth Knowles]

I'm confident that any choice will speed things up dramatically even beyond
a fast profile, even if the new tool runs all the extra stuff. But that is
a question that we can answer empirically anyhow. Let's see how it goes!

Incidentally, my experiments with Bazel have led me to the conclusion that
it is not the right choice for us so I'm not going to be proposing any
completed POC of that right now. I'm interested in the outcome of the
Gradle POC.

Kenn


On Fri, Nov 3, 2017 at 3:30 AM, Jean-Baptiste Onofré 
wrote:

> Hi
>
> It's what I said in a previous e-mail: I don't think that just changing
> the build tool will improve a lot the build time.
>
> We already know (and discussed while ago) that plugins like findbugs,
> checkstyle, etc are taking time.
>
> So, I think we can already have a fast profile.
>
> Regards
> JB
>
> On Nov 3, 2017, 11:16, at 11:16, Romain Manni-Bucau 
> wrote:
> >Hi guys,
> >
> >when you check the duration of each mojo of the build (almost since
> >python part of the build just breaks it locally) you see that there is
> >no real link with maven for the perf issues beam can encounter:
> >https://gist.github.com/rmannibucau/f65fdde28d5dab0fdac50633f84554c9
> >(generated from the profiling of tesla-profile and parsed with
> >https://gist.github.com/rmannibucau/e329d54b8af6c009f46fd151d10037ad)
> >
> >Before PoC-ing other tools which will end up to either have the same
> >issues if the other builds do the same things (test, checkstyle,
> >enforcer, findbugs, ...) or have a less reliable build (trying to skip
> >some parts of the build if "untouched" - note that this is a very hard
> >issue since static code anaylizis doesn't give you any guarantee of
> >what it does with modern code - then maybe some action can be taken on
> >the current build:
> >
> >- testing https://github.com/vackosar/gitflow-incremental-builder or
> >https://github.com/khmarbaise/incremental-module-builder maybe or do
> >the same kind of extension including the beam needs (/!\ the previous
> >warning is still accurate and requires a full run at some point to
> >validate the graph detection algorithm didn't get abused by some
> >indirect code dependency)
> >- maybe try to get rid of some shades (it is a bit crazy ATM to have
> >so much shades no?)
> >- the CI can have profiles based on a PR convention (name of the
> >branch?) to select the build profile, for instance
> >fb/elasticsearch_super-nice-PR would build only the elasticsearch
> >modules, jenkins/travis have this ability since they support scripting
> >- document how to setup a "fastBuild" profile in its settings.xml
> >which bypasses checkstyle, enforcer plugin, findbugs, etc... for fast
> >development iterations
> >
> >
> >
> >
> >Romain Manni-Bucau
> >@rmannibucau |  Blog | Old Blog | Github | LinkedIn
> >
> >
> >2017-11-01 21:02 GMT+01:00 Kenneth Knowles :
> >> I have started one, here:
> >https://github.com/kennknowles/beam/commits/bazel.
> >> It is not nearly as far along as Luke's. For the POC I am just
> >putting
> >> things in one root BUILD, and learning where we might find the
> >necessary
> >> plugins as I go. I am happy to grant push access to this branch. It
> >would
> >> be superb if you had some time to work through the Python steps.
> >>
> >> On Wed, Nov 1, 2017 at 10:09 AM, Ahmet Altay
> >
> >> wrote:
> >>
> >>> Has anyone started a POC with Bazel? I would be interested in
> >helping that
> >>> effort.
> >>>
> >>> On Wed, Nov 1, 2017 at 9:27 AM, Lukasz Cwik
> >
> >>> wrote:
> >>>
> >>> > I have started a POC for using Gradle here:
> >>> > https://github.com/lukecwik/incubator-beam/tree/gradle
> >>> >
> >>> > Things that work:
> >>> > * compiling all Java code (src/main and src/test)
> >>> > * generating source from protos
> >>> > * generating source from avro
> >>> > * running rat, checkstyle
> >>> >
> >>> > Partially working:
> >>> > * generating maven pom (albeit with wrong dependencies for some
> >>> > subprojects)
> >>> > * running tests (~80% pass, remainder seem to be dependency
> >related but
> >>> are
> >>> > uninvestigated)
> >>> >
> >>> > Things that don't work:
> >>> > * anything Python/Go/Docker compilation related
> >>> > * many tests fail because I messed up dependencies
> >>> > * anything shading related
> >>> > * minor plugins like eclipse code formatter/...
> >>> > * running @NeedsRunner/@ValidatesRunner/integration tests
> >>> >
> >>> > Feel free to reach out to me on Slack if you would like to try to
> >tackle
> >>> a
> >>> > piece of the POC to prevent duplication of effort from anyone
> >working on
> >>> > it.
> >>> >
> >>> >
> >>> >
> >>> > On Tue, Oct 31, 2017 at 10:25 PM, Jean-Baptiste Onofré
> >
> >>> > wrote:
> >>> >
> >>> > > Agree to move forward on a PoC.
> >>> > >
> >>> > > Thanks Reuven for bringing discussion on the mailing list !
> >>> > >
> >>> > > Regards
> >>> > > 

Re: [DISCUSS] Move away from Apache Maven as build tool

[Jean-Baptiste Onofré]

Hi

It's what I said in a previous e-mail: I don't think that just changing the 
build tool will improve a lot the build time.

We already know (and discussed while ago) that plugins like findbugs, 
checkstyle, etc are taking time.

So, I think we can already have a fast profile.

Regards
JB

On Nov 3, 2017, 11:16, at 11:16, Romain Manni-Bucau  
wrote:
>Hi guys,
>
>when you check the duration of each mojo of the build (almost since
>python part of the build just breaks it locally) you see that there is
>no real link with maven for the perf issues beam can encounter:
>https://gist.github.com/rmannibucau/f65fdde28d5dab0fdac50633f84554c9
>(generated from the profiling of tesla-profile and parsed with
>https://gist.github.com/rmannibucau/e329d54b8af6c009f46fd151d10037ad)
>
>Before PoC-ing other tools which will end up to either have the same
>issues if the other builds do the same things (test, checkstyle,
>enforcer, findbugs, ...) or have a less reliable build (trying to skip
>some parts of the build if "untouched" - note that this is a very hard
>issue since static code anaylizis doesn't give you any guarantee of
>what it does with modern code - then maybe some action can be taken on
>the current build:
>
>- testing https://github.com/vackosar/gitflow-incremental-builder or
>https://github.com/khmarbaise/incremental-module-builder maybe or do
>the same kind of extension including the beam needs (/!\ the previous
>warning is still accurate and requires a full run at some point to
>validate the graph detection algorithm didn't get abused by some
>indirect code dependency)
>- maybe try to get rid of some shades (it is a bit crazy ATM to have
>so much shades no?)
>- the CI can have profiles based on a PR convention (name of the
>branch?) to select the build profile, for instance
>fb/elasticsearch_super-nice-PR would build only the elasticsearch
>modules, jenkins/travis have this ability since they support scripting
>- document how to setup a "fastBuild" profile in its settings.xml
>which bypasses checkstyle, enforcer plugin, findbugs, etc... for fast
>development iterations
>
>
>
>
>Romain Manni-Bucau
>@rmannibucau |  Blog | Old Blog | Github | LinkedIn
>
>
>2017-11-01 21:02 GMT+01:00 Kenneth Knowles :
>> I have started one, here:
>https://github.com/kennknowles/beam/commits/bazel.
>> It is not nearly as far along as Luke's. For the POC I am just
>putting
>> things in one root BUILD, and learning where we might find the
>necessary
>> plugins as I go. I am happy to grant push access to this branch. It
>would
>> be superb if you had some time to work through the Python steps.
>>
>> On Wed, Nov 1, 2017 at 10:09 AM, Ahmet Altay
>
>> wrote:
>>
>>> Has anyone started a POC with Bazel? I would be interested in
>helping that
>>> effort.
>>>
>>> On Wed, Nov 1, 2017 at 9:27 AM, Lukasz Cwik
>
>>> wrote:
>>>
>>> > I have started a POC for using Gradle here:
>>> > https://github.com/lukecwik/incubator-beam/tree/gradle
>>> >
>>> > Things that work:
>>> > * compiling all Java code (src/main and src/test)
>>> > * generating source from protos
>>> > * generating source from avro
>>> > * running rat, checkstyle
>>> >
>>> > Partially working:
>>> > * generating maven pom (albeit with wrong dependencies for some
>>> > subprojects)
>>> > * running tests (~80% pass, remainder seem to be dependency
>related but
>>> are
>>> > uninvestigated)
>>> >
>>> > Things that don't work:
>>> > * anything Python/Go/Docker compilation related
>>> > * many tests fail because I messed up dependencies
>>> > * anything shading related
>>> > * minor plugins like eclipse code formatter/...
>>> > * running @NeedsRunner/@ValidatesRunner/integration tests
>>> >
>>> > Feel free to reach out to me on Slack if you would like to try to
>tackle
>>> a
>>> > piece of the POC to prevent duplication of effort from anyone
>working on
>>> > it.
>>> >
>>> >
>>> >
>>> > On Tue, Oct 31, 2017 at 10:25 PM, Jean-Baptiste Onofré
>
>>> > wrote:
>>> >
>>> > > Agree to move forward on a PoC.
>>> > >
>>> > > Thanks Reuven for bringing discussion on the mailing list !
>>> > >
>>> > > Regards
>>> > > JB
>>> > >
>>> > > On Nov 1, 2017, 03:20, at 03:20, Reuven Lax
>
>>> > > wrote:
>>> > > >Some good discussion here, and thanks to JB and Romain for
>adding to
>>> > > >it!
>>> > > >
>>> > > >JB makes the good point that we still need to release Maven
>artifacts,
>>> > > >as
>>> > > >many Beam users want to develop using Maven. So none of this
>>> discussion
>>> > > >will affect our release process, as we still need Maven
>"releases."
>>> > > >
>>> > > >At this point, if people are interested, I see no harm in
>prototyping.
>>> > > >Having working alternatives will give us a better basis for
>comparison
>>> > > >to
>>> > > >understand whether these other build systems give us anything
>over
>>> what
>>> > > >Maven does.
>>> > > >
>>> > > 

Re: [DISCUSS] Move away from Apache Maven as build tool

[Romain Manni-Bucau]

Hi guys,

when you check the duration of each mojo of the build (almost since
python part of the build just breaks it locally) you see that there is
no real link with maven for the perf issues beam can encounter:
https://gist.github.com/rmannibucau/f65fdde28d5dab0fdac50633f84554c9
(generated from the profiling of tesla-profile and parsed with
https://gist.github.com/rmannibucau/e329d54b8af6c009f46fd151d10037ad)

Before PoC-ing other tools which will end up to either have the same
issues if the other builds do the same things (test, checkstyle,
enforcer, findbugs, ...) or have a less reliable build (trying to skip
some parts of the build if "untouched" - note that this is a very hard
issue since static code anaylizis doesn't give you any guarantee of
what it does with modern code - then maybe some action can be taken on
the current build:

- testing https://github.com/vackosar/gitflow-incremental-builder or
https://github.com/khmarbaise/incremental-module-builder maybe or do
the same kind of extension including the beam needs (/!\ the previous
warning is still accurate and requires a full run at some point to
validate the graph detection algorithm didn't get abused by some
indirect code dependency)
- maybe try to get rid of some shades (it is a bit crazy ATM to have
so much shades no?)
- the CI can have profiles based on a PR convention (name of the
branch?) to select the build profile, for instance
fb/elasticsearch_super-nice-PR would build only the elasticsearch
modules, jenkins/travis have this ability since they support scripting
- document how to setup a "fastBuild" profile in its settings.xml
which bypasses checkstyle, enforcer plugin, findbugs, etc... for fast
development iterations




Romain Manni-Bucau
@rmannibucau |  Blog | Old Blog | Github | LinkedIn


2017-11-01 21:02 GMT+01:00 Kenneth Knowles :
> I have started one, here: https://github.com/kennknowles/beam/commits/bazel.
> It is not nearly as far along as Luke's. For the POC I am just putting
> things in one root BUILD, and learning where we might find the necessary
> plugins as I go. I am happy to grant push access to this branch. It would
> be superb if you had some time to work through the Python steps.
>
> On Wed, Nov 1, 2017 at 10:09 AM, Ahmet Altay 
> wrote:
>
>> Has anyone started a POC with Bazel? I would be interested in helping that
>> effort.
>>
>> On Wed, Nov 1, 2017 at 9:27 AM, Lukasz Cwik 
>> wrote:
>>
>> > I have started a POC for using Gradle here:
>> > https://github.com/lukecwik/incubator-beam/tree/gradle
>> >
>> > Things that work:
>> > * compiling all Java code (src/main and src/test)
>> > * generating source from protos
>> > * generating source from avro
>> > * running rat, checkstyle
>> >
>> > Partially working:
>> > * generating maven pom (albeit with wrong dependencies for some
>> > subprojects)
>> > * running tests (~80% pass, remainder seem to be dependency related but
>> are
>> > uninvestigated)
>> >
>> > Things that don't work:
>> > * anything Python/Go/Docker compilation related
>> > * many tests fail because I messed up dependencies
>> > * anything shading related
>> > * minor plugins like eclipse code formatter/...
>> > * running @NeedsRunner/@ValidatesRunner/integration tests
>> >
>> > Feel free to reach out to me on Slack if you would like to try to tackle
>> a
>> > piece of the POC to prevent duplication of effort from anyone working on
>> > it.
>> >
>> >
>> >
>> > On Tue, Oct 31, 2017 at 10:25 PM, Jean-Baptiste Onofré 
>> > wrote:
>> >
>> > > Agree to move forward on a PoC.
>> > >
>> > > Thanks Reuven for bringing discussion on the mailing list !
>> > >
>> > > Regards
>> > > JB
>> > >
>> > > On Nov 1, 2017, 03:20, at 03:20, Reuven Lax 
>> > > wrote:
>> > > >Some good discussion here, and thanks to JB and Romain for adding to
>> > > >it!
>> > > >
>> > > >JB makes the good point that we still need to release Maven artifacts,
>> > > >as
>> > > >many Beam users want to develop using Maven. So none of this
>> discussion
>> > > >will affect our release process, as we still need Maven "releases."
>> > > >
>> > > >At this point, if people are interested, I see no harm in prototyping.
>> > > >Having working alternatives will give us a better basis for comparison
>> > > >to
>> > > >understand whether these other build systems give us anything over
>> what
>> > > >Maven does.
>> > > >
>> > > >Reuven
>> > > >
>> > > >On Tue, Oct 31, 2017 at 11:05 AM, Charles Chen > >
>> > > >wrote:
>> > > >
>> > > >> As a contributor to the Beam Python SDK, I noticed that many of the
>> > > >points
>> > > >> above regarding Maven and Gradle pertain mostly to Java SDK
>> > > >development.
>> > > >> For Python development, Maven is much less natural, and we end up
>> > > >just
>> > > >> shelling out to perform builds and tests.  For Python SDK (and
>> > > >upcoming Go
>> > > >> SDK 

Re: [DISCUSS] Move away from Apache Maven as build tool

[Kenneth Knowles]

I have started one, here: https://github.com/kennknowles/beam/commits/bazel.
It is not nearly as far along as Luke's. For the POC I am just putting
things in one root BUILD, and learning where we might find the necessary
plugins as I go. I am happy to grant push access to this branch. It would
be superb if you had some time to work through the Python steps.

On Wed, Nov 1, 2017 at 10:09 AM, Ahmet Altay 
wrote:

> Has anyone started a POC with Bazel? I would be interested in helping that
> effort.
>
> On Wed, Nov 1, 2017 at 9:27 AM, Lukasz Cwik 
> wrote:
>
> > I have started a POC for using Gradle here:
> > https://github.com/lukecwik/incubator-beam/tree/gradle
> >
> > Things that work:
> > * compiling all Java code (src/main and src/test)
> > * generating source from protos
> > * generating source from avro
> > * running rat, checkstyle
> >
> > Partially working:
> > * generating maven pom (albeit with wrong dependencies for some
> > subprojects)
> > * running tests (~80% pass, remainder seem to be dependency related but
> are
> > uninvestigated)
> >
> > Things that don't work:
> > * anything Python/Go/Docker compilation related
> > * many tests fail because I messed up dependencies
> > * anything shading related
> > * minor plugins like eclipse code formatter/...
> > * running @NeedsRunner/@ValidatesRunner/integration tests
> >
> > Feel free to reach out to me on Slack if you would like to try to tackle
> a
> > piece of the POC to prevent duplication of effort from anyone working on
> > it.
> >
> >
> >
> > On Tue, Oct 31, 2017 at 10:25 PM, Jean-Baptiste Onofré 
> > wrote:
> >
> > > Agree to move forward on a PoC.
> > >
> > > Thanks Reuven for bringing discussion on the mailing list !
> > >
> > > Regards
> > > JB
> > >
> > > On Nov 1, 2017, 03:20, at 03:20, Reuven Lax 
> > > wrote:
> > > >Some good discussion here, and thanks to JB and Romain for adding to
> > > >it!
> > > >
> > > >JB makes the good point that we still need to release Maven artifacts,
> > > >as
> > > >many Beam users want to develop using Maven. So none of this
> discussion
> > > >will affect our release process, as we still need Maven "releases."
> > > >
> > > >At this point, if people are interested, I see no harm in prototyping.
> > > >Having working alternatives will give us a better basis for comparison
> > > >to
> > > >understand whether these other build systems give us anything over
> what
> > > >Maven does.
> > > >
> > > >Reuven
> > > >
> > > >On Tue, Oct 31, 2017 at 11:05 AM, Charles Chen  >
> > > >wrote:
> > > >
> > > >> As a contributor to the Beam Python SDK, I noticed that many of the
> > > >points
> > > >> above regarding Maven and Gradle pertain mostly to Java SDK
> > > >development.
> > > >> For Python development, Maven is much less natural, and we end up
> > > >just
> > > >> shelling out to perform builds and tests.  For Python SDK (and
> > > >upcoming Go
> > > >> SDK development), an option to use Bazel would be quite useful.
> > > >>
> > > >> On Tue, Oct 31, 2017 at 10:42 AM Robert Bradshaw
> > > >>  wrote:
> > > >>
> > > >> > +1, Maven is both a build tool and a repository, and the latter is
> > > >> > essential to keep. Both Gradel and Bazel can interface with this
> > > >> > repository.
> > > >> >
> > > >> > I am, however, very supportive of moving away from Maven to a tool
> > > >> > that supports correct incremental, hermetic, dependency-driven,
> > > >> > multi-langauge, and hopefully fast builds for our own development.
> > > >> >
> > > >> > On Tue, Oct 31, 2017 at 10:00 AM, Kenneth Knowles
> > > >> >  wrote:
> > > >> > > Echoing what JB and Reuven said, we absolutely must provide
> maven
> > > >> central
> > > >> > > artifacts for Java users, just as we provide pypi artifacts for
> > > >Python
> > > >> > > users.
> > > >> > >
> > > >> > > I see Maven as still a viable tool for single-module Java
> builds,
> > > >> > > especially considering its rich plugin ecosystem.
> > > >> > >
> > > >> > > On Mon, Oct 30, 2017 at 11:27 PM, Reuven Lax
> > > > > > >> >
> > > >> > > wrote:
> > > >> > >
> > > >> > >> I think that's a very good point. No matter what build system
> we
> > > >use
> > > >> for
> > > >> > >> our own personal development, we still need to release Maven
> > > >artifacts
> > > >> > and
> > > >> > >> releases as we need to support our users using Maven.
> > > >> > >>
> > > >> > >> On Mon, Oct 30, 2017 at 11:26 PM, Jean-Baptiste Onofré <
> > > >> j...@nanthrax.net
> > > >> > >
> > > >> > >> wrote:
> > > >> > >>
> > > >> > >> > Generally speaking, it's interesting to evaluate
> alternatives,
> > > >> > especially
> > > >> > >> > Gradle. My point is also to keep Maven artifacts and
> > > >"releases" as
> > > >> > most
> > > >> > >> of
> > > >> > >> > our users will use Maven.
> > > >> > >> > For 

Re: [DISCUSS] Move away from Apache Maven as build tool

[Ahmet Altay]

Has anyone started a POC with Bazel? I would be interested in helping that
effort.

On Wed, Nov 1, 2017 at 9:27 AM, Lukasz Cwik 
wrote:

> I have started a POC for using Gradle here:
> https://github.com/lukecwik/incubator-beam/tree/gradle
>
> Things that work:
> * compiling all Java code (src/main and src/test)
> * generating source from protos
> * generating source from avro
> * running rat, checkstyle
>
> Partially working:
> * generating maven pom (albeit with wrong dependencies for some
> subprojects)
> * running tests (~80% pass, remainder seem to be dependency related but are
> uninvestigated)
>
> Things that don't work:
> * anything Python/Go/Docker compilation related
> * many tests fail because I messed up dependencies
> * anything shading related
> * minor plugins like eclipse code formatter/...
> * running @NeedsRunner/@ValidatesRunner/integration tests
>
> Feel free to reach out to me on Slack if you would like to try to tackle a
> piece of the POC to prevent duplication of effort from anyone working on
> it.
>
>
>
> On Tue, Oct 31, 2017 at 10:25 PM, Jean-Baptiste Onofré 
> wrote:
>
> > Agree to move forward on a PoC.
> >
> > Thanks Reuven for bringing discussion on the mailing list !
> >
> > Regards
> > JB
> >
> > On Nov 1, 2017, 03:20, at 03:20, Reuven Lax 
> > wrote:
> > >Some good discussion here, and thanks to JB and Romain for adding to
> > >it!
> > >
> > >JB makes the good point that we still need to release Maven artifacts,
> > >as
> > >many Beam users want to develop using Maven. So none of this discussion
> > >will affect our release process, as we still need Maven "releases."
> > >
> > >At this point, if people are interested, I see no harm in prototyping.
> > >Having working alternatives will give us a better basis for comparison
> > >to
> > >understand whether these other build systems give us anything over what
> > >Maven does.
> > >
> > >Reuven
> > >
> > >On Tue, Oct 31, 2017 at 11:05 AM, Charles Chen 
> > >wrote:
> > >
> > >> As a contributor to the Beam Python SDK, I noticed that many of the
> > >points
> > >> above regarding Maven and Gradle pertain mostly to Java SDK
> > >development.
> > >> For Python development, Maven is much less natural, and we end up
> > >just
> > >> shelling out to perform builds and tests.  For Python SDK (and
> > >upcoming Go
> > >> SDK development), an option to use Bazel would be quite useful.
> > >>
> > >> On Tue, Oct 31, 2017 at 10:42 AM Robert Bradshaw
> > >>  wrote:
> > >>
> > >> > +1, Maven is both a build tool and a repository, and the latter is
> > >> > essential to keep. Both Gradel and Bazel can interface with this
> > >> > repository.
> > >> >
> > >> > I am, however, very supportive of moving away from Maven to a tool
> > >> > that supports correct incremental, hermetic, dependency-driven,
> > >> > multi-langauge, and hopefully fast builds for our own development.
> > >> >
> > >> > On Tue, Oct 31, 2017 at 10:00 AM, Kenneth Knowles
> > >> >  wrote:
> > >> > > Echoing what JB and Reuven said, we absolutely must provide maven
> > >> central
> > >> > > artifacts for Java users, just as we provide pypi artifacts for
> > >Python
> > >> > > users.
> > >> > >
> > >> > > I see Maven as still a viable tool for single-module Java builds,
> > >> > > especially considering its rich plugin ecosystem.
> > >> > >
> > >> > > On Mon, Oct 30, 2017 at 11:27 PM, Reuven Lax
> > > > >> >
> > >> > > wrote:
> > >> > >
> > >> > >> I think that's a very good point. No matter what build system we
> > >use
> > >> for
> > >> > >> our own personal development, we still need to release Maven
> > >artifacts
> > >> > and
> > >> > >> releases as we need to support our users using Maven.
> > >> > >>
> > >> > >> On Mon, Oct 30, 2017 at 11:26 PM, Jean-Baptiste Onofré <
> > >> j...@nanthrax.net
> > >> > >
> > >> > >> wrote:
> > >> > >>
> > >> > >> > Generally speaking, it's interesting to evaluate alternatives,
> > >> > especially
> > >> > >> > Gradle. My point is also to keep Maven artifacts and
> > >"releases" as
> > >> > most
> > >> > >> of
> > >> > >> > our users will use Maven.
> > >> > >> > For incremental build, afair, there's some enhancements on
> > >Maven
> > >> but I
> > >> > >> > have to take a look.
> > >> > >> >
> > >> > >> > Regards
> > >> > >> > JB
> > >> > >> >
> > >> > >> > On Oct 31, 2017, 07:22, at 07:22, Eugene Kirpichov
> > >> > >> >  wrote:
> > >> > >> > >Hi!
> > >> > >> > >
> > >> > >> > >Many of these points sound valid, but AFAICT Maven doesn't
> > >really
> > >> do
> > >> > >> > >incremental builds [1]. The best it can do is, it seems,
> > >recompile
> > >> > only
> > >> > >> > >changed files, but Java compilation is a tiny part of the
> > >overall
> > >> > >> > >build.
> > >> > >> > >
> > >> > >> > >Almost all time is taken by other plugins, 

Re: [DISCUSS] Move away from Apache Maven as build tool

[Lukasz Cwik]

I have started a POC for using Gradle here:
https://github.com/lukecwik/incubator-beam/tree/gradle

Things that work:
* compiling all Java code (src/main and src/test)
* generating source from protos
* generating source from avro
* running rat, checkstyle

Partially working:
* generating maven pom (albeit with wrong dependencies for some subprojects)
* running tests (~80% pass, remainder seem to be dependency related but are
uninvestigated)

Things that don't work:
* anything Python/Go/Docker compilation related
* many tests fail because I messed up dependencies
* anything shading related
* minor plugins like eclipse code formatter/...
* running @NeedsRunner/@ValidatesRunner/integration tests

Feel free to reach out to me on Slack if you would like to try to tackle a
piece of the POC to prevent duplication of effort from anyone working on it.



On Tue, Oct 31, 2017 at 10:25 PM, Jean-Baptiste Onofré 
wrote:

> Agree to move forward on a PoC.
>
> Thanks Reuven for bringing discussion on the mailing list !
>
> Regards
> JB
>
> On Nov 1, 2017, 03:20, at 03:20, Reuven Lax 
> wrote:
> >Some good discussion here, and thanks to JB and Romain for adding to
> >it!
> >
> >JB makes the good point that we still need to release Maven artifacts,
> >as
> >many Beam users want to develop using Maven. So none of this discussion
> >will affect our release process, as we still need Maven "releases."
> >
> >At this point, if people are interested, I see no harm in prototyping.
> >Having working alternatives will give us a better basis for comparison
> >to
> >understand whether these other build systems give us anything over what
> >Maven does.
> >
> >Reuven
> >
> >On Tue, Oct 31, 2017 at 11:05 AM, Charles Chen 
> >wrote:
> >
> >> As a contributor to the Beam Python SDK, I noticed that many of the
> >points
> >> above regarding Maven and Gradle pertain mostly to Java SDK
> >development.
> >> For Python development, Maven is much less natural, and we end up
> >just
> >> shelling out to perform builds and tests.  For Python SDK (and
> >upcoming Go
> >> SDK development), an option to use Bazel would be quite useful.
> >>
> >> On Tue, Oct 31, 2017 at 10:42 AM Robert Bradshaw
> >>  wrote:
> >>
> >> > +1, Maven is both a build tool and a repository, and the latter is
> >> > essential to keep. Both Gradel and Bazel can interface with this
> >> > repository.
> >> >
> >> > I am, however, very supportive of moving away from Maven to a tool
> >> > that supports correct incremental, hermetic, dependency-driven,
> >> > multi-langauge, and hopefully fast builds for our own development.
> >> >
> >> > On Tue, Oct 31, 2017 at 10:00 AM, Kenneth Knowles
> >> >  wrote:
> >> > > Echoing what JB and Reuven said, we absolutely must provide maven
> >> central
> >> > > artifacts for Java users, just as we provide pypi artifacts for
> >Python
> >> > > users.
> >> > >
> >> > > I see Maven as still a viable tool for single-module Java builds,
> >> > > especially considering its rich plugin ecosystem.
> >> > >
> >> > > On Mon, Oct 30, 2017 at 11:27 PM, Reuven Lax
> > >> >
> >> > > wrote:
> >> > >
> >> > >> I think that's a very good point. No matter what build system we
> >use
> >> for
> >> > >> our own personal development, we still need to release Maven
> >artifacts
> >> > and
> >> > >> releases as we need to support our users using Maven.
> >> > >>
> >> > >> On Mon, Oct 30, 2017 at 11:26 PM, Jean-Baptiste Onofré <
> >> j...@nanthrax.net
> >> > >
> >> > >> wrote:
> >> > >>
> >> > >> > Generally speaking, it's interesting to evaluate alternatives,
> >> > especially
> >> > >> > Gradle. My point is also to keep Maven artifacts and
> >"releases" as
> >> > most
> >> > >> of
> >> > >> > our users will use Maven.
> >> > >> > For incremental build, afair, there's some enhancements on
> >Maven
> >> but I
> >> > >> > have to take a look.
> >> > >> >
> >> > >> > Regards
> >> > >> > JB
> >> > >> >
> >> > >> > On Oct 31, 2017, 07:22, at 07:22, Eugene Kirpichov
> >> > >> >  wrote:
> >> > >> > >Hi!
> >> > >> > >
> >> > >> > >Many of these points sound valid, but AFAICT Maven doesn't
> >really
> >> do
> >> > >> > >incremental builds [1]. The best it can do is, it seems,
> >recompile
> >> > only
> >> > >> > >changed files, but Java compilation is a tiny part of the
> >overall
> >> > >> > >build.
> >> > >> > >
> >> > >> > >Almost all time is taken by other plugins, such as unit
> >testing or
> >> > >> > >findbugs
> >> > >> > >- and Maven does not seem to currently support features such
> >as "do
> >> > not
> >> > >> > >rerun unit tests of a module if the code didn't change".
> >> > >> > >
> >> > >> > >The fact that the surefire plugin has existed for >11 years
> >> (version
> >> > >> > >2.0
> >> > >> > >was released in 2006) and still doesn't have this feature
> >makes me
> >> > >> > >think
> 

Re: [DISCUSS] Move away from Apache Maven as build tool

[Jean-Baptiste Onofré]

Agree to move forward on a PoC.

Thanks Reuven for bringing discussion on the mailing list !

Regards
JB

On Nov 1, 2017, 03:20, at 03:20, Reuven Lax  wrote:
>Some good discussion here, and thanks to JB and Romain for adding to
>it!
>
>JB makes the good point that we still need to release Maven artifacts,
>as
>many Beam users want to develop using Maven. So none of this discussion
>will affect our release process, as we still need Maven "releases."
>
>At this point, if people are interested, I see no harm in prototyping.
>Having working alternatives will give us a better basis for comparison
>to
>understand whether these other build systems give us anything over what
>Maven does.
>
>Reuven
>
>On Tue, Oct 31, 2017 at 11:05 AM, Charles Chen 
>wrote:
>
>> As a contributor to the Beam Python SDK, I noticed that many of the
>points
>> above regarding Maven and Gradle pertain mostly to Java SDK
>development.
>> For Python development, Maven is much less natural, and we end up
>just
>> shelling out to perform builds and tests.  For Python SDK (and
>upcoming Go
>> SDK development), an option to use Bazel would be quite useful.
>>
>> On Tue, Oct 31, 2017 at 10:42 AM Robert Bradshaw
>>  wrote:
>>
>> > +1, Maven is both a build tool and a repository, and the latter is
>> > essential to keep. Both Gradel and Bazel can interface with this
>> > repository.
>> >
>> > I am, however, very supportive of moving away from Maven to a tool
>> > that supports correct incremental, hermetic, dependency-driven,
>> > multi-langauge, and hopefully fast builds for our own development.
>> >
>> > On Tue, Oct 31, 2017 at 10:00 AM, Kenneth Knowles
>> >  wrote:
>> > > Echoing what JB and Reuven said, we absolutely must provide maven
>> central
>> > > artifacts for Java users, just as we provide pypi artifacts for
>Python
>> > > users.
>> > >
>> > > I see Maven as still a viable tool for single-module Java builds,
>> > > especially considering its rich plugin ecosystem.
>> > >
>> > > On Mon, Oct 30, 2017 at 11:27 PM, Reuven Lax
>> >
>> > > wrote:
>> > >
>> > >> I think that's a very good point. No matter what build system we
>use
>> for
>> > >> our own personal development, we still need to release Maven
>artifacts
>> > and
>> > >> releases as we need to support our users using Maven.
>> > >>
>> > >> On Mon, Oct 30, 2017 at 11:26 PM, Jean-Baptiste Onofré <
>> j...@nanthrax.net
>> > >
>> > >> wrote:
>> > >>
>> > >> > Generally speaking, it's interesting to evaluate alternatives,
>> > especially
>> > >> > Gradle. My point is also to keep Maven artifacts and
>"releases" as
>> > most
>> > >> of
>> > >> > our users will use Maven.
>> > >> > For incremental build, afair, there's some enhancements on
>Maven
>> but I
>> > >> > have to take a look.
>> > >> >
>> > >> > Regards
>> > >> > JB
>> > >> >
>> > >> > On Oct 31, 2017, 07:22, at 07:22, Eugene Kirpichov
>> > >> >  wrote:
>> > >> > >Hi!
>> > >> > >
>> > >> > >Many of these points sound valid, but AFAICT Maven doesn't
>really
>> do
>> > >> > >incremental builds [1]. The best it can do is, it seems,
>recompile
>> > only
>> > >> > >changed files, but Java compilation is a tiny part of the
>overall
>> > >> > >build.
>> > >> > >
>> > >> > >Almost all time is taken by other plugins, such as unit
>testing or
>> > >> > >findbugs
>> > >> > >- and Maven does not seem to currently support features such
>as "do
>> > not
>> > >> > >rerun unit tests of a module if the code didn't change".
>> > >> > >
>> > >> > >The fact that the surefire plugin has existed for >11 years
>> (version
>> > >> > >2.0
>> > >> > >was released in 2006) and still doesn't have this feature
>makes me
>> > >> > >think
>> > >> > >that it's unlikely to be supported in the next few years
>either.
>> > >> > >
>> > >> > >I suspect most PRs affect a very small number of modules, so
>I
>> think
>> > >> > >the
>> > >> > >performance advantage of a build system truly supporting
>> incremental
>> > >> > >builds
>> > >> > >may be so overwhelming as to trump many other factors. Of
>course,
>> > we'd
>> > >> > >need
>> > >> > >to prototype and have hard numbers in hand to discuss this
>with
>> more
>> > >> > >substance.
>> > >> > >
>> > >> > >[1]
>> > >> > >https://stackoverflow.com/questions/8918165/does-maven-
>> > >> > support-incremental-builds
>> > >> > >
>> > >> > >On Mon, Oct 30, 2017 at 10:57 PM Romain Manni-Bucau
>> > >> > >
>> > >> > >wrote:
>> > >> > >
>> > >> > >> Hi
>> > >> > >>
>> > >> > >> Even if not a commiter or even PMC, I'd like to mention a
>few
>> > points
>> > >> > >from
>> > >> > >> an external eye:
>> > >> > >>
>> > >> > >> - Maven stays the most common build tool and easier one for
>any
>> > user.
>> > >> > >It
>> > >> > >> means it is the best one to hope contributions IMHO.
>> > >> > >> - Maven has incremental support but if there is any 

Re: [DISCUSS] Move away from Apache Maven as build tool

[Reuven Lax]

Some good discussion here, and thanks to JB and Romain for adding to it!

JB makes the good point that we still need to release Maven artifacts, as
many Beam users want to develop using Maven. So none of this discussion
will affect our release process, as we still need Maven "releases."

At this point, if people are interested, I see no harm in prototyping.
Having working alternatives will give us a better basis for comparison to
understand whether these other build systems give us anything over what
Maven does.

Reuven

On Tue, Oct 31, 2017 at 11:05 AM, Charles Chen 
wrote:

> As a contributor to the Beam Python SDK, I noticed that many of the points
> above regarding Maven and Gradle pertain mostly to Java SDK development.
> For Python development, Maven is much less natural, and we end up just
> shelling out to perform builds and tests.  For Python SDK (and upcoming Go
> SDK development), an option to use Bazel would be quite useful.
>
> On Tue, Oct 31, 2017 at 10:42 AM Robert Bradshaw
>  wrote:
>
> > +1, Maven is both a build tool and a repository, and the latter is
> > essential to keep. Both Gradel and Bazel can interface with this
> > repository.
> >
> > I am, however, very supportive of moving away from Maven to a tool
> > that supports correct incremental, hermetic, dependency-driven,
> > multi-langauge, and hopefully fast builds for our own development.
> >
> > On Tue, Oct 31, 2017 at 10:00 AM, Kenneth Knowles
> >  wrote:
> > > Echoing what JB and Reuven said, we absolutely must provide maven
> central
> > > artifacts for Java users, just as we provide pypi artifacts for Python
> > > users.
> > >
> > > I see Maven as still a viable tool for single-module Java builds,
> > > especially considering its rich plugin ecosystem.
> > >
> > > On Mon, Oct 30, 2017 at 11:27 PM, Reuven Lax  >
> > > wrote:
> > >
> > >> I think that's a very good point. No matter what build system we use
> for
> > >> our own personal development, we still need to release Maven artifacts
> > and
> > >> releases as we need to support our users using Maven.
> > >>
> > >> On Mon, Oct 30, 2017 at 11:26 PM, Jean-Baptiste Onofré <
> j...@nanthrax.net
> > >
> > >> wrote:
> > >>
> > >> > Generally speaking, it's interesting to evaluate alternatives,
> > especially
> > >> > Gradle. My point is also to keep Maven artifacts and "releases" as
> > most
> > >> of
> > >> > our users will use Maven.
> > >> > For incremental build, afair, there's some enhancements on Maven
> but I
> > >> > have to take a look.
> > >> >
> > >> > Regards
> > >> > JB
> > >> >
> > >> > On Oct 31, 2017, 07:22, at 07:22, Eugene Kirpichov
> > >> >  wrote:
> > >> > >Hi!
> > >> > >
> > >> > >Many of these points sound valid, but AFAICT Maven doesn't really
> do
> > >> > >incremental builds [1]. The best it can do is, it seems, recompile
> > only
> > >> > >changed files, but Java compilation is a tiny part of the overall
> > >> > >build.
> > >> > >
> > >> > >Almost all time is taken by other plugins, such as unit testing or
> > >> > >findbugs
> > >> > >- and Maven does not seem to currently support features such as "do
> > not
> > >> > >rerun unit tests of a module if the code didn't change".
> > >> > >
> > >> > >The fact that the surefire plugin has existed for >11 years
> (version
> > >> > >2.0
> > >> > >was released in 2006) and still doesn't have this feature makes me
> > >> > >think
> > >> > >that it's unlikely to be supported in the next few years either.
> > >> > >
> > >> > >I suspect most PRs affect a very small number of modules, so I
> think
> > >> > >the
> > >> > >performance advantage of a build system truly supporting
> incremental
> > >> > >builds
> > >> > >may be so overwhelming as to trump many other factors. Of course,
> > we'd
> > >> > >need
> > >> > >to prototype and have hard numbers in hand to discuss this with
> more
> > >> > >substance.
> > >> > >
> > >> > >[1]
> > >> > >https://stackoverflow.com/questions/8918165/does-maven-
> > >> > support-incremental-builds
> > >> > >
> > >> > >On Mon, Oct 30, 2017 at 10:57 PM Romain Manni-Bucau
> > >> > >
> > >> > >wrote:
> > >> > >
> > >> > >> Hi
> > >> > >>
> > >> > >> Even if not a commiter or even PMC, I'd like to mention a few
> > points
> > >> > >from
> > >> > >> an external eye:
> > >> > >>
> > >> > >> - Maven stays the most common build tool and easier one for any
> > user.
> > >> > >It
> > >> > >> means it is the best one to hope contributions IMHO.
> > >> > >> - Maven has incremental support but if there is any blocker the
> > >> > >community
> > >> > >> is probably ready to enhance it (has been done for compiler
> plugin
> > >> > >for
> > >> > >> instance)
> > >> > >> - Gradle hides issues easily with its daemon so a build without
> > >> > >daemon is
> > >> > >> needed
> > >> > >> - Gradle doesnt isolate plugins well enough so ensure your
> 

Re: [DISCUSS] Move away from Apache Maven as build tool

[Charles Chen]

As a contributor to the Beam Python SDK, I noticed that many of the points
above regarding Maven and Gradle pertain mostly to Java SDK development.
For Python development, Maven is much less natural, and we end up just
shelling out to perform builds and tests.  For Python SDK (and upcoming Go
SDK development), an option to use Bazel would be quite useful.

On Tue, Oct 31, 2017 at 10:42 AM Robert Bradshaw
 wrote:

> +1, Maven is both a build tool and a repository, and the latter is
> essential to keep. Both Gradel and Bazel can interface with this
> repository.
>
> I am, however, very supportive of moving away from Maven to a tool
> that supports correct incremental, hermetic, dependency-driven,
> multi-langauge, and hopefully fast builds for our own development.
>
> On Tue, Oct 31, 2017 at 10:00 AM, Kenneth Knowles
>  wrote:
> > Echoing what JB and Reuven said, we absolutely must provide maven central
> > artifacts for Java users, just as we provide pypi artifacts for Python
> > users.
> >
> > I see Maven as still a viable tool for single-module Java builds,
> > especially considering its rich plugin ecosystem.
> >
> > On Mon, Oct 30, 2017 at 11:27 PM, Reuven Lax 
> > wrote:
> >
> >> I think that's a very good point. No matter what build system we use for
> >> our own personal development, we still need to release Maven artifacts
> and
> >> releases as we need to support our users using Maven.
> >>
> >> On Mon, Oct 30, 2017 at 11:26 PM, Jean-Baptiste Onofré  >
> >> wrote:
> >>
> >> > Generally speaking, it's interesting to evaluate alternatives,
> especially
> >> > Gradle. My point is also to keep Maven artifacts and "releases" as
> most
> >> of
> >> > our users will use Maven.
> >> > For incremental build, afair, there's some enhancements on Maven but I
> >> > have to take a look.
> >> >
> >> > Regards
> >> > JB
> >> >
> >> > On Oct 31, 2017, 07:22, at 07:22, Eugene Kirpichov
> >> >  wrote:
> >> > >Hi!
> >> > >
> >> > >Many of these points sound valid, but AFAICT Maven doesn't really do
> >> > >incremental builds [1]. The best it can do is, it seems, recompile
> only
> >> > >changed files, but Java compilation is a tiny part of the overall
> >> > >build.
> >> > >
> >> > >Almost all time is taken by other plugins, such as unit testing or
> >> > >findbugs
> >> > >- and Maven does not seem to currently support features such as "do
> not
> >> > >rerun unit tests of a module if the code didn't change".
> >> > >
> >> > >The fact that the surefire plugin has existed for >11 years (version
> >> > >2.0
> >> > >was released in 2006) and still doesn't have this feature makes me
> >> > >think
> >> > >that it's unlikely to be supported in the next few years either.
> >> > >
> >> > >I suspect most PRs affect a very small number of modules, so I think
> >> > >the
> >> > >performance advantage of a build system truly supporting incremental
> >> > >builds
> >> > >may be so overwhelming as to trump many other factors. Of course,
> we'd
> >> > >need
> >> > >to prototype and have hard numbers in hand to discuss this with more
> >> > >substance.
> >> > >
> >> > >[1]
> >> > >https://stackoverflow.com/questions/8918165/does-maven-
> >> > support-incremental-builds
> >> > >
> >> > >On Mon, Oct 30, 2017 at 10:57 PM Romain Manni-Bucau
> >> > >
> >> > >wrote:
> >> > >
> >> > >> Hi
> >> > >>
> >> > >> Even if not a commiter or even PMC, I'd like to mention a few
> points
> >> > >from
> >> > >> an external eye:
> >> > >>
> >> > >> - Maven stays the most common build tool and easier one for any
> user.
> >> > >It
> >> > >> means it is the best one to hope contributions IMHO.
> >> > >> - Maven has incremental support but if there is any blocker the
> >> > >community
> >> > >> is probably ready to enhance it (has been done for compiler plugin
> >> > >for
> >> > >> instance)
> >> > >> - Gradle hides issues easily with its daemon so a build without
> >> > >daemon is
> >> > >> needed
> >> > >> - Gradle doesnt isolate plugins well enough so ensure your planned
> >> > >plugins
> >> > >> doesnt conflict
> >> > >> - Only Maven is correctly supported in mainstream and OS/free IDE
> >> > >>
> >> > >> This is the reasons why I think Maven is better - not even entering
> >> > >into
> >> > >> the ASF points.
> >> > >>
> >> > >> Now Maven is not perfect but some quick enhancements can be done:
> >> > >>
> >> > >> - A fast build profile can be created
> >> > >> - Takari scheduler can be used yo enhance the parallel build
> >> > >> - Scripts can be provided to build a subpart of the project
> >> > >> - A beam extension can surely be done to optimize or compute the
> >> > >reactors
> >> > >> more easily based on module names
> >> > >>
> >> > >> Romain
> >> > >>
> >> > >> Le 31 oct. 2017 06:42, "Jean-Baptiste Onofré"  a
> >> > >écrit :
> >> > >>
> >> > >> -0
> >> > >>
> >> > >> For 

Re: [DISCUSS] Move away from Apache Maven as build tool

[Robert Bradshaw]

+1, Maven is both a build tool and a repository, and the latter is
essential to keep. Both Gradel and Bazel can interface with this
repository.

I am, however, very supportive of moving away from Maven to a tool
that supports correct incremental, hermetic, dependency-driven,
multi-langauge, and hopefully fast builds for our own development.

On Tue, Oct 31, 2017 at 10:00 AM, Kenneth Knowles
 wrote:
> Echoing what JB and Reuven said, we absolutely must provide maven central
> artifacts for Java users, just as we provide pypi artifacts for Python
> users.
>
> I see Maven as still a viable tool for single-module Java builds,
> especially considering its rich plugin ecosystem.
>
> On Mon, Oct 30, 2017 at 11:27 PM, Reuven Lax 
> wrote:
>
>> I think that's a very good point. No matter what build system we use for
>> our own personal development, we still need to release Maven artifacts and
>> releases as we need to support our users using Maven.
>>
>> On Mon, Oct 30, 2017 at 11:26 PM, Jean-Baptiste Onofré 
>> wrote:
>>
>> > Generally speaking, it's interesting to evaluate alternatives, especially
>> > Gradle. My point is also to keep Maven artifacts and "releases" as most
>> of
>> > our users will use Maven.
>> > For incremental build, afair, there's some enhancements on Maven but I
>> > have to take a look.
>> >
>> > Regards
>> > JB
>> >
>> > On Oct 31, 2017, 07:22, at 07:22, Eugene Kirpichov
>> >  wrote:
>> > >Hi!
>> > >
>> > >Many of these points sound valid, but AFAICT Maven doesn't really do
>> > >incremental builds [1]. The best it can do is, it seems, recompile only
>> > >changed files, but Java compilation is a tiny part of the overall
>> > >build.
>> > >
>> > >Almost all time is taken by other plugins, such as unit testing or
>> > >findbugs
>> > >- and Maven does not seem to currently support features such as "do not
>> > >rerun unit tests of a module if the code didn't change".
>> > >
>> > >The fact that the surefire plugin has existed for >11 years (version
>> > >2.0
>> > >was released in 2006) and still doesn't have this feature makes me
>> > >think
>> > >that it's unlikely to be supported in the next few years either.
>> > >
>> > >I suspect most PRs affect a very small number of modules, so I think
>> > >the
>> > >performance advantage of a build system truly supporting incremental
>> > >builds
>> > >may be so overwhelming as to trump many other factors. Of course, we'd
>> > >need
>> > >to prototype and have hard numbers in hand to discuss this with more
>> > >substance.
>> > >
>> > >[1]
>> > >https://stackoverflow.com/questions/8918165/does-maven-
>> > support-incremental-builds
>> > >
>> > >On Mon, Oct 30, 2017 at 10:57 PM Romain Manni-Bucau
>> > >
>> > >wrote:
>> > >
>> > >> Hi
>> > >>
>> > >> Even if not a commiter or even PMC, I'd like to mention a few points
>> > >from
>> > >> an external eye:
>> > >>
>> > >> - Maven stays the most common build tool and easier one for any user.
>> > >It
>> > >> means it is the best one to hope contributions IMHO.
>> > >> - Maven has incremental support but if there is any blocker the
>> > >community
>> > >> is probably ready to enhance it (has been done for compiler plugin
>> > >for
>> > >> instance)
>> > >> - Gradle hides issues easily with its daemon so a build without
>> > >daemon is
>> > >> needed
>> > >> - Gradle doesnt isolate plugins well enough so ensure your planned
>> > >plugins
>> > >> doesnt conflict
>> > >> - Only Maven is correctly supported in mainstream and OS/free IDE
>> > >>
>> > >> This is the reasons why I think Maven is better - not even entering
>> > >into
>> > >> the ASF points.
>> > >>
>> > >> Now Maven is not perfect but some quick enhancements can be done:
>> > >>
>> > >> - A fast build profile can be created
>> > >> - Takari scheduler can be used yo enhance the parallel build
>> > >> - Scripts can be provided to build a subpart of the project
>> > >> - A beam extension can surely be done to optimize or compute the
>> > >reactors
>> > >> more easily based on module names
>> > >>
>> > >> Romain
>> > >>
>> > >> Le 31 oct. 2017 06:42, "Jean-Baptiste Onofré"  a
>> > >écrit :
>> > >>
>> > >> -0
>> > >>
>> > >> For the following reasons reasons:
>> > >> - maven is a Apache project and we can have support/improvement
>> > >> - I don't see how another build tool would speed up the build by
>> > >itself
>> > >> - Apache default release process is based on Maven
>> > >>
>> > >> On the other hand, Gradle could be interesting. Anyway it's something
>> > >to
>> > >> evaluate.
>> > >>
>> > >> Regards
>> > >> JB
>> > >>
>> > >>
>> > >> On Oct 30, 2017, 18:46, at 18:46, Ted Yu  wrote:
>> > >> >I agree with Ben's comment.
>> > >> >
>> > >> >Recently I have been using gradle in another Apache project and
>> > >found
>> > >> >it
>> > >> >interesting.
>> > >> >
>> > >> >Cheers
>> > >>
>> >
>>

Re: [DISCUSS] Move away from Apache Maven as build tool

[Kenneth Knowles]

Echoing what JB and Reuven said, we absolutely must provide maven central
artifacts for Java users, just as we provide pypi artifacts for Python
users.

I see Maven as still a viable tool for single-module Java builds,
especially considering its rich plugin ecosystem.

On Mon, Oct 30, 2017 at 11:27 PM, Reuven Lax 
wrote:

> I think that's a very good point. No matter what build system we use for
> our own personal development, we still need to release Maven artifacts and
> releases as we need to support our users using Maven.
>
> On Mon, Oct 30, 2017 at 11:26 PM, Jean-Baptiste Onofré 
> wrote:
>
> > Generally speaking, it's interesting to evaluate alternatives, especially
> > Gradle. My point is also to keep Maven artifacts and "releases" as most
> of
> > our users will use Maven.
> > For incremental build, afair, there's some enhancements on Maven but I
> > have to take a look.
> >
> > Regards
> > JB
> >
> > On Oct 31, 2017, 07:22, at 07:22, Eugene Kirpichov
> >  wrote:
> > >Hi!
> > >
> > >Many of these points sound valid, but AFAICT Maven doesn't really do
> > >incremental builds [1]. The best it can do is, it seems, recompile only
> > >changed files, but Java compilation is a tiny part of the overall
> > >build.
> > >
> > >Almost all time is taken by other plugins, such as unit testing or
> > >findbugs
> > >- and Maven does not seem to currently support features such as "do not
> > >rerun unit tests of a module if the code didn't change".
> > >
> > >The fact that the surefire plugin has existed for >11 years (version
> > >2.0
> > >was released in 2006) and still doesn't have this feature makes me
> > >think
> > >that it's unlikely to be supported in the next few years either.
> > >
> > >I suspect most PRs affect a very small number of modules, so I think
> > >the
> > >performance advantage of a build system truly supporting incremental
> > >builds
> > >may be so overwhelming as to trump many other factors. Of course, we'd
> > >need
> > >to prototype and have hard numbers in hand to discuss this with more
> > >substance.
> > >
> > >[1]
> > >https://stackoverflow.com/questions/8918165/does-maven-
> > support-incremental-builds
> > >
> > >On Mon, Oct 30, 2017 at 10:57 PM Romain Manni-Bucau
> > >
> > >wrote:
> > >
> > >> Hi
> > >>
> > >> Even if not a commiter or even PMC, I'd like to mention a few points
> > >from
> > >> an external eye:
> > >>
> > >> - Maven stays the most common build tool and easier one for any user.
> > >It
> > >> means it is the best one to hope contributions IMHO.
> > >> - Maven has incremental support but if there is any blocker the
> > >community
> > >> is probably ready to enhance it (has been done for compiler plugin
> > >for
> > >> instance)
> > >> - Gradle hides issues easily with its daemon so a build without
> > >daemon is
> > >> needed
> > >> - Gradle doesnt isolate plugins well enough so ensure your planned
> > >plugins
> > >> doesnt conflict
> > >> - Only Maven is correctly supported in mainstream and OS/free IDE
> > >>
> > >> This is the reasons why I think Maven is better - not even entering
> > >into
> > >> the ASF points.
> > >>
> > >> Now Maven is not perfect but some quick enhancements can be done:
> > >>
> > >> - A fast build profile can be created
> > >> - Takari scheduler can be used yo enhance the parallel build
> > >> - Scripts can be provided to build a subpart of the project
> > >> - A beam extension can surely be done to optimize or compute the
> > >reactors
> > >> more easily based on module names
> > >>
> > >> Romain
> > >>
> > >> Le 31 oct. 2017 06:42, "Jean-Baptiste Onofré"  a
> > >écrit :
> > >>
> > >> -0
> > >>
> > >> For the following reasons reasons:
> > >> - maven is a Apache project and we can have support/improvement
> > >> - I don't see how another build tool would speed up the build by
> > >itself
> > >> - Apache default release process is based on Maven
> > >>
> > >> On the other hand, Gradle could be interesting. Anyway it's something
> > >to
> > >> evaluate.
> > >>
> > >> Regards
> > >> JB
> > >>
> > >>
> > >> On Oct 30, 2017, 18:46, at 18:46, Ted Yu  wrote:
> > >> >I agree with Ben's comment.
> > >> >
> > >> >Recently I have been using gradle in another Apache project and
> > >found
> > >> >it
> > >> >interesting.
> > >> >
> > >> >Cheers
> > >>
> >
>

Re: [DISCUSS] Move away from Apache Maven as build tool

[Reuven Lax]

I think that's a very good point. No matter what build system we use for
our own personal development, we still need to release Maven artifacts and
releases as we need to support our users using Maven.

On Mon, Oct 30, 2017 at 11:26 PM, Jean-Baptiste Onofré 
wrote:

> Generally speaking, it's interesting to evaluate alternatives, especially
> Gradle. My point is also to keep Maven artifacts and "releases" as most of
> our users will use Maven.
> For incremental build, afair, there's some enhancements on Maven but I
> have to take a look.
>
> Regards
> JB
>
> On Oct 31, 2017, 07:22, at 07:22, Eugene Kirpichov
>  wrote:
> >Hi!
> >
> >Many of these points sound valid, but AFAICT Maven doesn't really do
> >incremental builds [1]. The best it can do is, it seems, recompile only
> >changed files, but Java compilation is a tiny part of the overall
> >build.
> >
> >Almost all time is taken by other plugins, such as unit testing or
> >findbugs
> >- and Maven does not seem to currently support features such as "do not
> >rerun unit tests of a module if the code didn't change".
> >
> >The fact that the surefire plugin has existed for >11 years (version
> >2.0
> >was released in 2006) and still doesn't have this feature makes me
> >think
> >that it's unlikely to be supported in the next few years either.
> >
> >I suspect most PRs affect a very small number of modules, so I think
> >the
> >performance advantage of a build system truly supporting incremental
> >builds
> >may be so overwhelming as to trump many other factors. Of course, we'd
> >need
> >to prototype and have hard numbers in hand to discuss this with more
> >substance.
> >
> >[1]
> >https://stackoverflow.com/questions/8918165/does-maven-
> support-incremental-builds
> >
> >On Mon, Oct 30, 2017 at 10:57 PM Romain Manni-Bucau
> >
> >wrote:
> >
> >> Hi
> >>
> >> Even if not a commiter or even PMC, I'd like to mention a few points
> >from
> >> an external eye:
> >>
> >> - Maven stays the most common build tool and easier one for any user.
> >It
> >> means it is the best one to hope contributions IMHO.
> >> - Maven has incremental support but if there is any blocker the
> >community
> >> is probably ready to enhance it (has been done for compiler plugin
> >for
> >> instance)
> >> - Gradle hides issues easily with its daemon so a build without
> >daemon is
> >> needed
> >> - Gradle doesnt isolate plugins well enough so ensure your planned
> >plugins
> >> doesnt conflict
> >> - Only Maven is correctly supported in mainstream and OS/free IDE
> >>
> >> This is the reasons why I think Maven is better - not even entering
> >into
> >> the ASF points.
> >>
> >> Now Maven is not perfect but some quick enhancements can be done:
> >>
> >> - A fast build profile can be created
> >> - Takari scheduler can be used yo enhance the parallel build
> >> - Scripts can be provided to build a subpart of the project
> >> - A beam extension can surely be done to optimize or compute the
> >reactors
> >> more easily based on module names
> >>
> >> Romain
> >>
> >> Le 31 oct. 2017 06:42, "Jean-Baptiste Onofré"  a
> >écrit :
> >>
> >> -0
> >>
> >> For the following reasons reasons:
> >> - maven is a Apache project and we can have support/improvement
> >> - I don't see how another build tool would speed up the build by
> >itself
> >> - Apache default release process is based on Maven
> >>
> >> On the other hand, Gradle could be interesting. Anyway it's something
> >to
> >> evaluate.
> >>
> >> Regards
> >> JB
> >>
> >>
> >> On Oct 30, 2017, 18:46, at 18:46, Ted Yu  wrote:
> >> >I agree with Ben's comment.
> >> >
> >> >Recently I have been using gradle in another Apache project and
> >found
> >> >it
> >> >interesting.
> >> >
> >> >Cheers
> >>
>

Re: [DISCUSS] Move away from Apache Maven as build tool

[Jean-Baptiste Onofré]

Generally speaking, it's interesting to evaluate alternatives, especially 
Gradle. My point is also to keep Maven artifacts and "releases" as most of our 
users will use Maven.
For incremental build, afair, there's some enhancements on Maven but I have to 
take a look.

Regards
JB

On Oct 31, 2017, 07:22, at 07:22, Eugene Kirpichov 
 wrote:
>Hi!
>
>Many of these points sound valid, but AFAICT Maven doesn't really do
>incremental builds [1]. The best it can do is, it seems, recompile only
>changed files, but Java compilation is a tiny part of the overall
>build.
>
>Almost all time is taken by other plugins, such as unit testing or
>findbugs
>- and Maven does not seem to currently support features such as "do not
>rerun unit tests of a module if the code didn't change".
>
>The fact that the surefire plugin has existed for >11 years (version
>2.0
>was released in 2006) and still doesn't have this feature makes me
>think
>that it's unlikely to be supported in the next few years either.
>
>I suspect most PRs affect a very small number of modules, so I think
>the
>performance advantage of a build system truly supporting incremental
>builds
>may be so overwhelming as to trump many other factors. Of course, we'd
>need
>to prototype and have hard numbers in hand to discuss this with more
>substance.
>
>[1]
>https://stackoverflow.com/questions/8918165/does-maven-support-incremental-builds
>
>On Mon, Oct 30, 2017 at 10:57 PM Romain Manni-Bucau
>
>wrote:
>
>> Hi
>>
>> Even if not a commiter or even PMC, I'd like to mention a few points
>from
>> an external eye:
>>
>> - Maven stays the most common build tool and easier one for any user.
>It
>> means it is the best one to hope contributions IMHO.
>> - Maven has incremental support but if there is any blocker the
>community
>> is probably ready to enhance it (has been done for compiler plugin
>for
>> instance)
>> - Gradle hides issues easily with its daemon so a build without
>daemon is
>> needed
>> - Gradle doesnt isolate plugins well enough so ensure your planned
>plugins
>> doesnt conflict
>> - Only Maven is correctly supported in mainstream and OS/free IDE
>>
>> This is the reasons why I think Maven is better - not even entering
>into
>> the ASF points.
>>
>> Now Maven is not perfect but some quick enhancements can be done:
>>
>> - A fast build profile can be created
>> - Takari scheduler can be used yo enhance the parallel build
>> - Scripts can be provided to build a subpart of the project
>> - A beam extension can surely be done to optimize or compute the
>reactors
>> more easily based on module names
>>
>> Romain
>>
>> Le 31 oct. 2017 06:42, "Jean-Baptiste Onofré"  a
>écrit :
>>
>> -0
>>
>> For the following reasons reasons:
>> - maven is a Apache project and we can have support/improvement
>> - I don't see how another build tool would speed up the build by
>itself
>> - Apache default release process is based on Maven
>>
>> On the other hand, Gradle could be interesting. Anyway it's something
>to
>> evaluate.
>>
>> Regards
>> JB
>>
>>
>> On Oct 30, 2017, 18:46, at 18:46, Ted Yu  wrote:
>> >I agree with Ben's comment.
>> >
>> >Recently I have been using gradle in another Apache project and
>found
>> >it
>> >interesting.
>> >
>> >Cheers
>>

Re: [DISCUSS] Move away from Apache Maven as build tool

[Eugene Kirpichov]

Hi!

Many of these points sound valid, but AFAICT Maven doesn't really do
incremental builds [1]. The best it can do is, it seems, recompile only
changed files, but Java compilation is a tiny part of the overall build.

Almost all time is taken by other plugins, such as unit testing or findbugs
- and Maven does not seem to currently support features such as "do not
rerun unit tests of a module if the code didn't change".

The fact that the surefire plugin has existed for >11 years (version 2.0
was released in 2006) and still doesn't have this feature makes me think
that it's unlikely to be supported in the next few years either.

I suspect most PRs affect a very small number of modules, so I think the
performance advantage of a build system truly supporting incremental builds
may be so overwhelming as to trump many other factors. Of course, we'd need
to prototype and have hard numbers in hand to discuss this with more
substance.

[1]
https://stackoverflow.com/questions/8918165/does-maven-support-incremental-builds

On Mon, Oct 30, 2017 at 10:57 PM Romain Manni-Bucau 
wrote:

> Hi
>
> Even if not a commiter or even PMC, I'd like to mention a few points from
> an external eye:
>
> - Maven stays the most common build tool and easier one for any user. It
> means it is the best one to hope contributions IMHO.
> - Maven has incremental support but if there is any blocker the community
> is probably ready to enhance it (has been done for compiler plugin for
> instance)
> - Gradle hides issues easily with its daemon so a build without daemon is
> needed
> - Gradle doesnt isolate plugins well enough so ensure your planned plugins
> doesnt conflict
> - Only Maven is correctly supported in mainstream and OS/free IDE
>
> This is the reasons why I think Maven is better - not even entering into
> the ASF points.
>
> Now Maven is not perfect but some quick enhancements can be done:
>
> - A fast build profile can be created
> - Takari scheduler can be used yo enhance the parallel build
> - Scripts can be provided to build a subpart of the project
> - A beam extension can surely be done to optimize or compute the reactors
> more easily based on module names
>
> Romain
>
> Le 31 oct. 2017 06:42, "Jean-Baptiste Onofré"  a écrit :
>
> -0
>
> For the following reasons reasons:
> - maven is a Apache project and we can have support/improvement
> - I don't see how another build tool would speed up the build by itself
> - Apache default release process is based on Maven
>
> On the other hand, Gradle could be interesting. Anyway it's something to
> evaluate.
>
> Regards
> JB
>
>
> On Oct 30, 2017, 18:46, at 18:46, Ted Yu  wrote:
> >I agree with Ben's comment.
> >
> >Recently I have been using gradle in another Apache project and found
> >it
> >interesting.
> >
> >Cheers
>

Re: [DISCUSS] Move away from Apache Maven as build tool

[Jean-Baptiste Onofré]

-0

For the following reasons reasons:
- maven is a Apache project and we can have support/improvement
- I don't see how another build tool would speed up the build by itself
- Apache default release process is based on Maven

On the other hand, Gradle could be interesting. Anyway it's something to 
evaluate.

Regards
JB


On Oct 30, 2017, 18:46, at 18:46, Ted Yu  wrote:
>I agree with Ben's comment.
>
>Recently I have been using gradle in another Apache project and found
>it
>interesting.
>
>Cheers

Re: [DISCUSS] Move away from Apache Maven as build tool

[Chamikara Jayalath]

+1 for exploring other build tools to improve the build time and for better
cross language support. But, as others mentioned, we should continue to
support Maven based builds for some time till things are fully migrated.
For example, we have updated PerfKitBenchmarker to execute Beam jobs
through Maven as a part of the performance testing framework.

https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/blob/eeb6b5a7eda28f681a6a5fb99fae1b71ea1e5615/perfkitbenchmarker/beam_benchmark_helper.py

https://beam.apache.org/documentation/io/testing/

Thanks,
Cham

On Mon, Oct 30, 2017 at 11:05 AM Reuven Lax 
wrote:

> One more comment: regardless of which build tool we use for development, we
> still need to publish Maven artifacts to support Maven users of Beam.
>
> On Mon, Oct 30, 2017 at 10:46 AM, Ted Yu  wrote:
>
> > I agree with Ben's comment.
> >
> > Recently I have been using gradle in another Apache project and found it
> > interesting.
> >
> > Cheers
> >
>

Re: [DISCUSS] Move away from Apache Maven as build tool

[Reuven Lax]

One more comment: regardless of which build tool we use for development, we
still need to publish Maven artifacts to support Maven users of Beam.

On Mon, Oct 30, 2017 at 10:46 AM, Ted Yu  wrote:

> I agree with Ben's comment.
>
> Recently I have been using gradle in another Apache project and found it
> interesting.
>
> Cheers
>

Re: [DISCUSS] Move away from Apache Maven as build tool

[Reuven Lax]

I wonder if we could prototype both Bazel and Gradle to do a better
comparison (and also to compare the results with our current Maven build).

On Mon, Oct 30, 2017 at 10:32 AM, Ben Chambers  wrote:

> I think both Gradle and Bazel are worth exploring. Gradle is definitely
> more common in the wild, but Bazel may be a better fit for the large
> mixture of languages being developed in one codebase within Beam. It might
> be helpful for us to list what functionality we want from such a tool, and
> then have someone who is interested in this dig in a bit and let us know
> more details how well Gradle or Bazel would fit our needs.
>
> On Mon, Oct 30, 2017 at 10:27 AM Kenneth Knowles 
> wrote:
>
> > I also support exploring a move away from Apache Maven for orchestrating
> > our build.
> >
> > For a single-module project, I still think it can be a good build tool,
> and
> > we could still use it for this sort of thing, but I think we are
> reaching a
> > multi-module scale where it does not work well. Almost all of our jobs
> > build things that are not needed and run tests that are redundant, and it
> > is not easy to do better, even with a sequence of maven commands.
> >
> > I'd like to lay out what we hope for from a tool. Here's a start:
> >
> > General:
> >
> >  - Dependency-driven build so devs working on one thing build & test only
> > what is needed
> >  - Supports orchestration across Protobuf, Java, Python, Go, Docker
> images
> >  - Meets devs where they are, letting folks in one language use familiar
> > tools
> >  - Caching across builds as much as possible
> >  - Easily extensible for when it doesn't have the feature we need
> (writing
> > a maven plugin is too much, using maven-exec-plugin is too crufty)
> >  - Preferably a declarative configuration language
> >
> > Java needs beyond the basics, which could be executed by the orchestrator
> > or my module-local mvn builds, etc.
> >
> >  - Pulling deps from maven central and alternate repos
> >  - Findbugs
> >  - RAT
> >  - Dependency rule enforcement
> >  - IWYU (Include What You Use)
> >  - Multiple Java versions in same project
> >  - ASF release workflow
> >
> > I probably missed some must-haves or nice-to-haves. I'd love to compile
> > thoughts on other languages' needs.
> >
> > Based on these, another project I would consider is Bazel. We could very
> > easily use it to orchestrate, but use Maven (or Gradle!) on the leaves. I
> > also think that Gradle is also more focused on the JVM ecosystem, so it
> is
> > not quite as neutral as Bazel, and uses Groovy which is a bit more
> esoteric
> > than Python for writing Bazel rules.
> >
> > Kenn
> >
> > On Mon, Oct 30, 2017 at 9:37 AM, Lukasz Cwik 
> > wrote:
> >
> > > I wanted to make this thread more visible. This discussion stems from
> > Ken's
> > > thread about Jenkins pre/post commit issues[1].
> > >
> > > I did some investigation as for ways to improve the quality of the
> signal
> > > from Jenkins by trying to modify the Jenkins jobs spawned from Groovy.
> I
> > > had limited success but everything I felt like I was doing was just
> > > patching symptoms of the problem which is that our build is just too
> > slow.
> > > For example, we keep adding all these profiles to Maven or tuning how a
> > > plugin runs to eek out a small decrease in build time. I believe
> swapping
> > > away from Apache Maven to a build tool which only builds the things
> which
> > > have changed in a PR would be the best approach.
> > >
> > > I would suggest that we migrate to Gradle as our build tool. I am
> > > suggesting Gradle because:
> > > * It is used in lots of open source projects and has a very large
> > community
> > > behind it.
> > > * It has better support for building languages other then Java
> > > (PyGradle[2], GoGradle[3], ...)
> > > * Its incremental build support works and only builds things that
> changed
> > > through the use of a build cache. Even without the build cache (or for
> > > clean builds), it is much faster.
> > > * Apache Infra already has Gradle v4.x installed on the Jenkins
> machines.
> > >
> > > Any alternatives that should be considered or should we stick with
> Apache
> > > Maven?
> > >
> > > 1:
> > > https://lists.apache.org/thread.html/25311e0e95be5c49afb168d9b4b4d3
> > > 57984c10c39c7b01da8ff3baaf@%3Cdev.beam.apache.org%3E
> > > 2: https://github.com/linkedin/pygradle
> > > 3: https://github.com/gogradle/gogradle
> > >
> >
>

Re: [DISCUSS] Move away from Apache Maven as build tool

[Ben Chambers]

I think both Gradle and Bazel are worth exploring. Gradle is definitely
more common in the wild, but Bazel may be a better fit for the large
mixture of languages being developed in one codebase within Beam. It might
be helpful for us to list what functionality we want from such a tool, and
then have someone who is interested in this dig in a bit and let us know
more details how well Gradle or Bazel would fit our needs.

On Mon, Oct 30, 2017 at 10:27 AM Kenneth Knowles 
wrote:

> I also support exploring a move away from Apache Maven for orchestrating
> our build.
>
> For a single-module project, I still think it can be a good build tool, and
> we could still use it for this sort of thing, but I think we are reaching a
> multi-module scale where it does not work well. Almost all of our jobs
> build things that are not needed and run tests that are redundant, and it
> is not easy to do better, even with a sequence of maven commands.
>
> I'd like to lay out what we hope for from a tool. Here's a start:
>
> General:
>
>  - Dependency-driven build so devs working on one thing build & test only
> what is needed
>  - Supports orchestration across Protobuf, Java, Python, Go, Docker images
>  - Meets devs where they are, letting folks in one language use familiar
> tools
>  - Caching across builds as much as possible
>  - Easily extensible for when it doesn't have the feature we need (writing
> a maven plugin is too much, using maven-exec-plugin is too crufty)
>  - Preferably a declarative configuration language
>
> Java needs beyond the basics, which could be executed by the orchestrator
> or my module-local mvn builds, etc.
>
>  - Pulling deps from maven central and alternate repos
>  - Findbugs
>  - RAT
>  - Dependency rule enforcement
>  - IWYU (Include What You Use)
>  - Multiple Java versions in same project
>  - ASF release workflow
>
> I probably missed some must-haves or nice-to-haves. I'd love to compile
> thoughts on other languages' needs.
>
> Based on these, another project I would consider is Bazel. We could very
> easily use it to orchestrate, but use Maven (or Gradle!) on the leaves. I
> also think that Gradle is also more focused on the JVM ecosystem, so it is
> not quite as neutral as Bazel, and uses Groovy which is a bit more esoteric
> than Python for writing Bazel rules.
>
> Kenn
>
> On Mon, Oct 30, 2017 at 9:37 AM, Lukasz Cwik 
> wrote:
>
> > I wanted to make this thread more visible. This discussion stems from
> Ken's
> > thread about Jenkins pre/post commit issues[1].
> >
> > I did some investigation as for ways to improve the quality of the signal
> > from Jenkins by trying to modify the Jenkins jobs spawned from Groovy. I
> > had limited success but everything I felt like I was doing was just
> > patching symptoms of the problem which is that our build is just too
> slow.
> > For example, we keep adding all these profiles to Maven or tuning how a
> > plugin runs to eek out a small decrease in build time. I believe swapping
> > away from Apache Maven to a build tool which only builds the things which
> > have changed in a PR would be the best approach.
> >
> > I would suggest that we migrate to Gradle as our build tool. I am
> > suggesting Gradle because:
> > * It is used in lots of open source projects and has a very large
> community
> > behind it.
> > * It has better support for building languages other then Java
> > (PyGradle[2], GoGradle[3], ...)
> > * Its incremental build support works and only builds things that changed
> > through the use of a build cache. Even without the build cache (or for
> > clean builds), it is much faster.
> > * Apache Infra already has Gradle v4.x installed on the Jenkins machines.
> >
> > Any alternatives that should be considered or should we stick with Apache
> > Maven?
> >
> > 1:
> > https://lists.apache.org/thread.html/25311e0e95be5c49afb168d9b4b4d3
> > 57984c10c39c7b01da8ff3baaf@%3Cdev.beam.apache.org%3E
> > 2: https://github.com/linkedin/pygradle
> > 3: https://github.com/gogradle/gogradle
> >
>

Re: [DISCUSS] Move away from Apache Maven as build tool

[Henning Rohde]

+1 to the initiative. It would great to have better support for Go and
Docker container images. The current Go maven integration in particular is
clunky [1], but I'll have to look into the details of the alternatives to
see if they are better.

Henning

[1] https://github.com/apache/beam/blob/master/sdks/go/BUILD.md

On Mon, Oct 30, 2017 at 10:27 AM, Kenneth Knowles 
wrote:

> I also support exploring a move away from Apache Maven for orchestrating
> our build.
>
> For a single-module project, I still think it can be a good build tool, and
> we could still use it for this sort of thing, but I think we are reaching a
> multi-module scale where it does not work well. Almost all of our jobs
> build things that are not needed and run tests that are redundant, and it
> is not easy to do better, even with a sequence of maven commands.
>
> I'd like to lay out what we hope for from a tool. Here's a start:
>
> General:
>
>  - Dependency-driven build so devs working on one thing build & test only
> what is needed
>  - Supports orchestration across Protobuf, Java, Python, Go, Docker images
>  - Meets devs where they are, letting folks in one language use familiar
> tools
>  - Caching across builds as much as possible
>  - Easily extensible for when it doesn't have the feature we need (writing
> a maven plugin is too much, using maven-exec-plugin is too crufty)
>  - Preferably a declarative configuration language
>
> Java needs beyond the basics, which could be executed by the orchestrator
> or my module-local mvn builds, etc.
>
>  - Pulling deps from maven central and alternate repos
>  - Findbugs
>  - RAT
>  - Dependency rule enforcement
>  - IWYU (Include What You Use)
>  - Multiple Java versions in same project
>  - ASF release workflow
>
> I probably missed some must-haves or nice-to-haves. I'd love to compile
> thoughts on other languages' needs.
>
> Based on these, another project I would consider is Bazel. We could very
> easily use it to orchestrate, but use Maven (or Gradle!) on the leaves. I
> also think that Gradle is also more focused on the JVM ecosystem, so it is
> not quite as neutral as Bazel, and uses Groovy which is a bit more esoteric
> than Python for writing Bazel rules.
>
> Kenn
>
> On Mon, Oct 30, 2017 at 9:37 AM, Lukasz Cwik 
> wrote:
>
> > I wanted to make this thread more visible. This discussion stems from
> Ken's
> > thread about Jenkins pre/post commit issues[1].
> >
> > I did some investigation as for ways to improve the quality of the signal
> > from Jenkins by trying to modify the Jenkins jobs spawned from Groovy. I
> > had limited success but everything I felt like I was doing was just
> > patching symptoms of the problem which is that our build is just too
> slow.
> > For example, we keep adding all these profiles to Maven or tuning how a
> > plugin runs to eek out a small decrease in build time. I believe swapping
> > away from Apache Maven to a build tool which only builds the things which
> > have changed in a PR would be the best approach.
> >
> > I would suggest that we migrate to Gradle as our build tool. I am
> > suggesting Gradle because:
> > * It is used in lots of open source projects and has a very large
> community
> > behind it.
> > * It has better support for building languages other then Java
> > (PyGradle[2], GoGradle[3], ...)
> > * Its incremental build support works and only builds things that changed
> > through the use of a build cache. Even without the build cache (or for
> > clean builds), it is much faster.
> > * Apache Infra already has Gradle v4.x installed on the Jenkins machines.
> >
> > Any alternatives that should be considered or should we stick with Apache
> > Maven?
> >
> > 1:
> > https://lists.apache.org/thread.html/25311e0e95be5c49afb168d9b4b4d3
> > 57984c10c39c7b01da8ff3baaf@%3Cdev.beam.apache.org%3E
> > 2: https://github.com/linkedin/pygradle
> > 3: https://github.com/gogradle/gogradle
> >
>

Re: [DISCUSS] Move away from Apache Maven as build tool

[Kenneth Knowles]

I also support exploring a move away from Apache Maven for orchestrating
our build.

For a single-module project, I still think it can be a good build tool, and
we could still use it for this sort of thing, but I think we are reaching a
multi-module scale where it does not work well. Almost all of our jobs
build things that are not needed and run tests that are redundant, and it
is not easy to do better, even with a sequence of maven commands.

I'd like to lay out what we hope for from a tool. Here's a start:

General:

 - Dependency-driven build so devs working on one thing build & test only
what is needed
 - Supports orchestration across Protobuf, Java, Python, Go, Docker images
 - Meets devs where they are, letting folks in one language use familiar
tools
 - Caching across builds as much as possible
 - Easily extensible for when it doesn't have the feature we need (writing
a maven plugin is too much, using maven-exec-plugin is too crufty)
 - Preferably a declarative configuration language

Java needs beyond the basics, which could be executed by the orchestrator
or my module-local mvn builds, etc.

 - Pulling deps from maven central and alternate repos
 - Findbugs
 - RAT
 - Dependency rule enforcement
 - IWYU (Include What You Use)
 - Multiple Java versions in same project
 - ASF release workflow

I probably missed some must-haves or nice-to-haves. I'd love to compile
thoughts on other languages' needs.

Based on these, another project I would consider is Bazel. We could very
easily use it to orchestrate, but use Maven (or Gradle!) on the leaves. I
also think that Gradle is also more focused on the JVM ecosystem, so it is
not quite as neutral as Bazel, and uses Groovy which is a bit more esoteric
than Python for writing Bazel rules.

Kenn

On Mon, Oct 30, 2017 at 9:37 AM, Lukasz Cwik 
wrote:

> I wanted to make this thread more visible. This discussion stems from Ken's
> thread about Jenkins pre/post commit issues[1].
>
> I did some investigation as for ways to improve the quality of the signal
> from Jenkins by trying to modify the Jenkins jobs spawned from Groovy. I
> had limited success but everything I felt like I was doing was just
> patching symptoms of the problem which is that our build is just too slow.
> For example, we keep adding all these profiles to Maven or tuning how a
> plugin runs to eek out a small decrease in build time. I believe swapping
> away from Apache Maven to a build tool which only builds the things which
> have changed in a PR would be the best approach.
>
> I would suggest that we migrate to Gradle as our build tool. I am
> suggesting Gradle because:
> * It is used in lots of open source projects and has a very large community
> behind it.
> * It has better support for building languages other then Java
> (PyGradle[2], GoGradle[3], ...)
> * Its incremental build support works and only builds things that changed
> through the use of a build cache. Even without the build cache (or for
> clean builds), it is much faster.
> * Apache Infra already has Gradle v4.x installed on the Jenkins machines.
>
> Any alternatives that should be considered or should we stick with Apache
> Maven?
>
> 1:
> https://lists.apache.org/thread.html/25311e0e95be5c49afb168d9b4b4d3
> 57984c10c39c7b01da8ff3baaf@%3Cdev.beam.apache.org%3E
> 2: https://github.com/linkedin/pygradle
> 3: https://github.com/gogradle/gogradle
>

Re: [DISCUSS] Move away from Apache Maven as build tool

[Reuven Lax]

If we're looking at other build options, I think we should also consider
Bazel. Some advantages I see of Bazel over Gradle:

Gradle is reportedly much slower than Bazel. I haven't measured this
myself, but there are many user reports of slow builds, memory leaks, etc.

Bazel's build language is based on Python, which more people are familiar
with than Groovy (which Gradle uses)

Bazel has build rules for building Docker images, which will be very useful
in the multi-language world.

Of course Gradle has been around longer, and as Luke mentioned is already
installed on the Jenkins machines.

In general, while Maven has done well for Beam (and prior to Beam for
Dataflow) for quite some time, Maven is primarily a Java build tool. As
Beam becomes a cross-language platform, I think we're seeing Maven become
more of an awkward fit. Our cross language "rules" are essentially just
shell scripts, where other build systems can natively manage those rules.
We're going to start wanting dependencies between components in different
languages, which is going to be difficult to do in Maven.

I don't think Maven can completely go away, we still depend on Maven
plugins for a number of things (RAT, FindBugs, etc.). Also an initial
approach might be to continue using Maven to build Java (for now at least),
but to use one of these other build systems to orchestrate the build across
all languages.

Reuven

On Mon, Oct 30, 2017 at 9:37 AM, Lukasz Cwik 
wrote:

> I wanted to make this thread more visible. This discussion stems from Ken's
> thread about Jenkins pre/post commit issues[1].
>
> I did some investigation as for ways to improve the quality of the signal
> from Jenkins by trying to modify the Jenkins jobs spawned from Groovy. I
> had limited success but everything I felt like I was doing was just
> patching symptoms of the problem which is that our build is just too slow.
> For example, we keep adding all these profiles to Maven or tuning how a
> plugin runs to eek out a small decrease in build time. I believe swapping
> away from Apache Maven to a build tool which only builds the things which
> have changed in a PR would be the best approach.
>
> I would suggest that we migrate to Gradle as our build tool. I am
> suggesting Gradle because:
> * It is used in lots of open source projects and has a very large community
> behind it.
> * It has better support for building languages other then Java
> (PyGradle[2], GoGradle[3], ...)
> * Its incremental build support works and only builds things that changed
> through the use of a build cache. Even without the build cache (or for
> clean builds), it is much faster.
> * Apache Infra already has Gradle v4.x installed on the Jenkins machines.
>
> Any alternatives that should be considered or should we stick with Apache
> Maven?
>
> 1:
> https://lists.apache.org/thread.html/25311e0e95be5c49afb168d9b4b4d3
> 57984c10c39c7b01da8ff3baaf@%3Cdev.beam.apache.org%3E
> 2: https://github.com/linkedin/pygradle
> 3: https://github.com/gogradle/gogradle
>

Re: [DISCUSS] Move away from Apache Maven as build tool

[Eugene Kirpichov]

This seems like a very good idea. With the effective complete lack of
incremental builds in Maven, it's frustrating to routinely spend several
minutes re-verifying a PR after fixing a checkstyle warning in an extension
module.

Another non-Apache alternative could be Bazel, which is even faster and
better at incremental builds [1], but the performance difference is not
*that* big and anyway both leave Maven far behind; and Gradle definitely
has better integration with other parts of the Apache ecosystem and is more
familiar to the community. So +1.

[1] https://www.pgrs.net/2015/09/01/migrating-from-gradle-to-bazel/

On Mon, Oct 30, 2017 at 9:37 AM Lukasz Cwik 
wrote:

> I wanted to make this thread more visible. This discussion stems from Ken's
> thread about Jenkins pre/post commit issues[1].
>
> I did some investigation as for ways to improve the quality of the signal
> from Jenkins by trying to modify the Jenkins jobs spawned from Groovy. I
> had limited success but everything I felt like I was doing was just
> patching symptoms of the problem which is that our build is just too slow.
> For example, we keep adding all these profiles to Maven or tuning how a
> plugin runs to eek out a small decrease in build time. I believe swapping
> away from Apache Maven to a build tool which only builds the things which
> have changed in a PR would be the best approach.
>
> I would suggest that we migrate to Gradle as our build tool. I am
> suggesting Gradle because:
> * It is used in lots of open source projects and has a very large community
> behind it.
> * It has better support for building languages other then Java
> (PyGradle[2], GoGradle[3], ...)
> * Its incremental build support works and only builds things that changed
> through the use of a build cache. Even without the build cache (or for
> clean builds), it is much faster.
> * Apache Infra already has Gradle v4.x installed on the Jenkins machines.
>
> Any alternatives that should be considered or should we stick with Apache
> Maven?
>
> 1:
>
> https://lists.apache.org/thread.html/25311e0e95be5c49afb168d9b4b4d357984c10c39c7b01da8ff3baaf@%3Cdev.beam.apache.org%3E
> 2: https://github.com/linkedin/pygradle
> 3: https://github.com/gogradle/gogradle
>