date:20171127

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-27 Thread Jean-Baptiste Onofré

Yeah, especially, I think it would have been great to have a vote before merging
on master.

Not a big deal, however, I'm really community focus ;)

Regards
JB

On 11/28/2017 07:36 AM, Reuven Lax wrote:
Agreed. I thinking having a formal vote before Luke had numbers and results
would have been too early. However now that we have such numbers, we should
think about having a vote.

Also, while I disagree with Romain that Gradle is not "enterprise ready" (it's
heavily used by Netflix, LinkedIn, Siemens, and is the default build framework
for Android apps), it would be interesting to see if any other ASF projects are
using it. I don't think that should not make or break the decision - we should
do what's best for the Beam project, and "everyone else is doing something" is
rarely a good argument - it will provide good data points for us to evaluate.

Reuven

On Mon, Nov 27, 2017 at 10:23 PM, Jean-Baptiste Onofré > wrote:

Hi Luke,

just curious (and maybe I missed it): did we do a formal vote to merge the
gradle build ?
Gradle is now on master, we have some Jira to update the release guide with
gradle. It's fine, but I remember only a discussion, not a vote.

In order to embrace the community and avoid to have some contributors
"frustrated" (meaning that "this project doesn't care about contributor,
they just do whatever they want"), I would have love to see a formal vote
about Gradle more than just a discussion.

My $0.01

Regards
JB

On 11/27/2017 07:46 PM, Lukasz Cwik wrote:

I have collected data by running several builds against master using
Gradle
and Maven without using Gradle's support for incremental builds.

Gradle (mins)
min: 25.04
max: 160.14
median: 45.78
average: 52.19
stdev: 30.80

Maven (mins)
min: 56.86
max: 216.55 (actually > 240 mins because this data does not include
timeouts)
median: 87.93
average: 109.10
stdev: 48.01

I excluded a few timeouts (240 mins) that happened during the Maven
build
from its numbers but we can see conclusively that Gradle is twice as
fast
for the build when compared to Maven when run using Jenkins.
On my desktop, I have enabled incremental builds and have seen a major
improvement on the above numbers but it doesn't yet work correctly
because
of incorrectly specified inputs/outputs for certain tasks.

The data is available here

https://docs.google.com/spreadsheets/d/1MHVjF-xoI49_NJqEQakUgnNIQ7Qbjzu8Y1q_h3dbF1M/edit?usp=sharing

With this data, I feel confident that we should swap and have opened the
following issue https://issues.apache.org/jira/browse/BEAM-3249
and related
sub-tasks.

On Sun, Nov 19, 2017 at 11:23 AM, Jean-Baptiste Onofré
mailto:j...@nanthrax.net>>
wrote:

Thanks for the update Luke.

I'm updating my local working copy to do new tests.

Regards
JB

On 11/19/2017 08:21 PM, Lukasz Cwik wrote:

The gradle build rules have been merged, I'm adding a
precommit[1] to
start
collecting data about the build times. It currently only mirrors
the Java
mvn install precommit. I'll gather data over the next two weeks
and
provide
a summary here.

You can rerun the precommit by issuing "Run Java Gradle
PreCommit"

1: https://github.com/apache/beam/pull/4146

On Mon, Nov 13, 2017 at 9:08 AM, Lukasz Cwik mailto:lc...@google.com>> wrote:

There has been plenty of time for comments on the PR and the
approach.

So far Ken Knowles has provided the most feedback on the PR,
Ken would
you
like to finish the review?

On Fri, Nov 10, 2017 at 1:22 PM, Romain Manni-Bucau <
rmannibu...@gmail.com

wrote:

This is only a setup thing and better to not break the
master history for

poc/tests, in particular when no very localized.
Alternative can be to
ask
another temp repo to infra and have a synchro between
both but dont
think
it does worth it personally.

Le 10 nov. 2017 1

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-27 Thread Reuven Lax

Agreed. I thinking having a formal vote before Luke had numbers and results
would have been too early. However now that we have such numbers, we should
think about having a vote.

Also, while I disagree with Romain that Gradle is not "enterprise ready"
(it's heavily used by Netflix, LinkedIn, Siemens, and is the default build
framework for Android apps), it would be interesting to see if any other
ASF projects are using it. I don't think that should not make or break the
decision - we should do what's best for the Beam project, and "everyone
else is doing something" is rarely a good argument - it will provide good
data points for us to evaluate.

Reuven

On Mon, Nov 27, 2017 at 10:23 PM, Jean-Baptiste Onofré 
wrote:

> Hi Luke,
>
> just curious (and maybe I missed it): did we do a formal vote to merge the
> gradle build ?
> Gradle is now on master, we have some Jira to update the release guide
> with gradle. It's fine, but I remember only a discussion, not a vote.
>
> In order to embrace the community and avoid to have some contributors
> "frustrated" (meaning that "this project doesn't care about contributor,
> they just do whatever they want"), I would have love to see a formal vote
> about Gradle more than just a discussion.
>
> My $0.01
>
> Regards
> JB
>
> On 11/27/2017 07:46 PM, Lukasz Cwik wrote:
>
>> I have collected data by running several builds against master using
>> Gradle
>> and Maven without using Gradle's support for incremental builds.
>>
>> Gradle (mins)
>> min: 25.04
>> max: 160.14
>> median: 45.78
>> average: 52.19
>> stdev: 30.80
>>
>> Maven (mins)
>> min: 56.86
>> max: 216.55 (actually > 240 mins because this data does not include
>> timeouts)
>> median: 87.93
>> average: 109.10
>> stdev: 48.01
>>
>> I excluded a few timeouts (240 mins) that happened during the Maven build
>> from its numbers but we can see conclusively that Gradle is twice as fast
>> for the build when compared to Maven when run using Jenkins.
>> On my desktop, I have enabled incremental builds and have seen a major
>> improvement on the above numbers but it doesn't yet work correctly because
>> of incorrectly specified inputs/outputs for certain tasks.
>>
>> The data is available here
>> https://docs.google.com/spreadsheets/d/1MHVjF-xoI49_NJqEQakU
>> gnNIQ7Qbjzu8Y1q_h3dbF1M/edit?usp=sharing
>>
>> With this data, I feel confident that we should swap and have opened the
>> following issue https://issues.apache.org/jira/browse/BEAM-3249 and
>> related
>> sub-tasks.
>>
>> On Sun, Nov 19, 2017 at 11:23 AM, Jean-Baptiste Onofré 
>> wrote:
>>
>> Thanks for the update Luke.
>>>
>>> I'm updating my local working copy to do new tests.
>>>
>>> Regards
>>> JB
>>>
>>> On 11/19/2017 08:21 PM, Lukasz Cwik wrote:
>>>
>>> The gradle build rules have been merged, I'm adding a precommit[1] to
 start
 collecting data about the build times. It currently only mirrors the
 Java
 mvn install precommit. I'll gather data over the next two weeks and
 provide
 a summary here.

 You can rerun the precommit by issuing "Run Java Gradle PreCommit"

 1: https://github.com/apache/beam/pull/4146


 On Mon, Nov 13, 2017 at 9:08 AM, Lukasz Cwik  wrote:

 There has been plenty of time for comments on the PR and the approach.

>
> So far Ken Knowles has provided the most feedback on the PR, Ken would
> you
> like to finish the review?
>
>
>
> On Fri, Nov 10, 2017 at 1:22 PM, Romain Manni-Bucau <
> rmannibu...@gmail.com
>
> wrote:
>>
>>
> This is only a setup thing and better to not break the master history
> for
>
>> poc/tests, in particular when no very localized. Alternative can be to
>> ask
>> another temp repo to infra and have a synchro between both but dont
>> think
>> it does worth it personally.
>>
>>
>>
>> Le 10 nov. 2017 18:57, "Lukasz Cwik"  a
>> écrit :
>>
>> The reason to get it on master is because that is where all the PRs
>>
>>>
>>> are. An
>>
>> upstream branch without any development means no data.
>>> Also, our Jenkins setup via job-dsl doesn't honor using the Jenkins
>>> configuration on the branch because the seed job always runs against
>>> master.
>>>
>>> On Thu, Nov 9, 2017 at 9:59 PM, Romain Manni-Bucau <
>>>
>>> rmannibu...@gmail.com>
>>
>> wrote:
>>>
>>> What about pushing it on a "upstream" branch and testing it for 1
>>>

 week in
>>>
>>
>> parallel of the maven reference build? If gradle is always 50% faster
>>>

 on
>>>
>>
>> jenkins then it could become master setup without much discussion I
>>>

 guess.
>>>
>>> We can even have 2 jenkins jobs: one with the daemon etc and one

 without.
>>>
>>
>>
>>> Also noticed yesterday that gradle build is killing my machine (al

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-27 Thread Jean-Baptiste Onofré

Hi Luke,

In order to embrace the community and avoid to have some contributors
"frustrated" (meaning that "this project doesn't care about contributor, they
just do whatever they want"), I would have love to see a formal vote about
Gradle more than just a discussion.

My $0.01

Regards
JB

On 11/27/2017 07:46 PM, Lukasz Cwik wrote:

I have collected data by running several builds against master using Gradle
and Maven without using Gradle's support for incremental builds.

Gradle (mins)
min: 25.04
max: 160.14
median: 45.78
average: 52.19
stdev: 30.80

Maven (mins)
min: 56.86
max: 216.55 (actually > 240 mins because this data does not include
timeouts)
median: 87.93
average: 109.10
stdev: 48.01

I excluded a few timeouts (240 mins) that happened during the Maven build
from its numbers but we can see conclusively that Gradle is twice as fast
for the build when compared to Maven when run using Jenkins.
On my desktop, I have enabled incremental builds and have seen a major
improvement on the above numbers but it doesn't yet work correctly because
of incorrectly specified inputs/outputs for certain tasks.

The data is available here
https://docs.google.com/spreadsheets/d/1MHVjF-xoI49_NJqEQakUgnNIQ7Qbjzu8Y1q_h3dbF1M/edit?usp=sharing

With this data, I feel confident that we should swap and have opened the
following issue https://issues.apache.org/jira/browse/BEAM-3249 and related
sub-tasks.

On Sun, Nov 19, 2017 at 11:23 AM, Jean-Baptiste Onofré
wrote:

Thanks for the update Luke.

I'm updating my local working copy to do new tests.

Regards
JB

On 11/19/2017 08:21 PM, Lukasz Cwik wrote:

The gradle build rules have been merged, I'm adding a precommit[1] to
start
collecting data about the build times. It currently only mirrors the Java
mvn install precommit. I'll gather data over the next two weeks and
provide
a summary here.

You can rerun the precommit by issuing "Run Java Gradle PreCommit"

1: https://github.com/apache/beam/pull/4146

On Mon, Nov 13, 2017 at 9:08 AM, Lukasz Cwik wrote:

There has been plenty of time for comments on the PR and the approach.

So far Ken Knowles has provided the most feedback on the PR, Ken would
you
like to finish the review?

On Fri, Nov 10, 2017 at 1:22 PM, Romain Manni-Bucau <
rmannibu...@gmail.com

wrote:

This is only a setup thing and better to not break the master history for

poc/tests, in particular when no very localized. Alternative can be to
ask
another temp repo to infra and have a synchro between both but dont
think
it does worth it personally.

Le 10 nov. 2017 18:57, "Lukasz Cwik" a
écrit :

The reason to get it on master is because that is where all the PRs

are. An

upstream branch without any development means no data.
Also, our Jenkins setup via job-dsl doesn't honor using the Jenkins
configuration on the branch because the seed job always runs against
master.

On Thu, Nov 9, 2017 at 9:59 PM, Romain Manni-Bucau <

rmannibu...@gmail.com>

wrote:

What about pushing it on a "upstream" branch and testing it for 1

week in

parallel of the maven reference build? If gradle is always 50% faster

jenkins then it could become master setup without much discussion I

guess.

We can even have 2 jenkins jobs: one with the daemon etc and one

without.

Also noticed yesterday that gradle build is killing my machine (all 8

cores

are 100%) during the first minutes vs maven build which let me do

something

else. Then all the consumed time which makes gradle not that fast is

about

python. Will try to send figures later today.

Le 10 nov. 2017 00:10, "Lukasz Cwik" a

écrit

I wouldn't mind merging this change in so I could setup those Gradle

Jenkins precommits.

As per our contribution guidelines, any committer willing to sign

off

the PR?

On Thu, Nov 9, 2017 at 2:12 PM, Romain Manni-Bucau <

rmannibu...@gmail.com>

wrote:

Le 9 nov. 2017 21:31, "Kenneth Knowles"

écrit :

Keep in mind that a clean build is unusual during development (it

common

for mvn use and that is a bug) and also not necessary for

precommits

the

build tool is correct enough that caching is safe. So while this

number

matters, it is not the most important.

Not sure, in dev you bypass the build tool most of the time

anyway -

thanks

to IDE or other shortcuts - but not on PR and CI. Keep in mind

that

not

doing a clean and killing gradle daemon makes the build not

reproducible

and therefore useful :(. Starting to build from a subpart of the

reactor

with the mentionned mvn plugin for instance - can be nice on some

travis if the caching is well configured but still not a guarantee

the

build is "green".

My trade

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-27 Thread Romain Manni-Bucau

Le 27 nov. 2017 23:07, "Eugene Kirpichov"  a
écrit :

On Mon, Nov 27, 2017 at 11:52 AM Romain Manni-Bucau 
wrote:

> 2017-11-27 20:26 GMT+01:00 Lukasz Cwik :
> > Romain, as mentioned earlier, I identified that Maven was slower because
> it
> > needed to finish building the entire module before dependent modules
> could
> > start which included running tests, performing checkstyle, etc...
> > Gradle is able to increase the parallelism of the build process since it
> > has task driven parallelism so as long as the files are compiled, the
> > dependent projects can start.
>
> This means we can implement a maven graph builder which is better than
> the default one - surely with a thread safe local repo - and
> contribute it back to solve it durably.
>

That would be indeed a wonderful contribution to the Java ecosystem as a
whole, but I'd like to reiterate that Maven has existed for very long, I'm
sure a lot of people have noticed its poor parallelization in the
meanwhile, and still nobody has even tried doing this AFAIK.

So it seems unlikely that this is doable in a reasonable timeframe, and I
don't think that the possibility of this is a strong argument against
switching to Gradle.


Agree. Just to clarify: my strong argument against gradle is that it is not
enterprise mainstream IMHO - dont forget github projects are a lot about
personal tests, small projects, so hard to use all the repos as a valid
stat - which leads to enhancing mvn rather than switching.

Again, as an ASF project, not even contacting our colleagues is not very
sane so I would go and check with them rather than just doing like a github
project maintained by a single company.

Side note on previous mail: mvn extensibility is "mature". Mojo let you do
what you want and you can customize the lifecycle as well. The main
difference is a standardized build (descriptive) vs a fully custom and "by
developer" styled script.



>
> >
> > Maven and Gradle are both heavily used since there are ~146k Maven
> projects
> > on Github while there are ~122k Gradle project on Github. Do you have
> data
> > which shows that Maven is significantly more "mainstream"?
>
> Yep, project i worked on in companies using gradle: 0, all were based
> on maven and maven was "tool-ed" versus gradle was "best effort" in
> term of plugins.
> Now  - with my EE background - I can guarantee you gradle is not able
> to handle properly its build since it flatten the classpath and
> plugins conflicts very quickly (their plugin dependency feature never
> worked and almost no plugin impl it correctly).
>
> Wonder if it is easy to have the ASF stats, anyone knows?
>
> >
> > I believe we want a rich multi-language SDK and community and feel as
> > though it would be unwise to treat non JDK based languages as second
> class.
>
> Hmm, not sure how it is related to the build tool since Maven and
> Gradle have the same level of support - actually surprsingly maven is
> better for js and surely as bad as gradle for others - or here again
> we can create plugin like the frontend-maven-plugin if needed for
> other languages.
>
> That said it can be an interesting other thread since people consuming
> these languages will probably want their mainstream build tool and a
> "standard" repository layout rather than a java one. But this is
> harder to measure.
>
> >
> > On Mon, Nov 27, 2017 at 11:00 AM, Romain Manni-Bucau <
> rmannibu...@gmail.com>
> > wrote:
> >
> >> Hi Lukasz,
> >>
> >> Did you manage to identify how maven was slower and test tesla stuff
> >> and potentially a few other fixes?
> >>
> >> Side note: figures without python can be interesting cause locally - =
> >> for me - python tends to flatten the figures whereas I get something
> >> close to your conclusions without python part.
> >>
> >> My point is mainly that switching now on gradle and being back on
> >> maven in a few months cause gradle ecosystem is far to support java 9
> >> - or any other volatile reason like this one - is probably not a good
> >> choice for a community. Maven is way more mainstream than gradle so
> >> helps to encourage people to contribute - vs gradle will increase the
> >> step to do it.
> >>
> >> I'd like to be sure before a switch that it is a one way decision and
> >> that the build tool was not just challenged by itself and its current
> >> state but also in the way it could be improved (= its community and
> >> potentially some local hacks).
> >>
> >> Romain Manni-Bucau
> >> @rmannibucau |  Blog | Old Blog | Github | LinkedIn
> >>
> >>
> >> 2017-11-27 19:46 GMT+01:00 Lukasz Cwik :
> >> > I have collected data by running several builds against master using
> >> Gradle
> >> > and Maven without using Gradle's support for incremental builds.
> >> >
> >> > Gradle (mins)
> >> > min: 25.04
> >> > max: 160.14
> >> > median: 45.78
> >> > average: 52.19
> >> > stdev: 30.80
> >> >
> >> > Maven (mins)
> >> > min: 56.86
> >> > max: 216.55 (actually > 240 mins because this data does not include
> >> > time

Re: [discuss] java profile

2017-11-27 Thread Romain Manni-Bucau

Hmm, no.

Incremental build is never correctly implemented cause there is just no way
to detect some dependencies statically with java code - or any dynamic
language.

Side note: same applies for gradle daemon usage BTW.

After if the list is not maintained it is a bug at the same level than
coding a toString() with "null.toString()". This is not very hard to handle
the list of modules and worse case a mvnextension can make it coded if you
feel more comfortable with this kind of solution.

Le 27 nov. 2017 23:12, "Lukasz Cwik"  a écrit :

> Manually whitelisting/blacklisting sub-modules is error prone since it
> hides issues due to incorrectly maintaining that list is the same argument
> as if the build process doesn't correctly invoke an incremental build
> process.
>
> On Mon, Nov 27, 2017 at 1:45 PM, Romain Manni-Bucau  >
> wrote:
>
> > Well for validation builds- pre PR, incremental support is pointless
> since
> > it easily hides issues die to caching so a solution saving half of the
> > build without loosing anuyhing would still be good IMHO.
> >
> > Le 27 nov. 2017 21:12, "Lukasz Cwik"  a écrit
> :
> >
> > > Incremental builds aren't correctly setup right now so your likely to
> see
> > > Python/Go rebuild even if there were no changes. See
> > > https://issues.apache.org/jira/browse/BEAM-3253
> > >
> > > On Mon, Nov 27, 2017 at 11:46 AM, Romain Manni-Bucau <
> > > rmannibu...@gmail.com>
> > > wrote:
> > >
> > > > that was the goal: validate there was no side effect of the changes
> on
> > > > the whole project. Now the "not java" part of the build will not be
> > > > impacted by java changed so this is the part i want to skip since it
> > > > takes a lot of time and I have guarantees it is safe to skip them.
> > > >
> > > > Romain Manni-Bucau
> > > > @rmannibucau |  Blog | Old Blog | Github | LinkedIn
> > > >
> > > >
> > > > 2017-11-27 20:28 GMT+01:00 Lukasz Cwik :
> > > > > Romain, that will build the entire project. I think you want to
> > execute
> > > > > (from the root of the project):
> > > > > ./gradlew :beam-sdks-parent:beam-sdks-python:build
> > > > >
> > > > > On Mon, Nov 27, 2017 at 11:25 AM, Romain Manni-Bucau <
> > > > rmannibu...@gmail.com>
> > > > > wrote:
> > > > >
> > > > >> gradle build --no-daemon
> > > > >>
> > > > >> (with gradle 4.2)
> > > > >>
> > > > >> Romain Manni-Bucau
> > > > >> @rmannibucau |  Blog | Old Blog | Github | LinkedIn
> > > > >>
> > > > >>
> > > > >> 2017-11-27 20:21 GMT+01:00 Kenneth Knowles  > >:
> > > > >> > What is the gradle command you are using to build just the
> Python
> > > SDK?
> > > > >> >
> > > > >> > On Mon, Nov 27, 2017 at 11:19 AM, Romain Manni-Bucau <
> > > > >> rmannibu...@gmail.com>
> > > > >> > wrote:
> > > > >> >
> > > > >> >> Hmm,
> > > > >> >>
> > > > >> >> issue is the same with gradle (locally python build takes 15mn
> > > alone
> > > > >> >> which is as much as the java build and it is not parallelized I
> > > > think)
> > > > >> >>
> > > > >> >> pl is not as smooth since it means doing it on each command
> > whereas
> > > > >> >> the proposal is automatically activated through settings.xml
> > > > >> >>
> > > > >> >> Romain Manni-Bucau
> > > > >> >> @rmannibucau |  Blog | Old Blog | Github | LinkedIn
> > > > >> >>
> > > > >> >>
> > > > >> >> 2017-11-27 20:07 GMT+01:00 Kenneth Knowles
> >  > > >:
> > > > >> >> > I think you can already mostly do this with mvn -pl sdks/XYZ
> > -am
> > > > >> -amd. I
> > > > >> >> > think that we have other work (gradle support) underway that
> > will
> > > > make
> > > > >> >> this
> > > > >> >> > a non-issue since gradle automatically does even better than
> > the
> > > > >> profile
> > > > >> >> or
> > > > >> >> > -am -amd.
> > > > >> >> >
> > > > >> >> > On Mon, Nov 27, 2017 at 11:01 AM, Romain Manni-Bucau <
> > > > >> >> rmannibu...@gmail.com>
> > > > >> >> > wrote:
> > > > >> >> >
> > > > >> >> >> Hi guys,
> > > > >> >> >>
> > > > >> >> >> java/python/go/xxx support is great but as a developer you
> > > rarely
> > > > >> hack
> > > > >> >> >> on them all.
> > > > >> >> >>
> > > > >> >> >> For that reason I opened https://github.com/apache/
> > > beam/pull/4173
> > > > .
> > > > >> >> >>
> > > > >> >> >> Goal is to give each developer a way to build the whole
> > project
> > > > and
> > > > >> >> >> all the code he can impact at once but without caring of the
> > > code
> > > > he
> > > > >> >> >> doesn't modify at all - other languages.
> > > > >> >> >>
> > > > >> >> >> Wdyt?
> > > > >> >> >>
> > > > >> >> >> Romain Manni-Bucau
> > > > >> >> >> @rmannibucau |  Blog | Old Blog | Github | LinkedIn
> > > > >> >> >>
> > > > >> >>
> > > > >>
> > > >
> > >
> >
>

Re: GitBox down

2017-11-27 Thread Jean-Baptiste Onofré


Good point.

I will prepare a PR for that.

Thanks !
Regards
JB

On 11/27/2017 08:58 PM, Reuven Lax wrote:

We need to update the contribution guide as well.

On Nov 27, 2017 10:30 AM, "Jean-Baptiste Onofré"  wrote:


No I don't think so,

In your case, it's because you still refering git-wip-us (probably from
): you have to use gitbox instead.

I already updated master branch but I didn't update the release branch.

Regards
JB

On 11/27/2017 07:02 PM, Reuven Lax wrote:


Does this have anything to do with the fact that I'm starting to get:

fatal: repository 'https://git-wip-us.apache.org/repos/asf/beam.git/' not
found

On Sun, Nov 26, 2017 at 10:37 PM, Jean-Baptiste Onofré 
wrote:

Hi,


it seems GitBox is down (timing out).

I don't see anything on status.apache.org, I will ping INFRA.

Regards
JB
--
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com





--
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com





--
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

1 2 >

1 - 100 of 140 matches

Mail list logo