Re: Summary of meeting about Maven performance improvements

2019-04-25 Thread Hervé BOUTEMY 
when your improvements are more related to some plugins, it's another type of 
issue: currently, the issues that are worked on are issues on Maven core 
preparing the full reactor model in memory, when there are many modules, 
dependencies, dependencyManagement, eventually with high depth.

But we're not yet at the stage of measuring the run after that preparation = 
when the plugins start working,. Key criterias here will probably be the 
number of source files per module, resources files, and so on. And perhaps 
some improvements here will come from core, but probably often more at plugin 
level.

Keeping a common view and common energy will be required to avoid someone 
exhausting alone on one aspect, staying misunderstood by others

Regards,

Hervé

Le mercredi 24 avril 2019, 13:29:29 CEST Jonathan Haber a écrit :
> > We need to find out who is interested in these kind improvements inside
> > the Maven community.
> 
> Just wanted to throw my two cents in. My company is a relatively large
> Maven user and we're very interested in these sorts of improvements. We've
> tried to upstream improvements in the past, but have been a bit discouraged
> by patches/PRs stagnating. So we mostly end up forking plugins and using
> those forks internally, which is a shame because no one else in the Maven
> community gets to benefit. It sounds like this customer did something
> similar with their performance improvements. So you have Maven users who
> are ready, willing, and able to contribute improvements but get defeated by
> the process, which is a shame. Obviously the Maven team has finite
> resources so I'm not suggesting that there's a trivial answer.
> 
> On Wed, Apr 24, 2019 at 4:51 AM Benedikt Ritter  wrote:
> > Hello,
> > 
> > this is a summary of a video conference call that happened yesterday
> > (April
> > 24).
> > 
> > Topic:
> > Discussion about performance improvements that have been proposed by
> > Stefan
> > Oehme, namely:
> > 
> > - [MNG-6638] - Prevent reparsing POMs in MavenMetadataSource (
> > https://github.com/apache/maven/pull/244)
> > 
> > - [MNG-6633] - Reduce memory usage of excludes (
> > https://github.com/apache/maven/pull/243)
> > 
> > - Speed up project discovery (
> > https://github.com/apache/maven/pull/242)
> > 
> > - Make location handling more
> > memory efficient (https://github.com/codehaus-plexus/modello/pull/31
> > 
> > )
> > 
> > The goal of this call was to give some more insights into how Stefan found
> > the improvements and to better understand what is missing before these
> > changes be merged.
> > 
> > Attendees of the call:
> > - Benedikt Ritter (Gradle Inc.)
> > - Stefan Oehme (Gradle Inc.)
> > - Robert Scholte (Apache Maven Team)
> > - Hervé Boutemy (Apache Maven Team; joined about half an hour after the
> > call started)
> > 
> > Summary:
> > 
> > Stefan gave some insights into how he discovered bottlenecks in Maven:
> > 
> > -
> > 
> > One of our customers has a huge Maven build:
> > -
> > 
> > Lots of sub projects (2000)
> > -
> > 
> > Lots of entries in dependency management (4000)
> > -
> > 
> > Results in a lot of garbage collection
> > -
> > 
> > Problems discovered in that build:
> > -
> > 
> > Re-parsing project POMs during dependency resolution
> > -
> > 
> > Model objects are too large because of location tracking
> > -
> > 
> > Low-level bottlenecks in project discovery (especially version
> > parsing)
> > -
> > 
> > Customer now has a Maven fork with the proposed changes included:
> > -
> > 
> > 1h 50min, 12GB RAM without changes
> > -
> > 
> > 45min, 8GB RAM with changes
> > 
> > 
> > Robert:
> > 
> > -
> > 
> > How to ensure that improvements are not broken?
> > -
> > 
> > No answer to how to test this
> > 
> > 
> > Stefan gave some insights into how performance testing works in the Gradle
> > project:
> > 
> > -
> > 
> > Build has a project generator
> > -
> > 
> > Create different projects in different shapes (e.g. lots of subprojects,
> > deeply nested projects) during the build
> > -
> > 
> > Download old Gradle version and run the build on generated projects
> > -
> > 
> > Run build again with current Gradle version
> > -
> > 
> > Compare results
> > -
> > 
> > use statistic methods to filter out variance
> > -
> > 
> > Downside to this approach is that it requires a lot of computing
> > resources
> > 
> > More information can be found on GitHub:
> > https://github.com/gradle/gradle/tree/master/subprojects/performance
> > 
> > The corresponding TeamCity build can be found here:
> > 
> > https://builds.gradle.org/viewLog.html?buildId=22179604=Gradle
> > _Check_PerformanceExperimentCoordinator=report_project941_Performance&
> > branch_Gradle_Check_Stage_ReadyforRelease=master
> >

Re: Summary of meeting about Maven performance improvements

2019-04-24 Thread Mickael Istria 
On Wed, Apr 24, 2019 at 9:20 PM Martijn Dashorst 
wrote:

> It has a typical long build time, but not a very complicated setup
> (e.g. the depth of nested projects is about 3 max)


I think Maven should aim (and generate dummy projects for
testing/benchmarking) for even more complex projects. Apache Camel for
example has a majority of modules with depth being 7 IIRC. The depth is the
cause of many performance issues (some of them got fixed in 3.6.1), depth
is one of the main criteria that should be taken into account in a
performance improvement plan.

Re: Summary of meeting about Maven performance improvements

2019-04-24 Thread Martijn Dashorst 
There's this umbrella of projects called Wicket Stuff at github.

https://github.com/wicketstuff/core

It has a typical long build time, but not a very complicated setup
(e.g. the depth of nested projects is about 3 max)

You might consider using this as a public benchmark.

Looking forward to improvements in build times.

Martijn

On Wed, Apr 24, 2019 at 8:30 PM Arnaud Héritier  wrote:
>
> Great findings then!
> It’s not easy to catch and my experience is that such big projects are
> often in corporate environments which aren’t open at all and not really
> working with oss projects thus it’s not surprising to have this surfacing
> on Gradle support side.
>
> Le mer. 24 avr. 2019 à 19:09, Karl Heinz Marbaise  a
> écrit :
>
> > Hi,
> >
> > On 24.04.19 17:52, Robert Scholte wrote:
> > > On Wed, 24 Apr 2019 12:45:42 +0200, Arnaud Héritier
> > >  wrote:
> > >
> > >> Just for my knowledge, is it a regression in a recent version or it is
> > >> like
> > >> this for ages ?
> > >
> > > It is like this for ages. AFAIK nobody of the team monitored memory
> > > consumption, focus was on correct + "readable" code.
> >
> > I had done some investigations a longer time ago...related to MNG-6030
> > https://github.com/khmarbaise/maven-test-project-generator
> >
> > and yes it's not a regression...
> >
> > Kind regards
> > Karl Heinz
> >
> > >
> > > Robert
> > >
> > >>
> > >>
> > >> Le mer. 24 avr. 2019 à 10:56, Benedikt Ritter  a
> > >> écrit :
> > >>
> > >>> Am Mi., 24. Apr. 2019 um 10:50 Uhr schrieb Benedikt Ritter <
> > >>> brit...@apache.org>:
> > >>>
> > >>> > Hello,
> > >>> >
> > >>> > this is a summary of a video conference call that happened yesterday
> > >>> > (April 24).
> > >>> >
> > >>>
> > >>> Sorry, actually yesterday was April 23... :o)
> > >>>
> > >>>
> > >>> >
> > >>> > Topic:
> > >>> > Discussion about performance improvements that have been proposed by
> > >>> > Stefan Oehme, namely:
> > >>> >
> > >>> > - [MNG-6638] - Prevent reparsing POMs in MavenMetadataSource (
> > >>> > https://github.com/apache/maven/pull/244)
> > >>> > - [MNG-6633] - Reduce memory usage of excludes (
> > >>> > https://github.com/apache/maven/pull/243) - Speed up project
> > >>> discovery (
> > >>> > https://github.com/apache/maven/pull/242) - Make location handling
> > >>> more
> > >>> > memory efficient (https://github.com/codehaus-plexus/modello/pull/31
> > )
> > >>> >
> > >>> > The goal of this call was to give some more insights into how Stefan
> > >>> found
> > >>> > the improvements and to better understand what is missing before
> > these
> > >>> > changes be merged.
> > >>> >
> > >>> > Attendees of the call:
> > >>> > - Benedikt Ritter (Gradle Inc.)
> > >>> > - Stefan Oehme (Gradle Inc.)
> > >>> > - Robert Scholte (Apache Maven Team)
> > >>> > - Hervé Boutemy (Apache Maven Team; joined about half an hour after
> > >>> the
> > >>> > call started)
> > >>> >
> > >>> > Summary:
> > >>> >
> > >>> > Stefan gave some insights into how he discovered bottlenecks in
> > Maven:
> > >>> >
> > >>> >-
> > >>> >
> > >>> >One of our customers has a huge Maven build:
> > >>> >-
> > >>> >
> > >>> >   Lots of sub projects (2000)
> > >>> >   -
> > >>> >
> > >>> >   Lots of entries in dependency management (4000)
> > >>> >   -
> > >>> >
> > >>> >   Results in a lot of garbage collection
> > >>> >   -
> > >>> >
> > >>> >Problems discovered in that build:
> > >>> >-
> > >>> >
> > >>> >   Re-parsing project POMs during dependency resolution
> > >>> >   -
> > >>> >
> > >>> >   Model objects are too large because of location tracking
> > >>> >   -
> > >>> >
> > >>> >   Low-level bottlenecks in project discovery (especially version
> > >>> >   parsing)
> > >>> >   -
> > >>> >
> > >>> >Customer now has a Maven fork with the proposed changes included:
> > >>> >-
> > >>> >
> > >>> >   1h 50min, 12GB RAM without changes
> > >>> >   -
> > >>> >
> > >>> >   45min, 8GB RAM with changes
> > >>> >
> > >>> >
> > >>> > Robert:
> > >>> >
> > >>> >-
> > >>> >
> > >>> >How to ensure that improvements are not broken?
> > >>> >-
> > >>> >
> > >>> >No answer to how to test this
> > >>> >
> > >>> >
> > >>> > Stefan gave some insights into how performance testing works in the
> > >>> Gradle
> > >>> > project:
> > >>> >
> > >>> >-
> > >>> >
> > >>> >Build has a project generator
> > >>> >-
> > >>> >
> > >>> >Create different projects in different shapes (e.g. lots of
> > >>> >subprojects, deeply nested projects) during the build
> > >>> >-
> > >>> >
> > >>> >Download old Gradle version and run the build on generated
> > projects
> > >>> >-
> > >>> >
> > >>> >Run build again with current Gradle version
> > >>> >-
> > >>> >
> > >>> >Compare results
> > >>> >-
> > >>> >
> > >>> >use statistic methods to filter out variance
> > >>> >-
> > >>> >
> > >>> >Downside to this approach is that it requires a lot

Re: Summary of meeting about Maven performance improvements

2019-04-24 Thread Anders Hammar 
Yes, there's been reports some time ago about performance degradation in
newer versions. I'm not sure if there was an actual jira ticket or just
mail to the list. Iirc there was no actual end result to resolve it from
that though.

/Anders (mobile)

Den ons 24 apr. 2019 19:09Karl Heinz Marbaise  skrev:

> Hi,
>
> On 24.04.19 17:52, Robert Scholte wrote:
> > On Wed, 24 Apr 2019 12:45:42 +0200, Arnaud Héritier
> >  wrote:
> >
> >> Just for my knowledge, is it a regression in a recent version or it is
> >> like
> >> this for ages ?
> >
> > It is like this for ages. AFAIK nobody of the team monitored memory
> > consumption, focus was on correct + "readable" code.
>
> I had done some investigations a longer time ago...related to MNG-6030
> https://github.com/khmarbaise/maven-test-project-generator
>
> and yes it's not a regression...
>
> Kind regards
> Karl Heinz
>
> >
> > Robert
> >
> >>
> >>
> >> Le mer. 24 avr. 2019 à 10:56, Benedikt Ritter  a
> >> écrit :
> >>
> >>> Am Mi., 24. Apr. 2019 um 10:50 Uhr schrieb Benedikt Ritter <
> >>> brit...@apache.org>:
> >>>
> >>> > Hello,
> >>> >
> >>> > this is a summary of a video conference call that happened yesterday
> >>> > (April 24).
> >>> >
> >>>
> >>> Sorry, actually yesterday was April 23... :o)
> >>>
> >>>
> >>> >
> >>> > Topic:
> >>> > Discussion about performance improvements that have been proposed by
> >>> > Stefan Oehme, namely:
> >>> >
> >>> > - [MNG-6638] - Prevent reparsing POMs in MavenMetadataSource (
> >>> > https://github.com/apache/maven/pull/244)
> >>> > - [MNG-6633] - Reduce memory usage of excludes (
> >>> > https://github.com/apache/maven/pull/243) - Speed up project
> >>> discovery (
> >>> > https://github.com/apache/maven/pull/242) - Make location handling
> >>> more
> >>> > memory efficient (https://github.com/codehaus-plexus/modello/pull/31
> )
> >>> >
> >>> > The goal of this call was to give some more insights into how Stefan
> >>> found
> >>> > the improvements and to better understand what is missing before
> these
> >>> > changes be merged.
> >>> >
> >>> > Attendees of the call:
> >>> > - Benedikt Ritter (Gradle Inc.)
> >>> > - Stefan Oehme (Gradle Inc.)
> >>> > - Robert Scholte (Apache Maven Team)
> >>> > - Hervé Boutemy (Apache Maven Team; joined about half an hour after
> >>> the
> >>> > call started)
> >>> >
> >>> > Summary:
> >>> >
> >>> > Stefan gave some insights into how he discovered bottlenecks in
> Maven:
> >>> >
> >>> >-
> >>> >
> >>> >One of our customers has a huge Maven build:
> >>> >-
> >>> >
> >>> >   Lots of sub projects (2000)
> >>> >   -
> >>> >
> >>> >   Lots of entries in dependency management (4000)
> >>> >   -
> >>> >
> >>> >   Results in a lot of garbage collection
> >>> >   -
> >>> >
> >>> >Problems discovered in that build:
> >>> >-
> >>> >
> >>> >   Re-parsing project POMs during dependency resolution
> >>> >   -
> >>> >
> >>> >   Model objects are too large because of location tracking
> >>> >   -
> >>> >
> >>> >   Low-level bottlenecks in project discovery (especially version
> >>> >   parsing)
> >>> >   -
> >>> >
> >>> >Customer now has a Maven fork with the proposed changes included:
> >>> >-
> >>> >
> >>> >   1h 50min, 12GB RAM without changes
> >>> >   -
> >>> >
> >>> >   45min, 8GB RAM with changes
> >>> >
> >>> >
> >>> > Robert:
> >>> >
> >>> >-
> >>> >
> >>> >How to ensure that improvements are not broken?
> >>> >-
> >>> >
> >>> >No answer to how to test this
> >>> >
> >>> >
> >>> > Stefan gave some insights into how performance testing works in the
> >>> Gradle
> >>> > project:
> >>> >
> >>> >-
> >>> >
> >>> >Build has a project generator
> >>> >-
> >>> >
> >>> >Create different projects in different shapes (e.g. lots of
> >>> >subprojects, deeply nested projects) during the build
> >>> >-
> >>> >
> >>> >Download old Gradle version and run the build on generated
> projects
> >>> >-
> >>> >
> >>> >Run build again with current Gradle version
> >>> >-
> >>> >
> >>> >Compare results
> >>> >-
> >>> >
> >>> >use statistic methods to filter out variance
> >>> >-
> >>> >
> >>> >Downside to this approach is that it requires a lot of computing
> >>> >resources
> >>> >
> >>> > More information can be found on GitHub:
> >>> > https://github.com/gradle/gradle/tree/master/subprojects/performance
> >>> > The corresponding TeamCity build can be found here:
> >>> >
> >>>
> https://builds.gradle.org/viewLog.html?buildId=22179604=Gradle_Check_PerformanceExperimentCoordinator=report_project941_Performance_Gradle_Check_Stage_ReadyforRelease=master
> >>>
> >>> (use
> >>> > "Login as guest" to view)
> >>> >
> >>> > Robert:
> >>> >
> >>> >-
> >>> >
> >>> >What about measuring performance using instruction calls?
> >>> >
> >>> >
> >>> > Stefan:
> >>> >
> >>> >-
> >>> >
> >>> >The performance improvements we found were mostly about

Re: Summary of meeting about Maven performance improvements

2019-04-24 Thread Arnaud Héritier 
Great findings then!
It’s not easy to catch and my experience is that such big projects are
often in corporate environments which aren’t open at all and not really
working with oss projects thus it’s not surprising to have this surfacing
on Gradle support side.

Le mer. 24 avr. 2019 à 19:09, Karl Heinz Marbaise  a
écrit :

> Hi,
>
> On 24.04.19 17:52, Robert Scholte wrote:
> > On Wed, 24 Apr 2019 12:45:42 +0200, Arnaud Héritier
> >  wrote:
> >
> >> Just for my knowledge, is it a regression in a recent version or it is
> >> like
> >> this for ages ?
> >
> > It is like this for ages. AFAIK nobody of the team monitored memory
> > consumption, focus was on correct + "readable" code.
>
> I had done some investigations a longer time ago...related to MNG-6030
> https://github.com/khmarbaise/maven-test-project-generator
>
> and yes it's not a regression...
>
> Kind regards
> Karl Heinz
>
> >
> > Robert
> >
> >>
> >>
> >> Le mer. 24 avr. 2019 à 10:56, Benedikt Ritter  a
> >> écrit :
> >>
> >>> Am Mi., 24. Apr. 2019 um 10:50 Uhr schrieb Benedikt Ritter <
> >>> brit...@apache.org>:
> >>>
> >>> > Hello,
> >>> >
> >>> > this is a summary of a video conference call that happened yesterday
> >>> > (April 24).
> >>> >
> >>>
> >>> Sorry, actually yesterday was April 23... :o)
> >>>
> >>>
> >>> >
> >>> > Topic:
> >>> > Discussion about performance improvements that have been proposed by
> >>> > Stefan Oehme, namely:
> >>> >
> >>> > - [MNG-6638] - Prevent reparsing POMs in MavenMetadataSource (
> >>> > https://github.com/apache/maven/pull/244)
> >>> > - [MNG-6633] - Reduce memory usage of excludes (
> >>> > https://github.com/apache/maven/pull/243) - Speed up project
> >>> discovery (
> >>> > https://github.com/apache/maven/pull/242) - Make location handling
> >>> more
> >>> > memory efficient (https://github.com/codehaus-plexus/modello/pull/31
> )
> >>> >
> >>> > The goal of this call was to give some more insights into how Stefan
> >>> found
> >>> > the improvements and to better understand what is missing before
> these
> >>> > changes be merged.
> >>> >
> >>> > Attendees of the call:
> >>> > - Benedikt Ritter (Gradle Inc.)
> >>> > - Stefan Oehme (Gradle Inc.)
> >>> > - Robert Scholte (Apache Maven Team)
> >>> > - Hervé Boutemy (Apache Maven Team; joined about half an hour after
> >>> the
> >>> > call started)
> >>> >
> >>> > Summary:
> >>> >
> >>> > Stefan gave some insights into how he discovered bottlenecks in
> Maven:
> >>> >
> >>> >-
> >>> >
> >>> >One of our customers has a huge Maven build:
> >>> >-
> >>> >
> >>> >   Lots of sub projects (2000)
> >>> >   -
> >>> >
> >>> >   Lots of entries in dependency management (4000)
> >>> >   -
> >>> >
> >>> >   Results in a lot of garbage collection
> >>> >   -
> >>> >
> >>> >Problems discovered in that build:
> >>> >-
> >>> >
> >>> >   Re-parsing project POMs during dependency resolution
> >>> >   -
> >>> >
> >>> >   Model objects are too large because of location tracking
> >>> >   -
> >>> >
> >>> >   Low-level bottlenecks in project discovery (especially version
> >>> >   parsing)
> >>> >   -
> >>> >
> >>> >Customer now has a Maven fork with the proposed changes included:
> >>> >-
> >>> >
> >>> >   1h 50min, 12GB RAM without changes
> >>> >   -
> >>> >
> >>> >   45min, 8GB RAM with changes
> >>> >
> >>> >
> >>> > Robert:
> >>> >
> >>> >-
> >>> >
> >>> >How to ensure that improvements are not broken?
> >>> >-
> >>> >
> >>> >No answer to how to test this
> >>> >
> >>> >
> >>> > Stefan gave some insights into how performance testing works in the
> >>> Gradle
> >>> > project:
> >>> >
> >>> >-
> >>> >
> >>> >Build has a project generator
> >>> >-
> >>> >
> >>> >Create different projects in different shapes (e.g. lots of
> >>> >subprojects, deeply nested projects) during the build
> >>> >-
> >>> >
> >>> >Download old Gradle version and run the build on generated
> projects
> >>> >-
> >>> >
> >>> >Run build again with current Gradle version
> >>> >-
> >>> >
> >>> >Compare results
> >>> >-
> >>> >
> >>> >use statistic methods to filter out variance
> >>> >-
> >>> >
> >>> >Downside to this approach is that it requires a lot of computing
> >>> >resources
> >>> >
> >>> > More information can be found on GitHub:
> >>> > https://github.com/gradle/gradle/tree/master/subprojects/performance
> >>> > The corresponding TeamCity build can be found here:
> >>> >
> >>>
> https://builds.gradle.org/viewLog.html?buildId=22179604=Gradle_Check_PerformanceExperimentCoordinator=report_project941_Performance_Gradle_Check_Stage_ReadyforRelease=master
> >>>
> >>> (use
> >>> > "Login as guest" to view)
> >>> >
> >>> > Robert:
> >>> >
> >>> >-
> >>> >
> >>> >What about measuring performance using instruction calls?
> >>> >
> >>> >
> >>> > Stefan:
> >>> >
> >>> >-
> >>> >
> >>> >The performance improvements we found were

Re: Summary of meeting about Maven performance improvements

2019-04-24 Thread Karl Heinz Marbaise 

Hi,

On 24.04.19 17:52, Robert Scholte wrote:

On Wed, 24 Apr 2019 12:45:42 +0200, Arnaud Héritier
 wrote:


Just for my knowledge, is it a regression in a recent version or it is
like
this for ages ?


It is like this for ages. AFAIK nobody of the team monitored memory
consumption, focus was on correct + "readable" code.


I had done some investigations a longer time ago...related to MNG-6030
https://github.com/khmarbaise/maven-test-project-generator

and yes it's not a regression...

Kind regards
Karl Heinz



Robert




Le mer. 24 avr. 2019 à 10:56, Benedikt Ritter  a
écrit :


Am Mi., 24. Apr. 2019 um 10:50 Uhr schrieb Benedikt Ritter <
brit...@apache.org>:

> Hello,
>
> this is a summary of a video conference call that happened yesterday
> (April 24).
>

Sorry, actually yesterday was April 23... :o)


>
> Topic:
> Discussion about performance improvements that have been proposed by
> Stefan Oehme, namely:
>
> - [MNG-6638] - Prevent reparsing POMs in MavenMetadataSource (
> https://github.com/apache/maven/pull/244)
> - [MNG-6633] - Reduce memory usage of excludes (
> https://github.com/apache/maven/pull/243) - Speed up project
discovery (
> https://github.com/apache/maven/pull/242) - Make location handling
more
> memory efficient (https://github.com/codehaus-plexus/modello/pull/31)
>
> The goal of this call was to give some more insights into how Stefan
found
> the improvements and to better understand what is missing before these
> changes be merged.
>
> Attendees of the call:
> - Benedikt Ritter (Gradle Inc.)
> - Stefan Oehme (Gradle Inc.)
> - Robert Scholte (Apache Maven Team)
> - Hervé Boutemy (Apache Maven Team; joined about half an hour after
the
> call started)
>
> Summary:
>
> Stefan gave some insights into how he discovered bottlenecks in Maven:
>
>    -
>
>    One of our customers has a huge Maven build:
>    -
>
>   Lots of sub projects (2000)
>   -
>
>   Lots of entries in dependency management (4000)
>   -
>
>   Results in a lot of garbage collection
>   -
>
>    Problems discovered in that build:
>    -
>
>   Re-parsing project POMs during dependency resolution
>   -
>
>   Model objects are too large because of location tracking
>   -
>
>   Low-level bottlenecks in project discovery (especially version
>   parsing)
>   -
>
>    Customer now has a Maven fork with the proposed changes included:
>    -
>
>   1h 50min, 12GB RAM without changes
>   -
>
>   45min, 8GB RAM with changes
>
>
> Robert:
>
>    -
>
>    How to ensure that improvements are not broken?
>    -
>
>    No answer to how to test this
>
>
> Stefan gave some insights into how performance testing works in the
Gradle
> project:
>
>    -
>
>    Build has a project generator
>    -
>
>    Create different projects in different shapes (e.g. lots of
>    subprojects, deeply nested projects) during the build
>    -
>
>    Download old Gradle version and run the build on generated projects
>    -
>
>    Run build again with current Gradle version
>    -
>
>    Compare results
>    -
>
>    use statistic methods to filter out variance
>    -
>
>    Downside to this approach is that it requires a lot of computing
>    resources
>
> More information can be found on GitHub:
> https://github.com/gradle/gradle/tree/master/subprojects/performance
> The corresponding TeamCity build can be found here:
>
https://builds.gradle.org/viewLog.html?buildId=22179604=Gradle_Check_PerformanceExperimentCoordinator=report_project941_Performance_Gradle_Check_Stage_ReadyforRelease=master

(use
> "Login as guest" to view)
>
> Robert:
>
>    -
>
>    What about measuring performance using instruction calls?
>
>
> Stefan:
>
>    -
>
>    The performance improvements we found were mostly about garbage
being
>    created
>    -
>
>    Measuring using instruction calls is interesting
>    -
>
>    ... but it is also very machine dependent
>
>
> Robert:
>
>    -
>
>    We need to find out who is interested in these kind improvements
>    inside the Maven community.
>    -
>
>    Build a community of people who would like to work on these kind of
>    things.
>
>
> Stefan:
>
>    -
>
>    It's easy to get started. We just used open source tools:
>    -
>
>    We used async-profiler for measuring things (
>    https://github.com/jvm-profiling-tools/async-profiler)
>    -
>
>    Heap dumps for analyzing memory usage
>
> To get started with performance tests in the maven project:
>
>    -
>
>    Start with only a few test projects
>    -
>
>    The Gradle generator is Apache License v2 and can be used as a
>    starting point to generate a big maven project
>
>
> Hervé:
>
>    -
>
>    PRs should be merged soon
>    -
>
>    Discussion need to be resolved
>    -
>
>    Why was the PR not merged after the discussion and resolving all
>    issues with the code?
>    -
>
>    Hervé will take care that the changes are merged soon
>
>
> Thank you!
> Benedikt
>

Re: Summary of meeting about Maven performance improvements

2019-04-24 Thread Robert Scholte 
On Wed, 24 Apr 2019 12:45:42 +0200, Arnaud Héritier   
wrote:


Just for my knowledge, is it a regression in a recent version or it is  
like

this for ages ?


It is like this for ages. AFAIK nobody of the team monitored memory  
consumption, focus was on correct + "readable" code.


Robert




Le mer. 24 avr. 2019 à 10:56, Benedikt Ritter  a  
écrit :



Am Mi., 24. Apr. 2019 um 10:50 Uhr schrieb Benedikt Ritter <
brit...@apache.org>:

> Hello,
>
> this is a summary of a video conference call that happened yesterday
> (April 24).
>

Sorry, actually yesterday was April 23... :o)


>
> Topic:
> Discussion about performance improvements that have been proposed by
> Stefan Oehme, namely:
>
> - [MNG-6638] - Prevent reparsing POMs in MavenMetadataSource (
> https://github.com/apache/maven/pull/244)
> - [MNG-6633] - Reduce memory usage of excludes (
> https://github.com/apache/maven/pull/243) - Speed up project  
discovery (
> https://github.com/apache/maven/pull/242) - Make location handling  
more

> memory efficient (https://github.com/codehaus-plexus/modello/pull/31)
>
> The goal of this call was to give some more insights into how Stefan
found
> the improvements and to better understand what is missing before these
> changes be merged.
>
> Attendees of the call:
> - Benedikt Ritter (Gradle Inc.)
> - Stefan Oehme (Gradle Inc.)
> - Robert Scholte (Apache Maven Team)
> - Hervé Boutemy (Apache Maven Team; joined about half an hour after  
the

> call started)
>
> Summary:
>
> Stefan gave some insights into how he discovered bottlenecks in Maven:
>
>-
>
>One of our customers has a huge Maven build:
>-
>
>   Lots of sub projects (2000)
>   -
>
>   Lots of entries in dependency management (4000)
>   -
>
>   Results in a lot of garbage collection
>   -
>
>Problems discovered in that build:
>-
>
>   Re-parsing project POMs during dependency resolution
>   -
>
>   Model objects are too large because of location tracking
>   -
>
>   Low-level bottlenecks in project discovery (especially version
>   parsing)
>   -
>
>Customer now has a Maven fork with the proposed changes included:
>-
>
>   1h 50min, 12GB RAM without changes
>   -
>
>   45min, 8GB RAM with changes
>
>
> Robert:
>
>-
>
>How to ensure that improvements are not broken?
>-
>
>No answer to how to test this
>
>
> Stefan gave some insights into how performance testing works in the
Gradle
> project:
>
>-
>
>Build has a project generator
>-
>
>Create different projects in different shapes (e.g. lots of
>subprojects, deeply nested projects) during the build
>-
>
>Download old Gradle version and run the build on generated projects
>-
>
>Run build again with current Gradle version
>-
>
>Compare results
>-
>
>use statistic methods to filter out variance
>-
>
>Downside to this approach is that it requires a lot of computing
>resources
>
> More information can be found on GitHub:
> https://github.com/gradle/gradle/tree/master/subprojects/performance
> The corresponding TeamCity build can be found here:
>
https://builds.gradle.org/viewLog.html?buildId=22179604=Gradle_Check_PerformanceExperimentCoordinator=report_project941_Performance_Gradle_Check_Stage_ReadyforRelease=master
(use
> "Login as guest" to view)
>
> Robert:
>
>-
>
>What about measuring performance using instruction calls?
>
>
> Stefan:
>
>-
>
>The performance improvements we found were mostly about garbage  
being

>created
>-
>
>Measuring using instruction calls is interesting
>-
>
>... but it is also very machine dependent
>
>
> Robert:
>
>-
>
>We need to find out who is interested in these kind improvements
>inside the Maven community.
>-
>
>Build a community of people who would like to work on these kind of
>things.
>
>
> Stefan:
>
>-
>
>It's easy to get started. We just used open source tools:
>-
>
>We used async-profiler for measuring things (
>https://github.com/jvm-profiling-tools/async-profiler)
>-
>
>Heap dumps for analyzing memory usage
>
> To get started with performance tests in the maven project:
>
>-
>
>Start with only a few test projects
>-
>
>The Gradle generator is Apache License v2 and can be used as a
>starting point to generate a big maven project
>
>
> Hervé:
>
>-
>
>PRs should be merged soon
>-
>
>Discussion need to be resolved
>-
>
>Why was the PR not merged after the discussion and resolving all
>issues with the code?
>-
>
>Hervé will take care that the changes are merged soon
>
>
> Thank you!
> Benedikt
>


-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org

Re: Summary of meeting about Maven performance improvements

2019-04-24 Thread Jonathan Haber 
>
> We need to find out who is interested in these kind improvements inside
> the Maven community.


Just wanted to throw my two cents in. My company is a relatively large
Maven user and we're very interested in these sorts of improvements. We've
tried to upstream improvements in the past, but have been a bit discouraged
by patches/PRs stagnating. So we mostly end up forking plugins and using
those forks internally, which is a shame because no one else in the Maven
community gets to benefit. It sounds like this customer did something
similar with their performance improvements. So you have Maven users who
are ready, willing, and able to contribute improvements but get defeated by
the process, which is a shame. Obviously the Maven team has finite
resources so I'm not suggesting that there's a trivial answer.

On Wed, Apr 24, 2019 at 4:51 AM Benedikt Ritter  wrote:

> Hello,
>
> this is a summary of a video conference call that happened yesterday (April
> 24).
>
> Topic:
> Discussion about performance improvements that have been proposed by Stefan
> Oehme, namely:
>
> - [MNG-6638] - Prevent reparsing POMs in MavenMetadataSource (
> https://github.com/apache/maven/pull/244)
> 
> - [MNG-6633] - Reduce memory usage of excludes (
> https://github.com/apache/maven/pull/243)
> 
> - Speed up project discovery (
> https://github.com/apache/maven/pull/242)
> 
> - Make location handling more
> memory efficient (https://github.com/codehaus-plexus/modello/pull/31
> 
> )
>
> The goal of this call was to give some more insights into how Stefan found
> the improvements and to better understand what is missing before these
> changes be merged.
>
> Attendees of the call:
> - Benedikt Ritter (Gradle Inc.)
> - Stefan Oehme (Gradle Inc.)
> - Robert Scholte (Apache Maven Team)
> - Hervé Boutemy (Apache Maven Team; joined about half an hour after the
> call started)
>
> Summary:
>
> Stefan gave some insights into how he discovered bottlenecks in Maven:
>
> -
>
> One of our customers has a huge Maven build:
> -
>
> Lots of sub projects (2000)
> -
>
> Lots of entries in dependency management (4000)
> -
>
> Results in a lot of garbage collection
> -
>
> Problems discovered in that build:
> -
>
> Re-parsing project POMs during dependency resolution
> -
>
> Model objects are too large because of location tracking
> -
>
> Low-level bottlenecks in project discovery (especially version
> parsing)
> -
>
> Customer now has a Maven fork with the proposed changes included:
> -
>
> 1h 50min, 12GB RAM without changes
> -
>
> 45min, 8GB RAM with changes
>
>
> Robert:
>
> -
>
> How to ensure that improvements are not broken?
> -
>
> No answer to how to test this
>
>
> Stefan gave some insights into how performance testing works in the Gradle
> project:
>
> -
>
> Build has a project generator
> -
>
> Create different projects in different shapes (e.g. lots of subprojects,
> deeply nested projects) during the build
> -
>
> Download old Gradle version and run the build on generated projects
> -
>
> Run build again with current Gradle version
> -
>
> Compare results
> -
>
> use statistic methods to filter out variance
> -
>
> Downside to this approach is that it requires a lot of computing
> resources
>
> More information can be found on GitHub:
> https://github.com/gradle/gradle/tree/master/subprojects/performance
> 
> The corresponding TeamCity build can be found here:
>
> https://builds.gradle.org/viewLog.html?buildId=22179604=Gradle_Check_PerformanceExperimentCoordinator=report_project941_Performance_Gradle_Check_Stage_ReadyforRelease=master
> 
> (use
> "Login as guest" to view)
>
> Robert:
>
> -
>
> What about measuring performance using instruction calls?
>
>
> Stefan:
>
> -
>
> The performance improvements we found were mostly about garbage being
> created
> -
>
> Measuring using instruction calls is interesting
> -
>
> ... but it is also very machine dependent
>
>
> Robert:
>
> -
>
> We need to find out who is interested in these kind improvements inside
> the Maven community.
> -
>
> Build a community of people who would like to work on these kind of
> things.
>
>
> Stefan:
>
> -
>
> It's easy to get started. We just used open source tools:
> -
>
> We used async-profiler for measuring things (
> https://github.com/jvm-profiling-tools/async-profiler
> 
> )
> -
>
> Heap dumps for analyzing memory usage
>
> To get started with performance tests in the maven project:
>
> -
>
> Start with only a few test projects
> -
>
> The Gradle generator is Apache License v2 and can be

Re: Summary of meeting about Maven performance improvements

2019-04-24 Thread Arnaud Héritier 
Just for my knowledge, is it a regression in a recent version or it is like
this for ages ?


Le mer. 24 avr. 2019 à 10:56, Benedikt Ritter  a écrit :

> Am Mi., 24. Apr. 2019 um 10:50 Uhr schrieb Benedikt Ritter <
> brit...@apache.org>:
>
> > Hello,
> >
> > this is a summary of a video conference call that happened yesterday
> > (April 24).
> >
>
> Sorry, actually yesterday was April 23... :o)
>
>
> >
> > Topic:
> > Discussion about performance improvements that have been proposed by
> > Stefan Oehme, namely:
> >
> > - [MNG-6638] - Prevent reparsing POMs in MavenMetadataSource (
> > https://github.com/apache/maven/pull/244)
> > - [MNG-6633] - Reduce memory usage of excludes (
> > https://github.com/apache/maven/pull/243) - Speed up project discovery (
> > https://github.com/apache/maven/pull/242) - Make location handling more
> > memory efficient (https://github.com/codehaus-plexus/modello/pull/31)
> >
> > The goal of this call was to give some more insights into how Stefan
> found
> > the improvements and to better understand what is missing before these
> > changes be merged.
> >
> > Attendees of the call:
> > - Benedikt Ritter (Gradle Inc.)
> > - Stefan Oehme (Gradle Inc.)
> > - Robert Scholte (Apache Maven Team)
> > - Hervé Boutemy (Apache Maven Team; joined about half an hour after the
> > call started)
> >
> > Summary:
> >
> > Stefan gave some insights into how he discovered bottlenecks in Maven:
> >
> >-
> >
> >One of our customers has a huge Maven build:
> >-
> >
> >   Lots of sub projects (2000)
> >   -
> >
> >   Lots of entries in dependency management (4000)
> >   -
> >
> >   Results in a lot of garbage collection
> >   -
> >
> >Problems discovered in that build:
> >-
> >
> >   Re-parsing project POMs during dependency resolution
> >   -
> >
> >   Model objects are too large because of location tracking
> >   -
> >
> >   Low-level bottlenecks in project discovery (especially version
> >   parsing)
> >   -
> >
> >Customer now has a Maven fork with the proposed changes included:
> >-
> >
> >   1h 50min, 12GB RAM without changes
> >   -
> >
> >   45min, 8GB RAM with changes
> >
> >
> > Robert:
> >
> >-
> >
> >How to ensure that improvements are not broken?
> >-
> >
> >No answer to how to test this
> >
> >
> > Stefan gave some insights into how performance testing works in the
> Gradle
> > project:
> >
> >-
> >
> >Build has a project generator
> >-
> >
> >Create different projects in different shapes (e.g. lots of
> >subprojects, deeply nested projects) during the build
> >-
> >
> >Download old Gradle version and run the build on generated projects
> >-
> >
> >Run build again with current Gradle version
> >-
> >
> >Compare results
> >-
> >
> >use statistic methods to filter out variance
> >-
> >
> >Downside to this approach is that it requires a lot of computing
> >resources
> >
> > More information can be found on GitHub:
> > https://github.com/gradle/gradle/tree/master/subprojects/performance
> > The corresponding TeamCity build can be found here:
> >
> https://builds.gradle.org/viewLog.html?buildId=22179604=Gradle_Check_PerformanceExperimentCoordinator=report_project941_Performance_Gradle_Check_Stage_ReadyforRelease=master
> (use
> > "Login as guest" to view)
> >
> > Robert:
> >
> >-
> >
> >What about measuring performance using instruction calls?
> >
> >
> > Stefan:
> >
> >-
> >
> >The performance improvements we found were mostly about garbage being
> >created
> >-
> >
> >Measuring using instruction calls is interesting
> >-
> >
> >... but it is also very machine dependent
> >
> >
> > Robert:
> >
> >-
> >
> >We need to find out who is interested in these kind improvements
> >inside the Maven community.
> >-
> >
> >Build a community of people who would like to work on these kind of
> >things.
> >
> >
> > Stefan:
> >
> >-
> >
> >It's easy to get started. We just used open source tools:
> >-
> >
> >We used async-profiler for measuring things (
> >https://github.com/jvm-profiling-tools/async-profiler)
> >-
> >
> >Heap dumps for analyzing memory usage
> >
> > To get started with performance tests in the maven project:
> >
> >-
> >
> >Start with only a few test projects
> >-
> >
> >The Gradle generator is Apache License v2 and can be used as a
> >starting point to generate a big maven project
> >
> >
> > Hervé:
> >
> >-
> >
> >PRs should be merged soon
> >-
> >
> >Discussion need to be resolved
> >-
> >
> >Why was the PR not merged after the discussion and resolving all
> >issues with the code?
> >-
> >
> >Hervé will take care that the changes are merged soon
> >
> >
> > Thank you!
> > Benedikt
> >
>
-- 
-
Arnaud Héritier
http://aheritier.net
Mail/GTalk: aheritier AT gmail DOT com

Re: Summary of meeting about Maven performance improvements

2019-04-24 Thread Benedikt Ritter 
Am Mi., 24. Apr. 2019 um 10:50 Uhr schrieb Benedikt Ritter <
brit...@apache.org>:

> Hello,
>
> this is a summary of a video conference call that happened yesterday
> (April 24).
>

Sorry, actually yesterday was April 23... :o)


>
> Topic:
> Discussion about performance improvements that have been proposed by
> Stefan Oehme, namely:
>
> - [MNG-6638] - Prevent reparsing POMs in MavenMetadataSource (
> https://github.com/apache/maven/pull/244)
> - [MNG-6633] - Reduce memory usage of excludes (
> https://github.com/apache/maven/pull/243) - Speed up project discovery (
> https://github.com/apache/maven/pull/242) - Make location handling more
> memory efficient (https://github.com/codehaus-plexus/modello/pull/31)
>
> The goal of this call was to give some more insights into how Stefan found
> the improvements and to better understand what is missing before these
> changes be merged.
>
> Attendees of the call:
> - Benedikt Ritter (Gradle Inc.)
> - Stefan Oehme (Gradle Inc.)
> - Robert Scholte (Apache Maven Team)
> - Hervé Boutemy (Apache Maven Team; joined about half an hour after the
> call started)
>
> Summary:
>
> Stefan gave some insights into how he discovered bottlenecks in Maven:
>
>-
>
>One of our customers has a huge Maven build:
>-
>
>   Lots of sub projects (2000)
>   -
>
>   Lots of entries in dependency management (4000)
>   -
>
>   Results in a lot of garbage collection
>   -
>
>Problems discovered in that build:
>-
>
>   Re-parsing project POMs during dependency resolution
>   -
>
>   Model objects are too large because of location tracking
>   -
>
>   Low-level bottlenecks in project discovery (especially version
>   parsing)
>   -
>
>Customer now has a Maven fork with the proposed changes included:
>-
>
>   1h 50min, 12GB RAM without changes
>   -
>
>   45min, 8GB RAM with changes
>
>
> Robert:
>
>-
>
>How to ensure that improvements are not broken?
>-
>
>No answer to how to test this
>
>
> Stefan gave some insights into how performance testing works in the Gradle
> project:
>
>-
>
>Build has a project generator
>-
>
>Create different projects in different shapes (e.g. lots of
>subprojects, deeply nested projects) during the build
>-
>
>Download old Gradle version and run the build on generated projects
>-
>
>Run build again with current Gradle version
>-
>
>Compare results
>-
>
>use statistic methods to filter out variance
>-
>
>Downside to this approach is that it requires a lot of computing
>resources
>
> More information can be found on GitHub:
> https://github.com/gradle/gradle/tree/master/subprojects/performance
> The corresponding TeamCity build can be found here:
> https://builds.gradle.org/viewLog.html?buildId=22179604=Gradle_Check_PerformanceExperimentCoordinator=report_project941_Performance_Gradle_Check_Stage_ReadyforRelease=master
>  (use
> "Login as guest" to view)
>
> Robert:
>
>-
>
>What about measuring performance using instruction calls?
>
>
> Stefan:
>
>-
>
>The performance improvements we found were mostly about garbage being
>created
>-
>
>Measuring using instruction calls is interesting
>-
>
>... but it is also very machine dependent
>
>
> Robert:
>
>-
>
>We need to find out who is interested in these kind improvements
>inside the Maven community.
>-
>
>Build a community of people who would like to work on these kind of
>things.
>
>
> Stefan:
>
>-
>
>It's easy to get started. We just used open source tools:
>-
>
>We used async-profiler for measuring things (
>https://github.com/jvm-profiling-tools/async-profiler)
>-
>
>Heap dumps for analyzing memory usage
>
> To get started with performance tests in the maven project:
>
>-
>
>Start with only a few test projects
>-
>
>The Gradle generator is Apache License v2 and can be used as a
>starting point to generate a big maven project
>
>
> Hervé:
>
>-
>
>PRs should be merged soon
>-
>
>Discussion need to be resolved
>-
>
>Why was the PR not merged after the discussion and resolving all
>issues with the code?
>-
>
>Hervé will take care that the changes are merged soon
>
>
> Thank you!
> Benedikt
>

Summary of meeting about Maven performance improvements

2019-04-24 Thread Benedikt Ritter 
Hello,

this is a summary of a video conference call that happened yesterday (April
24).

Topic:
Discussion about performance improvements that have been proposed by Stefan
Oehme, namely:

- [MNG-6638] - Prevent reparsing POMs in MavenMetadataSource (
https://github.com/apache/maven/pull/244)
- [MNG-6633] - Reduce memory usage of excludes (
https://github.com/apache/maven/pull/243) - Speed up project discovery (
https://github.com/apache/maven/pull/242) - Make location handling more
memory efficient (https://github.com/codehaus-plexus/modello/pull/31)

The goal of this call was to give some more insights into how Stefan found
the improvements and to better understand what is missing before these
changes be merged.

Attendees of the call:
- Benedikt Ritter (Gradle Inc.)
- Stefan Oehme (Gradle Inc.)
- Robert Scholte (Apache Maven Team)
- Hervé Boutemy (Apache Maven Team; joined about half an hour after the
call started)

Summary:

Stefan gave some insights into how he discovered bottlenecks in Maven:

   -

   One of our customers has a huge Maven build:
   -

  Lots of sub projects (2000)
  -

  Lots of entries in dependency management (4000)
  -

  Results in a lot of garbage collection
  -

   Problems discovered in that build:
   -

  Re-parsing project POMs during dependency resolution
  -

  Model objects are too large because of location tracking
  -

  Low-level bottlenecks in project discovery (especially version
  parsing)
  -

   Customer now has a Maven fork with the proposed changes included:
   -

  1h 50min, 12GB RAM without changes
  -

  45min, 8GB RAM with changes


Robert:

   -

   How to ensure that improvements are not broken?
   -

   No answer to how to test this


Stefan gave some insights into how performance testing works in the Gradle
project:

   -

   Build has a project generator
   -

   Create different projects in different shapes (e.g. lots of subprojects,
   deeply nested projects) during the build
   -

   Download old Gradle version and run the build on generated projects
   -

   Run build again with current Gradle version
   -

   Compare results
   -

   use statistic methods to filter out variance
   -

   Downside to this approach is that it requires a lot of computing
   resources

More information can be found on GitHub:
https://github.com/gradle/gradle/tree/master/subprojects/performance
The corresponding TeamCity build can be found here:
https://builds.gradle.org/viewLog.html?buildId=22179604=Gradle_Check_PerformanceExperimentCoordinator=report_project941_Performance_Gradle_Check_Stage_ReadyforRelease=master
(use
"Login as guest" to view)

Robert:

   -

   What about measuring performance using instruction calls?


Stefan:

   -

   The performance improvements we found were mostly about garbage being
   created
   -

   Measuring using instruction calls is interesting
   -

   ... but it is also very machine dependent


Robert:

   -

   We need to find out who is interested in these kind improvements inside
   the Maven community.
   -

   Build a community of people who would like to work on these kind of
   things.


Stefan:

   -

   It's easy to get started. We just used open source tools:
   -

   We used async-profiler for measuring things (
   https://github.com/jvm-profiling-tools/async-profiler)
   -

   Heap dumps for analyzing memory usage

To get started with performance tests in the maven project:

   -

   Start with only a few test projects
   -

   The Gradle generator is Apache License v2 and can be used as a starting
   point to generate a big maven project


Hervé:

   -

   PRs should be merged soon
   -

   Discussion need to be resolved
   -

   Why was the PR not merged after the discussion and resolving all issues
   with the code?
   -

   Hervé will take care that the changes are merged soon


Thank you!
Benedikt

Maven performance

2010-04-27 Thread Jörg Schaible 
Hi guys,

it's not the first time I complain about Maven performance, but it seems M3 
is heading the right direction. Yet, still not on the 2.0.x level:

 % =
$ MAVEN_OPTS=-Xmx768m -XX:MaxPermSize=128m mvn-2.0 eclipse:eclipse
Apache Maven 2.0.11
Java version: 1.6.0_20
Java home: /opt/sun-jdk-1.6.0.20/jre
Default locale: en_US, platform encoding: UTF-8
OS name: linux version: 2.6.32-gentoo-r7 arch: i386 Family: unix
snip
   
[INFO] Total time: 2 minutes 46 seconds 
  
[INFO] Finished at: Tue Apr 27 12:06:49 CEST 2010   
  
[INFO] Final Memory: 362M/573M  
  


$ MAVEN_OPTS=-Xmx768m -XX:MaxPermSize=128m mvn-2.2 eclipse:eclipse
Apache Maven 2.2.1 (r801777; 2009-08-06 21:16:01+0200)
Java version: 1.6.0_20
Java home: /opt/sun-jdk-1.6.0.20/jre
Default locale: en_US, platform encoding: UTF-8
OS name: linux version: 2.6.32-gentoo-r7 arch: i386 Family: unix
snip

[INFO] Total time: 6 minutes 46 seconds
[INFO] Finished at: Tue Apr 27 11:31:13 CEST 2010
[INFO] Final Memory: 434M/682M


$ MAVEN_OPTS=-Xmx768m -XX:MaxPermSize=128m mvn-3.0 eclipse:eclipse
Apache Maven 3.0-beta-1 (r935667; 2010-04-19 19:00:39+0200)
Java version: 1.6.0_20
Java home: /opt/sun-jdk-1.6.0.20/jre
Default locale: en_US, platform encoding: UTF-8
OS name: linux version: 2.6.32-gentoo-r7 arch: i386 Family: unix
snip

[INFO] Total time: 4:02.127s
[INFO] Finished at: Tue Apr 27 12:02:12 CEST 2010
[INFO] Final Memory: 344M/443M

 % =

The build processes ~400 projects. The version output is somewhat faked 
(esp. for M2.0), it's just to give you the details of the environment. M3 is 
way better than M2.2, but not (yet ?) at the level of 2.0.11 (and I bet 
2.0.9 will be a lot faster again).

Also have a look at the memory footprint. Since M2.2 it is quite no longer 
possible for me to build all the stuff at once. M3 looks very promising 
here.

- Jörg



-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org

RE: Maven performance

2010-04-27 Thread hermod.opstvedt 
Hi

Do the same test with  mvn org.twdata.maven:maven-cli-plugin:execute-phase - A 
regular compile with large projects cuts down the time drastically

Hermod 
-Original Message-
From: Jörg Schaible [mailto:joerg.schai...@gmx.de] 
Sent: Tuesday, April 27, 2010 2:11 PM
To: dev@maven.apache.org
Subject: Maven performance

Hi guys,

it's not the first time I complain about Maven performance, but it seems M3 is 
heading the right direction. Yet, still not on the 2.0.x level:

 % =
$ MAVEN_OPTS=-Xmx768m -XX:MaxPermSize=128m mvn-2.0 eclipse:eclipse Apache 
Maven 2.0.11 Java version: 1.6.0_20 Java home: /opt/sun-jdk-1.6.0.20/jre 
Default locale: en_US, platform encoding: UTF-8 OS name: linux version: 
2.6.32-gentoo-r7 arch: i386 Family: unix
snip
   
[INFO] Total time: 2 minutes 46 seconds 
  
[INFO] Finished at: Tue Apr 27 12:06:49 CEST 2010   
  
[INFO] Final Memory: 362M/573M  
  


$ MAVEN_OPTS=-Xmx768m -XX:MaxPermSize=128m mvn-2.2 eclipse:eclipse Apache 
Maven 2.2.1 (r801777; 2009-08-06 21:16:01+0200) Java version: 1.6.0_20 Java 
home: /opt/sun-jdk-1.6.0.20/jre Default locale: en_US, platform encoding: UTF-8 
OS name: linux version: 2.6.32-gentoo-r7 arch: i386 Family: unix
snip

[INFO] Total time: 6 minutes 46 seconds
[INFO] Finished at: Tue Apr 27 11:31:13 CEST 2010 [INFO] Final Memory: 434M/682M


$ MAVEN_OPTS=-Xmx768m -XX:MaxPermSize=128m mvn-3.0 eclipse:eclipse Apache 
Maven 3.0-beta-1 (r935667; 2010-04-19 19:00:39+0200) Java version: 1.6.0_20 
Java home: /opt/sun-jdk-1.6.0.20/jre Default locale: en_US, platform encoding: 
UTF-8 OS name: linux version: 2.6.32-gentoo-r7 arch: i386 Family: unix
snip

[INFO] Total time: 4:02.127s
[INFO] Finished at: Tue Apr 27 12:02:12 CEST 2010 [INFO] Final Memory: 344M/443M

 % =

The build processes ~400 projects. The version output is somewhat faked (esp. 
for M2.0), it's just to give you the details of the environment. M3 is way 
better than M2.2, but not (yet ?) at the level of 2.0.11 (and I bet
2.0.9 will be a lot faster again).

Also have a look at the memory footprint. Since M2.2 it is quite no longer 
possible for me to build all the stuff at once. M3 looks very promising here.

- Jörg



-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional 
commands, e-mail: dev-h...@maven.apache.org

* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *

This email with attachments is solely for the use of the individual or
entity to whom it is addressed. Please also be aware that the DnB NOR Group
cannot accept any payment orders or other legally binding correspondence with
customers as a part of an email. 

This email message has been virus checked by the anti virus programs used
in the DnB NOR Group.

* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *


-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org

Re: Maven performance

2010-04-27 Thread Arnaud Héritier 
I never had results like you.
I always noticed improvements between 2.02.23.0
I'll do some tests.

Where I have problems for now, it is on some OS. While with OSes like
Suse,OpensSolaris,Windows 7 we have good perfs results for the build, on
linux servers like ubuntu, fedora , ... we have results 4x slower. We are
studying these problems and if we succeed to explain the issue we'll share
it here (I suppose that many of us are using linux for continuous
integration servers)

Arnaud Héritier
Software Factory Manager
eXo platform - http://www.exoplatform.com
---
http://www.aheritier.net


On Tue, Apr 27, 2010 at 2:10 PM, Jörg Schaible joerg.schai...@gmx.dewrote:

 Hi guys,

 it's not the first time I complain about Maven performance, but it seems M3
 is heading the right direction. Yet, still not on the 2.0.x level:

  % =
 $ MAVEN_OPTS=-Xmx768m -XX:MaxPermSize=128m mvn-2.0 eclipse:eclipse
 Apache Maven 2.0.11
 Java version: 1.6.0_20
 Java home: /opt/sun-jdk-1.6.0.20/jre
 Default locale: en_US, platform encoding: UTF-8
 OS name: linux version: 2.6.32-gentoo-r7 arch: i386 Family: unix
 snip
 
 [INFO] Total time: 2 minutes 46 seconds
 [INFO] Finished at: Tue Apr 27 12:06:49 CEST 2010
 [INFO] Final Memory: 362M/573M
 

 $ MAVEN_OPTS=-Xmx768m -XX:MaxPermSize=128m mvn-2.2 eclipse:eclipse
 Apache Maven 2.2.1 (r801777; 2009-08-06 21:16:01+0200)
 Java version: 1.6.0_20
 Java home: /opt/sun-jdk-1.6.0.20/jre
 Default locale: en_US, platform encoding: UTF-8
 OS name: linux version: 2.6.32-gentoo-r7 arch: i386 Family: unix
 snip
 
 [INFO] Total time: 6 minutes 46 seconds
 [INFO] Finished at: Tue Apr 27 11:31:13 CEST 2010
 [INFO] Final Memory: 434M/682M
 

 $ MAVEN_OPTS=-Xmx768m -XX:MaxPermSize=128m mvn-3.0 eclipse:eclipse
 Apache Maven 3.0-beta-1 (r935667; 2010-04-19 19:00:39+0200)
 Java version: 1.6.0_20
 Java home: /opt/sun-jdk-1.6.0.20/jre
 Default locale: en_US, platform encoding: UTF-8
 OS name: linux version: 2.6.32-gentoo-r7 arch: i386 Family: unix
 snip
 
 [INFO] Total time: 4:02.127s
 [INFO] Finished at: Tue Apr 27 12:02:12 CEST 2010
 [INFO] Final Memory: 344M/443M
 
  % =

 The build processes ~400 projects. The version output is somewhat faked
 (esp. for M2.0), it's just to give you the details of the environment. M3
 is
 way better than M2.2, but not (yet ?) at the level of 2.0.11 (and I bet
 2.0.9 will be a lot faster again).

 Also have a look at the memory footprint. Since M2.2 it is quite no longer
 possible for me to build all the stuff at once. M3 looks very promising
 here.

 - Jörg



 -
 To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
 For additional commands, e-mail: dev-h...@maven.apache.org

Re: Maven performance

2010-04-27 Thread Jason van Zyl 
Jörg,

If you are going to use the maven-eclipse-plugin as your benchmark I think it's 
going to be very hard to pin point where the bottleneck is. It's a little 
universe unto itself. Could be the shared components, resolution code, the 
plugin itself. I see vast improvements in speed in Maven 3.x. I don't think you 
can use the maven-eclipse-plugin as the basis for measuring improvement. For 
example there is behavior in Maven 3.x that is more correct which may 
contribute to some slow down and that we're just going to have to live with.

As one user suggested it may be wise to provide execution timings for the core 
and individual plugins and phases. For example if you found that Maven's 
execution time remained constant over all versions of Maven where it was the 
maven-eclipse-plugin that varied then we know it's an interaction problem with 
the plugin and you can go from there.

Or use a profiler and find out where the actual problem is and tell us. We've 
done lots of profiling to get Maven 3.x and M2E running faster. But gigantic 
plugins like m-e-p and the assembler plugin will probably have problems 
entirely their own.

On Apr 27, 2010, at 8:10 AM, Jörg Schaible wrote:

 Hi guys,
 
 it's not the first time I complain about Maven performance, but it seems M3 
 is heading the right direction. Yet, still not on the 2.0.x level:
 
  % =
 $ MAVEN_OPTS=-Xmx768m -XX:MaxPermSize=128m mvn-2.0 eclipse:eclipse
 Apache Maven 2.0.11
 Java version: 1.6.0_20
 Java home: /opt/sun-jdk-1.6.0.20/jre
 Default locale: en_US, platform encoding: UTF-8
 OS name: linux version: 2.6.32-gentoo-r7 arch: i386 Family: unix
 snip
   
  
 [INFO] Total time: 2 minutes 46 seconds   
 
 [INFO] Finished at: Tue Apr 27 12:06:49 CEST 2010 
 
 [INFO] Final Memory: 362M/573M
 
 
 
 $ MAVEN_OPTS=-Xmx768m -XX:MaxPermSize=128m mvn-2.2 eclipse:eclipse
 Apache Maven 2.2.1 (r801777; 2009-08-06 21:16:01+0200)
 Java version: 1.6.0_20
 Java home: /opt/sun-jdk-1.6.0.20/jre
 Default locale: en_US, platform encoding: UTF-8
 OS name: linux version: 2.6.32-gentoo-r7 arch: i386 Family: unix
 snip
 
 [INFO] Total time: 6 minutes 46 seconds
 [INFO] Finished at: Tue Apr 27 11:31:13 CEST 2010
 [INFO] Final Memory: 434M/682M
 
 
 $ MAVEN_OPTS=-Xmx768m -XX:MaxPermSize=128m mvn-3.0 eclipse:eclipse
 Apache Maven 3.0-beta-1 (r935667; 2010-04-19 19:00:39+0200)
 Java version: 1.6.0_20
 Java home: /opt/sun-jdk-1.6.0.20/jre
 Default locale: en_US, platform encoding: UTF-8
 OS name: linux version: 2.6.32-gentoo-r7 arch: i386 Family: unix
 snip
 
 [INFO] Total time: 4:02.127s
 [INFO] Finished at: Tue Apr 27 12:02:12 CEST 2010
 [INFO] Final Memory: 344M/443M
 
  % =
 
 The build processes ~400 projects. The version output is somewhat faked 
 (esp. for M2.0), it's just to give you the details of the environment. M3 is 
 way better than M2.2, but not (yet ?) at the level of 2.0.11 (and I bet 
 2.0.9 will be a lot faster again).
 
 Also have a look at the memory footprint. Since M2.2 it is quite no longer 
 possible for me to build all the stuff at once. M3 looks very promising 
 here.
 
 - Jörg
 
 
 
 -
 To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
 For additional commands, e-mail: dev-h...@maven.apache.org
 

Thanks,

Jason

--
Jason van Zyl
Founder,  Apache Maven
http://twitter.com/jvanzyl
-

We know what we are, but know not what we may be.

  -- Shakespeare

Re: Maven performance

2010-04-27 Thread Jörg Schaible 
Hi Jason,

Jason van Zyl wrote:

 Jörg,
 
 If you are going to use the maven-eclipse-plugin as your benchmark I think
 it's going to be very hard to pin point where the bottleneck is. It's a
 little universe unto itself. Could be the shared components, resolution
 code, the plugin itself. I see vast improvements in speed in Maven 3.x. I
 don't think you can use the maven-eclipse-plugin as the basis for
 measuring improvement. For example there is behavior in Maven 3.x that is
 more correct which may contribute to some slow down and that we're just
 going to have to live with.

Well, it is an indication. Running a plugin that makes a lot usage of the 
Maven infrastructure does tell more than a build including compilation task 
and packaging time. From 2.0.8 to 2.2.1 there have been also changes to 
ensure more correct behaviour, but performance got worse. To be honest, I 
was positively surprised by M3 ;-)

However, Arnaud's remark for Linux is also interesting.

 As one user suggested it may be wise to provide execution timings for the
 core and individual plugins and phases. For example if you found that
 Maven's execution time remained constant over all versions of Maven where
 it was the maven-eclipse-plugin that varied then we know it's an
 interaction problem with the plugin and you can go from there.

Yes, that would be the best approach for a first examination in more detail. 
I'm not close enough to the Maven core, but isn't it possible to exchange 
the Plexus component that executes the mojos with a delegate that measures 
the time for each execution? Is there some kind of Plexus configuration that 
can be used to injected such a component from the outside e.g. with the 
m2.conf or with system properties?

 Or use a profiler and find out where the actual problem is and tell us.

A profiler gets interesting if you already can put a finger onto some part 
with such a complex application.

 We've done lots of profiling to get Maven 3.x and M2E running faster. But
 gigantic plugins like m-e-p and the assembler plugin will probably have
 problems entirely their own.

Actually I was really impressed by the memory footprint. Especially since it 
is a necessity e.g. with the Eclipse plugin to generate all Eclipse projects 
at once to get these projects linked.

- Jörg


-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org