date:20121212

Re: Logback in Maven Core

2012-12-12 Thread Ralph Goers

I checked out Maven and used its build as a comparison.  First, I ran the log4j 
2 build and it was taking around 59 seconds.  I then changed the log4j2.xml to 
remove the colors.  I then got an average time for Log4j 2 of  54.76s and for 
Logback I get an average of 55.225s.  I consider these differences to be 
meaningless.

For reference, the log4j2.xml I used is attached.

Ralph

 

log4j2.xml
Description: XML document








On Dec 11, 2012, at 11:19 PM, Stephen Connolly wrote:

 Well I am not going to tar and feather log4j2 based on one set of runs on
 my machine. I would like somebody else to repeat and confirm first as there
 could have been some background OS update or other process stealing CPU
 while doing the 3 log4j2 runs.
 
 Also I do not know if I am comparing the same things. Afaik the log back
 branch has the latest fixes in it, while the log4j2 branch is the colorized
 one from a few weeks back and likely has not got the fixes required for the
 issues you identified with the last 3.1.0 RC
 
 We need to compare like with like to make an informed decision... I am just
 putting some numbers down as a starting point
 
 -Stephen
 
 On Wednesday, 12 December 2012, Kristian Rosenvold wrote:
 
 Finally some interesting numbers, and if (heaven forbid) this decision
 should be based on
 technical grounds, this is one of the first significant pieces to come
 up in this discussion.
 
 Since I am quite unfamiliar with logging (I use loose coupling and
 tests instead ;), I took the opportunity to read
 http://logging.apache.org/log4j/2.x/performance.html Somehow the
 real-life results don't seem to match up with the advertising blurp on
 the log4j site. While it hardly surprises me, I was wondering if
 anyone actually knows why?
 
 Kristian
 
 
 
 2012/12/12 Stephen Connolly stephen.alan.conno...@gmail.comjavascript:;
 :
 The consistent times (i.e. repeated runs after discarding the first) are:
 
 3.0.4: 1min18sec
 logback: 1min13sec
 log4j2: 1min34sec
 
 The second test was building GIT hash
 85dd6e37456d30d2661e10b044efa9036c528023 of jszip-maven-plugin (@
 jszip.org)
 with the following command line:
 
 mvn -o -X clean verify -DskipTests -Dinvoker.skip
 
 [Testing heavy logging]
 
 3.0.4: 12.1sec
 logback: 12.2sec
 log4j2: 12.5sec
 
 -
 To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org javascript:;
 For additional commands, e-mail: dev-h...@maven.apache.org javascript:;
 
 



-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org

Re: Logback in Maven Core

2012-12-12 Thread Ralph Goers

Oh - FWIW  I'm running on OSX 10.7.5 with a 2.5 GHz Intel Core i7 with 16GB of 
memory.

Ralph

On Dec 12, 2012, at 12:27 AM, Ralph Goers wrote:

 I checked out Maven and used its build as a comparison.  First, I ran the 
 log4j 2 build and it was taking around 59 seconds.  I then changed the 
 log4j2.xml to remove the colors.  I then got an average time for Log4j 2 of  
 54.76s and for Logback I get an average of 55.225s.  I consider these 
 differences to be meaningless.
 
 For reference, the log4j2.xml I used is attached.
 
 Ralph
 
 log4j2.xml
 
 
 
 
 
 
 
 On Dec 11, 2012, at 11:19 PM, Stephen Connolly wrote:
 
 Well I am not going to tar and feather log4j2 based on one set of runs on
 my machine. I would like somebody else to repeat and confirm first as there
 could have been some background OS update or other process stealing CPU
 while doing the 3 log4j2 runs.
 
 Also I do not know if I am comparing the same things. Afaik the log back
 branch has the latest fixes in it, while the log4j2 branch is the colorized
 one from a few weeks back and likely has not got the fixes required for the
 issues you identified with the last 3.1.0 RC
 
 We need to compare like with like to make an informed decision... I am just
 putting some numbers down as a starting point
 
 -Stephen
 
 On Wednesday, 12 December 2012, Kristian Rosenvold wrote:
 
 Finally some interesting numbers, and if (heaven forbid) this decision
 should be based on
 technical grounds, this is one of the first significant pieces to come
 up in this discussion.
 
 Since I am quite unfamiliar with logging (I use loose coupling and
 tests instead ;), I took the opportunity to read
 http://logging.apache.org/log4j/2.x/performance.html Somehow the
 real-life results don't seem to match up with the advertising blurp on
 the log4j site. While it hardly surprises me, I was wondering if
 anyone actually knows why?
 
 Kristian
 
 
 
 2012/12/12 Stephen Connolly stephen.alan.conno...@gmail.comjavascript:;
 :
 The consistent times (i.e. repeated runs after discarding the first) are:
 
 3.0.4: 1min18sec
 logback: 1min13sec
 log4j2: 1min34sec
 
 The second test was building GIT hash
 85dd6e37456d30d2661e10b044efa9036c528023 of jszip-maven-plugin (@
 jszip.org)
 with the following command line:
 
 mvn -o -X clean verify -DskipTests -Dinvoker.skip
 
 [Testing heavy logging]
 
 3.0.4: 12.1sec
 logback: 12.2sec
 log4j2: 12.5sec
 
 -
 To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org javascript:;
 For additional commands, e-mail: dev-h...@maven.apache.org javascript:;
 
 
 


-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org

Re: A very interesting performance regression in 3.1 embedded mode

2012-12-12 Thread Kristian Rosenvold

2012/12/12 Milos Kleint mkle...@gmail.com:
 how much memory will be freed by the soft reference? if it's not a big
 chunk, most likely not worth it, soft references are released only
 when your VM is really, really in trouble. by the time it gets
 released, you've been slowed down by repeatedly hitting the ceiling of
 your memory and CGed frequently.

I think the appropriate question to ask is how much memory would be
freed by an eviction strategy? and maybe who needs it?.

The answer to how much can precisely be formulated as a truckload.
The surefire IT's in embedded mode require 350MB permgen and 500MB
memory. When things are evicted those numbers are more or less halved
(I seem to believe they ran in 128MB permgen!). I will come up with
some hard numbers on that at some point...

An embedded core used to hold on to a *lot* of classloaders, both for
plugins, extensions and projects. Some of these are never freed and
will hence leak for as long as the embedded container is being kept
alive. So for someone IDE-like embedding maven it would definitely
make sense to release and reallocate the embedded maven instance every
time a project is loaded/reloaded. Just recently it was changed to
always deallocate the plugins (but still leaking extension/project
realms), which is probably too much or too little. From an IDE point
of view 0.5 seconds added overhead /may/ be ok (while certainly not
world class performance).

For something like an embedded test-run it might make sense to just
ditch the embedded container every N runs and start with a fresh one.

The real problem starts when you decide to hold onto the embedded instance and
declare that the instance should release class realms according to
some eviction strategy. Like so many times before, neither
softreferences nor weakreferences really seem appropriate; one is too
liberal in letting go, the other too conservative. Currently core does
not track actual usage of each class realm (although it would be
fairly simple to implement) ; if it did we could do all sorts of cool
stuff.

I prefer a /predictable/ eviction strategy. We *could* do something like
a round-robin eviction (evict a rotating 25% of all classrealms every
time) or a usage based eviction (evict all unused and every N usages
of a given class realm). But then again maybe this whole issue is best
solved by restoring the old strategy of no eviction and tell the
clients to evict us every now and then. That would work very well for
embedded test runs, unsure what something like m2e would do about
it

Kristian









 just an illustrative story, most likely not related to this usecase
 but still interesting
 while embedding maven in netbeans, we've SoftReferenced MavenProject
 at some point in time. MavenProjects can hold a big object tree,
 mostly via the cache in ProjectBuildingRequest. So once you run out of
 memory, the SRs get dropped however sometimes the MP instances are
 again immediately required, so they are loaded again. And dropped. And
 loaded and the IDE grinds to a halt. In some cases, one gets OOME
 fairly fast, but in others it can take fairly long.

 BTW we don't embed maven builds, just MavenProject loading in netbeans

 Milos

 On Sun, Dec 9, 2012 at 11:04 PM, Christian Schulte c...@schulte.it wrote:
 Am 12/09/12 22:58, schrieb Kristian Rosenvold:

 Anyone else have any ideas about eviction strategies ?


 Without having looked at the code. GC driven by using soft references.

 --
 Christian


 -
 To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
 For additional commands, e-mail: dev-h...@maven.apache.org


 -
 To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
 For additional commands, e-mail: dev-h...@maven.apache.org


-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org

Re: Logback in Maven Core

2012-12-12 Thread Kristian Rosenvold

Yeah, measuring performance on modern cpu's is totally borked. To get
any real measurements one probably needs to to average of 100 non-stop
builds or similar, to counter for all the dark magics intel do with
temperature-based overclocking.

I think I've seen somewhere that it's possible to disable all the
cpu-voodo in the bios. But what a pain to reboot to change those
settings ! Or dig out the old Pentium4 from the closet.

Kristian

2012/12/12 Ralph Goers ralph.go...@dslextreme.com:
 I checked out Maven and used its build as a comparison.  First, I ran the 
 log4j 2 build and it was taking around 59 seconds.  I then changed the 
 log4j2.xml to remove the colors.  I then got an average time for Log4j 2 of  
 54.76s and for Logback I get an average of 55.225s.  I consider these 
 differences to be meaningless.

 For reference, the log4j2.xml I used is attached.

 Ralph










 On Dec 11, 2012, at 11:19 PM, Stephen Connolly wrote:

 Well I am not going to tar and feather log4j2 based on one set of runs on
 my machine. I would like somebody else to repeat and confirm first as there
 could have been some background OS update or other process stealing CPU
 while doing the 3 log4j2 runs.

 Also I do not know if I am comparing the same things. Afaik the log back
 branch has the latest fixes in it, while the log4j2 branch is the colorized
 one from a few weeks back and likely has not got the fixes required for the
 issues you identified with the last 3.1.0 RC

 We need to compare like with like to make an informed decision... I am just
 putting some numbers down as a starting point

 -Stephen

 On Wednesday, 12 December 2012, Kristian Rosenvold wrote:

 Finally some interesting numbers, and if (heaven forbid) this decision
 should be based on
 technical grounds, this is one of the first significant pieces to come
 up in this discussion.

 Since I am quite unfamiliar with logging (I use loose coupling and
 tests instead ;), I took the opportunity to read
 http://logging.apache.org/log4j/2.x/performance.html Somehow the
 real-life results don't seem to match up with the advertising blurp on
 the log4j site. While it hardly surprises me, I was wondering if
 anyone actually knows why?

 Kristian



 2012/12/12 Stephen Connolly stephen.alan.conno...@gmail.comjavascript:;
 :
 The consistent times (i.e. repeated runs after discarding the first) are:

 3.0.4: 1min18sec
 logback: 1min13sec
 log4j2: 1min34sec

 The second test was building GIT hash
 85dd6e37456d30d2661e10b044efa9036c528023 of jszip-maven-plugin (@
 jszip.org)
 with the following command line:

 mvn -o -X clean verify -DskipTests -Dinvoker.skip

 [Testing heavy logging]

 3.0.4: 12.1sec
 logback: 12.2sec
 log4j2: 12.5sec

 -
 To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org javascript:;
 For additional commands, e-mail: dev-h...@maven.apache.org javascript:;






 -
 To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
 For additional commands, e-mail: dev-h...@maven.apache.org

-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org

Re: Logback in Maven Core

2012-12-12 Thread Stephen Connolly

Can we get a set of baseline git hashes for any versions of maven that we
are comparing. I might see if I can pull logback out of the latest RC and
put log4j2 and some other impls in its place so we can get some real apples
for apples comparisons going


On 12 December 2012 08:35, Kristian Rosenvold
kristian.rosenv...@gmail.comwrote:

 Yeah, measuring performance on modern cpu's is totally borked. To get
 any real measurements one probably needs to to average of 100 non-stop
 builds or similar, to counter for all the dark magics intel do with
 temperature-based overclocking.

 I think I've seen somewhere that it's possible to disable all the
 cpu-voodo in the bios. But what a pain to reboot to change those
 settings ! Or dig out the old Pentium4 from the closet.

 Kristian

 2012/12/12 Ralph Goers ralph.go...@dslextreme.com:
  I checked out Maven and used its build as a comparison.  First, I ran
 the log4j 2 build and it was taking around 59 seconds.  I then changed the
 log4j2.xml to remove the colors.  I then got an average time for Log4j 2 of
  54.76s and for Logback I get an average of 55.225s.  I consider these
 differences to be meaningless.
 
  For reference, the log4j2.xml I used is attached.
 
  Ralph
 
 
 
 
 
 
 
 
 
 
  On Dec 11, 2012, at 11:19 PM, Stephen Connolly wrote:
 
  Well I am not going to tar and feather log4j2 based on one set of runs
 on
  my machine. I would like somebody else to repeat and confirm first as
 there
  could have been some background OS update or other process stealing CPU
  while doing the 3 log4j2 runs.
 
  Also I do not know if I am comparing the same things. Afaik the log back
  branch has the latest fixes in it, while the log4j2 branch is the
 colorized
  one from a few weeks back and likely has not got the fixes required for
 the
  issues you identified with the last 3.1.0 RC
 
  We need to compare like with like to make an informed decision... I am
 just
  putting some numbers down as a starting point
 
  -Stephen
 
  On Wednesday, 12 December 2012, Kristian Rosenvold wrote:
 
  Finally some interesting numbers, and if (heaven forbid) this decision
  should be based on
  technical grounds, this is one of the first significant pieces to come
  up in this discussion.
 
  Since I am quite unfamiliar with logging (I use loose coupling and
  tests instead ;), I took the opportunity to read
  http://logging.apache.org/log4j/2.x/performance.html Somehow the
  real-life results don't seem to match up with the advertising blurp on
  the log4j site. While it hardly surprises me, I was wondering if
  anyone actually knows why?
 
  Kristian
 
 
 
  2012/12/12 Stephen Connolly stephen.alan.conno...@gmail.com
 javascript:;
  :
  The consistent times (i.e. repeated runs after discarding the first)
 are:
 
  3.0.4: 1min18sec
  logback: 1min13sec
  log4j2: 1min34sec
 
  The second test was building GIT hash
  85dd6e37456d30d2661e10b044efa9036c528023 of jszip-maven-plugin (@
  jszip.org)
  with the following command line:
 
  mvn -o -X clean verify -DskipTests -Dinvoker.skip
 
  [Testing heavy logging]
 
  3.0.4: 12.1sec
  logback: 12.2sec
  log4j2: 12.5sec
 
  -
  To unsubscribe, e-mail: dev-unsubscr...@maven.apache.orgjavascript:;
  For additional commands, e-mail: dev-h...@maven.apache.orgjavascript:;
 
 
 
 
 
 
  -
  To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
  For additional commands, e-mail: dev-h...@maven.apache.org

 -
 To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
 For additional commands, e-mail: dev-h...@maven.apache.org

Re: Logback in Maven Core

2012-12-12 Thread Mark Struberg

I only do those benchmarks on a desktop linux box with the cpu governer set to 
a fixed frequency. 
Imo that's the only way to get reproducible numbers.
Especially when working on a notebook it also depends how much heat the 
notebook can dissipate. I've seen my MBP not using his full turbo when he is 
really hot after benchmarks (even singlethreaded).


LieGrue,
strub




- Original Message -
 From: Kristian Rosenvold kristian.rosenv...@gmail.com
 To: Maven Developers List dev@maven.apache.org
 Cc: 
 Sent: Wednesday, December 12, 2012 9:35 AM
 Subject: Re: Logback in Maven Core
 
 Yeah, measuring performance on modern cpu's is totally borked. To get
 any real measurements one probably needs to to average of 100 non-stop
 builds or similar, to counter for all the dark magics intel do with
 temperature-based overclocking.
 
 I think I've seen somewhere that it's possible to disable all the
 cpu-voodo in the bios. But what a pain to reboot to change those
 settings ! Or dig out the old Pentium4 from the closet.
 
 Kristian
 
 2012/12/12 Ralph Goers ralph.go...@dslextreme.com:
  I checked out Maven and used its build as a comparison.  First, I ran the 
 log4j 2 build and it was taking around 59 seconds.  I then changed the 
 log4j2.xml to remove the colors.  I then got an average time for Log4j 2 of  
 54.76s and for Logback I get an average of 55.225s.  I consider these 
 differences to be meaningless.
 
  For reference, the log4j2.xml I used is attached.
 
  Ralph
 
 
 
 
 
 
 
 
 
 
  On Dec 11, 2012, at 11:19 PM, Stephen Connolly wrote:
 
  Well I am not going to tar and feather log4j2 based on one set of runs 
 on
  my machine. I would like somebody else to repeat and confirm first as 
 there
  could have been some background OS update or other process stealing CPU
  while doing the 3 log4j2 runs.
 
  Also I do not know if I am comparing the same things. Afaik the log 
 back
  branch has the latest fixes in it, while the log4j2 branch is the 
 colorized
  one from a few weeks back and likely has not got the fixes required for 
 the
  issues you identified with the last 3.1.0 RC
 
  We need to compare like with like to make an informed decision... I am 
 just
  putting some numbers down as a starting point
 
  -Stephen
 
  On Wednesday, 12 December 2012, Kristian Rosenvold wrote:
 
  Finally some interesting numbers, and if (heaven forbid) this 
 decision
  should be based on
  technical grounds, this is one of the first significant pieces to 
 come
  up in this discussion.
 
  Since I am quite unfamiliar with logging (I use loose coupling and
  tests instead ;), I took the opportunity to read
  http://logging.apache.org/log4j/2.x/performance.html Somehow the
  real-life results don't seem to match up with the advertising 
 blurp on
  the log4j site. While it hardly surprises me, I was wondering if
  anyone actually knows why?
 
  Kristian
 
 
 
  2012/12/12 Stephen Connolly 
 stephen.alan.conno...@gmail.comjavascript:;
  :
  The consistent times (i.e. repeated runs after discarding the 
 first) are:
 
  3.0.4: 1min18sec
  logback: 1min13sec
  log4j2: 1min34sec
 
  The second test was building GIT hash
  85dd6e37456d30d2661e10b044efa9036c528023 of jszip-maven-plugin 
 (@
  jszip.org)
  with the following command line:
 
  mvn -o -X clean verify -DskipTests -Dinvoker.skip
 
  [Testing heavy logging]
 
  3.0.4: 12.1sec
  logback: 12.2sec
  log4j2: 12.5sec
 
 
 -
  To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org 
 javascript:;
  For additional commands, e-mail: dev-h...@maven.apache.org 
 javascript:;
 
 
 
 
 
 
  -
  To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
  For additional commands, e-mail: dev-h...@maven.apache.org
 
 -
 To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
 For additional commands, e-mail: dev-h...@maven.apache.org


-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org

Re: git commit: Basic experiment with log4j 1.2.

2012-12-12 Thread Olivier Lamy

note you must use %p{WARN=WARNING}
by default as slf4j-simple, log4j2 use WARN instead of WARNING.
It's just to prevent change in log output (and help it to pass as
there is an it which use the WARNING to verify logs content).
log4j2 community add this feature recently as I was complaining :-)
(https://issues.apache.org/jira/browse/LOG4J2-105)

2012/12/12  steph...@apache.org:
 Updated Branches:
   refs/heads/logging/slf4j-log4j [created] 262d644d4


 Basic experiment with log4j 1.2.

 - Should be the same as 7f9e280522379fc0f3ac09f4d81e8188cdb54192 only with 
 log4j 1.2 as the backing impl


 Project: http://git-wip-us.apache.org/repos/asf/maven/repo
 Commit: http://git-wip-us.apache.org/repos/asf/maven/commit/262d644d
 Tree: http://git-wip-us.apache.org/repos/asf/maven/tree/262d644d
 Diff: http://git-wip-us.apache.org/repos/asf/maven/diff/262d644d

 Branch: refs/heads/logging/slf4j-log4j
 Commit: 262d644d4d166040cbd4371c14ef6f720fe64345
 Parents: c8536bf
 Author: Stephen Connolly stephen.alan.conno...@gmail.com
 Authored: Wed Dec 12 10:02:13 2012 +
 Committer: Stephen Connolly stephen.alan.conno...@gmail.com
 Committed: Wed Dec 12 10:02:13 2012 +

 --
  apache-maven/src/conf/logging/log4j.xml |   32 ++
  1 files changed, 32 insertions(+), 0 deletions(-)
 --


 http://git-wip-us.apache.org/repos/asf/maven/blob/262d644d/apache-maven/src/conf/logging/log4j.xml
 --
 diff --git a/apache-maven/src/conf/logging/log4j.xml 
 b/apache-maven/src/conf/logging/log4j.xml
 new file mode 100644
 index 000..d21c86e
 --- /dev/null
 +++ b/apache-maven/src/conf/logging/log4j.xml
 @@ -0,0 +1,32 @@
 +?xml version=1.0 encoding=UTF-8 ?
 +!--
 +  ~ Licensed to the Apache Software Foundation (ASF) under one
 +  ~ or more contributor license agreements.  See the NOTICE file
 +  ~ distributed with this work for additional information
 +  ~ regarding copyright ownership.  The ASF licenses this file
 +  ~ to you under the Apache License, Version 2.0 (the
 +  ~ License); you may not use this file except in compliance
 +  ~ with the License.  You may obtain a copy of the License at
 +  ~
 +  ~   http://www.apache.org/licenses/LICENSE-2.0
 +  ~
 +  ~ Unless required by applicable law or agreed to in writing,
 +  ~ software distributed under the License is distributed on an
 +  ~ AS IS BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
 +  ~ KIND, either express or implied.  See the License for the
 +  ~ specific language governing permissions and limitations
 +  ~ under the License.
 +  --
 +
 +!DOCTYPE log4j:configuration PUBLIC -//APACHE//DTD LOG4J 1.2//EN 
 log4j.dtd
 +log4j:configuration xmlns:log4j='http://jakarta.apache.org/log4j/'
 +  appender name=console class=org.apache.log4j.ConsoleAppender
 +layout class=org.apache.log4j.PatternLayout
 +  param name=ConversionPattern value=[%p] %m%n/
 +/layout
 +  /appender
 +root
 +  priority value=INFO/
 +  appender-ref ref=console/
 +/root
 +/log4j:configuration




-- 
Olivier Lamy
Talend: http://coders.talend.com
http://twitter.com/olamy | http://linkedin.com/in/olamy

-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org

Re: git commit: Basic experiment with log4j 1.2.

2012-12-12 Thread Olivier Lamy

oups it looks to be a log4j 1.2 based branch.
sorry for noise.

2012/12/12 Olivier Lamy ol...@apache.org:
 note you must use %p{WARN=WARNING}
 by default as slf4j-simple, log4j2 use WARN instead of WARNING.
 It's just to prevent change in log output (and help it to pass as
 there is an it which use the WARNING to verify logs content).
 log4j2 community add this feature recently as I was complaining :-)
 (https://issues.apache.org/jira/browse/LOG4J2-105)

 2012/12/12  steph...@apache.org:
 Updated Branches:
   refs/heads/logging/slf4j-log4j [created] 262d644d4


 Basic experiment with log4j 1.2.

 - Should be the same as 7f9e280522379fc0f3ac09f4d81e8188cdb54192 only with 
 log4j 1.2 as the backing impl


 Project: http://git-wip-us.apache.org/repos/asf/maven/repo
 Commit: http://git-wip-us.apache.org/repos/asf/maven/commit/262d644d
 Tree: http://git-wip-us.apache.org/repos/asf/maven/tree/262d644d
 Diff: http://git-wip-us.apache.org/repos/asf/maven/diff/262d644d

 Branch: refs/heads/logging/slf4j-log4j
 Commit: 262d644d4d166040cbd4371c14ef6f720fe64345
 Parents: c8536bf
 Author: Stephen Connolly stephen.alan.conno...@gmail.com
 Authored: Wed Dec 12 10:02:13 2012 +
 Committer: Stephen Connolly stephen.alan.conno...@gmail.com
 Committed: Wed Dec 12 10:02:13 2012 +

 --
  apache-maven/src/conf/logging/log4j.xml |   32 ++
  1 files changed, 32 insertions(+), 0 deletions(-)
 --


 http://git-wip-us.apache.org/repos/asf/maven/blob/262d644d/apache-maven/src/conf/logging/log4j.xml
 --
 diff --git a/apache-maven/src/conf/logging/log4j.xml 
 b/apache-maven/src/conf/logging/log4j.xml
 new file mode 100644
 index 000..d21c86e
 --- /dev/null
 +++ b/apache-maven/src/conf/logging/log4j.xml
 @@ -0,0 +1,32 @@
 +?xml version=1.0 encoding=UTF-8 ?
 +!--
 +  ~ Licensed to the Apache Software Foundation (ASF) under one
 +  ~ or more contributor license agreements.  See the NOTICE file
 +  ~ distributed with this work for additional information
 +  ~ regarding copyright ownership.  The ASF licenses this file
 +  ~ to you under the Apache License, Version 2.0 (the
 +  ~ License); you may not use this file except in compliance
 +  ~ with the License.  You may obtain a copy of the License at
 +  ~
 +  ~   http://www.apache.org/licenses/LICENSE-2.0
 +  ~
 +  ~ Unless required by applicable law or agreed to in writing,
 +  ~ software distributed under the License is distributed on an
 +  ~ AS IS BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
 +  ~ KIND, either express or implied.  See the License for the
 +  ~ specific language governing permissions and limitations
 +  ~ under the License.
 +  --
 +
 +!DOCTYPE log4j:configuration PUBLIC -//APACHE//DTD LOG4J 1.2//EN 
 log4j.dtd
 +log4j:configuration xmlns:log4j='http://jakarta.apache.org/log4j/'
 +  appender name=console class=org.apache.log4j.ConsoleAppender
 +layout class=org.apache.log4j.PatternLayout
 +  param name=ConversionPattern value=[%p] %m%n/
 +/layout
 +  /appender
 +root
 +  priority value=INFO/
 +  appender-ref ref=console/
 +/root
 +/log4j:configuration




 --
 Olivier Lamy
 Talend: http://coders.talend.com
 http://twitter.com/olamy | http://linkedin.com/in/olamy



-- 
Olivier Lamy
Talend: http://coders.talend.com
http://twitter.com/olamy | http://linkedin.com/in/olamy

-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org

Re: Logback in Maven Core

2012-12-12 Thread Stephen Connolly

I am working on getting some branches for different options.

I think I have the logging/slf4j-jul branch done...

I think my logging/slf4j-log4j2 branch is correct

I am fairly certain my logging/slf4j-logback branch is correct

My logging/slf4j-log4j (i.e. 1.2) branch needs some tweaks

If somebody can run the integration tests on those branches and report the
results here that would be great.


On 12 December 2012 08:48, Stephen Connolly stephen.alan.conno...@gmail.com
 wrote:

 Can we get a set of baseline git hashes for any versions of maven that we
 are comparing. I might see if I can pull logback out of the latest RC and
 put log4j2 and some other impls in its place so we can get some real apples
 for apples comparisons going


 On 12 December 2012 08:35, Kristian Rosenvold 
 kristian.rosenv...@gmail.com wrote:

 Yeah, measuring performance on modern cpu's is totally borked. To get
 any real measurements one probably needs to to average of 100 non-stop
 builds or similar, to counter for all the dark magics intel do with
 temperature-based overclocking.

 I think I've seen somewhere that it's possible to disable all the
 cpu-voodo in the bios. But what a pain to reboot to change those
 settings ! Or dig out the old Pentium4 from the closet.

 Kristian

 2012/12/12 Ralph Goers ralph.go...@dslextreme.com:
  I checked out Maven and used its build as a comparison.  First, I ran
 the log4j 2 build and it was taking around 59 seconds.  I then changed the
 log4j2.xml to remove the colors.  I then got an average time for Log4j 2 of
  54.76s and for Logback I get an average of 55.225s.  I consider these
 differences to be meaningless.
 
  For reference, the log4j2.xml I used is attached.
 
  Ralph
 
 
 
 
 
 
 
 
 
 
  On Dec 11, 2012, at 11:19 PM, Stephen Connolly wrote:
 
  Well I am not going to tar and feather log4j2 based on one set of runs
 on
  my machine. I would like somebody else to repeat and confirm first as
 there
  could have been some background OS update or other process stealing CPU
  while doing the 3 log4j2 runs.
 
  Also I do not know if I am comparing the same things. Afaik the log
 back
  branch has the latest fixes in it, while the log4j2 branch is the
 colorized
  one from a few weeks back and likely has not got the fixes required
 for the
  issues you identified with the last 3.1.0 RC
 
  We need to compare like with like to make an informed decision... I am
 just
  putting some numbers down as a starting point
 
  -Stephen
 
  On Wednesday, 12 December 2012, Kristian Rosenvold wrote:
 
  Finally some interesting numbers, and if (heaven forbid) this decision
  should be based on
  technical grounds, this is one of the first significant pieces to come
  up in this discussion.
 
  Since I am quite unfamiliar with logging (I use loose coupling and
  tests instead ;), I took the opportunity to read
  http://logging.apache.org/log4j/2.x/performance.html Somehow the
  real-life results don't seem to match up with the advertising blurp on
  the log4j site. While it hardly surprises me, I was wondering if
  anyone actually knows why?
 
  Kristian
 
 
 
  2012/12/12 Stephen Connolly stephen.alan.conno...@gmail.com
 javascript:;
  :
  The consistent times (i.e. repeated runs after discarding the first)
 are:
 
  3.0.4: 1min18sec
  logback: 1min13sec
  log4j2: 1min34sec
 
  The second test was building GIT hash
  85dd6e37456d30d2661e10b044efa9036c528023 of jszip-maven-plugin (@
  jszip.org)
  with the following command line:
 
  mvn -o -X clean verify -DskipTests -Dinvoker.skip
 
  [Testing heavy logging]
 
  3.0.4: 12.1sec
  logback: 12.2sec
  log4j2: 12.5sec
 
  -
  To unsubscribe, e-mail: dev-unsubscr...@maven.apache.orgjavascript:;
  For additional commands, e-mail: dev-h...@maven.apache.orgjavascript:;
 
 
 
 
 
 
  -
  To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
  For additional commands, e-mail: dev-h...@maven.apache.org

 -
 To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
 For additional commands, e-mail: dev-h...@maven.apache.org

Re: Logback in Maven Core

2012-12-12 Thread Stephen Connolly

I have built some binaries for people to play with:

http://people.apache.org/~stephenc/

MD5 (apache-maven-3.1.0-pre-jul.tar.gz) = dd40afbfa64ab53f614ede19385e4a48
MD5 (apache-maven-3.1.0-pre-jul.zip) = 731fe7136e96e2027d145d993d917f20
MD5 (apache-maven-3.1.0-pre-log4j.tar.gz) = 63fead6accb60ca52c8a299c39d172f5
MD5 (apache-maven-3.1.0-pre-log4j.zip) = 64982b6395a80bb6f2d27020c9f439be
MD5 (apache-maven-3.1.0-pre-log4j2.tar.gz) =
cc38e6f2110d9e76f6fc9feb29c48500
MD5 (apache-maven-3.1.0-pre-log4j2.zip) = 77eda0a4bbb6f82cfdff8e8afb408dd5
MD5 (apache-maven-3.1.0-pre-logback.tar.gz) =
b51215f3e1c5589772b4d65c68f28654
MD5 (apache-maven-3.1.0-pre-logback.zip) = 6af89e67b1dca4699ef0ce57c3db6a5f

They should all behave roughly similarly... though the log4j 1.2 version
will probably output [WARN] and not [WARNING]



On 12 December 2012 10:40, Stephen Connolly stephen.alan.conno...@gmail.com
 wrote:

 I am working on getting some branches for different options.

 I think I have the logging/slf4j-jul branch done...

 I think my logging/slf4j-log4j2 branch is correct

 I am fairly certain my logging/slf4j-logback branch is correct

 My logging/slf4j-log4j (i.e. 1.2) branch needs some tweaks

 If somebody can run the integration tests on those branches and report the
 results here that would be great.


 On 12 December 2012 08:48, Stephen Connolly 
 stephen.alan.conno...@gmail.com wrote:

 Can we get a set of baseline git hashes for any versions of maven that we
 are comparing. I might see if I can pull logback out of the latest RC and
 put log4j2 and some other impls in its place so we can get some real apples
 for apples comparisons going


 On 12 December 2012 08:35, Kristian Rosenvold 
 kristian.rosenv...@gmail.com wrote:

 Yeah, measuring performance on modern cpu's is totally borked. To get
 any real measurements one probably needs to to average of 100 non-stop
 builds or similar, to counter for all the dark magics intel do with
 temperature-based overclocking.

 I think I've seen somewhere that it's possible to disable all the
 cpu-voodo in the bios. But what a pain to reboot to change those
 settings ! Or dig out the old Pentium4 from the closet.

 Kristian

 2012/12/12 Ralph Goers ralph.go...@dslextreme.com:
  I checked out Maven and used its build as a comparison.  First, I ran
 the log4j 2 build and it was taking around 59 seconds.  I then changed the
 log4j2.xml to remove the colors.  I then got an average time for Log4j 2 of
  54.76s and for Logback I get an average of 55.225s.  I consider these
 differences to be meaningless.
 
  For reference, the log4j2.xml I used is attached.
 
  Ralph
 
 
 
 
 
 
 
 
 
 
  On Dec 11, 2012, at 11:19 PM, Stephen Connolly wrote:
 
  Well I am not going to tar and feather log4j2 based on one set of
 runs on
  my machine. I would like somebody else to repeat and confirm first as
 there
  could have been some background OS update or other process stealing
 CPU
  while doing the 3 log4j2 runs.
 
  Also I do not know if I am comparing the same things. Afaik the log
 back
  branch has the latest fixes in it, while the log4j2 branch is the
 colorized
  one from a few weeks back and likely has not got the fixes required
 for the
  issues you identified with the last 3.1.0 RC
 
  We need to compare like with like to make an informed decision... I
 am just
  putting some numbers down as a starting point
 
  -Stephen
 
  On Wednesday, 12 December 2012, Kristian Rosenvold wrote:
 
  Finally some interesting numbers, and if (heaven forbid) this
 decision
  should be based on
  technical grounds, this is one of the first significant pieces to
 come
  up in this discussion.
 
  Since I am quite unfamiliar with logging (I use loose coupling and
  tests instead ;), I took the opportunity to read
  http://logging.apache.org/log4j/2.x/performance.html Somehow the
  real-life results don't seem to match up with the advertising blurp
 on
  the log4j site. While it hardly surprises me, I was wondering if
  anyone actually knows why?
 
  Kristian
 
 
 
  2012/12/12 Stephen Connolly stephen.alan.conno...@gmail.com
 javascript:;
  :
  The consistent times (i.e. repeated runs after discarding the
 first) are:
 
  3.0.4: 1min18sec
  logback: 1min13sec
  log4j2: 1min34sec
 
  The second test was building GIT hash
  85dd6e37456d30d2661e10b044efa9036c528023 of jszip-maven-plugin (@
  jszip.org)
  with the following command line:
 
  mvn -o -X clean verify -DskipTests -Dinvoker.skip
 
  [Testing heavy logging]
 
  3.0.4: 12.1sec
  logback: 12.2sec
  log4j2: 12.5sec
 
  -
  To unsubscribe, e-mail: dev-unsubscr...@maven.apache.orgjavascript:;
  For additional commands, e-mail: dev-h...@maven.apache.orgjavascript:;
 
 
 
 
 
 
  -
  To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
  For additional commands, e-mail: dev-h...@maven.apache.org

Re: Logback in Maven Core

2012-12-12 Thread Stephen Connolly

The idea behind these binaries is to give everyone the opportunity to make
decisions with regards to the technical gates:

1. Does it pass the integration tests
2. Is it roughly equivalent in terms of performance to 3.0.4 (can be better
or up to say 5% worse)

There are additional criteria people can consider as well. These proof of
concept builds should at least allow everyone to compare like with like...
as I have made a bit of effort to ensure that the diffs between the
branches are as minimal as I can get them (i.e. just what is required to
support the logging framework, no colorised loggers)

For people who see being able to implement colorised logging:

1. Logback and Log4j2 support this natively
2. JUL could be written to support this (i.e. we need to write a custom
Formatter anyway, no reason we couldn't add the ANSI support too)
3. Log4j 1.2 doesn't support, but hey if we want to we could always write
and maintain that! (If we want to have [WARNING] as before we would have to
write a formatter anyway... once again, while we are in there!)


On 12 December 2012 11:40, Stephen Connolly stephen.alan.conno...@gmail.com
 wrote:

 I have built some binaries for people to play with:

 http://people.apache.org/~stephenc/

 MD5 (apache-maven-3.1.0-pre-jul.tar.gz) = dd40afbfa64ab53f614ede19385e4a48
 MD5 (apache-maven-3.1.0-pre-jul.zip) = 731fe7136e96e2027d145d993d917f20
 MD5 (apache-maven-3.1.0-pre-log4j.tar.gz) =
 63fead6accb60ca52c8a299c39d172f5
 MD5 (apache-maven-3.1.0-pre-log4j.zip) = 64982b6395a80bb6f2d27020c9f439be
 MD5 (apache-maven-3.1.0-pre-log4j2.tar.gz) =
 cc38e6f2110d9e76f6fc9feb29c48500
 MD5 (apache-maven-3.1.0-pre-log4j2.zip) = 77eda0a4bbb6f82cfdff8e8afb408dd5
 MD5 (apache-maven-3.1.0-pre-logback.tar.gz) =
 b51215f3e1c5589772b4d65c68f28654
 MD5 (apache-maven-3.1.0-pre-logback.zip) = 6af89e67b1dca4699ef0ce57c3db6a5f

 They should all behave roughly similarly... though the log4j 1.2 version
 will probably output [WARN] and not [WARNING]



 On 12 December 2012 10:40, Stephen Connolly 
 stephen.alan.conno...@gmail.com wrote:

 I am working on getting some branches for different options.

 I think I have the logging/slf4j-jul branch done...

 I think my logging/slf4j-log4j2 branch is correct

 I am fairly certain my logging/slf4j-logback branch is correct

 My logging/slf4j-log4j (i.e. 1.2) branch needs some tweaks

 If somebody can run the integration tests on those branches and report
 the results here that would be great.


 On 12 December 2012 08:48, Stephen Connolly 
 stephen.alan.conno...@gmail.com wrote:

 Can we get a set of baseline git hashes for any versions of maven that
 we are comparing. I might see if I can pull logback out of the latest RC
 and put log4j2 and some other impls in its place so we can get some real
 apples for apples comparisons going


 On 12 December 2012 08:35, Kristian Rosenvold 
 kristian.rosenv...@gmail.com wrote:

 Yeah, measuring performance on modern cpu's is totally borked. To get
 any real measurements one probably needs to to average of 100 non-stop
 builds or similar, to counter for all the dark magics intel do with
 temperature-based overclocking.

 I think I've seen somewhere that it's possible to disable all the
 cpu-voodo in the bios. But what a pain to reboot to change those
 settings ! Or dig out the old Pentium4 from the closet.

 Kristian

 2012/12/12 Ralph Goers ralph.go...@dslextreme.com:
  I checked out Maven and used its build as a comparison.  First, I ran
 the log4j 2 build and it was taking around 59 seconds.  I then changed the
 log4j2.xml to remove the colors.  I then got an average time for Log4j 2 of
  54.76s and for Logback I get an average of 55.225s.  I consider these
 differences to be meaningless.
 
  For reference, the log4j2.xml I used is attached.
 
  Ralph
 
 
 
 
 
 
 
 
 
 
  On Dec 11, 2012, at 11:19 PM, Stephen Connolly wrote:
 
  Well I am not going to tar and feather log4j2 based on one set of
 runs on
  my machine. I would like somebody else to repeat and confirm first
 as there
  could have been some background OS update or other process stealing
 CPU
  while doing the 3 log4j2 runs.
 
  Also I do not know if I am comparing the same things. Afaik the log
 back
  branch has the latest fixes in it, while the log4j2 branch is the
 colorized
  one from a few weeks back and likely has not got the fixes required
 for the
  issues you identified with the last 3.1.0 RC
 
  We need to compare like with like to make an informed decision... I
 am just
  putting some numbers down as a starting point
 
  -Stephen
 
  On Wednesday, 12 December 2012, Kristian Rosenvold wrote:
 
  Finally some interesting numbers, and if (heaven forbid) this
 decision
  should be based on
  technical grounds, this is one of the first significant pieces to
 come
  up in this discussion.
 
  Since I am quite unfamiliar with logging (I use loose coupling and
  tests instead ;), I took the opportunity to read

Re: Logback in Maven Core

2012-12-12 Thread Daniel Kulp


I ran the CXF build like:

mvn -Pfastinstall -T8 -o

using the latest on the two branches as of 11pm last night:

3.0.4:  1:16
log4j2:  1:12
logback:  1:19

Ran each 3 times and results were fairly consistent.Thus, for me using 
parallel builds, log4j2 is the fastest, but the difference is almost irrelevant.

I'll repeat the tests later without the -T8, but this may show that 
multi-threaded logging is faster with log4j2.

Dan


On Dec 12, 2012, at 1:48 AM, Kristian Rosenvold kristian.rosenv...@gmail.com 
wrote:

 Finally some interesting numbers, and if (heaven forbid) this decision
 should be based on
 technical grounds, this is one of the first significant pieces to come
 up in this discussion.
 
 Since I am quite unfamiliar with logging (I use loose coupling and
 tests instead ;), I took the opportunity to read
 http://logging.apache.org/log4j/2.x/performance.html Somehow the
 real-life results don't seem to match up with the advertising blurp on
 the log4j site. While it hardly surprises me, I was wondering if
 anyone actually knows why?
 
 Kristian
 
 
 
 2012/12/12 Stephen Connolly stephen.alan.conno...@gmail.com:
 The consistent times (i.e. repeated runs after discarding the first) are:
 
 3.0.4: 1min18sec
 logback: 1min13sec
 log4j2: 1min34sec
 
 The second test was building GIT hash
 85dd6e37456d30d2661e10b044efa9036c528023 of jszip-maven-plugin (@jszip.org)
 with the following command line:
 
 mvn -o -X clean verify -DskipTests -Dinvoker.skip
 
 [Testing heavy logging]
 
 3.0.4: 12.1sec
 logback: 12.2sec
 log4j2: 12.5sec
 
 -
 To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
 For additional commands, e-mail: dev-h...@maven.apache.org
 

-- 
Daniel Kulp
dk...@apache.org - http://dankulp.com/blog
Talend Community Coder - http://coders.talend.com


-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org

Re: Logback in Maven Core

2012-12-12 Thread Stephen Connolly

Another criteria that people should pay attention to is whether the
implementation supports Mapped Diagnostic Contexts
http://logback.qos.ch/manual/mdc.html

AFAIU this may rule out JUL as a serious implementation... In other words
when we want to start using MDCs to make it easier to navigate the logs, if
the implementation we choose does not support MDCs then we would need to
choose again.

According to the slf4j website logback and log4j support MDCs (I am
presuming that log4j2 does also therefore)

On Tuesday, 11 December 2012, Stephen Connolly wrote:

 Given that some people are already confused as to what the exact question
 is
 I think we should clarify exactly what is the decision that is being asked.

 There has already been a decision to use the slf4j API for logging within
 core.

 *  The vast majority of plugins will still use the Maven Log interface for
 their logging.

 *  A small limited number of plugins with specific needs will use the
 slf4j API that
is exposed by core directly, but those plugins will have to do some
 work in
order to ensure that they work with Maven pre 3.1.0 as well as Maven
 post 3.1.0

My understanding is that they will have to add a slf4j implementation
 to their
dependencies... on Maven 3.1.0+ the implementation will be ignored as
 the
slf4j-api that they see on their classpath will already have been
 initialized with
core's implementation. On Maven pre-3.1.0 their slf4j-api will be
 initialized and
find their slf4j implementation.

 *  A smaller subset of plugins need to control the slf4j implementation
 and
cannot have a different implementation. Those plugins will have to add
 a
metadata flag requesting a classloader that is isolated from core's
 slf4j API.
Such plugins might be redirecting log output for processing, or some
 other
need that we have not foreseen.

On Maven 3.1.0 if the metadata flag is not present the plugin will
 likely be
borked and a newer version with the metadata flag will need to be
 released
[Though the precise behaviour in the absence of this flag is yet to be
 defined]
When the flag is enabled, the Maven Log interface will be the way the
 plugin
is required to route the information it wants logged to the user
 through to
the user.

On Maven pre-3.1.0 things will be as they always were, i.e. the Maven
 Log
interface is the way we prefer information to be logged through to the
 user.

 What is being discussed now is which *implementation* to use for the Maven
 CLI
 commands by default in core starting with Maven 3.1.0.

 There are a number of possible implementations:

 *  SLF4J Simple was initially considered. This is a very limited logger
 but would
be able to produce logging output formatted the same as Maven currently
provides. *However* there is a regression caused for some of the
 integration
tests when using SLF4J and fixing those issues currently produces
 performance
regressions as well as the current uncertainty as to whether those
 fixes will
be accepted by Ceki (the SLF4J maintainer). For this reason SLF4J Simple
is currently being rejected on technical grounds (but if sufficient
 developers
really want to go with the we don't want to choose a specific
 implementation
choice, this is the implementation you would be selecting.

 *  Logback is being proposed by Jason. AFAIK this implementation passes on
 the
technical criteria. In other words no failing integration tests and no
 significant
performance regressions.

 *  Log4j2 has been championed by Olivier. AFAIK this implementation passes
 on
the integration tests. I am not sure of the performance technical test.
 One should
note that Log4j2 is a new implementation and is not Log4j

I hope that Olivier or somebody else can confirm the integration test
 status of Log4j2
as a back end implementation and provide information on the performance
 status
of Log4j2.

 To my knowledge no other SLF4J implementations have been proposed, but if
 there are others that people want considered please provide an update here.

 The primary concern that Maven committers should apply is the technical
 merit of
 any proposed implementation.

 An implementation should be able to pass all the integration tests and not
 produce
 a significant performance regression.

 Developers should not concern themselves with the licensing terms of the
 implementation provided that the implementation is available under one of
 the
 ASL compatible licenses (i.e. Category A or Category B). If a Category B
 licensed
 implementation is chosen then for legal reasons the PMC must approve the
 use
 of that dependency. (Personally speaking, if that decision is purely based
 on
 technical grounds, I do not see why the PMC would object)

 Maven Committers can use other criteria in making th

Re: Logback in Maven Core

2012-12-12 Thread Stephen Connolly

Another criteria that people should pay attention to is whether the
implementation supports Mapped Diagnostic Contexts
http://logback.qos.ch/manual/mdc.html

AFAIU this may rule out JUL as a serious implementation... In other words
when we want to start using MDCs to make it easier to navigate the logs, if
the implementation we choose does not support MDCs then we would need to
choose again.

According to the slf4j website logback and log4j support MDCs (I am
presuming that log4j2 does also therefore)

On Tuesday, 11 December 2012, Stephen Connolly wrote:

 Given that some people are already confused as to what the exact question
 is
 I think we should clarify exactly what is the decision that is being asked.

 There has already been a decision to use the slf4j API for logging within
 core.

 *  The vast majority of plugins will still use the Maven Log interface for
 their logging.

 *  A small limited number of plugins with specific needs will use the
 slf4j API that
is exposed by core directly, but those plugins will have to do some
 work in
order to ensure that they work with Maven pre 3.1.0 as well as Maven
 post 3.1.0

My understanding is that they will have to add a slf4j implementation
 to their
dependencies... on Maven 3.1.0+ the implementation will be ignored as
 the
slf4j-api that they see on their classpath will already have been
 initialized with
core's implementation. On Maven pre-3.1.0 their slf4j-api will be
 initialized and
find their slf4j implementation.

 *  A smaller subset of plugins need to control the slf4j implementation
 and
cannot have a different implementation. Those plugins will have to add
 a
metadata flag requesting a classloader that is isolated from core's
 slf4j API.
Such plugins might be redirecting log output for processing, or some
 other
need that we have not foreseen.

On Maven 3.1.0 if the metadata flag is not present the plugin will
 likely be
borked and a newer version with the metadata flag will need to be
 released
[Though the precise behaviour in the absence of this flag is yet to be
 defined]
When the flag is enabled, the Maven Log interface will be the way the
 plugin
is required to route the information it wants logged to the user
 through to
the user.

On Maven pre-3.1.0 things will be as they always were, i.e. the Maven
 Log
interface is the way we prefer information to be logged through to the
 user.

 What is being discussed now is which *implementation* to use for the Maven
 CLI
 commands by default in core starting with Maven 3.1.0.

 There are a number of possible implementations:

 *  SLF4J Simple was initially considered. This is a very limited logger
 but would
be able to produce logging output formatted the same as Maven currently
provides. *However* there is a regression caused for some of the
 integration
tests when using SLF4J and fixing those issues currently produces
 performance
regressions as well as the current uncertainty as to whether those
 fixes will
be accepted by Ceki (the SLF4J maintainer). For this reason SLF4J Simple
is currently being rejected on technical grounds (but if sufficient
 developers
really want to go with the we don't want to choose a specific
 implementation
choice, this is the implementation you would be selecting.

 *  Logback is being proposed by Jason. AFAIK this implementation passes on
 the
technical criteria. In other words no failing integration tests and no
 significant
performance regressions.

 *  Log4j2 has been championed by Olivier. AFAIK this implementation passes
 on
the integration tests. I am not sure of the performance technical test.
 One should
note that Log4j2 is a new implementation and is not Log4j

I hope that Olivier or somebody else can confirm the integration test
 status of Log4j2
as a back end implementation and provide information on the performance
 status
of Log4j2.

 To my knowledge no other SLF4J implementations have been proposed, but if
 there are others that people want considered please provide an update here.

 The primary concern that Maven committers should apply is the technical
 merit of
 any proposed implementation.

 An implementation should be able to pass all the integration tests and not
 produce
 a significant performance regression.

 Developers should not concern themselves with the licensing terms of the
 implementation provided that the implementation is available under one of
 the
 ASL compatible licenses (i.e. Category A or Category B). If a Category B
 licensed
 implementation is chosen then for legal reasons the PMC must approve the
 use
 of that dependency. (Personally speaking, if that decision is purely based
 on
 technical grounds, I do not see why the PMC would object)

 Maven Committers can use other criteria in making th

Re: Logback in Maven Core

2012-12-12 Thread Daniel Kulp


On Dec 12, 2012, at 8:45 AM, Stephen Connolly stephen.alan.conno...@gmail.com 
wrote:

 Another criteria that people should pay attention to is whether the
 implementation supports Mapped Diagnostic Contexts
 http://logback.qos.ch/manual/mdc.html
 
 AFAIU this may rule out JUL as a serious implementation... In other words
 when we want to start using MDCs to make it easier to navigate the logs, if
 the implementation we choose does not support MDCs then we would need to
 choose again.

Well, if have to write our jul formatter to handle the WARN - WARNING thing, 
we could handle this as well.   slf4j-jdk14 sticks a BasicMDCAdapater on there 
that just stores the context in a ThreadLocal.   Nothing uses it, but the data 
would be there.   Our formatter could grab the MDC and use it if we wanted it 
to.

Not saying that we SHOULD do this, just that we could, particularly if we did 
write a formatter.

Dan




 
 According to the slf4j website logback and log4j support MDCs (I am
 presuming that log4j2 does also therefore)
 
 On Tuesday, 11 December 2012, Stephen Connolly wrote:
 
 Given that some people are already confused as to what the exact question
 is
 I think we should clarify exactly what is the decision that is being asked.
 
 There has already been a decision to use the slf4j API for logging within
 core.
 
 *  The vast majority of plugins will still use the Maven Log interface for
 their logging.
 
 *  A small limited number of plugins with specific needs will use the
 slf4j API that
   is exposed by core directly, but those plugins will have to do some
 work in
   order to ensure that they work with Maven pre 3.1.0 as well as Maven
 post 3.1.0
 
   My understanding is that they will have to add a slf4j implementation
 to their
   dependencies... on Maven 3.1.0+ the implementation will be ignored as
 the
   slf4j-api that they see on their classpath will already have been
 initialized with
   core's implementation. On Maven pre-3.1.0 their slf4j-api will be
 initialized and
   find their slf4j implementation.
 
 *  A smaller subset of plugins need to control the slf4j implementation
 and
   cannot have a different implementation. Those plugins will have to add
 a
   metadata flag requesting a classloader that is isolated from core's
 slf4j API.
   Such plugins might be redirecting log output for processing, or some
 other
   need that we have not foreseen.
 
   On Maven 3.1.0 if the metadata flag is not present the plugin will
 likely be
   borked and a newer version with the metadata flag will need to be
 released
   [Though the precise behaviour in the absence of this flag is yet to be
 defined]
   When the flag is enabled, the Maven Log interface will be the way the
 plugin
   is required to route the information it wants logged to the user
 through to
   the user.
 
   On Maven pre-3.1.0 things will be as they always were, i.e. the Maven
 Log
   interface is the way we prefer information to be logged through to the
 user.
 
 What is being discussed now is which *implementation* to use for the Maven
 CLI
 commands by default in core starting with Maven 3.1.0.
 
 There are a number of possible implementations:
 
 *  SLF4J Simple was initially considered. This is a very limited logger
 but would
   be able to produce logging output formatted the same as Maven currently
   provides. *However* there is a regression caused for some of the
 integration
   tests when using SLF4J and fixing those issues currently produces
 performance
   regressions as well as the current uncertainty as to whether those
 fixes will
   be accepted by Ceki (the SLF4J maintainer). For this reason SLF4J Simple
   is currently being rejected on technical grounds (but if sufficient
 developers
   really want to go with the we don't want to choose a specific
 implementation
   choice, this is the implementation you would be selecting.
 
 *  Logback is being proposed by Jason. AFAIK this implementation passes on
 the
   technical criteria. In other words no failing integration tests and no
 significant
   performance regressions.
 
 *  Log4j2 has been championed by Olivier. AFAIK this implementation passes
 on
   the integration tests. I am not sure of the performance technical test.
 One should
   note that Log4j2 is a new implementation and is not Log4j
 
   I hope that Olivier or somebody else can confirm the integration test
 status of Log4j2
   as a back end implementation and provide information on the performance
 status
   of Log4j2.
 
 To my knowledge no other SLF4J implementations have been proposed, but if
 there are others that people want considered please provide an update here.
 
 The primary concern that Maven committers should apply is the technical
 merit of
 any proposed implementation.
 
 An implementation should be able to pass all the integration tests and not
 produce
 a significant performance regression.
 
 Developers should not concern themselves with the licensing terms of the
 implementation

Re: Logback in Maven Core

2012-12-12 Thread Gary Gregory

On Wed, Dec 12, 2012 at 8:38 AM, Stephen Connolly 
stephen.alan.conno...@gmail.com wrote:

 Another criteria that people should pay attention to is whether the
 implementation supports Mapped Diagnostic Contexts
 http://logback.qos.ch/manual/mdc.html

 AFAIU this may rule out JUL as a serious implementation... In other words
 when we want to start using MDCs to make it easier to navigate the logs, if
 the implementation we choose does not support MDCs then we would need to
 choose again.

 According to the slf4j website logback and log4j support MDCs (I am
 presuming that log4j2 does also therefore)


Yes, please see
https://logging.apache.org/log4j/2.x/manual/thread-context.html

Gary



 On Tuesday, 11 December 2012, Stephen Connolly wrote:

  Given that some people are already confused as to what the exact question
  is
  I think we should clarify exactly what is the decision that is being
 asked.
 
  There has already been a decision to use the slf4j API for logging within
  core.
 
  *  The vast majority of plugins will still use the Maven Log interface
 for
  their logging.
 
  *  A small limited number of plugins with specific needs will use the
  slf4j API that
 is exposed by core directly, but those plugins will have to do some
  work in
 order to ensure that they work with Maven pre 3.1.0 as well as Maven
  post 3.1.0
 
 My understanding is that they will have to add a slf4j implementation
  to their
 dependencies... on Maven 3.1.0+ the implementation will be ignored as
  the
 slf4j-api that they see on their classpath will already have been
  initialized with
 core's implementation. On Maven pre-3.1.0 their slf4j-api will be
  initialized and
 find their slf4j implementation.
 
  *  A smaller subset of plugins need to control the slf4j implementation
  and
 cannot have a different implementation. Those plugins will have to add
  a
 metadata flag requesting a classloader that is isolated from core's
  slf4j API.
 Such plugins might be redirecting log output for processing, or some
  other
 need that we have not foreseen.
 
 On Maven 3.1.0 if the metadata flag is not present the plugin will
  likely be
 borked and a newer version with the metadata flag will need to be
  released
 [Though the precise behaviour in the absence of this flag is yet to be
  defined]
 When the flag is enabled, the Maven Log interface will be the way the
  plugin
 is required to route the information it wants logged to the user
  through to
 the user.
 
 On Maven pre-3.1.0 things will be as they always were, i.e. the Maven
  Log
 interface is the way we prefer information to be logged through to the
  user.
 
  What is being discussed now is which *implementation* to use for the
 Maven
  CLI
  commands by default in core starting with Maven 3.1.0.
 
  There are a number of possible implementations:
 
  *  SLF4J Simple was initially considered. This is a very limited logger
  but would
 be able to produce logging output formatted the same as Maven
 currently
 provides. *However* there is a regression caused for some of the
  integration
 tests when using SLF4J and fixing those issues currently produces
  performance
 regressions as well as the current uncertainty as to whether those
  fixes will
 be accepted by Ceki (the SLF4J maintainer). For this reason SLF4J
 Simple
 is currently being rejected on technical grounds (but if sufficient
  developers
 really want to go with the we don't want to choose a specific
  implementation
 choice, this is the implementation you would be selecting.
 
  *  Logback is being proposed by Jason. AFAIK this implementation passes
 on
  the
 technical criteria. In other words no failing integration tests and no
  significant
 performance regressions.
 
  *  Log4j2 has been championed by Olivier. AFAIK this implementation
 passes
  on
 the integration tests. I am not sure of the performance technical
 test.
  One should
 note that Log4j2 is a new implementation and is not Log4j
 
 I hope that Olivier or somebody else can confirm the integration test
  status of Log4j2
 as a back end implementation and provide information on the
 performance
  status
 of Log4j2.
 
  To my knowledge no other SLF4J implementations have been proposed, but if
  there are others that people want considered please provide an update
 here.
 
  The primary concern that Maven committers should apply is the technical
  merit of
  any proposed implementation.
 
  An implementation should be able to pass all the integration tests and
 not
  produce
  a significant performance regression.
 
  Developers should not concern themselves with the licensing terms of the
  implementation provided that the implementation is available under one of
  the
  ASL compatible licenses (i.e. Category A or Category B). If a Category B
  licensed
  implementation is chosen then for legal reasons the PMC

Re: Logback in Maven Core

2012-12-12 Thread Stephen Connolly

On Wednesday, 12 December 2012, Daniel Kulp wrote:


 On Dec 12, 2012, at 8:45 AM, Stephen Connolly 
 stephen.alan.conno...@gmail.com javascript:; wrote:

  Another criteria that people should pay attention to is whether the
  implementation supports Mapped Diagnostic Contexts
  http://logback.qos.ch/manual/mdc.html
 
  AFAIU this may rule out JUL as a serious implementation... In other words
  when we want to start using MDCs to make it easier to navigate the logs,
 if
  the implementation we choose does not support MDCs then we would need to
  choose again.

 Well, if have to write our jul formatter to handle the WARN - WARNING
 thing, we could handle this as well.   slf4j-jdk14 sticks a
 BasicMDCAdapater on there that just stores the context in a ThreadLocal.
 Nothing uses it, but the data would be there.   Our formatter could grab
 the MDC and use it if we wanted it to.


I cannot recall if JUL guarantees that the formatter will be called from
the same thread as the log record was generated on.

If it maintains that as an invariant then we could rescue JUL as a possible
implementation

If not then I fear it us dead in the water

-Stephen


 Not saying that we SHOULD do this, just that we could, particularly if we
 did write a formatter.

 Dan




 
  According to the slf4j website logback and log4j support MDCs (I am
  presuming that log4j2 does also therefore)
 
  On Tuesday, 11 December 2012, Stephen Connolly wrote:
 
  Given that some people are already confused as to what the exact
 question
  is
  I think we should clarify exactly what is the decision that is being
 asked.
 
  There has already been a decision to use the slf4j API for logging
 within
  core.
 
  *  The vast majority of plugins will still use the Maven Log interface
 for
  their logging.
 
  *  A small limited number of plugins with specific needs will use the
  slf4j API that
is exposed by core directly, but those plugins will have to do some
  work in
order to ensure that they work with Maven pre 3.1.0 as well as Maven
  post 3.1.0
 
My understanding is that they will have to add a slf4j implementation
  to their
dependencies... on Maven 3.1.0+ the implementation will be ignored as
  the
slf4j-api that they see on their classpath will already have been
  initialized with
core's implementation. On Maven pre-3.1.0 their slf4j-api will be
  initialized and
find their slf4j implementation.
 
  *  A smaller subset of plugins need to control the slf4j implementation
  and
cannot have a different implementation. Those plugins will have to add
  a
metadata flag requesting a classloader that is isolated from core's
  slf4j API.
Such plugins might be redirecting log output for processing, or some
  other
need that we have not foreseen.
 
On Maven 3.1.0 if the metadata flag is not present the plugin will
  likely be
borked and a newer version with the metadata flag will need to be
  released
[Though the precise behaviour in the absence of this flag is yet to be
  defined]
When the flag is enabled, the Maven Log interface will be the way the
  plugin
is required to route the information it wants logged to the user
  through to
the user.
 
On Maven pre-3.1.0 things will be as they always were, i.e. the Maven
  Log
interface is the way we prefer information to be logged through to the
  user.
 
  What is being discussed now is which *implementation* to use for the
 Maven
  CLI
  commands by default in core starting with Maven 3.1.0.
 
  There are a number of possible implementations:
 
  *  SLF4J Simple was initially considered. This is a very limited logger
  but would
be able to produce logging output formatted the same as Maven
 currently
provides. *However* there is a regression caused for some of the
  integration
tests when using SLF4J and fixing those issues currently produces
  performance
regressions as well as the current uncertainty as to whether those
  fixes will
be accepted by Ceki (the SLF4J maintainer). For this reason SLF4J
 Simple
is currently being rejected on technical grounds (but if sufficient
  developers
really want to go with the we don't want to choose a specific
  implementation
choice, this is the implementation you would be selecting.
 
  *  Logback is being proposed by Jason. AFAIK this implementation passes
 on
  the
technical criteria. In other words no failing integration tests and no
  significant
performance regressions.
 
  *  Log4j2 has been championed by Olivier. AFAIK this implementation
 passes
  on
the integration tests. I am not sure of the performance technical
 test.
  One should
note that Log4j2 is a new implementation and is not Log4j
 
I hope that Olivier or somebody else can confirm the integration test
  status of Log4j2
as a back end implementation and provide information on the
 performance
 --
 Daniel Kulp
 dk...@apache.org javascript:; - http://dankulp.com/blog

Re: Logback in Maven Core

2012-12-12 Thread Jason van Zyl

I'll let folks focus on their evaluations and I'll work on the sample plugins 
and classloader isolation.

On Dec 12, 2012, at 9:23 AM, Stephen Connolly stephen.alan.conno...@gmail.com 
wrote:

 On Wednesday, 12 December 2012, Daniel Kulp wrote:
 
 
 On Dec 12, 2012, at 8:45 AM, Stephen Connolly 
 stephen.alan.conno...@gmail.com javascript:; wrote:
 
 Another criteria that people should pay attention to is whether the
 implementation supports Mapped Diagnostic Contexts
 http://logback.qos.ch/manual/mdc.html
 
 AFAIU this may rule out JUL as a serious implementation... In other words
 when we want to start using MDCs to make it easier to navigate the logs,
 if
 the implementation we choose does not support MDCs then we would need to
 choose again.
 
 Well, if have to write our jul formatter to handle the WARN - WARNING
 thing, we could handle this as well.   slf4j-jdk14 sticks a
 BasicMDCAdapater on there that just stores the context in a ThreadLocal.
 Nothing uses it, but the data would be there.   Our formatter could grab
 the MDC and use it if we wanted it to.
 
 
 I cannot recall if JUL guarantees that the formatter will be called from
 the same thread as the log record was generated on.
 
 If it maintains that as an invariant then we could rescue JUL as a possible
 implementation
 
 If not then I fear it us dead in the water
 
 -Stephen
 
 
 Not saying that we SHOULD do this, just that we could, particularly if we
 did write a formatter.
 
 Dan
 
 
 
 
 
 According to the slf4j website logback and log4j support MDCs (I am
 presuming that log4j2 does also therefore)
 
 On Tuesday, 11 December 2012, Stephen Connolly wrote:
 
 Given that some people are already confused as to what the exact
 question
 is
 I think we should clarify exactly what is the decision that is being
 asked.
 
 There has already been a decision to use the slf4j API for logging
 within
 core.
 
 *  The vast majority of plugins will still use the Maven Log interface
 for
 their logging.
 
 *  A small limited number of plugins with specific needs will use the
 slf4j API that
  is exposed by core directly, but those plugins will have to do some
 work in
  order to ensure that they work with Maven pre 3.1.0 as well as Maven
 post 3.1.0
 
  My understanding is that they will have to add a slf4j implementation
 to their
  dependencies... on Maven 3.1.0+ the implementation will be ignored as
 the
  slf4j-api that they see on their classpath will already have been
 initialized with
  core's implementation. On Maven pre-3.1.0 their slf4j-api will be
 initialized and
  find their slf4j implementation.
 
 *  A smaller subset of plugins need to control the slf4j implementation
 and
  cannot have a different implementation. Those plugins will have to add
 a
  metadata flag requesting a classloader that is isolated from core's
 slf4j API.
  Such plugins might be redirecting log output for processing, or some
 other
  need that we have not foreseen.
 
  On Maven 3.1.0 if the metadata flag is not present the plugin will
 likely be
  borked and a newer version with the metadata flag will need to be
 released
  [Though the precise behaviour in the absence of this flag is yet to be
 defined]
  When the flag is enabled, the Maven Log interface will be the way the
 plugin
  is required to route the information it wants logged to the user
 through to
  the user.
 
  On Maven pre-3.1.0 things will be as they always were, i.e. the Maven
 Log
  interface is the way we prefer information to be logged through to the
 user.
 
 What is being discussed now is which *implementation* to use for the
 Maven
 CLI
 commands by default in core starting with Maven 3.1.0.
 
 There are a number of possible implementations:
 
 *  SLF4J Simple was initially considered. This is a very limited logger
 but would
  be able to produce logging output formatted the same as Maven
 currently
  provides. *However* there is a regression caused for some of the
 integration
  tests when using SLF4J and fixing those issues currently produces
 performance
  regressions as well as the current uncertainty as to whether those
 fixes will
  be accepted by Ceki (the SLF4J maintainer). For this reason SLF4J
 Simple
  is currently being rejected on technical grounds (but if sufficient
 developers
  really want to go with the we don't want to choose a specific
 implementation
  choice, this is the implementation you would be selecting.
 
 *  Logback is being proposed by Jason. AFAIK this implementation passes
 on
 the
  technical criteria. In other words no failing integration tests and no
 significant
  performance regressions.
 
 *  Log4j2 has been championed by Olivier. AFAIK this implementation
 passes
 on
  the integration tests. I am not sure of the performance technical
 test.
 One should
  note that Log4j2 is a new implementation and is not Log4j
 
  I hope that Olivier or somebody else can confirm the integration test
 status of Log4j2
  as a back end implementation and provide

Re: A very interesting performance regression in 3.1 embedded mode

2012-12-12 Thread Igor Fedorenko


For tests, I think the easiest is to check number of realms at the end
of each test and drop plexus container if it grew over certain number of
realms. Pick the number large enough to fit in 128M of permgen and I
think this will provide good tradeoff between performance and memory
usage. This does mean separate container per each test thread, but I
think this is required for other reasons.

For IDEs, I would not try to guess what they need and let IDE developers
come with proposed improvements to caching. I can't speak for other
IDEs, but m2e for example, already deals with project realms properly
and plugin realms are not usually an issue, so we don't really have any
issues we need to fix.

--
Regards,
Igor

On 2012-12-12 3:31 AM, Kristian Rosenvold wrote:

2012/12/12 Milos Kleint mkle...@gmail.com:

how much memory will be freed by the soft reference? if it's not a big
chunk, most likely not worth it, soft references are released only
when your VM is really, really in trouble. by the time it gets
released, you've been slowed down by repeatedly hitting the ceiling of
your memory and CGed frequently.


I think the appropriate question to ask is how much memory would be
freed by an eviction strategy? and maybe who needs it?.

The answer to how much can precisely be formulated as a truckload.
The surefire IT's in embedded mode require 350MB permgen and 500MB
memory. When things are evicted those numbers are more or less halved
(I seem to believe they ran in 128MB permgen!). I will come up with
some hard numbers on that at some point...

An embedded core used to hold on to a *lot* of classloaders, both for
plugins, extensions and projects. Some of these are never freed and
will hence leak for as long as the embedded container is being kept
alive. So for someone IDE-like embedding maven it would definitely
make sense to release and reallocate the embedded maven instance every
time a project is loaded/reloaded. Just recently it was changed to
always deallocate the plugins (but still leaking extension/project
realms), which is probably too much or too little. From an IDE point
of view 0.5 seconds added overhead /may/ be ok (while certainly not
world class performance).

For something like an embedded test-run it might make sense to just
ditch the embedded container every N runs and start with a fresh one.

The real problem starts when you decide to hold onto the embedded instance and
declare that the instance should release class realms according to
some eviction strategy. Like so many times before, neither
softreferences nor weakreferences really seem appropriate; one is too
liberal in letting go, the other too conservative. Currently core does
not track actual usage of each class realm (although it would be
fairly simple to implement) ; if it did we could do all sorts of cool
stuff.

I prefer a /predictable/ eviction strategy. We *could* do something like
a round-robin eviction (evict a rotating 25% of all classrealms every
time) or a usage based eviction (evict all unused and every N usages
of a given class realm). But then again maybe this whole issue is best
solved by restoring the old strategy of no eviction and tell the
clients to evict us every now and then. That would work very well for
embedded test runs, unsure what something like m2e would do about
it

Kristian










just an illustrative story, most likely not related to this usecase
but still interesting
while embedding maven in netbeans, we've SoftReferenced MavenProject
at some point in time. MavenProjects can hold a big object tree,
mostly via the cache in ProjectBuildingRequest. So once you run out of
memory, the SRs get dropped however sometimes the MP instances are
again immediately required, so they are loaded again. And dropped. And
loaded and the IDE grinds to a halt. In some cases, one gets OOME
fairly fast, but in others it can take fairly long.

BTW we don't embed maven builds, just MavenProject loading in netbeans

Milos

On Sun, Dec 9, 2012 at 11:04 PM, Christian Schulte c...@schulte.it wrote:

Am 12/09/12 22:58, schrieb Kristian Rosenvold:


Anyone else have any ideas about eviction strategies ?



Without having looked at the code. GC driven by using soft references.

--
Christian


-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org



-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org



-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org



-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail:

Re: A very interesting performance regression in 3.1 embedded mode

2012-12-12 Thread Kristian Rosenvold

Does that mean m2e ditches the container every now and then?

In that case the whole unloading jason implemented can be reverted...?

K

Den 12. des. 2012 kl. 16:44 skrev Igor Fedorenko i...@ifedorenko.com:

 For tests, I think the easiest is to check number of realms at the end
 of each test and drop plexus container if it grew over certain number of
 realms. Pick the number large enough to fit in 128M of permgen and I
 think this will provide good tradeoff between performance and memory
 usage. This does mean separate container per each test thread, but I
 think this is required for other reasons.

 For IDEs, I would not try to guess what they need and let IDE developers
 come with proposed improvements to caching. I can't speak for other
 IDEs, but m2e for example, already deals with project realms properly
 and plugin realms are not usually an issue, so we don't really have any
 issues we need to fix.

 --
 Regards,
 Igor

 On 2012-12-12 3:31 AM, Kristian Rosenvold wrote:
 2012/12/12 Milos Kleint mkle...@gmail.com:
 how much memory will be freed by the soft reference? if it's not a big
 chunk, most likely not worth it, soft references are released only
 when your VM is really, really in trouble. by the time it gets
 released, you've been slowed down by repeatedly hitting the ceiling of
 your memory and CGed frequently.

 I think the appropriate question to ask is how much memory would be
 freed by an eviction strategy? and maybe who needs it?.

 The answer to how much can precisely be formulated as a truckload.
 The surefire IT's in embedded mode require 350MB permgen and 500MB
 memory. When things are evicted those numbers are more or less halved
 (I seem to believe they ran in 128MB permgen!). I will come up with
 some hard numbers on that at some point...

 An embedded core used to hold on to a *lot* of classloaders, both for
 plugins, extensions and projects. Some of these are never freed and
 will hence leak for as long as the embedded container is being kept
 alive. So for someone IDE-like embedding maven it would definitely
 make sense to release and reallocate the embedded maven instance every
 time a project is loaded/reloaded. Just recently it was changed to
 always deallocate the plugins (but still leaking extension/project
 realms), which is probably too much or too little. From an IDE point
 of view 0.5 seconds added overhead /may/ be ok (while certainly not
 world class performance).

 For something like an embedded test-run it might make sense to just
 ditch the embedded container every N runs and start with a fresh one.

 The real problem starts when you decide to hold onto the embedded instance 
 and
 declare that the instance should release class realms according to
 some eviction strategy. Like so many times before, neither
 softreferences nor weakreferences really seem appropriate; one is too
 liberal in letting go, the other too conservative. Currently core does
 not track actual usage of each class realm (although it would be
 fairly simple to implement) ; if it did we could do all sorts of cool
 stuff.

 I prefer a /predictable/ eviction strategy. We *could* do something like
 a round-robin eviction (evict a rotating 25% of all classrealms every
 time) or a usage based eviction (evict all unused and every N usages
 of a given class realm). But then again maybe this whole issue is best
 solved by restoring the old strategy of no eviction and tell the
 clients to evict us every now and then. That would work very well for
 embedded test runs, unsure what something like m2e would do about
 it

 Kristian









 just an illustrative story, most likely not related to this usecase
 but still interesting
 while embedding maven in netbeans, we've SoftReferenced MavenProject
 at some point in time. MavenProjects can hold a big object tree,
 mostly via the cache in ProjectBuildingRequest. So once you run out of
 memory, the SRs get dropped however sometimes the MP instances are
 again immediately required, so they are loaded again. And dropped. And
 loaded and the IDE grinds to a halt. In some cases, one gets OOME
 fairly fast, but in others it can take fairly long.

 BTW we don't embed maven builds, just MavenProject loading in netbeans

 Milos

 On Sun, Dec 9, 2012 at 11:04 PM, Christian Schulte c...@schulte.it wrote:
 Am 12/09/12 22:58, schrieb Kristian Rosenvold:

 Anyone else have any ideas about eviction strategies ?

 Without having looked at the code. GC driven by using soft references.

 --
 Christian


 -
 To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
 For additional commands, e-mail: dev-h...@maven.apache.org

 -
 To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
 For additional commands, e-mail: dev-h...@maven.apache.org

 -
 To unsubscribe, e-mail:

Re: A very interesting performance regression in 3.1 embedded mode

2012-12-12 Thread Jason van Zyl

In embedded mode the ITs were failing without the change. I can't remember what 
you were looking at but what problem is it causing? 

jvz

On 2012-12-12, at 11:35 AM, Kristian Rosenvold kristian.rosenv...@zenior.no 
wrote:

 Does that mean m2e ditches the container every now and then?
 
 In that case the whole unloading jason implemented can be reverted...?
 
 K
 
 Den 12. des. 2012 kl. 16:44 skrev Igor Fedorenko i...@ifedorenko.com:
 
 For tests, I think the easiest is to check number of realms at the end
 of each test and drop plexus container if it grew over certain number of
 realms. Pick the number large enough to fit in 128M of permgen and I
 think this will provide good tradeoff between performance and memory
 usage. This does mean separate container per each test thread, but I
 think this is required for other reasons.
 
 For IDEs, I would not try to guess what they need and let IDE developers
 come with proposed improvements to caching. I can't speak for other
 IDEs, but m2e for example, already deals with project realms properly
 and plugin realms are not usually an issue, so we don't really have any
 issues we need to fix.
 
 --
 Regards,
 Igor
 
 On 2012-12-12 3:31 AM, Kristian Rosenvold wrote:
 2012/12/12 Milos Kleint mkle...@gmail.com:
 how much memory will be freed by the soft reference? if it's not a big
 chunk, most likely not worth it, soft references are released only
 when your VM is really, really in trouble. by the time it gets
 released, you've been slowed down by repeatedly hitting the ceiling of
 your memory and CGed frequently.
 
 I think the appropriate question to ask is how much memory would be
 freed by an eviction strategy? and maybe who needs it?.
 
 The answer to how much can precisely be formulated as a truckload.
 The surefire IT's in embedded mode require 350MB permgen and 500MB
 memory. When things are evicted those numbers are more or less halved
 (I seem to believe they ran in 128MB permgen!). I will come up with
 some hard numbers on that at some point...
 
 An embedded core used to hold on to a *lot* of classloaders, both for
 plugins, extensions and projects. Some of these are never freed and
 will hence leak for as long as the embedded container is being kept
 alive. So for someone IDE-like embedding maven it would definitely
 make sense to release and reallocate the embedded maven instance every
 time a project is loaded/reloaded. Just recently it was changed to
 always deallocate the plugins (but still leaking extension/project
 realms), which is probably too much or too little. From an IDE point
 of view 0.5 seconds added overhead /may/ be ok (while certainly not
 world class performance).
 
 For something like an embedded test-run it might make sense to just
 ditch the embedded container every N runs and start with a fresh one.
 
 The real problem starts when you decide to hold onto the embedded instance 
 and
 declare that the instance should release class realms according to
 some eviction strategy. Like so many times before, neither
 softreferences nor weakreferences really seem appropriate; one is too
 liberal in letting go, the other too conservative. Currently core does
 not track actual usage of each class realm (although it would be
 fairly simple to implement) ; if it did we could do all sorts of cool
 stuff.
 
 I prefer a /predictable/ eviction strategy. We *could* do something like
 a round-robin eviction (evict a rotating 25% of all classrealms every
 time) or a usage based eviction (evict all unused and every N usages
 of a given class realm). But then again maybe this whole issue is best
 solved by restoring the old strategy of no eviction and tell the
 clients to evict us every now and then. That would work very well for
 embedded test runs, unsure what something like m2e would do about
 it
 
 Kristian
 
 
 
 
 
 
 
 
 
 just an illustrative story, most likely not related to this usecase
 but still interesting
 while embedding maven in netbeans, we've SoftReferenced MavenProject
 at some point in time. MavenProjects can hold a big object tree,
 mostly via the cache in ProjectBuildingRequest. So once you run out of
 memory, the SRs get dropped however sometimes the MP instances are
 again immediately required, so they are loaded again. And dropped. And
 loaded and the IDE grinds to a halt. In some cases, one gets OOME
 fairly fast, but in others it can take fairly long.
 
 BTW we don't embed maven builds, just MavenProject loading in netbeans
 
 Milos
 
 On Sun, Dec 9, 2012 at 11:04 PM, Christian Schulte c...@schulte.it wrote:
 Am 12/09/12 22:58, schrieb Kristian Rosenvold:
 
 Anyone else have any ideas about eviction strategies ?
 
 Without having looked at the code. GC driven by using soft references.
 
 --
 Christian
 
 
 -
 To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
 For additional commands, e-mail: dev-h...@maven.apache.org

Re: A very interesting performance regression in 3.1 embedded mode

2012-12-12 Thread Igor Fedorenko




On 2012-12-12 11:35 AM, Kristian Rosenvold wrote:

Does that mean m2e ditches the container every now and then?



No. m2e keeps the same container but injects its own cache
implementations that allows purging project-specific cache entries
whenever workspace project is re-read or removed from workspace. There
is also logic to dispose project realms.

m2e never releases plugin realms, but as I mentioned, this does not
cause any immediate problems so I have not spent much time on a solution.


In that case the whole unloading jason implemented can be reverted...?



That was actually my change Jason committed for me. I could not remember
my svn password at the time :-) The change was meant to deal with
classloader leaks during maven core IT execution in embedded mode. I was
getting OOME permgen pretty much immediately without this change. The
change did not cause any meaningful performance problems for me either
(I am still curious to know where the time difference comes from).

--
Regards,
Igor


K

Den 12. des. 2012 kl. 16:44 skrev Igor Fedorenkoi...@ifedorenko.com:


For tests, I think the easiest is to check number of realms at the end
of each test and drop plexus container if it grew over certain number of
realms. Pick the number large enough to fit in 128M of permgen and I
think this will provide good tradeoff between performance and memory
usage. This does mean separate container per each test thread, but I
think this is required for other reasons.

For IDEs, I would not try to guess what they need and let IDE developers
come with proposed improvements to caching. I can't speak for other
IDEs, but m2e for example, already deals with project realms properly
and plugin realms are not usually an issue, so we don't really have any
issues we need to fix.



-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org

Re: A very interesting performance regression in 3.1 embedded mode

2012-12-12 Thread Jason van Zyl


On Dec 12, 2012, at 12:09 PM, Igor Fedorenko i...@ifedorenko.com wrote:

 
 
 On 2012-12-12 11:35 AM, Kristian Rosenvold wrote:
 Does that mean m2e ditches the container every now and then?
 
 
 No. m2e keeps the same container but injects its own cache
 implementations that allows purging project-specific cache entries
 whenever workspace project is re-read or removed from workspace. There
 is also logic to dispose project realms.
 
 m2e never releases plugin realms, but as I mentioned, this does not
 cause any immediate problems so I have not spent much time on a solution.
 
 In that case the whole unloading jason implemented can be reverted...?
 
 
 That was actually my change Jason committed for me. I could not remember
 my svn password at the time :-)

I applied the patch as you gave it to me which had your name in it. If git 
apply strips it out I didn't know that, I just assumed it would show up as you.

 --
 Regards,
 Igor
 
 K
 
 Den 12. des. 2012 kl. 16:44 skrev Igor Fedorenkoi...@ifedorenko.com:
 
 For tests, I think the easiest is to check number of realms at the end
 of each test and drop plexus container if it grew over certain number of
 realms. Pick the number large enough to fit in 128M of permgen and I
 think this will provide good tradeoff between performance and memory
 usage. This does mean separate container per each test thread, but I
 think this is required for other reasons.
 
 For IDEs, I would not try to guess what they need and let IDE developers
 come with proposed improvements to caching. I can't speak for other
 IDEs, but m2e for example, already deals with project realms properly
 and plugin realms are not usually an issue, so we don't really have any
 issues we need to fix.
 
 
 -
 To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
 For additional commands, e-mail: dev-h...@maven.apache.org
 

Thanks,

Jason

--
Jason van Zyl
Founder  CTO, Sonatype
Founder,  Apache Maven
http://twitter.com/jvanzyl
-

A man enjoys his work when he understands the whole and when he
is responsible for the quality of the whole

 -- Christopher Alexander, A Pattern Language

Re: A very interesting performance regression in 3.1 embedded mode

2012-12-12 Thread Christian Schulte

Am 12/12/12 08:23, schrieb Milos Kleint:
 how much memory will be freed by the soft reference? if it's not a big
 chunk, most likely not worth it, soft references are released only
 when your VM is really, really in trouble. by the time it gets
 released, you've been slowed down by repeatedly hitting the ceiling of
 your memory and CGed frequently.

Driving the VM constantly at its memory limits soft references won't
help, of course.

 
 just an illustrative story, most likely not related to this usecase
 but still interesting
 while embedding maven in netbeans, we've SoftReferenced MavenProject
 at some point in time. MavenProjects can hold a big object tree,
 mostly via the cache in ProjectBuildingRequest. So once you run out of
 memory, the SRs get dropped however sometimes the MP instances are
 again immediately required, so they are loaded again. And dropped. And
 loaded and the IDE grinds to a halt. In some cases, one gets OOME
 fairly fast, but in others it can take fairly long.

That's exactly what happens when memory requirements are larger than
what is available. Soft/weak references can be of great help when used
in a fine-grained manner. Using some third party library with large
memory requirements there is nearly no way to manage its memory
requirements from client code efficiently. Exposing some cache
infrastructure by providing cache interfaces with get/put methods will
not work efficiently most of the time also. Going that road, those
interfaces will evolve and at some time they will look like callback
interfaces of the garbage collector. At that point the library could as
well use soft references internally and not expose any memory management
related interfaces at all. This already matches with what Kristian is
suggesting.

[...]
B) Cache eviction mechanism in DefaultPluginRealmCache and maybe
DefaultProjectRealmCache. Maybe ditch everything
every now and then or half the plugins every 5 invocations or similar.
[...]


Don't re-invent the garbage collector - just use it. I know this can
lead to major refactorings internally. I still haven't looked at the code.


-- 
Christian


-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org

Re: A very interesting performance regression in 3.1 embedded mode

2012-12-12 Thread Kristian Rosenvold

2012/12/12 Igor Fedorenko i...@ifedorenko.com:


 On 2012-12-12 11:35 AM, Kristian Rosenvold wrote:

 Does that mean m2e ditches the container every now and then?


 No. m2e keeps the same container but injects its own cache
 implementations that allows purging project-specific cache entries
 whenever workspace project is re-read or removed from workspace. There
 is also logic to dispose project realms.

Excellent, that was the piece of information I was missing!

 That was actually my change Jason committed for me. I could not remember
 my svn password at the time :-) The change was meant to deal with
 classloader leaks during maven core IT execution in embedded mode. I was
 getting OOME permgen pretty much immediately without this change. The
 change did not cause any meaningful performance problems for me either
 (I am still curious to know where the time difference comes from).

Equally excellent! All of this means there is really no hard-pressing use-case
for solving the cache eviction problem for anyone but embedded
test-runs. And that's just
Computer Programming, I know how to do that!

Kristian

-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org

Re: Logback in Maven Core

Re: Logback in Maven Core

Re: A very interesting performance regression in 3.1 embedded mode

Re: Logback in Maven Core

Re: Logback in Maven Core

Re: Logback in Maven Core

Re: git commit: Basic experiment with log4j 1.2.

Re: git commit: Basic experiment with log4j 1.2.

Re: Logback in Maven Core

Re: Logback in Maven Core

Re: Logback in Maven Core

Re: Logback in Maven Core

Re: Logback in Maven Core

Re: Logback in Maven Core

Re: Logback in Maven Core

Re: Logback in Maven Core

Re: Logback in Maven Core

Re: Logback in Maven Core

Re: A very interesting performance regression in 3.1 embedded mode

Re: A very interesting performance regression in 3.1 embedded mode

Re: A very interesting performance regression in 3.1 embedded mode

Re: A very interesting performance regression in 3.1 embedded mode

Re: A very interesting performance regression in 3.1 embedded mode

Re: A very interesting performance regression in 3.1 embedded mode

Re: A very interesting performance regression in 3.1 embedded mode

25 matches

Site Navigation

Mail list logo

Footer information