Re: [jira] [Commented] (GROOVY-8592) Migrate command line tools to picocli version of CliBuilder

[Paul King]

2.5.9

On Fri, May 25, 2018 at 11:35 AM, Daniel Sun (JIRA)  wrote:
>
> [ 
> https://issues.apache.org/jira/browse/GROOVY-8592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16490099#comment-16490099
>  ]
>
> Daniel Sun commented on GROOVY-8592:
> 
>
> [~paulk] 2.5.0-rc-4?
>
>> Migrate command line tools to picocli version of CliBuilder
>> ---
>>
>> Key: GROOVY-8592
>> URL: https://issues.apache.org/jira/browse/GROOVY-8592
>> Project: Groovy
>>  Issue Type: Improvement
>>  Components: command line processing
>>Affects Versions: 2.5.0-rc-3
>>Reporter: Remko Popma
>>Priority: Major
>> Fix For: 2.5.0
>>
>>
>> The following command line tools use either 
>> {{groovy.cli.commons.CliBuilder}} or implicitly use 
>> {{groovy.util.CliBuilder}} which delegates to the commons-cli version of 
>> CliBuilder:
>> * groovyConsole (groovy.ui.Console)
>> * groovydoc (org.codehaus.groovy.tools.groovydoc.Main)
>> * groovysh (org.codehaus.groovy.tools.shell.Main)
>> This ticket proposes to use {{groovy.cli.picocli.CliBuilder}} instead, so 
>> that all command line tools in the {{bin}} directory of the Groovy 
>> distribution show usage help in ANSI colors.
>
>
>
> --
> This message was sent by Atlassian JIRA
> (v7.6.3#76005)

Re: Breaking changes in 3.0 [was: Re: 2.5.0-rc-3]

[Jesper Steen Møller]

On 23 May 2018, at 13.24, Cédric Champeau  wrote:
> 
> +1, regarding indy by default, I wonder if we could provide the "old" MOP as 
> a backward compatibility runtime jar...
> 

Yes, but that would be pretty tricky if we want to target JPMS too, due to 
split packages.

I wonder if it'd be possible to split it all up into (JPMS) modules like this:

groovy-core
groovy-compiler
groovy-xml
groovy-cli
...

And then provide a "Groovy 2.x" compatibility JARs for stuff like MOP1 
compatibility, modules that have moved, etc. and provide that in the groovy-all 
jar?

-Jesper

> Le mer. 23 mai 2018 à 13:11, Jesper Steen Møller  > a écrit :
> 
> > On 23 May 2018, at 12.23, Russel Winder  > > wrote:
> > 
> > On Wed, 2018-05-23 at 00:28 +1000, Paul King wrote:
> >> No plans to go to 18/19 model at this stage.
> >> 
> >> If we push for an early 3.0, some of the breaking changes will have to be
> >> deferred.
> >> A very quick release after 3.0 could easily be a 3.1 if it was needed.
> >> 
> >> The next major release (4.0) would be when we had tackled (a significant
> >> chunk of) the remaining breaking changes.
> >> 
> > 
> > I am not sure a very rapid 3.0 → 3.1 is actually any sort of problem per se.
> > 
> > In the current climate is a 3.0 now, 4.0 in the next year actually a 
> > problem?
> > 
> > Given the very volunteer nature of the Groovy project, pragmatism more than
> > anything has to be the order of the day. However I think there is a 
> > marketing
> > element here. Having new 2.4.X releases doesn't really achieve anything
> > marketing-wise. Releasing 2.5.0 actually doesn't much either really, though
> > clearly we do it as we are already at RC-3, and it can be "milked". But 2.5 
> > is
> > just backward compatible extensions to 2.4, is this Groovy actually
> > progressing?
> > 
> > I suggest we want to get a 3.0 out to show Groovy is progressing and with 
> > some
> > breaking changes, in particular JDK8+ only, not to mention a parser based on
> > somewhat newer technology!  "Groovy drops support for JDK7" is probably a 
> > good
> > thing marketing wise.
> > 
> > My feeling is that Groovy does not have enough resource to go for big major
> > version number releases, but that it needs something to combat the "it's old
> > technology" feel given the rise of Kotlin. I'd push for a 3.0 release soon 
> > and
> > worry about the metamodel changes later – unless they are already in place?
> > 
> 
> I agree with the soft factors around numbering.
> 
> 
> As I understand, the metamodel changes are not in place yet, and not on 
> anybody's immediate schedule.
> 
> A) However, as I understand it, MOP2 could be made backwards compatible, so 
> that a new MOP2 runtime could still honor the MOP1 protocol, from some 
> version 3.x onwards. We could then deprecate the MOP1 way of doing things.
> 
> B) As for the indy stuff, we should choose to produce indy-only bytecodes now 
> that we require Java 8 for Groovy 3.0. Again, we still need runtime support 
> for the existing CallSite caching, or we'd have to force everybody to 
> recompile every Groovy library they use. That's hardly a good idea!
> 
> C) Finally, for 3.0 we could switch generation of closures to be 
> bootstrap-driven, i.e. by keeping the closure body as a method of the 
> enclosing class, but by generating the closure in the fashion of 
> LambdaMetafactory. That way, we would keep the Groovy semantics, but reduce 
> the cost of using closures (all those extra classfiles). We could possibly 
> introduce a ClosureInterface for forwards compatibility, and deprecate the 
> use of the Closure class directly.
> 
> Then, in some future majorversion 4.0, we could stop supporting MOP1, we 
> could drop support for the old callsite caching, and we could make 
> 
> Was that about right?
> 
> -Jesper
> 
> 

Re: Performance of the compiler

[Jesper Steen Møller]

Interesting! So let me get this straight: Are we using an actual "in-JVM" 
classloader to load classes examined by the Groovy Compiler itself?
In the Eclipse Java compiler, we don't actually load the classes into the JVM, 
instead we have our own implementation of classpath traversal and read it using 
ClassFileReader. Would a similar approach not work for constructing ClassNodes 
in Groovy? (obviousluy using ASM to do the bytecode parsing instead of rolling 
it ourselves).

JPMS would naturally make complicate things further, but again, ECJ has cracked 
this, too.

-Jesper

> On 24 May 2018, at 08.30, Cédric Champeau  wrote:
> 
> Hi folks,
> 
> I just wanted to share the result of profiling the performance of the 
> compiler on "real world" classes, namely compiling the tests of Gradle. We 
> have a lot of tests, so compilation times becomes really a pain point, so I 
> have checked where we spend time. I have attached the export of hotspots from 
> a real compilation session.
> 
> It's no surprise to me, most of the time (70%) is spent in the resolve 
> visitor, and most of this time itself is spent in loading classes. We made 
> some improvements in the past, by not initializing those classes, but it's 
> still a crazy amount of time.
> 
> Similarly, we spend around 10% of our time in filling stack traces which are 
> used for flow control. Unfortunately we don't have the opportunity to change 
> this because it's either ClassNotFoundException (during resolution) sent by 
> the classloader, or ANTLR recognition exception, used for flow control (duh!).
> 
> I remember that for Gradle I had implemented a custom ResolveVisitor that 
> adds some assumptions for Gradle scripts to avoid too many lookups for 
> classes which would obvisouly not exist, and it significantly improved the 
> performance of compiling scripts, but that was because there were lots of 
> implicit imports. For regular classes I'm not sure it's as simple.
> 
> Resolution is also very easy to break... Anyway, any change in this area 
> would probably make the lives of our users better!
> 
> 
> 

Performance of the compiler

[Cédric Champeau]

Hi folks,

I just wanted to share the result of profiling the performance of the
compiler on "real world" classes, namely compiling the tests of Gradle. We
have a lot of tests, so compilation times becomes really a pain point, so I
have checked where we spend time. I have attached the export of hotspots
from a real compilation session.

It's no surprise to me, most of the time (70%) is spent in the resolve
visitor, and most of this time itself is spent in loading classes. We made
some improvements in the past, by not initializing those classes, but it's
still a crazy amount of time.

Similarly, we spend around 10% of our time in filling stack traces which
are used for flow control. Unfortunately we don't have the opportunity to
change this because it's either ClassNotFoundException (during resolution)
sent by the classloader, or ANTLR recognition exception, used for flow
control (duh!).

I remember that for Gradle I had implemented a custom ResolveVisitor that
adds some assumptions for Gradle scripts to avoid too many lookups for
classes which would obvisouly not exist, and it significantly improved the
performance of compiling scripts, but that was because there were lots of
implicit imports. For regular classes I'm not sure it's as simple.

Resolution is also very easy to break... Anyway, any change in this area
would probably make the lives of our users better!
CPU hot spots

+---++
|  
Method  
 |   Time (ms)|
+---++
|  java.net.URLClassLoader.findClass(String) URLClassLoader.java
|  
6,531   54 %  |
|  java.security.AccessController.doPrivileged(PrivilegedExceptionAction, 
AccessControlContext) AccessController.java (native)
  |  6,090   50 %  |
|  java.net.URLClassLoader$1.run() URLClassLoader.java  
|  
5,987   50 %  |
|  java.net.URLClassLoader$1.run() URLClassLoader.java  
|  
5,987   50 %  |
|  org.codehaus.groovy.control.ResolveVisitor.startResolving(ClassNode, 
SourceUnit) ResolveVisitor.java 
|  4,711   39 %  |
|  org.codehaus.groovy.control.ResolveVisitor.visitClass(ClassNode) 
ResolveVisitor.java 
|  4,711   39 %  |
|  org.codehaus.groovy.control.ResolveVisitor.resolve(ClassNode, boolean, 
boolean, boolean) ResolveVisitor.java   
  |  4,486   37 %  |
|  org.codehaus.groovy.control.ClassNodeResolver.findClassNode(String, 
CompilationUnit) ClassNodeResolver.java 
 |  4,363   36 %  |
|  org.codehaus.groovy.control.ClassNodeResolver.resolveName(String, 
CompilationUnit) ClassNodeResolver.java 
   |  4,363   36 %  |
|  
org.codehaus.groovy.control.ClassNodeResolver.tryAsLoaderClassOrScript(String, 
CompilationUnit) ClassNodeResolver.java   |  
4,363   36 %  |
|  org.codehaus.groovy.control.ResolveVisitor.resolveToOuter(ClassNode) 
ResolveVisitor.java 
|  4,363   36 %  |
|  groovy.lang.GroovyClassLoader.loadClass(String, boolean, boolean, boolean) 
GroovyClassLoader.java| 
 4,183   35 %  |
|  groovy.lang.GroovyClassLoader.loadClass(String, boolean, boolean) 
GroovyClassLoader.java  
   |  3,984   33 %  |
|  org.codehaus.groovy.tools.javac.JavaAwareCompilationUnit$1.call(SourceUnit, 
GeneratorContext, ClassNode) JavaAwareCompilationUnit.java   |  
3,438   28 %  |
|  java.net.URLClassLoader.defineClass(String, Resource) URLClassLoader.java
|  
3,070   25 %  |
|  sun.misc.URLClassPath.getResource(String, boolean) URLClassPath.java 
|  
2,978   25 %  |
|  java.lang.ClassLoader.defineClass(String, byte[], int, int, 
ProtectionDomain) ClassLoader.java  
 |  2,292   19 %  |
|  java.lang.ClassLoader.defineClass1(String, byte[], int,