Groovy 2.5 @Macro ?

[MG]

Hi guys,

giving the new Groovy 2.5 macro functionality a spin, and would have 
expected the code below to replace the "call" to nv(x) with the AST 
expression created in the method, i.e. returning the name of the 
"passed" variable. Instead no macro magic happens, and the compilation 
accordingly fails with "groovy.lang.MissingMethodException: No signature 
of method: groovy.GroovyMacroSpike.nv() is applicable for argument 
types: (Integer) values: [123]":


import org.codehaus.groovy.ast.expr.Expression
import org.codehaus.groovy.ast.expr.VariableExpression
import org.codehaus.groovy.macro.runtime.Macro import 
org.codehaus.groovy.macro.runtime.MacroContext
import org.junit.Ignore import org.junit.Test import static 
org.codehaus.groovy.ast.tools.GeneralUtils.constX

class GroovyMacroSpike {
  @Test @Ignore void nvTest() {
final x =123 assert x ==123 assert nv(x) =="x" }

  @Macro Expression nv(MacroContext ctx, VariableExpression variable) {
return constX(variable.getName());
  }
}

What is missing to make this work ?
mg

Re: Performance of the compiler

[Daniel.Sun]

Hi Jochen,

> but this means we will have to manually update the list for java.lang,
> java.util, java.io and java.net

 In order to avoid updating the list manually, I tried to find the way
to get the list via reflection or classloader but failed...

 However, even if we add this check to return early, the whole resolving
performance will not be improved a lot...

Cheers,
Daniel.Sun




--
Sent from: http://groovy.329449.n5.nabble.com/Groovy-Dev-f372993.html

Re: Performance of the compiler

[Cédric Champeau]

Tens of thousands of tests, and ~4 minutes compile time (with --parallel).

Le ven. 25 mai 2018 à 14:50, mg  a écrit :

> Hi Cedric,
>
> to put this in context and to better understand the ongoing discussion:
> How many tests and what absolute compile time are we talking about here ?
>
> Cheers,
> mg
>
>
>  Ursprüngliche Nachricht 
> Von: Cédric Champeau 
> Datum: 24.05.18 08:30 (GMT+01:00)
> An: dev@groovy.apache.org
> Betreff: Performance of the compiler
>
> Hi folks,
>
> I just wanted to share the result of profiling the performance of the
> compiler on "real world" classes, namely compiling the tests of Gradle. We
> have a lot of tests, so compilation times becomes really a pain point, so I
> have checked where we spend time. I have attached the export of hotspots
> from a real compilation session.
>
> It's no surprise to me, most of the time (70%) is spent in the resolve
> visitor, and most of this time itself is spent in loading classes. We made
> some improvements in the past, by not initializing those classes, but it's
> still a crazy amount of time.
>
> Similarly, we spend around 10% of our time in filling stack traces which
> are used for flow control. Unfortunately we don't have the opportunity to
> change this because it's either ClassNotFoundException (during resolution)
> sent by the classloader, or ANTLR recognition exception, used for flow
> control (duh!).
>
> I remember that for Gradle I had implemented a custom ResolveVisitor that
> adds some assumptions for Gradle scripts to avoid too many lookups for
> classes which would obvisouly not exist, and it significantly improved the
> performance of compiling scripts, but that was because there were lots of
> implicit imports. For regular classes I'm not sure it's as simple.
>
> Resolution is also very easy to break... Anyway, any change in this area
> would probably make the lives of our users better!
>
>
>

Re: Performance of the compiler

[Cédric Champeau]

The problem is not the performance of the test, it's the performance of
_compiling_ the test. @CompileStatic wouldn't help there.

Le ven. 25 mai 2018 à 14:24, Thibault Kruse  a
écrit :

> Would the test performance be improved if @CompileStatic were used? I
> think gradle uses Spock, and last time I checked Spock could not be
> used with @CompileStatic. But Spock could also be removed with some
> effort...
>
> On Fri, May 25, 2018 at 8:52 PM, Jochen Theodorou 
> wrote:
> >
> >
> > Am 25.05.2018 um 12:51 schrieb Daniel.Sun:
> >>
> >> Hi Cédric,
> >>
> >>   I am not going to cache ClassNode instance(just cache class names,
> >> which are `String`), but I want to add a check whether the name of the
> >> ClassNode being resolved is possibly in the default imported packages,
> >> e.g.
> >> If a ClassNode instance's name is `Foobar`(apparently it could not be in
> >> any
> >> default imported packages), then we can `return false` immediately and
> the
> >> further resolving can be eliminated.
> >
> >
> > but this means we will have to manually update the list for java.lang,
> > java.util, java.io and java.net
> >
> > Take for example Module. It is new in Java 9 and is in java.lang. If we
> had
> > this logic already in say Groovy 2.0 I am pretty sure the last versions
> till
> > Groovy 2.3 would not be able to resolve this class anymore then.
> >
> > I think there would be no problem with Java10, but think of Java 11...
> we do
> > not know yet.
> >
> > bye Jochen
>

Re: Performance of the compiler

[mg]

Hi Cedric,
to put this in context and to better understand the ongoing discussion: How 
many tests and what absolute compile time are we talking about here ?
Cheers,mg

 Ursprüngliche Nachricht Von: Cédric Champeau 
 Datum: 24.05.18  08:30  (GMT+01:00) An: 
dev@groovy.apache.org Betreff: Performance of the compiler 
Hi folks,

I just wanted to share the result of profiling the performance of the compiler 
on "real world" classes, namely compiling the tests of Gradle. We have a lot of 
tests, so compilation times becomes really a pain point, so I have checked 
where we spend time. I have attached the export of hotspots from a real 
compilation session.

It's no surprise to me, most of the time (70%) is spent in the resolve visitor, 
and most of this time itself is spent in loading classes. We made some 
improvements in the past, by not initializing those classes, but it's still a 
crazy amount of time.

Similarly, we spend around 10% of our time in filling stack traces which are 
used for flow control. Unfortunately we don't have the opportunity to change 
this because it's either ClassNotFoundException (during resolution) sent by the 
classloader, or ANTLR recognition exception, used for flow control (duh!).

I remember that for Gradle I had implemented a custom ResolveVisitor that adds 
some assumptions for Gradle scripts to avoid too many lookups for classes which 
would obvisouly not exist, and it significantly improved the performance of 
compiling scripts, but that was because there were lots of implicit imports. 
For regular classes I'm not sure it's as simple.

Resolution is also very easy to break... Anyway, any change in this area would 
probably make the lives of our users better!

Re: Performance of the compiler

[Thibault Kruse]

Would the test performance be improved if @CompileStatic were used? I
think gradle uses Spock, and last time I checked Spock could not be
used with @CompileStatic. But Spock could also be removed with some
effort...

On Fri, May 25, 2018 at 8:52 PM, Jochen Theodorou  wrote:
>
>
> Am 25.05.2018 um 12:51 schrieb Daniel.Sun:
>>
>> Hi Cédric,
>>
>>   I am not going to cache ClassNode instance(just cache class names,
>> which are `String`), but I want to add a check whether the name of the
>> ClassNode being resolved is possibly in the default imported packages,
>> e.g.
>> If a ClassNode instance's name is `Foobar`(apparently it could not be in
>> any
>> default imported packages), then we can `return false` immediately and the
>> further resolving can be eliminated.
>
>
> but this means we will have to manually update the list for java.lang,
> java.util, java.io and java.net
>
> Take for example Module. It is new in Java 9 and is in java.lang. If we had
> this logic already in say Groovy 2.0 I am pretty sure the last versions till
> Groovy 2.3 would not be able to resolve this class anymore then.
>
> I think there would be no problem with Java10, but think of Java 11... we do
> not know yet.
>
> bye Jochen

Re: Performance of the compiler

[Jochen Theodorou]

Am 25.05.2018 um 12:51 schrieb Daniel.Sun:

Hi Cédric,

  I am not going to cache ClassNode instance(just cache class names,
which are `String`), but I want to add a check whether the name of the
ClassNode being resolved is possibly in the default imported packages, e.g.
If a ClassNode instance's name is `Foobar`(apparently it could not be in any
default imported packages), then we can `return false` immediately and the
further resolving can be eliminated.


but this means we will have to manually update the list for java.lang, 
java.util, java.io and java.net


Take for example Module. It is new in Java 9 and is in java.lang. If we 
had this logic already in say Groovy 2.0 I am pretty sure the last 
versions till Groovy 2.3 would not be able to resolve this class anymore 
then.


I think there would be no problem with Java10, but think of Java 11... 
we do not know yet.


bye Jochen

Re: Troubles with IDEA setup

[Keegan Witt]

I'm with Russel: I've seen some flakiness with Gradle Idea plugin.  On my
current client's project I actually saw Gradle Idea plugin add the wrong
version of a dependency to the project, but IntelliJ's import chose the
version that was mentioned in the Gradle file.  That did it for me -- I
permanently distrust Gradle Idea plugin now.

On Wed, May 23, 2018 at 7:33 AM, Cédric Champeau 
wrote:

> Thanks for checking, Marcin. For reference, stub generation takes long but
> passes. It's really the semantic analysis which gets stuck somehow.
>
> Le mer. 23 mai 2018 à 13:32, Marcin Erdmann  a
> écrit :
>
>> Cedric,
>>
>> I've just pulled master then I got rid of all idea config files and build
>> direcotries to start with a clean state:
>>
>> rm groovy.i*
>> find . -name "*.iml" -type f | xargs rm -f
>> ./gradlew clean
>>
>> Then I run the same command as you:
>> ./gradlew jar idea --rerun-tasks
>>
>> Finally I opened the project in IntelliJ and tried rebuilding it (Build >
>> Rebuild project). The fan is spinning like crazy, CPU usage is high and it
>> seems to be stuck on "Groovy stub generator: initialization [tests of
>> groovy-xml and 8 more]". After then killing the IDE and trying to
>> run DownUpStepTest I'm seeing the same symptoms as you are - stuck in
>> "Groovyc: semantic analysis [tests of groovy-xml and 8 more]" with CPU is
>> not busy.
>>
>> For reference, I'm using IntelliJ 2018.1 181.4203.550.
>>
>> On Tue, May 22, 2018 at 8:02 PM, Cédric Champeau <
>> cedric.champ...@gmail.com> wrote:
>>
>>> Hi team,
>>>
>>> I've been trying to figure out what is happening for hours now, but I
>>> cannot find any reason. I'm working on master, after removing all IDEA
>>> state (.iml, .ipr, .iws), regenerating the IDEA files using:
>>>
>>> ./gradlew jar idea --rerun-tasks
>>>
>>> And then I open the project in IDEA and try to run a unit test. But then
>>> the IDE is stuck in "Groovyc: semantic analysis [tests of groovy-xml and 8
>>> more]". CPU is not busy, nothing happens.
>>>
>>> I thought it could be a CPU issue, so I increased the memory for the
>>> IDEA compiler process, without success.
>>>
>>> Any idea what could be going on? Am I the only one seeing this?
>>>
>>>
>>

Re: Performance of the compiler

[Cédric Champeau]

I have no idea!

Le ven. 25 mai 2018 à 11:05, Jesper Steen Møller  a
écrit :

> Oh - I see now that this is implemented already (optimizationOptions
> w/asmResolvingin:true).
> Is there any reason this is not the default? Is it slower? Incorrect?
>
> -Jesper
>
> > On 24 May 2018, at 10.08, Jesper Steen Møller 
> wrote:
> >
> > Interesting! So let me get this straight: Are we using an actual
> "in-JVM" classloader to load classes examined by the Groovy Compiler itself?
> > In the Eclipse Java compiler, we don't actually load the classes into
> the JVM, instead we have our own implementation of classpath traversal and
> read it using ClassFileReader. Would a similar approach not work for
> constructing ClassNodes in Groovy? (obviousluy using ASM to do the bytecode
> parsing instead of rolling it ourselves).
> >
> > JPMS would naturally make complicate things further, but again, ECJ has
> cracked this, too.
> >
> > -Jesper
> >
> >> On 24 May 2018, at 08.30, Cédric Champeau  wrote:
> >>
> >> Hi folks,
> >>
> >> I just wanted to share the result of profiling the performance of the
> compiler on "real world" classes, namely compiling the tests of Gradle. We
> have a lot of tests, so compilation times becomes really a pain point, so I
> have checked where we spend time. I have attached the export of hotspots
> from a real compilation session.
> >>
> >> It's no surprise to me, most of the time (70%) is spent in the resolve
> visitor, and most of this time itself is spent in loading classes. We made
> some improvements in the past, by not initializing those classes, but it's
> still a crazy amount of time.
> >>
> >> Similarly, we spend around 10% of our time in filling stack traces
> which are used for flow control. Unfortunately we don't have the
> opportunity to change this because it's either ClassNotFoundException
> (during resolution) sent by the classloader, or ANTLR recognition
> exception, used for flow control (duh!).
> >>
> >> I remember that for Gradle I had implemented a custom ResolveVisitor
> that adds some assumptions for Gradle scripts to avoid too many lookups for
> classes which would obvisouly not exist, and it significantly improved the
> performance of compiling scripts, but that was because there were lots of
> implicit imports. For regular classes I'm not sure it's as simple.
> >>
> >> Resolution is also very easy to break... Anyway, any change in this
> area would probably make the lives of our users better!
> >>
> >>
> >> 
> >
>
>

Re: Performance of the compiler

[Jesper Steen Møller]

Oh - I see now that this is implemented already (optimizationOptions 
w/asmResolvingin:true).
Is there any reason this is not the default? Is it slower? Incorrect?

-Jesper

> On 24 May 2018, at 10.08, Jesper Steen Møller  wrote:
> 
> Interesting! So let me get this straight: Are we using an actual "in-JVM" 
> classloader to load classes examined by the Groovy Compiler itself?
> In the Eclipse Java compiler, we don't actually load the classes into the 
> JVM, instead we have our own implementation of classpath traversal and read 
> it using ClassFileReader. Would a similar approach not work for constructing 
> ClassNodes in Groovy? (obviousluy using ASM to do the bytecode parsing 
> instead of rolling it ourselves).
> 
> JPMS would naturally make complicate things further, but again, ECJ has 
> cracked this, too.
> 
> -Jesper
> 
>> On 24 May 2018, at 08.30, Cédric Champeau  wrote:
>> 
>> Hi folks,
>> 
>> I just wanted to share the result of profiling the performance of the 
>> compiler on "real world" classes, namely compiling the tests of Gradle. We 
>> have a lot of tests, so compilation times becomes really a pain point, so I 
>> have checked where we spend time. I have attached the export of hotspots 
>> from a real compilation session.
>> 
>> It's no surprise to me, most of the time (70%) is spent in the resolve 
>> visitor, and most of this time itself is spent in loading classes. We made 
>> some improvements in the past, by not initializing those classes, but it's 
>> still a crazy amount of time.
>> 
>> Similarly, we spend around 10% of our time in filling stack traces which are 
>> used for flow control. Unfortunately we don't have the opportunity to change 
>> this because it's either ClassNotFoundException (during resolution) sent by 
>> the classloader, or ANTLR recognition exception, used for flow control 
>> (duh!).
>> 
>> I remember that for Gradle I had implemented a custom ResolveVisitor that 
>> adds some assumptions for Gradle scripts to avoid too many lookups for 
>> classes which would obvisouly not exist, and it significantly improved the 
>> performance of compiling scripts, but that was because there were lots of 
>> implicit imports. For regular classes I'm not sure it's as simple.
>> 
>> Resolution is also very easy to break... Anyway, any change in this area 
>> would probably make the lives of our users better!
>> 
>> 
>> 
> 

Re: upcoming 2.5.0 release

[Cédric Champeau]

I realize I have pushed the groovydoc improvements to 2_5_X. This means the
branch differs from the last rc. Does it mean it's going to be in 2.5.1, or
that the final 2.5.0 is going to be different from the rc2 (uh!) or, that
we need rc3?

Le ven. 25 mai 2018 à 10:53, Paul King  a écrit :

>
> Last call for any additions - otherwise I hope to prepare a candidate over
> the weekend.
>
> Cheers, Paul.
>
>

Re: Performance of the compiler

[Cédric Champeau]

Be warned that caching may prove to be complicated: what if a class in the
same compile unit is named `List`, or a class in the same package? This is
one of the reasons the lookup is not cached for this.

Le ven. 25 mai 2018 à 10:17, Daniel.Sun  a écrit :

> I am going to cache the class names of default imported packages, e.g.
> `List`, `ArrayList`, `Map`, `HashMap`, etc.
>
> And try to find the name of ClassNode instance in the class names cache, if
> not found, `return false`(add the check at the line[1]). The polished logic
> can avoid unnecessary resolving and improve the resolving performance
> somehow.
>
> It's a pity that I have not found any approach to get all classes of the
> given package, e.g. `java.lang`, `java.util`
> Any help is appreciated. (Note: We can not use 3rd party library excluding
> antlr, asm, cli)
>
> Cheers,
> Daniel.Sun
> [1]
>
> https://github.com/apache/groovy/blob/master/src/main/java/org/codehaus/groovy/control/ResolveVisitor.java#L513
>
>
>
> --
> Sent from: http://groovy.329449.n5.nabble.com/Groovy-Dev-f372993.html
>

upcoming 2.5.0 release

[Paul King]

Last call for any additions - otherwise I hope to prepare a candidate over
the weekend.

Cheers, Paul.

Re: Performance of the compiler

[Daniel.Sun]

I am going to cache the class names of default imported packages, e.g.
`List`, `ArrayList`, `Map`, `HashMap`, etc.

And try to find the name of ClassNode instance in the class names cache, if
not found, `return false`(add the check at the line[1]). The polished logic
can avoid unnecessary resolving and improve the resolving performance
somehow.

It's a pity that I have not found any approach to get all classes of the
given package, e.g. `java.lang`, `java.util`
Any help is appreciated. (Note: We can not use 3rd party library excluding
antlr, asm, cli)

Cheers,
Daniel.Sun
[1]
https://github.com/apache/groovy/blob/master/src/main/java/org/codehaus/groovy/control/ResolveVisitor.java#L513



--
Sent from: http://groovy.329449.n5.nabble.com/Groovy-Dev-f372993.html