Re: [PROPOSAL]Provide an option to generate stubs in in-momery file system for better compiling performance
Gotcha. Thanks for your explanation, Rémi :-) Cheers, Daniel.Sun - Apache Groovy committer Blog: http://blog.sunlan.me Twitter: @daniel_sun -- Sent from: http://groovy.329449.n5.nabble.com/Groovy-Dev-f372993.html
Re: [PROPOSAL]Provide an option to generate stubs in in-momery file system for better compiling performance
- Mail original - > De: "Daniel.Sun" > À: "dev" > Envoyé: Samedi 19 Janvier 2019 16:44:43 > Objet: Re: [PROPOSAL]Provide an option to generate stubs in in-momery file > system for better compiling performance > Use `ClassLoader.getPlatformClassLoader()` instead to solve the issue. > > GroovyClassLoader dependencyLoader = new GroovyClassLoader(new > URLClassLoader(urls, (ClassLoader)ClassLoader.getPlatformClassLoader())) > > class loading changed in Java 9+. WTF! ClassLoading have not changed but classes that were previously loaded by the boot loader (the one written in C so reported as null from the Java side) have been moved to the platform loader for security reason. Being loaded by the boot loader make you a privileged class. https://bugs.openjdk.java.net/browse/JDK-8189116?jql=labels%20%3D%20deprivilege > > > Cheers, > Daniel.Sun cheers, Rémi > > > > > - > Apache Groovy committer > Blog: http://blog.sunlan.me > Twitter: @daniel_sun > > -- > Sent from: http://groovy.329449.n5.nabble.com/Groovy-Dev-f372993.html
Re: [PROPOSAL]Provide an option to generate stubs in in-momery file system for better compiling performance
Use `ClassLoader.getPlatformClassLoader()` instead to solve the issue. GroovyClassLoader dependencyLoader = new GroovyClassLoader(new URLClassLoader(urls, (ClassLoader)ClassLoader.getPlatformClassLoader())) class loading changed in Java 9+. WTF! Cheers, Daniel.Sun - Apache Groovy committer Blog: http://blog.sunlan.me Twitter: @daniel_sun -- Sent from: http://groovy.329449.n5.nabble.com/Groovy-Dev-f372993.html
Re: [PROPOSAL]Provide an option to generate stubs in in-momery file system for better compiling performance
I found the issue is related to the specified parent classloader: https://github.com/apache/groovy/blob/master/src/test/org/codehaus/groovy/transform/classloading/TransformsAndCustomClassLoadersTest.groovy#L46 Looking for the way to expose javax.tools classes to the classloader... - Apache Groovy committer Blog: http://blog.sunlan.me Twitter: @daniel_sun -- Sent from: http://groovy.329449.n5.nabble.com/Groovy-Dev-f372993.html
Re: [PROPOSAL]Provide an option to generate stubs in in-momery file system for better compiling performance
[Help wanted] Even if I add "--add-modules java.compiler --add-opens java.compiler/javax.tools=ALL-UNNAMED" to JVM, the following error( Type javax.tools.JavaFileObject not present ) still happens when running on Java9+... Any advice to fix the issue? Thanks in advance! java.lang.TypeNotPresentException: Type javax.tools.JavaFileObject not present at java.base/sun.reflect.generics.factory.CoreReflectionFactory.makeNamedType(CoreReflectionFactory.java:117) at java.base/sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:125) at java.base/sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49) at java.base/sun.reflect.generics.visitor.Reifier.reifyTypeArguments(Reifier.java:68) at java.base/sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:138) at java.base/sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49) at java.base/sun.reflect.generics.repository.ConstructorRepository.computeParameterTypes(ConstructorRepository.java:111) at java.base/sun.reflect.generics.repository.ConstructorRepository.getParameterTypes(ConstructorRepository.java:87) at java.base/java.lang.reflect.Executable.getGenericParameterTypes(Executable.java:279) at java.base/java.lang.reflect.Method.getGenericParameterTypes(Method.java:330) at java.desktop/java.beans.FeatureDescriptor.getParameterTypes(FeatureDescriptor.java:391) at java.desktop/java.beans.MethodDescriptor.setMethod(MethodDescriptor.java:118) at java.desktop/java.beans.MethodDescriptor.(MethodDescriptor.java:74) at java.desktop/java.beans.MethodDescriptor.(MethodDescriptor.java:58) at java.desktop/java.beans.Introspector.getTargetMethodInfo(Introspector.java:1047) at java.desktop/java.beans.Introspector.getBeanInfo(Introspector.java:462) at java.desktop/java.beans.Introspector.getBeanInfo(Introspector.java:205) at groovy.lang.MetaClassImpl$15.run(MetaClassImpl.java:3417) at java.base/java.security.AccessController.doPrivileged(Native Method) at groovy.lang.MetaClassImpl.addProperties(MetaClassImpl.java:3415) at groovy.lang.MetaClassImpl.initialize(MetaClassImpl.java:3392) at org.codehaus.groovy.reflection.ClassInfo.getMetaClassUnderLock(ClassInfo.java:289) at org.codehaus.groovy.reflection.ClassInfo.getMetaClass(ClassInfo.java:331) at org.codehaus.groovy.runtime.metaclass.MetaClassRegistryImpl.getMetaClass(MetaClassRegistryImpl.java:271) at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:984) at org.codehaus.groovy.runtime.typehandling.DefaultTypeTransformation.compareEqual(DefaultTypeTransformation.java:653) at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.compareEqual(ScriptBytecodeAdapter.java:710) at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.compareNotEqual(ScriptBytecodeAdapter.java:714) at org.codehaus.groovy.transform.classloading.TransformsAndCustomClassLoadersTest.setUp(TransformsAndCustomClassLoadersTest.groovy:60) at junit.framework.TestCase.runBare(TestCase.java:139) at junit.framework.TestResult$1.protect(TestResult.java:122) at junit.framework.TestResult.runProtected(TestResult.java:142) at junit.framework.TestResult.run(TestResult.java:125) at junit.framework.TestCase.run(TestCase.java:129) at junit.framework.TestSuite.runTest(TestSuite.java:252) at junit.framework.TestSuite.run(TestSuite.java:247) at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:86) at org.junit.runner.JUnitCore.run(JUnitCore.java:137) at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68) at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47) at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242) at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70) Caused by: java.lang.ClassNotFoundException: javax.tools.JavaFileObject at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:471) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:588) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521) at java.base/java.lang.Class.forName0(Native Method) at java.base/java.lang.Class.forName(Class.java:398) at java.base/sun.reflect.generics.factory.CoreReflectionFactory.makeNamedType(CoreReflectionFactory.java:114) ... 41 more ( https://travis-ci.org/apache/groovy/builds/481727005?utm_source=github_status_medium=notification ) Cheers, Daniel.Sun - Apache Groovy committer Blog: http://blog.sunlan.me Twitter: @daniel_sun -- Sent from:
Re: [PROPOSAL]Provide an option to generate stubs in in-momery file system for better compiling performance
I found gradle's Java compiler(not reuse Groovy's `JavacJavaCompiler`) relies on the stubs on disk... When I enabled generating stub in memory, Groovy's `JavacJavaCompiler` is refined so it can find stubs smartly, and everything is OK, but gradle's Java compiler can not find stub files and throws errors because stub class not found. If the new feature is accepted, I wish gradle can refine its own Java compiler as well(add an option too). Cheers, Daniel.Sun - Apache Groovy committer Blog: http://blog.sunlan.me Twitter: @daniel_sun -- Sent from: http://groovy.329449.n5.nabble.com/Groovy-Dev-f372993.html
Re: [PROPOSAL]Provide an option to generate stubs in in-momery file system for better compiling performance
Yep ;-) The stub files normally are very small. Given 8K by average per stub file, Groovy project contains about 2900 groovy files(normally one groovy file contains only one groovy class), keeping all the stubs in memory occupies at most 30M space. In addition, we can clean stubs in memory when compilation is complete, so the impact on memory is temporary :-) Cheers, Daniel.Sun - Apache Groovy committer Blog: http://blog.sunlan.me Twitter: @daniel_sun -- Sent from: http://groovy.329449.n5.nabble.com/Groovy-Dev-f372993.html
Re: [PROPOSAL]Provide an option to generate stubs in in-momery file system for better compiling performance
The only minus on this is the memory consumption... though I did not measure it, so maybe it is not that bad. On 18.01.19 09:44, Daniel Sun wrote: Hi all, [Background] Groovy's joint compiler will generate a lot of stubs for groovy source files, many of the stubs are useless and written to disk and clean later. When project contains a lot of groovy source files, the performance of compiling is not good. [Proposal] I propose to add an option(e.g. `groovy.generate.stubs.in.memory`) to generate stubs in in-momery file system[1]. We can get the generated stub files from the the in-memory file system with `StandardJavaFileManager`[2] . Here is the `JavaCompiler` usage[3]. [Benefits] We can avoid writing lots of stub files to disk and subsequent cleaning, in addition, reading stub files from memory will be much faster than reading from disk. Any thoughts? Cheers, Daniel.Sun [1] https://github.com/google/jimfs [2] https://docs.oracle.com/javase/7/docs/api/javax/tools/StandardJavaFileManager.html [3] https://github.com/danielsun1106/SmartASMifier/blob/master/SmartASMifier.groovy#L81-L88 -- Sent from: http://groovy.329449.n5.nabble.com/Groovy-Dev-f372993.html
Re: [PROPOSAL]Provide an option to generate stubs in in-momery file system for better compiling performance
Hi Cédric, It seems that gradle has its own java compiler and does not invoke the following method of groovy, right? https://github.com/apache/groovy/blob/master/src/main/java/org/codehaus/groovy/tools/javac/JavacJavaCompiler.java#L54 Cheers, Daniel.Sun - Apache Groovy committer Blog: http://blog.sunlan.me Twitter: @daniel_sun -- Sent from: http://groovy.329449.n5.nabble.com/Groovy-Dev-f372993.html
Re: [PROPOSAL]Provide an option to generate stubs in in-momery file system for better compiling performance
If we keep stubs in memory, IDE's incremental compiler can reuse them if no changes happen. I think builds of large groovy projects like Grails, Gradle and Groovy itself, etc will get more benefits from the new feature. Cheers, Daniel.Sun - Apache Groovy committer Blog: http://blog.sunlan.me Twitter: @daniel_sun -- Sent from: http://groovy.329449.n5.nabble.com/Groovy-Dev-f372993.html
Re: [PROPOSAL]Provide an option to generate stubs in in-momery file system for better compiling performance
Could IDEs like Intellij benefit from this? p On Fri, Jan 18, 2019 at 7:53 PM Daniel.Sun wrote: > Fix the PR link: https://github.com/apache/groovy/pull/855 > > > > - > Apache Groovy committer > Blog: http://blog.sunlan.me > Twitter: @daniel_sun > > -- > Sent from: http://groovy.329449.n5.nabble.com/Groovy-Dev-f372993.html >
Re: [PROPOSAL]Provide an option to generate stubs in in-momery file system for better compiling performance
Fix the PR link: https://github.com/apache/groovy/pull/855 - Apache Groovy committer Blog: http://blog.sunlan.me Twitter: @daniel_sun -- Sent from: http://groovy.329449.n5.nabble.com/Groovy-Dev-f372993.html
Re: [PROPOSAL]Provide an option to generate stubs in in-momery file system for better compiling performance
The proposed PR is here: https://issues.apache.org/jira/browse/GROOVY-8962 Merged in 72 hours if no one rejects. Any advice is appreciated! Cheers, Daniel.Sun - Apache Groovy committer Blog: http://blog.sunlan.me Twitter: @daniel_sun -- Sent from: http://groovy.329449.n5.nabble.com/Groovy-Dev-f372993.html
Re: [PROPOSAL]Provide an option to generate stubs in in-momery file system for better compiling performance
Agreed.
I find we can get rid of the 3rd-part lib and just rely on `javax.tools`
APIs, e.g.
```
compilationUnits.add(new MemJavaFileObject(new
URI("string:///labs/Hello.java"), srcStr));
```
I'm doing some experiment.
Cheers,
Daniel.Sun
-
Apache Groovy committer
Blog: http://blog.sunlan.me
Twitter: @daniel_sun
--
Sent from: http://groovy.329449.n5.nabble.com/Groovy-Dev-f372993.html
Re: [PROPOSAL]Provide an option to generate stubs in in-momery file system for better compiling performance
This would be indeed very useful. The current join compilation tend to be time consuming. p On Fri, Jan 18, 2019 at 9:44 AM Daniel Sun wrote: > Hi all, > > [Background] >Groovy's joint compiler will generate a lot of stubs for groovy > source files, many of the stubs are useless and written to disk and clean > later. When project contains a lot of groovy source files, the performance > of compiling is not good. > > [Proposal] > I propose to add an option(e.g. `groovy.generate.stubs.in.memory`) > to > generate stubs in in-momery file system[1]. We can get the generated stub > files from the the in-memory file system with `StandardJavaFileManager`[2] > . > Here is the `JavaCompiler` usage[3]. > > [Benefits] > We can avoid writing lots of stub files to disk and subsequent > cleaning, in addition, reading stub files from memory will be much faster > than reading from disk. > > > Any thoughts? > > Cheers, > Daniel.Sun > [1] https://github.com/google/jimfs > [2] > > https://docs.oracle.com/javase/7/docs/api/javax/tools/StandardJavaFileManager.html > [3] > > https://github.com/danielsun1106/SmartASMifier/blob/master/SmartASMifier.groovy#L81-L88 > > > > > -- > Sent from: http://groovy.329449.n5.nabble.com/Groovy-Dev-f372993.html >
Re: [PROPOSAL]Provide an option to generate stubs in in-momery file system for better compiling performance
It's an interesting idea. I like that it's hidden behind a flag, because sometimes you want to keep the stubs. That must be the case for IDEs that implement "some kind" of incremental compilation. Le ven. 18 janv. 2019 à 09:44, Daniel Sun a écrit : > Hi all, > > [Background] >Groovy's joint compiler will generate a lot of stubs for groovy > source files, many of the stubs are useless and written to disk and clean > later. When project contains a lot of groovy source files, the performance > of compiling is not good. > > [Proposal] > I propose to add an option(e.g. `groovy.generate.stubs.in.memory`) > to > generate stubs in in-momery file system[1]. We can get the generated stub > files from the the in-memory file system with `StandardJavaFileManager`[2] > . > Here is the `JavaCompiler` usage[3]. > > [Benefits] > We can avoid writing lots of stub files to disk and subsequent > cleaning, in addition, reading stub files from memory will be much faster > than reading from disk. > > > Any thoughts? > > Cheers, > Daniel.Sun > [1] https://github.com/google/jimfs > [2] > > https://docs.oracle.com/javase/7/docs/api/javax/tools/StandardJavaFileManager.html > [3] > > https://github.com/danielsun1106/SmartASMifier/blob/master/SmartASMifier.groovy#L81-L88 > > > > > -- > Sent from: http://groovy.329449.n5.nabble.com/Groovy-Dev-f372993.html >
[PROPOSAL]Provide an option to generate stubs in in-momery file system for better compiling performance
Hi all, [Background] Groovy's joint compiler will generate a lot of stubs for groovy source files, many of the stubs are useless and written to disk and clean later. When project contains a lot of groovy source files, the performance of compiling is not good. [Proposal] I propose to add an option(e.g. `groovy.generate.stubs.in.memory`) to generate stubs in in-momery file system[1]. We can get the generated stub files from the the in-memory file system with `StandardJavaFileManager`[2] . Here is the `JavaCompiler` usage[3]. [Benefits] We can avoid writing lots of stub files to disk and subsequent cleaning, in addition, reading stub files from memory will be much faster than reading from disk. Any thoughts? Cheers, Daniel.Sun [1] https://github.com/google/jimfs [2] https://docs.oracle.com/javase/7/docs/api/javax/tools/StandardJavaFileManager.html [3] https://github.com/danielsun1106/SmartASMifier/blob/master/SmartASMifier.groovy#L81-L88 -- Sent from: http://groovy.329449.n5.nabble.com/Groovy-Dev-f372993.html
