Re: JDK 9 Early Access with Project Jigsaw, build 135 on 09-14-2016 (#5500)
On 02/10/2016 13:26, Jochen Theodorou wrote: Project Panama is going to be in JDK9? That is new to me. I didn't say this. I was just making the point that in the future that using system calls and native code will be a lot better (simpler, more reliable, faster, ...). But yes, I too suggest the gradle people bring that to core-lib-dev Thanks, that is the right place to discuss this issue. hmm... if I look at https://github.com/gradle/gradle/blob/master/subprojects/base-services/src/main/java/org/gradle/internal/classloader/MultiParentClassLoader.java and I imagine one parent being a MultiParentClassLoader... getDefinedPackages is final, so you cannot do what they did for getPackages, unless you do it from outside. And for that you need to know about multiple parents... I don´t know if it is working well enough for Gradle, but no, getDefinedPackages is no suitable replacement for getPackages at all if you have to consider multiparent class loaders. If MultiParentClassLoader defines a method to get its parents then you can walk the hierarchy, special case MultiParentClassLoader, and calling getDefinedPackages on each one. It's not going to help if there are other custom multi-parent loaders from elsewhere in the hierarchy but I can't tell if this is possible here. -Alan
Re: JDK 9 Early Access with Project Jigsaw, build 135 on 09-14-2016 (#5500)
On 02.10.2016 14:43, Claes Redestad wrote: On 2016-10-02 14:26, Jochen Theodorou wrote: Do you know anything about the ClassLoader::getPackages issue? Is this a case where the (new) public getDefinedPackages returns the Packages that Gradle is looking for? hmm... if I look at https://github.com/gradle/gradle/blob/master/subprojects/base-services/src/main/java/org/gradle/internal/classloader/MultiParentClassLoader.java and I imagine one parent being a MultiParentClassLoader... getDefinedPackages is final, so you cannot do what they did for getPackages, unless you do it from outside. And for that you need to know about multiple parents... I don´t know if it is working well enough for Gradle, but no, getDefinedPackages is no suitable replacement for getPackages at all if you have to consider multiparent class loaders. Since the methods reflected upon in ClassLoader here are all protected, couldn't this specific code be made reflection-free by wrapping all parent classloaders in a delegating classloader that overrides getPackage/getPackages as public? How is the delegating classloader going to access getPackage of the loader it delegates to? Just being a classloader subclass does not solve the problem. And you cannot be in the same package as well... not to mention, that your class is probably defined in a different classloader as well. Or is here a Java-trick I do not know of? bye Jochen
Re: JDK 9 Early Access with Project Jigsaw, build 135 on 09-14-2016 (#5500)
On 02.10.2016 11:42, Alan Bateman wrote: [...] In prior releases when updates to the Java Language would have required to IDEs and some tools but wouldn't have wide impact on tools. This is the reason for early access releases and the ongoing outreach to create awkward of issues and impact. it really depends on the project of if things are affecting you or not. I think since JDK 1.2 only JDK6 did no require us making changes somewhere... sometimes because of things changed in maintenance versions, sometimes because of new features conflicting with ours. On Groovy then Jochen has brought up issues here several times. I get the impression that there is a lot more to this, often it seems like issues that should be discussed on mlvm-dev. We have not seen anything like these issues with Nashorn (Javascript), it is of course based on method handles and a different design. Well, if you have a 10 year old project you can expect it depending much more on Reflection than something like Nashorn... especially when it is coupled tightly to Java and does not build up its own independent universe bye Jochen
Re: JDK 9 Early Access with Project Jigsaw, build 135 on 09-14-2016 (#5500)
On 02.10.2016 12:12, Alan Bateman wrote: On 02/10/2016 10:20, Jochen Theodorou wrote: [...] All I can suggest is bring up the topic on core-libs-dev. I don't wish to get into the discussion here on whether it's a good idea or not to add such a method. Also with Project Panama looking good then one has to wonder if it make sense to add methods like this. Project Panama is going to be in JDK9? That is new to me. But even if, it makes no major difference, since everything you could have done there, you could have done with JNI/JNA and you will still need platform specific code for that... unless you use a more independent library, which then has to be added in different versions for multiple platforms again. But yes, I too suggest the gradle people bring that to core-lib-dev On the Process API then just to say that it has been updated significantly for Java SE 9 to support managing of child processes (and trees of processes). Details in JEP 102. This might not be exactly what you are looking for here but I just mention that the API has been updated to address many long standing shortcomings. for setting environment variables JEP 102 does not look useful. You can applaud the effort for fixing many long standing issues, but if you need a certain feature and it becomes "disabled" without having a replacement at hand... well you are not in the mood to applaud then. I am sure the Gradle people will solve all those problems with one hack or another. It just means 3.0 and 3.1 will no longer be jdk9 ready people will have to wait for 3.2 or even 3.3 Do you know anything about the ClassLoader::getPackages issue? Is this a case where the (new) public getDefinedPackages returns the Packages that Gradle is looking for? hmm... if I look at https://github.com/gradle/gradle/blob/master/subprojects/base-services/src/main/java/org/gradle/internal/classloader/MultiParentClassLoader.java and I imagine one parent being a MultiParentClassLoader... getDefinedPackages is final, so you cannot do what they did for getPackages, unless you do it from outside. And for that you need to know about multiple parents... I don´t know if it is working well enough for Gradle, but no, getDefinedPackages is no suitable replacement for getPackages at all if you have to consider multiparent class loaders. bye Jochen
Re: JDK 9 Early Access with Project Jigsaw, build 135 on 09-14-2016 (#5500)
On 02/10/2016 10:20, Jochen Theodorou wrote: The real issue is probably Gradle trying to "fork" the process to a running daemon. That includes environment variables for example. Java is not very "fork"-friendly. Not in the classic posix sense, and not by passing the execution to an already running daemon either. Then you find things like this: http://stackoverflow.com/questions/318239/how-do-i-set-environment-variables-from-java And remember, there is no ProcessBuilder, if you delegate execution to a daemon. Only without the daemon startup times may matter much more. And even if in JDK9 the JVM starts faster now, that does not mean you get a ready setup to work in from that point on... not if you have to load another 10k classes and data structures for example. Gradle is struggling with this for years already and I am talking about comparing execution speeds of gradle with something like make or scons, not with ant, maven or sbt. So now I would be curious as of what you suggest on how to fix the issue. I mean even if you start using native code to set the environment, will the java process notice those changes? There is no guarantee for that and it may break again even if it works now, right? java.lang.System doesn't define a method that is the equivalent of setenv(3). That seems to what the stackoverflow post is about. All I can suggest is bring up the topic on core-libs-dev. I don't wish to get into the discussion here on whether it's a good idea or not to add such a method. Also with Project Panama looking good then one has to wonder if it make sense to add methods like this. On the Process API then just to say that it has been updated significantly for Java SE 9 to support managing of child processes (and trees of processes). Details in JEP 102. This might not be exactly what you are looking for here but I just mention that the API has been updated to address many long standing shortcomings. Do you know anything about the ClassLoader::getPackages issue? Is this a case where the (new) public getDefinedPackages returns the Packages that Gradle is looking for? -Alan
Re: JDK 9 Early Access with Project Jigsaw, build 135 on 09-14-2016 (#5500)
On 02/10/2016 09:04, Cédric Champeau wrote: As a member of both the Gradle and Groovy teams, I must say I am very worried by this late change in strong encapsulation at runtime. As Jochen explained, it severely broke Gradle and Groovy, but more importantly, it breaks a lot of libraries. And it breaks for things that do _not_ care about modules. They are all running "old fashioned" with good old classpath. I could understand something like that happening as soon as we have one module on classpath, but this is not the case (the only modules are those from the JDK itself). When you are using the class path then you are on a modular platform so there are always modules in the picture. The platform modules do not export their internals and so will be problematic for code on the class path that relies on being able to access JDK internals. Yes, it is very disruptive change but one that just extends the proposal from 2015 - the proposal in 2015 attempted to strike a balance but leaves the hole that is #AwkwardStrongEncapsulation. To mitigate the impact then there are command line options, equivalent attributes for executable JARs, and a partial solution for JNLP apps too. It's also possible to use JDK 9 command line options with JDK 7 and JDK 8 if you are willing to run with the option to ignore unrecognized options. It might be late in JDK 9 to fix this hole but there is still time for the important tools and libraries in the eco system to identify and address issues. As we've been saying for years, JDK 9 is the first release that requires most of the main stream tools and libraries to work with us in order to make this release usable. In prior releases when updates to the Java Language would have required to IDEs and some tools but wouldn't have wide impact on tools. This is the reason for early access releases and the ongoing outreach to create awkward of issues and impact. On Groovy then Jochen has brought up issues here several times. I get the impression that there is a lot more to this, often it seems like issues that should be discussed on mlvm-dev. We have not seen anything like these issues with Nashorn (Javascript), it is of course based on method handles and a different design. -Alan
Re: JDK 9 Early Access with Project Jigsaw, build 135 on 09-14-2016 (#5500)
On 02.10.2016 09:53, Alan Bateman wrote: On 02/10/2016 01:25, Malachi de Ælfweald wrote: : This appears to come from a setAccessible from inside getEnv https://github.com/adammurdoch/native-platform/blob/master/src/main/java/net/rubygrapefruit/platform/internal/WrapperProcess.java#L113 Indeed, as we've said in other mails, this change is going to expose a lot hacks. In this specific case then System.getenv() is specified to return an unmodifiable map but a library that Gradle uses seems to want to hack into the underlying map so that it can modify it. Stack trace below. Jochen may be able to find out more about this, maybe there is a description somewhere on what the real issue is. It might be something that can be tackled in other ways. The real issue is probably Gradle trying to "fork" the process to a running daemon. That includes environment variables for example. Java is not very "fork"-friendly. Not in the classic posix sense, and not by passing the execution to an already running daemon either. Then you find things like this: http://stackoverflow.com/questions/318239/how-do-i-set-environment-variables-from-java And remember, there is no ProcessBuilder, if you delegate execution to a daemon. Only without the daemon startup times may matter much more. And even if in JDK9 the JVM starts faster now, that does not mean you get a ready setup to work in from that point on... not if you have to load another 10k classes and data structures for example. Gradle is struggling with this for years already and I am talking about comparing execution speeds of gradle with something like make or scons, not with ant, maven or sbt. So now I would be curious as of what you suggest on how to fix the issue. I mean even if you start using native code to set the environment, will the java process notice those changes? There is no guarantee for that and it may break again even if it works now, right? bye Jochen
Re: JDK 9 Early Access with Project Jigsaw, build 135 on 09-14-2016 (#5500)
As a member of both the Gradle and Groovy teams, I must say I am very worried by this late change in strong encapsulation at runtime. As Jochen explained, it severely broke Gradle and Groovy, but more importantly, it breaks a lot of libraries. And it breaks for things that do _not_ care about modules. They are all running "old fashioned" with good old classpath. I could understand something like that happening as soon as we have one module on classpath, but this is not the case (the only modules are those from the JDK itself). And Jochen explained this could have a very large impact on the performance of Groovy, making it effectively unusable (some would argue that this is a good thing, but let's not go into that debate). Asciidoctor is broken too, due to its dependency on JRuby. That's just one example over many. The truth is that we have no clue what the impact on this change has on the larger JVM ecosystem. Transitive dependencies are a big issue, because you cannot ask all your dependencies to upgrade their code to JDK 9 as fast as we can. And for Gradle at least, we must absolutely run on older JDKs with the same code, so we cannot rely on JDK 9 specific classes or hacks to start the JVM (like add-exports). In short I am very worried that this change could make upgrading to JDK 9 so complex that nobody would do it. 2016-10-02 9:53 GMT+02:00 Alan Bateman: > On 02/10/2016 01:25, Malachi de Ælfweald wrote: > > : >> >> This appears to come from a setAccessible from inside getEnv >> https://github.com/adammurdoch/native-platform/blob/master/ >> src/main/java/net/rubygrapefruit/platform/internal/ >> WrapperProcess.java#L113 >> > Indeed, as we've said in other mails, this change is going to expose a lot > hacks. In this specific case then System.getenv() is specified to return an > unmodifiable map but a library that Gradle uses seems to want to hack into > the underlying map so that it can modify it. Stack trace below. > > Jochen may be able to find out more about this, maybe there is a > description somewhere on what the real issue is. It might be something that > can be tackled in other ways. > > -Alan > > > * Exception is: > net.rubygrapefruit.platform.NativeException: Unable to get mutable > environment variable map. > at net.rubygrapefruit.platform.internal.WrapperProcess.getEnv( > WrapperProcess.java:113) > at net.rubygrapefruit.platform.internal.WrapperProcess.removeEn > vInternal(WrapperProcess.java:91) > at net.rubygrapefruit.platform.internal.WrapperProcess.setEnvir > onmentVariable(WrapperProcess.java:82) > at org.gradle.internal.nativeintegration.processenvironment.Nat > ivePlatformBackedProcessEnvironment.removeNativeEnvironmentV > ariable(NativePlatformBackedProcessEnvironment.java:31) > at org.gradle.internal.nativeintegration.processenvironment.Abs > tractProcessEnvironment.removeEnvironmentVariable(AbstractPr > ocessEnvironment.java:47) > at org.gradle.internal.nativeintegration.processenvironment.Abs > tractProcessEnvironment.maybeSetEnvironment(AbstractProcessE > nvironment.java:37) > at org.gradle.launcher.daemon.server.exec.EstablishBuildEnviron > ment.doBuild(EstablishBuildEnvironment.java:65) > at org.gradle.launcher.daemon.server.exec.BuildCommandOnly.exec > ute(BuildCommandOnly.java:36) > at org.gradle.launcher.daemon.server.api.DaemonCommandExecution > .proceed(DaemonCommandExecution.java:120) > at org.gradle.launcher.daemon.server.exec.HintGCAfterBuild.exec > ute(HintGCAfterBuild.java:44) > at org.gradle.launcher.daemon.server.api.DaemonCommandExecution > .proceed(DaemonCommandExecution.java:120) > at org.gradle.launcher.daemon.server.exec.StartBuildOrRespondWi > thBusy$1.run(StartBuildOrRespondWithBusy.java:50) > at org.gradle.launcher.daemon.server.DaemonStateCoordinator$1. > run(DaemonStateCoordinator.java:293) > at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecord > Failures.onExecute(ExecutorPolicy.java:54) > at org.gradle.internal.concurrent.StoppableExecutorImpl$1.run(S > toppableExecutorImpl.java:40) > at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1161) > at java.base/java.util.concurrent.ThreadPoolExecutor$Worker. > run(ThreadPoolExecutor.java:635) > at java.base/java.lang.Thread.run(Thread.java:843) > Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make > field private final java.util.Map java.util.Collections$UnmodifiableMap.m > accessible: module java.base does not "exports private java.util" to > unnamed module @78123e82 > at java.base/jdk.internal.reflect.Reflection.throwInaccessibleO > bjectException(Reflection.java:414) > at java.base/java.lang.reflect.AccessibleObject.checkCanSetAcce > ssible(AccessibleObject.java:198) > at java.base/java.lang.reflect.Field.checkCanSetAccessible(Fiel >
Re: JDK 9 Early Access with Project Jigsaw, build 135 on 09-14-2016 (#5500)
On 02.10.2016 02:25, Malachi de Ælfweald wrote: With build 125, Gradle 3.1 works fine. With build 138, I have not yet figured out how to get it running. The Gradle guys had been very proud of their "we support jdk9" story, but #AwkwardStrongEncapsulation really destroyed that for them... for now at least. Trying to run 'gradle init' in a bare directory results in 'java.lang.ExceptionInInitializerError' Adding '-Dsun.reflect.debugModuleAccessChecks=true' it reported: 16:56:47.501 [ERROR] [system.err] java.lang.reflect.InaccessibleObjectException: Unable to make protected java.lang.Package[] java.lang.ClassLoader.getPackages() accessible: module java.base does not "exports private java.lang" to unnamed module @352c1b98 I changed the _JAVA_OPTIONS to '--add-exports-private=java.base/java.lang=ALL-UNNAMED --add-modules=ALL-SYSTEM --add-exports=java.base/sun.nio.ch=ALL-UNNAMED' (the last two are for Dagger and Neo4j compat) That resulted in: Unable to get mutable environment variable map. This appears to come from a setAccessible from inside getEnv https://github.com/adammurdoch/native-platform/blob/master/src/main/java/net/rubygrapefruit/platform/internal/WrapperProcess.java#L113 adding --add-exports-private java.base/java.util=ALL-UNNAMED can help a bit here and there. Adding the same options above to the "org.gradle.jvmargs" in gradle.properties if you fork might too. And even though this will solve the problem in parts of Gradle, it will not solve the problem in the Groovy code called by Gradle. Because before I had a logic, that if not all methods can be made accessible, I skip the class when creating the meta class and use only the superclass and interface information. This logic worked fine with SecurityManagers and with the module logic before #AwkwardStrongEncapsulation. Now with #AwkwardStrongEncapsulation I have to go through each method and make it accessible piece by piece. Without this change the --add-exports-private is going to be quite long I imagine... and I am not convinced yet, that this solution will work in all relevant cases. It for sure is no solution for modules written in Groovy - should they ever come. An option to completely turn off #AwkwardStrongEncapsulation would be nice. So how do we get past #AwkwardStrongEncapsulation? Do I need to fork Gradle's dependencies and rebuild both of them since I have no code of my own in this use case? you can try making Gradle use Groovy master plus the above. I can´t say if it will work then though, haven´t tried that yet. bye Jochen
Re: JDK 9 Early Access with Project Jigsaw, build 135 on 09-14-2016 (#5500)
With build 125, Gradle 3.1 works fine. With build 138, I have not yet figured out how to get it running. Trying to run 'gradle init' in a bare directory results in 'java.lang.ExceptionInInitializerError' Adding '-Dsun.reflect.debugModuleAccessChecks=true' it reported: 16:56:47.501 [ERROR] [system.err] java.lang.reflect.InaccessibleObjectException: Unable to make protected java.lang.Package[] java.lang.ClassLoader.getPackages() accessible: module java.base does not "exports private java.lang" to unnamed module @352c1b98 I changed the _JAVA_OPTIONS to '--add-exports-private=java.base/java.lang=ALL-UNNAMED --add-modules=ALL-SYSTEM --add-exports=java.base/sun.nio.ch=ALL-UNNAMED' (the last two are for Dagger and Neo4j compat) That resulted in: Unable to get mutable environment variable map. This appears to come from a setAccessible from inside getEnv https://github.com/adammurdoch/native-platform/blob/master/src/main/java/net/rubygrapefruit/platform/internal/WrapperProcess.java#L113 So how do we get past #AwkwardStrongEncapsulation? Do I need to fork Gradle's dependencies and rebuild both of them since I have no code of my own in this use case? Malachi de Ælfweald http://www.google.com/profiles/malachid On Thu, Sep 15, 2016 at 10:07 AM, Alan Batemanwrote: > The EA download [1] has been refreshed with new builds that are mostly > aligned with the current proposals for JPMS issues [2]. Not everything is > there yet of course (and some areas need a few more iterations) but there > is enough to test and try things out. > > For #ReflectiveAccessToNonExportedTypes and #AwkwardStrongEncapsulation > then the initial support for `weak module` and `exports private` is in > place. These are probably the highest priority changes to try out. > > The changes to setAccessible in #AwkwardStrongEncapsulation is going to be > disruptive and will no doubt expose hacks in many areas. We've run into > several of these already. The command line option to allow existing code to > break into non-public types/members is --add-exports-private. The format of > the value passed to this option is the same--add-exports. With > #AddExportsInManifest then there are equivalents in the application JAR > main manifest to try out too. > > As part of the #ClassLoaderNames then the format of the stack trace > elements has changed. > > For #ServiceLoaderEnhancements then the implementation is mostly aligned, > except for the static factory method which needs an update to javac and SL > to align with the proposal (should be there soon, just not in this build). > > Several people have brought up #ResourceEncapsulation and > #ClassFilesAsResources here so it would be good to try this out. > > As always, the most valuable feedback is from those trying out the builds > so please let us know what you have tried and what works or doesn't work. > > -Alan > > [1] https://jdk9.java.net/jigsaw/ > [2] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-Septe > mber/009365.html >
Re: JDK 9 Early Access with Project Jigsaw, build 135 on 09-14-2016 (#5500)
Guice 4.1.0 throws an exception from within the embedded cglib with this release: >java -version java version "9-ea" Java(TM) SE Runtime Environment (build 9-ea+135-jigsaw-nightly-h5500-20160914) Java HotSpot(TM) 64-Bit Server VM (build 9-ea+135-jigsaw-nightly-h5500-20160914, mixed mode) > [apptest] > Caused by: java.lang.NoClassDefFoundError: Could not initialize class com.google.inject.internal.cglib.core.$ReflectUtils >at com.google.inject.internal.cglib.reflect.$FastClass$Generator.getProtectionDomain(FastClass.java:73) >at com.google.inject.internal.cglib.core.$AbstractClassGenerator.create(AbstractClassGenerator.java:206) >at com.google.inject.internal.cglib.reflect.$FastClass$Generator.create(FastClass.java:65) >at com.google.inject.internal.BytecodeGen.newFastClassForMember(BytecodeGen.java:252) >at com.google.inject.internal.BytecodeGen.newFastClassForMember(BytecodeGen.java:203) >at com.google.inject.internal.DefaultConstructionProxyFactory.create(DefaultConstructionProxyFactory.java:53) >at com.google.inject.internal.ProxyFactory.create(ProxyFactory.java:158) >at com.google.inject.internal.ConstructorInjectorStore.createConstructor(ConstructorInjectorStore.java:90) >at com.google.inject.internal.ConstructorInjectorStore.access$000(ConstructorInjectorStore.java:29) >at com.google.inject.internal.ConstructorInjectorStore$1.create(ConstructorInjectorStore.java:37) >at com.google.inject.internal.ConstructorInjectorStore$1.create(ConstructorInjectorStore.java:33) >at com.google.inject.internal.FailableCache$1.load(FailableCache.java:37) >at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3542) >at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2323) >at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2286) >at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2201) We had it working just fine with: Java(TM) SE Runtime Environment (build 9-ea+129-jigsaw-nightly-h5343-20160802) Java HotSpot(TM) 64-Bit Server VM (build 9-ea+129-jigsaw-nightly-h5343-20160802, mixed mode) Dawid On Thu, Sep 15, 2016 at 7:07 PM, Alan Batemanwrote: > The EA download [1] has been refreshed with new builds that are mostly > aligned with the current proposals for JPMS issues [2]. Not everything is > there yet of course (and some areas need a few more iterations) but there is > enough to test and try things out. > > For #ReflectiveAccessToNonExportedTypes and #AwkwardStrongEncapsulation then > the initial support for `weak module` and `exports private` is in place. > These are probably the highest priority changes to try out. > > The changes to setAccessible in #AwkwardStrongEncapsulation is going to be > disruptive and will no doubt expose hacks in many areas. We've run into > several of these already. The command line option to allow existing code to > break into non-public types/members is --add-exports-private. The format of > the value passed to this option is the same--add-exports. With > #AddExportsInManifest then there are equivalents in the application JAR main > manifest to try out too. > > As part of the #ClassLoaderNames then the format of the stack trace elements > has changed. > > For #ServiceLoaderEnhancements then the implementation is mostly aligned, > except for the static factory method which needs an update to javac and SL > to align with the proposal (should be there soon, just not in this build). > > Several people have brought up #ResourceEncapsulation and > #ClassFilesAsResources here so it would be good to try this out. > > As always, the most valuable feedback is from those trying out the builds so > please let us know what you have tried and what works or doesn't work. > > -Alan > > [1] https://jdk9.java.net/jigsaw/ > [2] > http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-September/009365.html