Hi,
Sorry my message was incorrect: There are 2 bugs in Lucene and Java: - In Lucene the unmap hack’s exception handling falls over the stupid InaccessibleObjectException does not extend ReflectiveOperationException issue (I complained about that several times). I can fix this, but it’s horrible. - Java now hides internal of ByteBuffer completely, although the JEP about Unsafe & Co. said which Java APIs are excluded from the lock down. Next to Unsafe, also ByteBuffer was on the list of “still allowed APIs where the Java 9 backwards layer helps”. MappedByteBuffer unmapping is one of the most important parts in Lucene that must be supported, otherwise it is impossible to use Lucene, Solr, or Elasticsearch with Java 9 in a performant way. We can also not require from our users to add strange command line options to their JVMs. Uwe ----- Uwe Schindler Achterdiek 19, D-28357 Bremen <http://www.thetaphi.de/> http://www.thetaphi.de eMail: u...@thetaphi.de From: Uwe Schindler [mailto:u...@thetaphi.de] Sent: Friday, December 9, 2016 5:53 PM To: dev@lucene.apache.org; 'Rory O'Donnell' <rory.odonn...@oracle.com> Cc: 'Dalibor Topic' <dalibor.to...@oracle.com>; balchandra.vai...@oracle.com; abdul.kolarku...@oracle.com; 'Dawid Weiss' <dawid.we...@cs.put.poznan.pl> Subject: RE: JDK 9 b148 including a refresh of the module system is available on java.net Hi, build 148 broke the unmap hack. The problem is an additional bug in Java: The unmap hack should be more unspecific on the Exception Type (InaccessibleObjectException does not extend ReflectiveOperationException). I will change this in an issue, but the general problem persists: With the buil 148 version we can no longer unmap byte buffers: java.lang.reflect.InaccessibleObjectException: Unable to make public jdk.internal.ref.Cleaner java.nio.DirectByteBuffer.cleaner() accessible: module java.base does not "opens java.nio" to unnamed module @10b73cf [junit4] > at java.base/jdk.internal.reflect.Reflection.throwInaccessibleObjectException(Reflection.java:427) [junit4] > at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:201) [junit4] > at java.base/java.lang.reflect.Method.checkCanSetAccessible(Method.java:192) [junit4] > at java.base/java.lang.reflect.Method.setAccessible(Method.java:186) [junit4] > at org.apache.lucene.store.MMapDirectory.unmapHackImpl(MMapDirectory.java:345) [junit4] > at java.base/java.security.AccessController.doPrivileged(Native Method) [junit4] > at org.apache.lucene.store.MMapDirectory.<clinit>(MMapDirectory.java:326) [junit4] > ... 42 moreThrowable #2: java.lang.NullPointerException [junit4] > at org.apache.lucene.search.TestDateSort.tearDown(TestDateSort.java:71) [junit4] > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [junit4] > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [junit4] > at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [junit4] > at java.base/java.lang.reflect.Method.invoke(Method.java:538) [junit4] > at java.base/java.lang.Thread.run(Thread.java:844) ----- Uwe Schindler Achterdiek 19, D-28357 Bremen http://www.thetaphi.de <http://www.thetaphi.de/> eMail: <mailto:u...@thetaphi.de> u...@thetaphi.de From: Uwe Schindler [ <mailto:u...@thetaphi.de> mailto:u...@thetaphi.de] Sent: Friday, December 9, 2016 4:35 PM To: 'Rory O'Donnell' < <mailto:rory.odonn...@oracle.com> rory.odonn...@oracle.com> Cc: 'Dalibor Topic' < <mailto:dalibor.to...@oracle.com> dalibor.to...@oracle.com>; <mailto:balchandra.vai...@oracle.com> balchandra.vai...@oracle.com; <mailto:abdul.kolarku...@oracle.com> abdul.kolarku...@oracle.com; 'Dawid Weiss' < <mailto:dawid.we...@cs.put.poznan.pl> dawid.we...@cs.put.poznan.pl>; <mailto:dev@lucene.apache.org> dev@lucene.apache.org Subject: RE: JDK 9 b148 including a refresh of the module system is available on java.net Hi Rory, thanks for the message! I put build 148 in the Jenkins test rotation. Let’s see what happens. Uwe ----- Uwe Schindler Achterdiek 19, D-28357 Bremen http://www.thetaphi.de <http://www.thetaphi.de/> eMail: <mailto:u...@thetaphi.de> u...@thetaphi.de From: Rory O'Donnell [ <mailto:rory.odonn...@oracle.com> mailto:rory.odonn...@oracle.com] Sent: Friday, December 9, 2016 11:34 AM To: <mailto:u...@thetaphi.de> u...@thetaphi.de Cc: <mailto:rory.odonn...@oracle.com> rory.odonn...@oracle.com; Dalibor Topic < <mailto:dalibor.to...@oracle.com> dalibor.to...@oracle.com>; <mailto:balchandra.vai...@oracle.com> balchandra.vai...@oracle.com; <mailto:abdul.kolarku...@oracle.com> abdul.kolarku...@oracle.com; Dawid Weiss < <mailto:dawid.we...@cs.put.poznan.pl> dawid.we...@cs.put.poznan.pl>; <mailto:dev@lucene.apache.org> dev@lucene.apache.org Subject: JDK 9 b148 including a refresh of the module system is available on java.net Hi Uwe & Dawid, JDK 9 build b148 <https://jdk9.java.net/download/> includes an important Refresh of the module system [1] , summary of changes are listed here <http://download.java.net/java/jdk9/changes/jdk-9+148.html> . This refresh includes a disruptive change that is important to understand. For those that have been trying out modules with regular JDK 9 builds then be aware that `requires public` changes to `requires transitive`. In addition, the binary representation of the module declaration (module-info.class) has changed so that you need to recompile any modules that were compiled with previous JDK 9 builds. As things stand today in JDK 9 then you use setAccessible to break into non-public elements of any type in exported packages. However, it cannot be used to break into any type in non-exported package. The current specified behavior was a compromise for the initial integration of the module system. It is of course not very satisfactory, hence the #AwkwardStrongEncapsulation issue [2] on the JSR 376 issues list. With the updated proposal in the JSR, this refresh changes setAccessible further so that it cannot be used to break into non-public types, or non-public elements of public types, in exported packages. Code that uses setAccessible to hack into the private constructor of java.lang.invoke.MethodHandles.Lookup will be disappointed for example. This change will expose hacks in many existing libraries and tools. As a workaround then a new command line option `--add-opens` can be used to open specific packages for "deep reflection". For example, a really popular build tool fails with this refresh because it uses setAccessible + core reflection to hack into a private field of an unmodifiable collection so that it can mutate it, facepalm! This code will continue to work as before when run with `--add-opens java.base/java.util=ALL-UNNAMED` to open the package java.util in module java.base to "all unnamed modules" (think class path). Any help reporting issues to popular tools and libraries would be appreciated. A debugging aid that is useful to identify issues is to run with -Dsun.reflect.debugModuleAccessChecks=true to get a stack trace when setAccessible fails, this is particularly useful when code swallows exceptions without any logging. Rgds,Rory [1] http://mail.openjdk.java.net/pipermail/jdk9-dev/2016-November/005276.html <http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-October/000430.html> [2] http://openjdk.java.net/projects/jigsaw/spec/issues/#AwkwardStrongEncapsulation -- Rgds,Rory O'Donnell Quality Engineering Manager Oracle EMEA , Dublin, Ireland
