RE: Java 9 build 148 causes trouble in Apache Lucene/Solr/Elasticsearch

[Uwe Schindler]

Hi,

sorry for the delay! I updated Lucene's build system to use the JFrog snapshort 
artifacts and build succeeds with JDK 9 b148+:

That's the one that was choosen by Apache Ant's Ivy downloader: 
groovy-all-2.4.8-20161220.101835-40.jar

So we are waiting for a release!
Uwe

-
Uwe Schindler
Achterdiek 19, D-28357 Bremen
http://www.thetaphi.de
eMail: u...@thetaphi.de

> -Original Message-
> From: core-libs-dev [mailto:core-libs-dev-boun...@openjdk.java.net] On
> Behalf Of Jochen Theodorou
> Sent: Friday, December 16, 2016 2:11 PM
> To: d...@groovy.apache.org; jigsaw-...@openjdk.java.net; 'Core-Libs-Dev'
> <core-libs-dev@openjdk.java.net>
> Subject: Re: Java 9 build 148 causes trouble in Apache
> Lucene/Solr/Elasticsearch
> 
> Hi,
> 
> I strongly hope Paul and Cedric will be able to start the release
> process next week, if not we will have to do it the old way I think.
> 
> what would help us a lot would be you testing the GROOVY_2_4_X branch
> with your build system to see if it really does solve your problem. Even
> if it is only locally on your computer
> 
> bye Jochen
> 
> On 16.12.2016 10:58, Uwe Schindler wrote:
> > Hi Jochen,
> >
> > thank you for the information! Is there any plan about a release? I also
> found no JIRA issue about this issue to link it against our JIRA:
> https://issues.apache.org/jira/browse/LUCENE-7596
> >
> > The problem makes our build system unusable, so it would be very
> important to have a fix quite soon! As our Ant/Ivy-based build relies on
> Maven Central, it would be good to have the bugfix release available there,
> which requires a release. I think the same applies for Gradle users
> (Elasticsearch).
> >
> > As a temporary workaround we might be able to use the Apache Snapshot
> repository, but this is not allowed if we do a release of Lucene.
> >
> > Uwe
> >
> > -
> > Uwe Schindler
> > uschind...@apache.org
> > ASF Member, Apache Lucene PMC / Committer
> > Bremen, Germany
> > http://lucene.apache.org/
> >
> >> -Original Message-
> >> From: Jochen Theodorou [mailto:blackd...@gmx.org]
> >> Sent: Saturday, December 10, 2016 9:23 AM
> >> To: Uwe Schindler <uschind...@apache.org>; jigsaw-
> d...@openjdk.java.net;
> >> Core-Libs-Dev <core-libs-dev@openjdk.java.net>
> >> Subject: Re: Java 9 build 148 causes trouble in Apache
> >> Lucene/Solr/Elasticsearch
> >>
> >> On 09.12.2016 23:32, Uwe Schindler wrote:
> >>> Hi,
> >>>
> >>> I updated our Jenkins server for the JDK 9 preview testing to use build
> 148.
> >> Previously we had build 140 and build 147, which both worked without
> any
> >> issues. But after the update the following stuff goes wrong:
> >>>
> >>> (1) Unmapping of direct buffers no longer works, although this API was
> >> marked as critical because there is no replacement up to now, so code can
> >> unmap memory mapped files, which is one of the most important things
> >> Apache Lucene needs to use to access huge random access files while
> >> reading the index. Without memory mapping, the slowdown for Lucene
> >> users will be huge
> >>>
> >>> This is caused by the recent Jigsaw changes, published in build 148.
> >> Unfortunately we did not test the Jigsaw builds, so we would have noticed
> >> that earlier. Basically the following peace of code fails now (with or
> without
> >> doPrivileged and with/without security manager):
> >>>
> >>>final Class directBufferClass =
> >> Class.forName("java.nio.DirectByteBuffer");
> >>>
> >>>final Method m = directBufferClass.getMethod("cleaner");
> >>>m.setAccessible(true);
> >>>MethodHandle directBufferCleanerMethod = lookup.unreflect(m);
> >>>Class cleanerClass =
> >> directBufferCleanerMethod.type().returnType();
> >>>// build method handle for unmapping, full code is here:
> >> https://goo.gl/TfQWl6
> >>
> >> I guess that is the effect of #AwkwardStrongEncapsulation. I would
> >> advise doing regular checks against the jigsaw builds to know about such
> >> problems in the future earlier... but seeing your code break without an
> >> obvious good solution sure is stressful. I feel with you.
> >>
> >> [...]
> >>> (2) A second thing  we noticed is that Groovy no longer works and dies
> with
> >> strange error messages.
> >>
> >> That is because

Re: Java 9 build 148 causes trouble in Apache Lucene/Solr/Elasticsearch

[Chris Hegarty]

> On 19 Dec 2016, at 19:21, Uwe Schindler <uschind...@apache.org> wrote:
> 
> Hi,
> 
> will there be an update for JEP 260, so this is documented?

Yes, working on it.

-Chris.

> Uwe
> 
> -
> Uwe Schindler
> uschind...@apache.org 
> ASF Member, Apache Lucene PMC / Committer
> Bremen, Germany
> http://lucene.apache.org/
> 
>> -Original Message-
>> From: Chris Hegarty [mailto:chris.hega...@oracle.com]
>> Sent: Friday, December 16, 2016 6:39 PM
>> To: Peter Levart <peter.lev...@gmail.com>; Core-Libs-Dev > d...@openjdk.java.net>; jigsaw-dev <jigsaw-...@openjdk.java.net>; Uwe
>> Schindler <uschind...@apache.org>
>> Subject: Re: Java 9 build 148 causes trouble in Apache
>> Lucene/Solr/Elasticsearch
>> 
>> Pushed to jdk9/dev. Should make b150.
>> 
>>  https://bugs.openjdk.java.net/browse/JDK-8171377
>> 
>> -Chris.
>> 
>>> On 14 Dec 2016, at 11:58, Chris Hegarty <chris.hega...@oracle.com>
>> wrote:
>>> 
>>> Webrev updated in-place.
>>> http://cr.openjdk.java.net/~chegar/Unsafe_invokeCleaner/
>>> 
>>> -Chris.
>>> 
>>> On 13/12/16 21:18, Peter Levart wrote:
>>>> I think this is OK.
>>>> 
>>>> Just a couple of nits in test:
>>>> 
>>>> 1. You create a static Path bob = Paths.get("bob") field, but then you
>>>> don't use it in:
>>>> 
>>>> 56 try (FileChannel fc = FileChannel.open(Paths.get("bob"),
>>>> CREATE, WRITE)) {
>>>> 
>>>> 2. badBuffers could include a duplicate and a slice of a direct buffer
>>>> allocated with ByteBuffer.allocateDirect()
>>>> 
>>>> 3. The comment in the test is referencing the old method name:
>>>> 
>>>> 26  * @summary Basic test for Unsafe::deallocate
>>>> 
>>>> 
>>>> Regards, Peter
>>>> 
>>>> On 12/13/2016 08:47 PM, Chris Hegarty wrote:
>>>>> Taking into account the feedback so far, and changing the method name
>> ( since
>>>>> it is an attractive nuisance ), here is where I think we ended up.
>>>>> 
>>>>> http://cr.openjdk.java.net/~chegar/Unsafe_invokeCleaner/
>>>>> 
>>>>> If this is agreeable, I’ll file an issue in JIRA to track the code 
>>>>> changes, and
>>>>> update JEP 260.
>>>>> 
>>>>> -Chris.
>>>> 
> 

RE: Java 9 build 148 causes trouble in Apache Lucene/Solr/Elasticsearch

[Uwe Schindler]

Hi,

will there be an update for JEP 260, so this is documented?

Uwe

-
Uwe Schindler
uschind...@apache.org 
ASF Member, Apache Lucene PMC / Committer
Bremen, Germany
http://lucene.apache.org/

> -Original Message-
> From: Chris Hegarty [mailto:chris.hega...@oracle.com]
> Sent: Friday, December 16, 2016 6:39 PM
> To: Peter Levart <peter.lev...@gmail.com>; Core-Libs-Dev  d...@openjdk.java.net>; jigsaw-dev <jigsaw-...@openjdk.java.net>; Uwe
> Schindler <uschind...@apache.org>
> Subject: Re: Java 9 build 148 causes trouble in Apache
> Lucene/Solr/Elasticsearch
> 
> Pushed to jdk9/dev. Should make b150.
> 
>   https://bugs.openjdk.java.net/browse/JDK-8171377
> 
> -Chris.
> 
> > On 14 Dec 2016, at 11:58, Chris Hegarty <chris.hega...@oracle.com>
> wrote:
> >
> > Webrev updated in-place.
> >  http://cr.openjdk.java.net/~chegar/Unsafe_invokeCleaner/
> >
> > -Chris.
> >
> > On 13/12/16 21:18, Peter Levart wrote:
> >> I think this is OK.
> >>
> >> Just a couple of nits in test:
> >>
> >> 1. You create a static Path bob = Paths.get("bob") field, but then you
> >> don't use it in:
> >>
> >>  56 try (FileChannel fc = FileChannel.open(Paths.get("bob"),
> >> CREATE, WRITE)) {
> >>
> >> 2. badBuffers could include a duplicate and a slice of a direct buffer
> >> allocated with ByteBuffer.allocateDirect()
> >>
> >> 3. The comment in the test is referencing the old method name:
> >>
> >>  26  * @summary Basic test for Unsafe::deallocate
> >>
> >>
> >> Regards, Peter
> >>
> >> On 12/13/2016 08:47 PM, Chris Hegarty wrote:
> >>> Taking into account the feedback so far, and changing the method name
> ( since
> >>> it is an attractive nuisance ), here is where I think we ended up.
> >>>
> >>> http://cr.openjdk.java.net/~chegar/Unsafe_invokeCleaner/
> >>>
> >>> If this is agreeable, I’ll file an issue in JIRA to track the code 
> >>> changes, and
> >>> update JEP 260.
> >>>
> >>> -Chris.
> >>

RE: Java 9 build 148 causes trouble in Apache Lucene/Solr/Elasticsearch

[Uwe Schindler]

Hi Chris,

thanks, works perfectly. I compiled a JDK with this commit and Lucene's 
unmapping works. Thanks.
https://github.com/apache/lucene-solr/compare/LUCENE-6989-v2

Uwe

-
Uwe Schindler
uschind...@apache.org 
ASF Member, Apache Lucene PMC / Committer
Bremen, Germany
http://lucene.apache.org/

> -Original Message-
> From: Chris Hegarty [mailto:chris.hega...@oracle.com]
> Sent: Friday, December 16, 2016 6:39 PM
> To: Peter Levart <peter.lev...@gmail.com>; Core-Libs-Dev  d...@openjdk.java.net>; jigsaw-dev <jigsaw-...@openjdk.java.net>; Uwe
> Schindler <uschind...@apache.org>
> Subject: Re: Java 9 build 148 causes trouble in Apache
> Lucene/Solr/Elasticsearch
> 
> Pushed to jdk9/dev. Should make b150.
> 
>   https://bugs.openjdk.java.net/browse/JDK-8171377
> 
> -Chris.
> 
> > On 14 Dec 2016, at 11:58, Chris Hegarty <chris.hega...@oracle.com>
> wrote:
> >
> > Webrev updated in-place.
> >  http://cr.openjdk.java.net/~chegar/Unsafe_invokeCleaner/
> >
> > -Chris.
> >
> > On 13/12/16 21:18, Peter Levart wrote:
> >> I think this is OK.
> >>
> >> Just a couple of nits in test:
> >>
> >> 1. You create a static Path bob = Paths.get("bob") field, but then you
> >> don't use it in:
> >>
> >>  56 try (FileChannel fc = FileChannel.open(Paths.get("bob"),
> >> CREATE, WRITE)) {
> >>
> >> 2. badBuffers could include a duplicate and a slice of a direct buffer
> >> allocated with ByteBuffer.allocateDirect()
> >>
> >> 3. The comment in the test is referencing the old method name:
> >>
> >>  26  * @summary Basic test for Unsafe::deallocate
> >>
> >>
> >> Regards, Peter
> >>
> >> On 12/13/2016 08:47 PM, Chris Hegarty wrote:
> >>> Taking into account the feedback so far, and changing the method name
> ( since
> >>> it is an attractive nuisance ), here is where I think we ended up.
> >>>
> >>> http://cr.openjdk.java.net/~chegar/Unsafe_invokeCleaner/
> >>>
> >>> If this is agreeable, I’ll file an issue in JIRA to track the code 
> >>> changes, and
> >>> update JEP 260.
> >>>
> >>> -Chris.
> >>

Re: Java 9 build 148 causes trouble in Apache Lucene/Solr/Elasticsearch

[Chris Hegarty]

Pushed to jdk9/dev. Should make b150.

  https://bugs.openjdk.java.net/browse/JDK-8171377

-Chris.

> On 14 Dec 2016, at 11:58, Chris Hegarty  wrote:
> 
> Webrev updated in-place.
>  http://cr.openjdk.java.net/~chegar/Unsafe_invokeCleaner/
> 
> -Chris.
> 
> On 13/12/16 21:18, Peter Levart wrote:
>> I think this is OK.
>> 
>> Just a couple of nits in test:
>> 
>> 1. You create a static Path bob = Paths.get("bob") field, but then you
>> don't use it in:
>> 
>>  56 try (FileChannel fc = FileChannel.open(Paths.get("bob"),
>> CREATE, WRITE)) {
>> 
>> 2. badBuffers could include a duplicate and a slice of a direct buffer
>> allocated with ByteBuffer.allocateDirect()
>> 
>> 3. The comment in the test is referencing the old method name:
>> 
>>  26  * @summary Basic test for Unsafe::deallocate
>> 
>> 
>> Regards, Peter
>> 
>> On 12/13/2016 08:47 PM, Chris Hegarty wrote:
>>> Taking into account the feedback so far, and changing the method name ( 
>>> since
>>> it is an attractive nuisance ), here is where I think we ended up.
>>> 
>>> http://cr.openjdk.java.net/~chegar/Unsafe_invokeCleaner/
>>> 
>>> If this is agreeable, I’ll file an issue in JIRA to track the code changes, 
>>> and
>>> update JEP 260.
>>> 
>>> -Chris.
>> 

Re: Java 9 build 148 causes trouble in Apache Lucene/Solr/Elasticsearch

[Jochen Theodorou]

Hi,

I strongly hope Paul and Cedric will be able to start the release 
process next week, if not we will have to do it the old way I think.


what would help us a lot would be you testing the GROOVY_2_4_X branch 
with your build system to see if it really does solve your problem. Even 
if it is only locally on your computer


bye Jochen

On 16.12.2016 10:58, Uwe Schindler wrote:

Hi Jochen,

thank you for the information! Is there any plan about a release? I also found 
no JIRA issue about this issue to link it against our JIRA: 
https://issues.apache.org/jira/browse/LUCENE-7596

The problem makes our build system unusable, so it would be very important to 
have a fix quite soon! As our Ant/Ivy-based build relies on Maven Central, it 
would be good to have the bugfix release available there, which requires a 
release. I think the same applies for Gradle users (Elasticsearch).

As a temporary workaround we might be able to use the Apache Snapshot 
repository, but this is not allowed if we do a release of Lucene.

Uwe

-
Uwe Schindler
uschind...@apache.org
ASF Member, Apache Lucene PMC / Committer
Bremen, Germany
http://lucene.apache.org/


-Original Message-
From: Jochen Theodorou [mailto:blackd...@gmx.org]
Sent: Saturday, December 10, 2016 9:23 AM
To: Uwe Schindler <uschind...@apache.org>; jigsaw-...@openjdk.java.net;
Core-Libs-Dev <core-libs-dev@openjdk.java.net>
Subject: Re: Java 9 build 148 causes trouble in Apache
Lucene/Solr/Elasticsearch

On 09.12.2016 23:32, Uwe Schindler wrote:

Hi,

I updated our Jenkins server for the JDK 9 preview testing to use build 148.

Previously we had build 140 and build 147, which both worked without any
issues. But after the update the following stuff goes wrong:


(1) Unmapping of direct buffers no longer works, although this API was

marked as critical because there is no replacement up to now, so code can
unmap memory mapped files, which is one of the most important things
Apache Lucene needs to use to access huge random access files while
reading the index. Without memory mapping, the slowdown for Lucene
users will be huge


This is caused by the recent Jigsaw changes, published in build 148.

Unfortunately we did not test the Jigsaw builds, so we would have noticed
that earlier. Basically the following peace of code fails now (with or without
doPrivileged and with/without security manager):


   final Class directBufferClass =

Class.forName("java.nio.DirectByteBuffer");


   final Method m = directBufferClass.getMethod("cleaner");
   m.setAccessible(true);
   MethodHandle directBufferCleanerMethod = lookup.unreflect(m);
   Class cleanerClass =

directBufferCleanerMethod.type().returnType();

   // build method handle for unmapping, full code is here:

https://goo.gl/TfQWl6

I guess that is the effect of #AwkwardStrongEncapsulation. I would
advise doing regular checks against the jigsaw builds to know about such
problems in the future earlier... but seeing your code break without an
obvious good solution sure is stressful. I feel with you.

[...]

(2) A second thing  we noticed is that Groovy no longer works and dies with

strange error messages.

That is because versions including Groovy 2.4.7 are using
setAccessible(AccessibleObject[] array, true), and the array will also
include private methods or fields. This worked till
#AwkwardStrongEncapsulation because will then a class was either
exported and its method can all be made accessible or not. For example
on GAE or earlier versions of the module system. Now an exported class
may break this, since its private methods can no longer be made
accessible using setAccessible.

A fix for this is already committed, we are only waiting for release of
Groovy 2.4.8. Of course even with the fix Groovy code can possibly
break... for example if you did the direct buffer access in Groovy.

Btw, do not hesitate to ask about such problems on groovy-user, please.

bye Jochen

RE: Java 9 build 148 causes trouble in Apache Lucene/Solr/Elasticsearch

[Uwe Schindler]

Hi Jochen,

thank you for the information! Is there any plan about a release? I also found 
no JIRA issue about this issue to link it against our JIRA: 
https://issues.apache.org/jira/browse/LUCENE-7596

The problem makes our build system unusable, so it would be very important to 
have a fix quite soon! As our Ant/Ivy-based build relies on Maven Central, it 
would be good to have the bugfix release available there, which requires a 
release. I think the same applies for Gradle users (Elasticsearch).

As a temporary workaround we might be able to use the Apache Snapshot 
repository, but this is not allowed if we do a release of Lucene.

Uwe

-
Uwe Schindler
uschind...@apache.org 
ASF Member, Apache Lucene PMC / Committer
Bremen, Germany
http://lucene.apache.org/

> -Original Message-
> From: Jochen Theodorou [mailto:blackd...@gmx.org]
> Sent: Saturday, December 10, 2016 9:23 AM
> To: Uwe Schindler <uschind...@apache.org>; jigsaw-...@openjdk.java.net;
> Core-Libs-Dev <core-libs-dev@openjdk.java.net>
> Subject: Re: Java 9 build 148 causes trouble in Apache
> Lucene/Solr/Elasticsearch
> 
> On 09.12.2016 23:32, Uwe Schindler wrote:
> > Hi,
> >
> > I updated our Jenkins server for the JDK 9 preview testing to use build 148.
> Previously we had build 140 and build 147, which both worked without any
> issues. But after the update the following stuff goes wrong:
> >
> > (1) Unmapping of direct buffers no longer works, although this API was
> marked as critical because there is no replacement up to now, so code can
> unmap memory mapped files, which is one of the most important things
> Apache Lucene needs to use to access huge random access files while
> reading the index. Without memory mapping, the slowdown for Lucene
> users will be huge
> >
> > This is caused by the recent Jigsaw changes, published in build 148.
> Unfortunately we did not test the Jigsaw builds, so we would have noticed
> that earlier. Basically the following peace of code fails now (with or without
> doPrivileged and with/without security manager):
> >
> >final Class directBufferClass =
> Class.forName("java.nio.DirectByteBuffer");
> >
> >final Method m = directBufferClass.getMethod("cleaner");
> >m.setAccessible(true);
> >MethodHandle directBufferCleanerMethod = lookup.unreflect(m);
> >Class cleanerClass =
> directBufferCleanerMethod.type().returnType();
> >// build method handle for unmapping, full code is here:
> https://goo.gl/TfQWl6
> 
> I guess that is the effect of #AwkwardStrongEncapsulation. I would
> advise doing regular checks against the jigsaw builds to know about such
> problems in the future earlier... but seeing your code break without an
> obvious good solution sure is stressful. I feel with you.
> 
> [...]
> > (2) A second thing  we noticed is that Groovy no longer works and dies with
> strange error messages.
> 
> That is because versions including Groovy 2.4.7 are using
> setAccessible(AccessibleObject[] array, true), and the array will also
> include private methods or fields. This worked till
> #AwkwardStrongEncapsulation because will then a class was either
> exported and its method can all be made accessible or not. For example
> on GAE or earlier versions of the module system. Now an exported class
> may break this, since its private methods can no longer be made
> accessible using setAccessible.
> 
> A fix for this is already committed, we are only waiting for release of
> Groovy 2.4.8. Of course even with the fix Groovy code can possibly
> break... for example if you did the direct buffer access in Groovy.
> 
> Btw, do not hesitate to ask about such problems on groovy-user, please.
> 
> bye Jochen

Re: Java 9 build 148 causes trouble in Apache Lucene/Solr/Elasticsearch

[Alan Bateman]

On 13/12/2016 21:18, Peter Levart wrote:

I think this is OK.

Just a couple of nits in test:

1. You create a static Path bob = Paths.get("bob") field, but then you 
don't use it in:


  56 try (FileChannel fc = FileChannel.open(Paths.get("bob"), 
CREATE, WRITE)) {


Adding to Peter's comment, this can be further changed to use 
Files.write(bob, srcData).


Otherwise I think the patch looks okay although it does feel like 
invokeCleaner needs a warning in the javadoc, maybe being in Unsafe is 
enough.


-Alan

RE: Java 9 build 148 causes trouble in Apache Lucene/Solr/Elasticsearch

[Uwe Schindler]

Hi Chris,

looks good to me. I already created a patch / branch of Apache Lucene that uses 
your proposed API and removes the Runnable hack. You can see it and check it 
out on Github:

<https://github.com/apache/lucene-solr/compare/LUCENE-6989-v2>

For your info the issue description and description is:
<https://issues.apache.org/jira/browse/LUCENE-6989?focusedCommentId=15748848=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15748848>

If you would like to test it against your patch, check out this branch, run 
"ant ivy-bootstrap" and then cd into the lucene/core dirto execute:
$ ant test -Dtestcase=TestMmapDirectory

Those tests should run without any error message (disabled test because 
unmapping not supported):

[junit4] Started J0 PID(3464@localhost).
   [junit4] Suite: org.apache.lucene.store.TestMmapDirectory
   [junit4] OK  0.07s | TestMmapDirectory.testIllegalEOF
   [junit4] OK  0.01s | TestMmapDirectory.testShort
   [junit4] OK  0.06s | TestMmapDirectory.testRandomInt
   [junit4] OK  0.02s | TestMmapDirectory.testFsyncDoesntCreateNewFiles
   [junit4] OK  0.01s | TestMmapDirectory.testSetOfStrings
   [junit4] OK  0.01s | TestMmapDirectory.testLargeWrites
   [junit4] OK  0.01s | TestMmapDirectory.testMapOfStrings
   [junit4] OK  0.01s | TestMmapDirectory.testSliceOfSlice
   [junit4] OK  0.01s | TestMmapDirectory.testByte
   [junit4] OK  0.01s | TestMmapDirectory.testVInt
   [junit4] OK  0.01s | TestMmapDirectory.testChecksum
   [junit4] OK  0.02s | TestMmapDirectory.testZLong
   [junit4] OK  0.06s | TestMmapDirectory.testRandomShort
   [junit4] OK  0.01s | TestMmapDirectory.testRename
   [junit4] OK  0.10s | TestMmapDirectory.testCreateTempOutput
   [junit4] OK  0.01s | TestMmapDirectory.testZInt
   [junit4] OK  0.01s | TestMmapDirectory.testIndexOutputToString
   [junit4] OK  0.01s | TestMmapDirectory.testDetectClose
   [junit4] OK  0.03s | TestMmapDirectory.testListAllIsSorted
   [junit4] OK  0.01s | TestMmapDirectory.testDoubleCloseDirectory
   [junit4] OK  0.00s | TestMmapDirectory.testDirectoryFilter
   [junit4] OK  0.11s | TestMmapDirectory.testPendingDeletions
   [junit4] OK  0.04s | TestMmapDirectory.testCopyFromDestination
   [junit4] OK  0.01s | TestMmapDirectory.testDeleteFile
   [junit4] OK  0.04s | TestMmapDirectory.testCopyBytesWithThreads
   [junit4] OK  0.03s | TestMmapDirectory.testRandomByte
   [junit4] IGNORED 0.00s | TestMmapDirectory.testAceWithThreads
   [junit4]> Cause: Annotated @Ignore(This test is for JVM testing 
purposes. There are no guarantees that it may not fail with SIGSEGV!)
   [junit4] OK  0.02s | TestMmapDirectory.testNoDir
   [junit4] OK  0.01s | TestMmapDirectory.testSeekBeyondEndOfFile
   [junit4] OK  0.13s | TestMmapDirectory.testRandomLong
   [junit4] OK  0.01s | TestMmapDirectory.testDoubleCloseInput
   [junit4] OK  0.01s | TestMmapDirectory.testSliceOutOfBounds
   [junit4] OK  2.75s | TestMmapDirectory.testThreadSafety
   [junit4] OK  0.01s | TestMmapDirectory.testSeekToEndOfFile
   [junit4] OK  0.00s | TestMmapDirectory.testLong
   [junit4] OK  0.01s | TestMmapDirectory.testCopyFrom
   [junit4] OK  0.00s | TestMmapDirectory.testString
   [junit4] OK  0.01s | TestMmapDirectory.testSeekToEOFThenBack
   [junit4] OK  0.00s | TestMmapDirectory.testInt
   [junit4] OK  0.01s | TestMmapDirectory.testSeekPastEOF
   [junit4] OK  0.02s | TestMmapDirectory.testCopyBytes
   [junit4] OK  0.01s | TestMmapDirectory.testVLong
   [junit4] OK  0.01s | TestMmapDirectory.testDoubleCloseOutput
   [junit4] Completed [1/1] in 5.14s, 43 tests, 1 skipped

With Java 9 build 148 it currently looks like this:

   [junit4] Suite: org.apache.lucene.store.TestMmapDirectory
   [junit4] IGNOR/A 0.04s | TestMmapDirectory.testShort
   [junit4]> Assumption #1: test requires a jre that supports unmapping: 
Unmapping is not supported on this platform, because internal Java APIs are not 
compatible to this Lucene version: 
java.lang.reflect.InaccessibleObjectException: Unable to make public 
jdk.internal.ref.Cleaner java.nio.DirectByteBuffer.cleaner() accessible: module 
java.base does not "opens java.nio" to unnamed module @45c7ad7f

Don't care about the Groovy error printed later, this one is known, but does 
not affect testing.

Uwe

-
Uwe Schindler
uschind...@apache.org 
ASF Member, Apache Lucene PMC / Committer
Bremen, Germany
http://lucene.apache.org/

> -Original Message-
> From: Chris Hegarty [mailto:chris.hega...@oracle.com]
> Sent: Wednesday, December 14, 2016 12:58 PM
> To: Peter Levart <peter.lev...@gmail.com>; Core-Libs-Dev  d...@openjdk.java.net>; jigsaw-dev <jigsaw-...@openjdk.java.net>; Uwe
> Schindler <uschind...@apache.org>
> Subject: Re: Java 9 build 148 causes trouble 

Re: Java 9 build 148 causes trouble in Apache Lucene/Solr/Elasticsearch

[Chris Hegarty]

Webrev updated in-place.
  http://cr.openjdk.java.net/~chegar/Unsafe_invokeCleaner/

-Chris.

On 13/12/16 21:18, Peter Levart wrote:

I think this is OK.

Just a couple of nits in test:

1. You create a static Path bob = Paths.get("bob") field, but then you
don't use it in:

  56 try (FileChannel fc = FileChannel.open(Paths.get("bob"),
CREATE, WRITE)) {

2. badBuffers could include a duplicate and a slice of a direct buffer
allocated with ByteBuffer.allocateDirect()

3. The comment in the test is referencing the old method name:

  26  * @summary Basic test for Unsafe::deallocate


Regards, Peter

On 12/13/2016 08:47 PM, Chris Hegarty wrote:

Taking into account the feedback so far, and changing the method name ( since
it is an attractive nuisance ), here is where I think we ended up.

http://cr.openjdk.java.net/~chegar/Unsafe_invokeCleaner/

If this is agreeable, I’ll file an issue in JIRA to track the code changes, and
update JEP 260.

-Chris.

Re: Java 9 build 148 causes trouble in Apache Lucene/Solr/Elasticsearch

[Peter Levart]

I think this is OK.

Just a couple of nits in test:

1. You create a static Path bob = Paths.get("bob") field, but then you 
don't use it in:


  56 try (FileChannel fc = FileChannel.open(Paths.get("bob"), 
CREATE, WRITE)) {


2. badBuffers could include a duplicate and a slice of a direct buffer 
allocated with ByteBuffer.allocateDirect()


3. The comment in the test is referencing the old method name:

  26  * @summary Basic test for Unsafe::deallocate


Regards, Peter

On 12/13/2016 08:47 PM, Chris Hegarty wrote:

Taking into account the feedback so far, and changing the method name ( since
it is an attractive nuisance ), here is where I think we ended up.

http://cr.openjdk.java.net/~chegar/Unsafe_invokeCleaner/

If this is agreeable, I’ll file an issue in JIRA to track the code changes, and
update JEP 260.

-Chris.

RE: Java 9 build 148 causes trouble in Apache Lucene/Solr/Elasticsearch

[Uwe Schindler]

Hi,

+1 to this approach.

I can create a PR for Apache Lucene to test this! With our current code it is 
very easy to add support for this - which is great (at the end I just need a 
MethodHandle with a MethodType "(ByteBuffer)void")!

Unfortunately I am not good in compiling OpenJDK, so if somebody could provide 
me a patched build for windows or linux, I'd be happy.

Uwe

-
Uwe Schindler
uschind...@apache.org 
ASF Member, Apache Lucene PMC / Committer
Bremen, Germany
http://lucene.apache.org/

> -Original Message-
> From: Chris Hegarty [mailto:chris.hega...@oracle.com]
> Sent: Tuesday, December 13, 2016 8:47 PM
> To: Core-Libs-Dev <core-libs-dev@openjdk.java.net>; jigsaw-dev  d...@openjdk.java.net>; Uwe Schindler <uschind...@apache.org>; Peter
> Levart <peter.lev...@gmail.com>
> Subject: Re: Java 9 build 148 causes trouble in Apache
> Lucene/Solr/Elasticsearch
> 
> Taking into account the feedback so far, and changing the method name (
> since
> it is an attractive nuisance ), here is where I think we ended up.
> 
> http://cr.openjdk.java.net/~chegar/Unsafe_invokeCleaner/
> 
> If this is agreeable, I’ll file an issue in JIRA to track the code changes, 
> and
> update JEP 260.
> 
> -Chris.=

Re: Java 9 build 148 causes trouble in Apache Lucene/Solr/Elasticsearch

[Chris Hegarty]

Taking into account the feedback so far, and changing the method name ( since 
it is an attractive nuisance ), here is where I think we ended up.

http://cr.openjdk.java.net/~chegar/Unsafe_invokeCleaner/

If this is agreeable, I’ll file an issue in JIRA to track the code changes, and 
update JEP 260.

-Chris.

Re: Java 9 build 148 causes trouble in Apache Lucene/Solr/Elasticsearch

[Eric Johnson]

On 12/9/16 3:21 PM, Uwe Schindler wrote:
> The -Dsun.reflect.debugModuleAccessChecks=true options help to debug, indeed, 
> but it does not solve the underlying issue. Apache Solr/Lucene and 
> Elasticsearch will no longer work with Java 9 unless you require users to add 
> those strange options. Elasticsearch already runs with a SecurityManager by 
> default, so the question is: why is this not handled by a security manager 
> and a new permission like "crossModuleAccess/module/package"? Why must it be 
> done on command line? This makes it impossible to ship something like Lucene 
> that it work out of box together with correct policy files?
I've asked the question multiple times on this list, about what the
threat model analysis is behind this new level of runtime "security".

Alas, no answers so far.

Eric.

Re: Java 9 build 148 causes trouble in Apache Lucene/Solr/Elasticsearch

[Carsten Varming]

Dear Alan,

On Sun, Dec 11, 2016 at 6:16 AM, Alan Bateman 
wrote:

>
> The alternative is of course:
>
> ByteBuffer wrap(long address, int capacity)
> void unmap(MappedByteBuffer)
>
> The wrap method allow be similar to JNI's NewDirectByteBuffer for those
> that are managing the underlying memory themselves. This makes it a more
> advanced method to avoid too much temptation to free the memory underlying
> a buffer created with ByteBuffer.allocateDirect. We can't do much with
> unmap but that at least won't be widely used.


I previously patched Netty to use the Runnable cleaner, so I have some
interesting in this discussion.

Having a public method "ByteBuffer wrap(long adddress, int capacity) in the
standard would simplify Netty code. Netty currently use the cleaner on
ByteBuffers allocated by ByteBuffer.allocateDirect, but I believe that can
be changed.

Carsten

Re: Java 9 build 148 causes trouble in Apache Lucene/Solr/Elasticsearch

[Peter Levart]

On 12/11/2016 09:59 PM, Peter Levart wrote:

The alternative is of course:

ByteBuffer wrap(long address, int capacity)
void unmap(MappedByteBuffer)

The wrap method allow be similar to JNI's NewDirectByteBuffer for 
those that are managing the underlying memory themselves. This makes 
it a more advanced method to avoid too much temptation to free the 
memory underlying a buffer created with ByteBuffer.allocateDirect. We 
can't do much with unmap but that at least won't be widely used.


-Alan.


So, If I understand correctly, you are proposing users to do their own 
memory allocation/management/deallocation with Unsafe and use wrap() 
just to create a view of the allocated memory in the form of 
ByteBuffer. Wouldn't this force them to use a different approach to 
managing ByteBuffer(s) from what they do now - they would have to keep 
the allocated memory's address somewhere outside the buffer in order 
to free the memory later... It is doable, but I think Uwe will not 
like it very much.


Sorry, I just realized that Uwe is interested only in the unmapping case...



About unmap()... Is it just the name and signature that would 
discourage freeing memory underlying a buffer created with 
ByteBuffer.allocateDirect() or do you propose to disallow such use and 
only allow actual unmapping?

Re: Java 9 build 148 causes trouble in Apache Lucene/Solr/Elasticsearch

[Peter Levart]

Hi Alan,


On 12/11/2016 12:16 PM, Alan Bateman wrote:



On 10/12/2016 17:11, Chris Hegarty wrote:

:

How about: Unsafe::deallocate(ByteBuffer directBuffer)?
   http://cr.openjdk.java.net/~chegar/Unsafe_deallocate/


The alternative is of course:

ByteBuffer wrap(long address, int capacity)
void unmap(MappedByteBuffer)

The wrap method allow be similar to JNI's NewDirectByteBuffer for 
those that are managing the underlying memory themselves. This makes 
it a more advanced method to avoid too much temptation to free the 
memory underlying a buffer created with ByteBuffer.allocateDirect. We 
can't do much with unmap but that at least won't be widely used.


-Alan.


So, If I understand correctly, you are proposing users to do their own 
memory allocation/management/deallocation with Unsafe and use wrap() 
just to create a view of the allocated memory in the form of ByteBuffer. 
Wouldn't this force them to use a different approach to managing 
ByteBuffer(s) from what they do now - they would have to keep the 
allocated memory's address somewhere outside the buffer in order to free 
the memory later... It is doable, but I think Uwe will not like it very 
much.


About unmap()... Is it just the name and signature that would discourage 
freeing memory underlying a buffer created with 
ByteBuffer.allocateDirect() or do you propose to disallow such use and 
only allow actual unmapping?


Regards, Peter

Re: Java 9 build 148 causes trouble in Apache Lucene/Solr/Elasticsearch

[Peter Levart]

Hi Uwe,


On 12/11/2016 01:57 PM, Uwe Schindler wrote:


Hi,

How about the following:

-Check that the buffer is direct, if not throw IAE(“not direct buffer”)

-Check that buffer has attachment==null (this tells you that it’s not 
a slice/dup), if not throw IAE(“not allowed to free duplicates/slices”)


-Finally do the standard if (cleaner!=null) cleaner.clean(), but don’t 
throw any exceptions if cleaner is null (as this is implementation detail)


This allows for empty buffers without cleaner that are still marked as 
direct. But it disallows all slices or duplicates.




Yes, this would be the right logic I agree. It would silently ignore the 
requests to free memory for buffers constructed via JNI's 
NewDirectByteBuffer calls, but I suppose this would not be a problem in 
practice.


I am fine with Alan’s proposal to restrict to MappedByteBuffer but 
that’s out of my interest – I am happy to unmap mapped byte buffers. I 
would also place the method in the legacy sun.misc.Unsafe only, the 
JDK-internal private one is not accessible to the outside. Of course 
for consistency it could be in both, but primarily it must be in 
sun.misc.Unsafe – that’s also what most code is using anyways.




Yes, internally, at least in java.base, the code can always directly 
invoke the DirectBuffer's and Cleaner's methods...


Regards, Peter



Uwe

*From:*Peter Levart [mailto:peter.lev...@gmail.com]
*Sent:* Saturday, December 10, 2016 10:23 PM
*To:* Uwe Schindler <uschind...@apache.org>; Chris Hegarty 
<chris.hega...@oracle.com>
*Cc:* jigsaw-...@openjdk.java.net; Core-Libs-Dev 
<core-libs-dev@openjdk.java.net>
*Subject:* Re: Java 9 build 148 causes trouble in Apache 
Lucene/Solr/Elasticsearch


On 12/10/2016 09:00 PM, Uwe Schindler wrote:

Hi,

We noticed that buffers with zero length also have no cleaner.
This is why we also have the null check in our code (see Github)
and the guardWithTest in the MethodHandle, although we never free
duplicates. So a noop is better imho.


Oh yes, good catch. Then what about being noop just for zero length? I 
don't know, maybe I'm just being paranoid and those who would use this 
API know perfectly well what they are doing. I'm just imagining a 
situation where one would create and keep just a duplicate of a direct 
buffer and afterwards use it to try to deallocate the native memory. 
This would be a noop, but the developer would think it works as GC 
would finally do it for him. I think it's better to throw an exception 
to prevent such situations...


Regards, Peter



I like the Unsafe approach. To me both variants are fine.

Uwe

Am 10. Dezember 2016 20:47:46 MEZ schrieb Peter Levart
<peter.lev...@gmail.com> <mailto:peter.lev...@gmail.com>:

Hi Chris,

On 12/10/2016 06:11 PM, Chris Hegarty wrote:

How about: Unsafe::deallocate(ByteBuffer directBuffer)?

   http://cr.openjdk.java.net/~chegar/Unsafe_deallocate/
<http://cr.openjdk.java.net/%7Echegar/Unsafe_deallocate/>


Apart from the fact that Unsafe is (was?) reserved for
low-level stuff, I think this approach is reasonable. Is the
method in jdk.internal.misc.Unsafe needed? You could add the
method just to the sun.misc.Unsafe (to keep internal Unsafe
free from hacks) and export the two packages selectively to
jdk.unsupported.


We could attempt to limit this to the direct buffer that "owns" the

memory, i.e. not a duplicate or a slice, but I'm not sure it is 
worth

it.


What you have here *is* limited to direct ByteBuffer(s) that
"own" the memory. Derived buffer(s) (duplicated or sliced) do
not have a Cleaner instance (they have an 'attachment' to keep
the 1st-level buffer reachable while they are reachable). I
would even make it more unforgiving by throwing an IAE if the
passed-in buffer didn't have a Cleaner. In addition I would
specify this behavior. For example:

"Deallocates the underlying memory associated with given
directBuffer if the buffer was obtained from either {@link
ByteBuffer#allocateDirect} or {@link FileChannel#map} methods.
In any other case (when the buffer is not a direct buffer or
was obtained by  {@link ByteBuffer#duplicate() duplicating} or
{@link ByteBuffer#slice(int, int) slicing} a direct buffer),
the method throws {@code IllegalArgumentException}.

Regards, Peter

Re: Java 9 build 148 causes trouble in Apache Lucene/Solr/Elasticsearch

[Peter Levart]

Hi Chris,


On 12/11/2016 10:26 AM, Chris Hegarty wrote:

>"Deallocates the underlying memory associated with given directBuffer if the 
buffer was obtained from either {@link ByteBuffer#allocateDirect} or {@link 
FileChannel#map} methods. In any other case (when the buffer is not a direct buffer or 
was obtained by  {@link ByteBuffer#duplicate() duplicating} or {@link 
ByteBuffer#slice(int, int) slicing} a direct buffer), the method throws {@code 
IllegalArgumentException}.

Yes, but given a ByteBuffer it is not possible to determine if it “owns” the
memory, or not. So users of the API would have to have full knowledge of
the buffers they pass to it. Maybe this is ok?

-Chris.


In order for deallocation to be effective and, above all, safe, user has 
to know the whole story of a buffer (s)he intends to deallocate and the 
story of all possible derived buffers. I don't believe one can create a 
safe program by choosing to deallocate a direct buffer for which (s)he 
does not know where it came from, because then (s)he also doesn't know 
what other buffers might still be using the same piece of memory.


Regards, Peter

RE: Java 9 build 148 causes trouble in Apache Lucene/Solr/Elasticsearch

[Uwe Schindler]

Hi,

How about the following:

 

-  Check that the buffer is direct, if not throw IAE(“not direct 
buffer”)

-  Check that buffer has attachment==null (this tells you that it’s not 
a slice/dup), if not throw IAE(“not allowed to free duplicates/slices”)

-  Finally do the standard if (cleaner!=null) cleaner.clean(), but 
don’t throw any exceptions if cleaner is null (as this is implementation detail)

 

This allows for empty buffers without cleaner that are still marked as direct. 
But it disallows all slices or duplicates. 

 

I am fine with Alan’s proposal to restrict to MappedByteBuffer but that’s out 
of my interest – I am happy to unmap mapped byte buffers. I would also place 
the method in the legacy sun.misc.Unsafe only, the JDK-internal private one is 
not accessible to the outside. Of course for consistency it could be in both, 
but primarily it must be in sun.misc.Unsafe – that’s also what most code is 
using anyways.

 

Uwe

 

 

From: Peter Levart [mailto:peter.lev...@gmail.com] 
Sent: Saturday, December 10, 2016 10:23 PM
To: Uwe Schindler <uschind...@apache.org>; Chris Hegarty 
<chris.hega...@oracle.com>
Cc: jigsaw-...@openjdk.java.net; Core-Libs-Dev <core-libs-dev@openjdk.java.net>
Subject: Re: Java 9 build 148 causes trouble in Apache Lucene/Solr/Elasticsearch

 

 

 

On 12/10/2016 09:00 PM, Uwe Schindler wrote:

Hi,

We noticed that buffers with zero length also have no cleaner. This is why we 
also have the null check in our code (see Github) and the guardWithTest in the 
MethodHandle, although we never free duplicates. So a noop is better imho.


Oh yes, good catch. Then what about being noop just for zero length? I don't 
know, maybe I'm just being paranoid and those who would use this API know 
perfectly well what they are doing. I'm just imagining a situation where one 
would create and keep just a duplicate of a direct buffer and afterwards use it 
to try to deallocate the native memory. This would be a noop, but the developer 
would think it works as GC would finally do it for him. I think it's better to 
throw an exception to prevent such situations...

Regards, Peter





I like the Unsafe approach. To me both variants are fine.

Uwe

Am 10. Dezember 2016 20:47:46 MEZ schrieb Peter Levart  
<mailto:peter.lev...@gmail.com> <peter.lev...@gmail.com>: 

Hi Chris,

 

On 12/10/2016 06:11 PM, Chris Hegarty wrote:

How about: Unsafe::deallocate(ByteBuffer directBuffer)?
  http://cr.openjdk.java.net/~chegar/Unsafe_deallocate/ 
<http://cr.openjdk.java.net/%7Echegar/Unsafe_deallocate/> 


Apart from the fact that Unsafe is (was?) reserved for low-level stuff, I think 
this approach is reasonable. Is the method in jdk.internal.misc.Unsafe needed? 
You could add the method just to the sun.misc.Unsafe (to keep internal Unsafe 
free from hacks) and export the two packages selectively to jdk.unsupported.
 



We could attempt to limit this to the direct buffer that "owns" the
memory, i.e. not a duplicate or a slice, but I'm not sure it is worth
it.


What you have here *is* limited to direct ByteBuffer(s) that "own" the memory. 
Derived buffer(s) (duplicated or sliced) do not have a Cleaner instance (they 
have an 'attachment' to keep the 1st-level buffer reachable while they are 
reachable). I would even make it more unforgiving by throwing an IAE if the 
passed-in buffer didn't have a Cleaner. In addition I would specify this 
behavior. For example:

"Deallocates the underlying memory associated with given directBuffer if the 
buffer was obtained from either {@link ByteBuffer#allocateDirect} or {@link 
FileChannel#map} methods. In any other case (when the buffer is not a direct 
buffer or was obtained by  {@link ByteBuffer#duplicate() duplicating} or {@link 
ByteBuffer#slice(int, int) slicing} a direct buffer), the method throws {@code 
IllegalArgumentException}.

Regards, Peter

 

Re: Java 9 build 148 causes trouble in Apache Lucene/Solr/Elasticsearch

[Alan Bateman]

On 10/12/2016 17:11, Chris Hegarty wrote:

:

How about: Unsafe::deallocate(ByteBuffer directBuffer)?
   http://cr.openjdk.java.net/~chegar/Unsafe_deallocate/


The alternative is of course:

ByteBuffer wrap(long address, int capacity)
void unmap(MappedByteBuffer)

The wrap method allow be similar to JNI's NewDirectByteBuffer for those 
that are managing the underlying memory themselves. This makes it a more 
advanced method to avoid too much temptation to free the memory 
underlying a buffer created with ByteBuffer.allocateDirect. We can't do 
much with unmap but that at least won't be widely used.


-Alan.

Re: Java 9 build 148 causes trouble in Apache Lucene/Solr/Elasticsearch

[Chris Hegarty]

> On 10 Dec 2016, at 19:47, Peter Levart  wrote:
> 
> Hi Chris,
> 
> On 12/10/2016 06:11 PM, Chris Hegarty wrote:
>> How about: Unsafe::deallocate(ByteBuffer directBuffer)?
>>   
>> http://cr.openjdk.java.net/~chegar/Unsafe_deallocate/
> 
> Apart from the fact that Unsafe is (was?) reserved for low-level stuff, I 
> think this approach is reasonable. Is the method in jdk.internal.misc.Unsafe 
> needed? You could add the method just to the sun.misc.Unsafe (to keep 
> internal Unsafe free from hacks) and export the two packages selectively to 
> jdk.unsupported.

Yes, possibly.

>> We could attempt to limit this to the direct buffer that "owns" the
>> memory, i.e. not a duplicate or a slice, but I'm not sure it is worth
>> it.
>> 
> 
> What you have here *is* limited to direct ByteBuffer(s) that "own" the memory.

Understood, what I meant was throwing an exception if the given buffer
does not “own” the memory.

> Derived buffer(s) (duplicated or sliced) do not have a Cleaner instance (they 
> have an 'attachment' to keep the 1st-level buffer reachable while they are 
> reachable). I would even make it more unforgiving by throwing an IAE if the 
> passed-in buffer didn't have a Cleaner. In addition I would specify this 
> behavior. For example:
> 
> "Deallocates the underlying memory associated with given directBuffer if the 
> buffer was obtained from either {@link ByteBuffer#allocateDirect} or {@link 
> FileChannel#map} methods. In any other case (when the buffer is not a direct 
> buffer or was obtained by  {@link ByteBuffer#duplicate() duplicating} or 
> {@link ByteBuffer#slice(int, int) slicing} a direct buffer), the method 
> throws {@code IllegalArgumentException}.

Yes, but given a ByteBuffer it is not possible to determine if it “owns” the
memory, or not. So users of the API would have to have full knowledge of
the buffers they pass to it. Maybe this is ok?

-Chris.

Re: Java 9 build 148 causes trouble in Apache Lucene/Solr/Elasticsearch

[Peter Levart]

On 12/10/2016 09:00 PM, Uwe Schindler wrote:

Hi,

We noticed that buffers with zero length also have no cleaner. This is 
why we also have the null check in our code (see Github) and the 
guardWithTest in the MethodHandle, although we never free duplicates. 
So a noop is better imho.


Oh yes, good catch. Then what about being noop just for zero length? I 
don't know, maybe I'm just being paranoid and those who would use this 
API know perfectly well what they are doing. I'm just imagining a 
situation where one would create and keep just a duplicate of a direct 
buffer and afterwards use it to try to deallocate the native memory. 
This would be a noop, but the developer would think it works as GC would 
finally do it for him. I think it's better to throw an exception to 
prevent such situations...


Regards, Peter



I like the Unsafe approach. To me both variants are fine.

Uwe

Am 10. Dezember 2016 20:47:46 MEZ schrieb Peter Levart 
:


Hi Chris,


On 12/10/2016 06:11 PM, Chris Hegarty wrote:

How about: Unsafe::deallocate(ByteBuffer directBuffer)?
   http://cr.openjdk.java.net/~chegar/Unsafe_deallocate/


Apart from the fact that Unsafe is (was?) reserved for low-level
stuff, I think this approach is reasonable. Is the method in
jdk.internal.misc.Unsafe needed? You could add the method just to
the sun.misc.Unsafe (to keep internal Unsafe free from hacks) and
export the two packages selectively to jdk.unsupported.


We could attempt to limit this to the direct buffer that "owns" the
memory, i.e. not a duplicate or a slice, but I'm not sure it is worth
it.


What you have here *is* limited to direct ByteBuffer(s) that "own"
the memory. Derived buffer(s) (duplicated or sliced) do not have a
Cleaner instance (they have an 'attachment' to keep the 1st-level
buffer reachable while they are reachable). I would even make it
more unforgiving by throwing an IAE if the passed-in buffer didn't
have a Cleaner. In addition I would specify this behavior. For
example:

"Deallocates the underlying memory associated with given
directBuffer if the buffer was obtained from either {@link
ByteBuffer#allocateDirect} or {@link FileChannel#map} methods. In
any other case (when the buffer is not a direct buffer or was
obtained by  {@link ByteBuffer#duplicate() duplicating} or {@link
ByteBuffer#slice(int, int) slicing} a direct buffer), the method
throws {@code IllegalArgumentException}.

Regards, Peter

Re: Java 9 build 148 causes trouble in Apache Lucene/Solr/Elasticsearch

[Uwe Schindler]

Hi,

We noticed that buffers with zero length also have no cleaner. This is why we 
also have the null check in our code (see Github) and the guardWithTest in the 
MethodHandle, although we never free duplicates. So a noop is better imho.

I like the Unsafe approach. To me both variants are fine.

Uwe

Am 10. Dezember 2016 20:47:46 MEZ schrieb Peter Levart :
>Hi Chris,
>
>
>On 12/10/2016 06:11 PM, Chris Hegarty wrote:
>> How about: Unsafe::deallocate(ByteBuffer directBuffer)?
>>http://cr.openjdk.java.net/~chegar/Unsafe_deallocate/
>
>Apart from the fact that Unsafe is (was?) reserved for low-level stuff,
>
>I think this approach is reasonable. Is the method in 
>jdk.internal.misc.Unsafe needed? You could add the method just to the 
>sun.misc.Unsafe (to keep internal Unsafe free from hacks) and export
>the 
>two packages selectively to jdk.unsupported.
>
>> We could attempt to limit this to the direct buffer that "owns" the
>> memory, i.e. not a duplicate or a slice, but I'm not sure it is worth
>> it.
>
>What you have here *is* limited to direct ByteBuffer(s) that "own" the 
>memory. Derived buffer(s) (duplicated or sliced) do not have a Cleaner 
>instance (they have an 'attachment' to keep the 1st-level buffer 
>reachable while they are reachable). I would even make it more 
>unforgiving by throwing an IAE if the passed-in buffer didn't have a 
>Cleaner. In addition I would specify this behavior. For example:
>
>"Deallocates the underlying memory associated with given directBuffer
>if 
>the buffer was obtained from either {@link ByteBuffer#allocateDirect}
>or 
>{@link FileChannel#map} methods. In any other case (when the buffer is 
>not a direct buffer or was obtained by  {@link ByteBuffer#duplicate() 
>duplicating} or {@link ByteBuffer#slice(int, int) slicing} a direct 
>buffer), the method throws {@code IllegalArgumentException}.
>
>Regards, Peter

Re: Java 9 build 148 causes trouble in Apache Lucene/Solr/Elasticsearch

[Peter Levart]

Hi Chris,


On 12/10/2016 06:11 PM, Chris Hegarty wrote:

How about: Unsafe::deallocate(ByteBuffer directBuffer)?
   http://cr.openjdk.java.net/~chegar/Unsafe_deallocate/


Apart from the fact that Unsafe is (was?) reserved for low-level stuff, 
I think this approach is reasonable. Is the method in 
jdk.internal.misc.Unsafe needed? You could add the method just to the 
sun.misc.Unsafe (to keep internal Unsafe free from hacks) and export the 
two packages selectively to jdk.unsupported.



We could attempt to limit this to the direct buffer that "owns" the
memory, i.e. not a duplicate or a slice, but I'm not sure it is worth
it.


What you have here *is* limited to direct ByteBuffer(s) that "own" the 
memory. Derived buffer(s) (duplicated or sliced) do not have a Cleaner 
instance (they have an 'attachment' to keep the 1st-level buffer 
reachable while they are reachable). I would even make it more 
unforgiving by throwing an IAE if the passed-in buffer didn't have a 
Cleaner. In addition I would specify this behavior. For example:


"Deallocates the underlying memory associated with given directBuffer if 
the buffer was obtained from either {@link ByteBuffer#allocateDirect} or 
{@link FileChannel#map} methods. In any other case (when the buffer is 
not a direct buffer or was obtained by  {@link ByteBuffer#duplicate() 
duplicating} or {@link ByteBuffer#slice(int, int) slicing} a direct 
buffer), the method throws {@code IllegalArgumentException}.


Regards, Peter

Re: Java 9 build 148 causes trouble in Apache Lucene/Solr/Elasticsearch

[Chris Hegarty]

I think it best to keep whatever we do here simple and straight forward.
It is, after all, just a stop gap until a public API can be put in place
in a future release ( which I believe is a realistic possibility
given some of the discussions that I have heard about, but that is for
another day).

If we add a method to Unsafe, then we get the benefit of being 
protected by the security manager for free ( you need reflective access
to get "theUnsafe" field). Most of the code I've seen in this area delves
into Unsafe anyway.

How about: Unsafe::deallocate(ByteBuffer directBuffer)?
  http://cr.openjdk.java.net/~chegar/Unsafe_deallocate/

We could attempt to limit this to the direct buffer that "owns" the
memory, i.e. not a duplicate or a slice, but I'm not sure it is worth
it.

Once we reach agreement on the technical solution, I will add appropriate 
wording to JEP 260 to cover this case.

-Chris.



> On 10 Dec 2016, at 13:08, Peter Levart <peter.lev...@gmail.com> wrote:
> 
> Hi Uwe,
> 
> 
> On 12/10/2016 01:33 PM, Uwe Schindler wrote:
>> 
>> Hi Peter,
>> 
>> this would be a great fix! Thanks!!!
>> 
>> I also think the non-static method is superior to my original proposal, 
>> because it allows us to do the security check **once**, which is really 
>> needed for Lucene. I am still fine if the permission is still checked on 
>> every unmapping, but we need to do the check up-front. If you look at our 
>> current unmapping code (https://goo.gl/TfQWl6), you will the that the 
>> detector checks for the extra runtime permission upfront, so we can be sure 
>> that the actual unmapping will work for sure. This is also the reason why we 
>> use MethodHandles: As those are compiled on investigation of possible 
>> unmapping variants depending on the VM, we can “compile” the MethodHandle 
>> and later call it as often as we like, without the risk that it breaks for 
>> incompatibility reasons. The MethodHandle makes sure that all types are 
>> checked up front.
>> 
>> About MappedByteBuffer vs ByteBuffer (or maybe just java.nio.Buffer!?): I’d 
>> make it generic so it works with any direct buffer (maybe also non-byte 
>> ones). For Lucene it does not matter, but other projects (I know Cassandra 
>> or other off-Heap frameworks) do the same with buffers that were allocated 
>> direct (not only mmapped). The method signature in your proposal is also 
>> compatible to our requirements: We can create the DirectBufferDeallocator up 
>> front and then produce a MH which is bound to the allocator.
>> 
> 
> I choose to limit the method to ByteBuffer type because this is the public 
> static type used in programs for instances that are possibly "owning" the 
> underlying native memory. Other-typed buffers or even 2nd-level direct 
> ByteBuffers obtained by duplicating or slicing are just views and do not 
> "own" the underlying memory. While it would be possible to trigger 
> deallocation / unmapping via any buffer that references the owning buffer, I 
> think this might be prone to bugs. By limiting the method to 1st-level direct 
> ByteBuffer(s), the programmer is forced to think about ownership and lifetime 
> of derived buffers and consequently write better code.
> 
> So on 2nd thought, the API might be even better to reject non-direct and 
> 2nd-level direct ByteBuffer(s) by throwing an exception rather than silently 
> ignoring the deallocation request.
> 
>> I will make a pull request to Lucene using your current proposal so you have 
>> a “patch” to test this with Lucene before you commit something like this.
>> 
> 
> Let us first wait for a proposal from Oracle to see what they have in mind...
> 
> Regards, Peter
> 
>> Uwe
>> 
>> -
>> 
>> Uwe Schindler
>> 
>> uschind...@apache.org
>> 
>> ASF Member, Apache Lucene PMC / Committer
>> 
>> Bremen, Germany
>> 
>> http://lucene.apache.org/
>> 
>> *From:*Peter Levart [mailto:peter.lev...@gmail.com]
>> *Sent:* Saturday, December 10, 2016 12:10 PM
>> *To:* Alan Bateman <alan.bate...@oracle.com>; Uwe Schindler 
>> <uschind...@apache.org>; jigsaw-...@openjdk.java.net; Core-Libs-Dev 
>> <core-libs-dev@openjdk.java.net>
>> *Subject:* Re: Java 9 build 148 causes trouble in Apache 
>> Lucene/Solr/Elasticsearch
>> 
>> Hi,
>> 
>> On 12/10/2016 06:14 AM, Alan Bateman wrote:
>> 
>>On 09/12/2016 22:32, Uwe Schindler wrote:
>> 
>> 
>>Hi,
>> 
>>I updated our Jenkins server for the JDK 9 preview testing to
>>use build 148. Previously we had build 140

Re: Java 9 build 148 causes trouble in Apache Lucene/Solr/Elasticsearch

[Peter Levart]

Hi Uwe,


On 12/10/2016 01:33 PM, Uwe Schindler wrote:


Hi Peter,

this would be a great fix! Thanks!!!

I also think the non-static method is superior to my original 
proposal, because it allows us to do the security check **once**, 
which is really needed for Lucene. I am still fine if the permission 
is still checked on every unmapping, but we need to do the check 
up-front. If you look at our current unmapping code 
(https://goo.gl/TfQWl6), you will the that the detector checks for the 
extra runtime permission upfront, so we can be sure that the actual 
unmapping will work for sure. This is also the reason why we use 
MethodHandles: As those are compiled on investigation of possible 
unmapping variants depending on the VM, we can “compile” the 
MethodHandle and later call it as often as we like, without the risk 
that it breaks for incompatibility reasons. The MethodHandle makes 
sure that all types are checked up front.


About MappedByteBuffer vs ByteBuffer (or maybe just 
java.nio.Buffer!?): I’d make it generic so it works with any direct 
buffer (maybe also non-byte ones). For Lucene it does not matter, but 
other projects (I know Cassandra or other off-Heap frameworks) do the 
same with buffers that were allocated direct (not only mmapped). The 
method signature in your proposal is also compatible to our 
requirements: We can create the DirectBufferDeallocator up front and 
then produce a MH which is bound to the allocator.




I choose to limit the method to ByteBuffer type because this is the 
public static type used in programs for instances that are possibly 
"owning" the underlying native memory. Other-typed buffers or even 
2nd-level direct ByteBuffers obtained by duplicating or slicing are just 
views and do not "own" the underlying memory. While it would be possible 
to trigger deallocation / unmapping via any buffer that references the 
owning buffer, I think this might be prone to bugs. By limiting the 
method to 1st-level direct ByteBuffer(s), the programmer is forced to 
think about ownership and lifetime of derived buffers and consequently 
write better code.


So on 2nd thought, the API might be even better to reject non-direct and 
2nd-level direct ByteBuffer(s) by throwing an exception rather than 
silently ignoring the deallocation request.


I will make a pull request to Lucene using your current proposal so 
you have a “patch” to test this with Lucene before you commit 
something like this.




Let us first wait for a proposal from Oracle to see what they have in 
mind...


Regards, Peter


Uwe

-

Uwe Schindler

uschind...@apache.org

ASF Member, Apache Lucene PMC / Committer

Bremen, Germany

http://lucene.apache.org/

*From:*Peter Levart [mailto:peter.lev...@gmail.com]
*Sent:* Saturday, December 10, 2016 12:10 PM
*To:* Alan Bateman <alan.bate...@oracle.com>; Uwe Schindler 
<uschind...@apache.org>; jigsaw-...@openjdk.java.net; Core-Libs-Dev 
<core-libs-dev@openjdk.java.net>
*Subject:* Re: Java 9 build 148 causes trouble in Apache 
Lucene/Solr/Elasticsearch


Hi,

On 12/10/2016 06:14 AM, Alan Bateman wrote:

On 09/12/2016 22:32, Uwe Schindler wrote:


Hi,

I updated our Jenkins server for the JDK 9 preview testing to
use build 148. Previously we had build 140 and build 147,
which both worked without any issues. But after the update the
following stuff goes wrong:

(1) Unmapping of direct buffers no longer works, although this
API was marked as critical because there is no replacement up
to now, so code can unmap memory mapped files, which is one of
the most important things Apache Lucene needs to use to access
huge random access files while reading the index. Without
memory mapping, the slowdown for Lucene users will be huge

sun.misc.Cleaner was indeed on the original list of APIs for JEP
260 to identify as a "critical internal API". It turned out not to
be useful because it would have required some way to get the
Cleaner in the first place. That lead to the "new" hack that is
reading the private "cleaner" field from DBB and treating it as a
Runnable. That hack now breaks because setAccessible has changed
in jdk-9+148 to align with the JSR 376 proposal tracked as
#AwkwardStrongEncapsulation.

No need to panic though, there is an update to JEP 260 coming soon
for this specific need. Details TDB but it will probably be a
method in jdk.unsupported module. It does mean that libraries
using the old (or "new") hacks will need to change. I hope it will
be seen as a reasonable compromise for this generally awkward issue.

-Alan


Something like the following?

http://cr.openjdk.java.net/~plevart/jdk9-dev/DirectBufferDeallocator/webrev.01/ 
<http://cr.openjdk.java.net/%7Eplevart/jdk9-dev/DirectBufferDeallocator/webrev.01/>



Regards, Peter

RE: Java 9 build 148 causes trouble in Apache Lucene/Solr/Elasticsearch

[Uwe Schindler]

Hi Peter,

 

this would be a great fix! Thanks!!!

 

I also think the non-static method is superior to my original proposal, because 
it allows us to do the security check *once*, which is really needed for 
Lucene. I am still fine if the permission is still checked on every unmapping, 
but we need to do the check up-front. If you look at our current unmapping code 
(https://goo.gl/TfQWl6), you will the that the detector checks for the extra 
runtime permission upfront, so we can be sure that the actual unmapping will 
work for sure. This is also the reason why we use MethodHandles: As those are 
compiled on investigation of possible unmapping variants depending on the VM, 
we can “compile” the MethodHandle and later call it as often as we like, 
without the risk that it breaks for incompatibility reasons. The MethodHandle 
makes sure that all types are checked up front.

 

About MappedByteBuffer vs ByteBuffer (or maybe just java.nio.Buffer!?): I’d 
make it generic so it works with any direct buffer (maybe also non-byte ones). 
For Lucene it does not matter, but other projects (I know Cassandra or other 
off-Heap frameworks) do the same with buffers that were allocated direct (not 
only mmapped). The method signature in your proposal is also compatible to our 
requirements: We can create the DirectBufferDeallocator up front and then 
produce a MH which is bound to the allocator.

 

I will make a pull request to Lucene using your current proposal so you have a 
“patch” to test this with Lucene before you commit something like this.

 

Uwe

 

-

Uwe Schindler

uschind...@apache.org 

ASF Member, Apache Lucene PMC / Committer

Bremen, Germany

http://lucene.apache.org/

 

From: Peter Levart [mailto:peter.lev...@gmail.com] 
Sent: Saturday, December 10, 2016 12:10 PM
To: Alan Bateman <alan.bate...@oracle.com>; Uwe Schindler 
<uschind...@apache.org>; jigsaw-...@openjdk.java.net; Core-Libs-Dev 
<core-libs-dev@openjdk.java.net>
Subject: Re: Java 9 build 148 causes trouble in Apache Lucene/Solr/Elasticsearch

 

Hi,

 

On 12/10/2016 06:14 AM, Alan Bateman wrote:

On 09/12/2016 22:32, Uwe Schindler wrote: 




Hi, 

I updated our Jenkins server for the JDK 9 preview testing to use build 148. 
Previously we had build 140 and build 147, which both worked without any 
issues. But after the update the following stuff goes wrong: 

(1) Unmapping of direct buffers no longer works, although this API was marked 
as critical because there is no replacement up to now, so code can unmap memory 
mapped files, which is one of the most important things Apache Lucene needs to 
use to access huge random access files while reading the index. Without memory 
mapping, the slowdown for Lucene users will be huge 

sun.misc.Cleaner was indeed on the original list of APIs for JEP 260 to 
identify as a "critical internal API". It turned out not to be useful because 
it would have required some way to get the Cleaner in the first place. That 
lead to the "new" hack that is reading the private "cleaner" field from DBB and 
treating it as a Runnable. That hack now breaks because setAccessible has 
changed in jdk-9+148 to align with the JSR 376 proposal tracked as 
#AwkwardStrongEncapsulation. 

No need to panic though, there is an update to JEP 260 coming soon for this 
specific need. Details TDB but it will probably be a method in jdk.unsupported 
module. It does mean that libraries using the old (or "new") hacks will need to 
change. I hope it will be seen as a reasonable compromise for this generally 
awkward issue. 

-Alan 


Something like the following?

 
<http://cr.openjdk.java.net/~plevart/jdk9-dev/DirectBufferDeallocator/webrev.01/>
 http://cr.openjdk.java.net/~plevart/jdk9-dev/DirectBufferDeallocator/webrev.01/


Regards, Peter

Re: Java 9 build 148 causes trouble in Apache Lucene/Solr/Elasticsearch

[Alan Bateman]

On 10/12/2016 08:22, Jochen Theodorou wrote:


:

A fix for this is already committed, we are only waiting for release 
of Groovy 2.4.8. Of course even with the fix Groovy code can possibly 
break... for example if you did the direct buffer access in Groovy.

Thanks for sharing, that is very useful to know.

-Alan

Re: Java 9 build 148 causes trouble in Apache Lucene/Solr/Elasticsearch

[Alan Bateman]

On 10/12/2016 11:09, Peter Levart wrote:


:

Something like the following?

http://cr.openjdk.java.net/~plevart/jdk9-dev/DirectBufferDeallocator/webrev.01/
Sort of although I think the proposal will be more specific, as in 
unmap(MappedByteBuffer) on an existing class.


The other update to JEP 260 that is needed is to mention the additional 
methods in sun.reflect.RefectionFactory that are needed by custom 
serialization libraries to work with strong encapsulation.


-Alan

RE: Java 9 build 148 causes trouble in Apache Lucene/Solr/Elasticsearch

[Uwe Schindler]

Thanks Alan,

I will look forward to see a solution for this! THANKS! A static method 
somewhere in jdk.internal to trigger the unmapping of a MappedByteBuffer would 
be fine. This is easy to change in our code, we just need a MethodHandle to a 
function with the following signature: static void unmap(MappedByteBuffer)

Our current code just generates the MethodHandle with exactly that signature 
and functionality by some guardWithTest, filterArguments,... and using the 
Cleaner as Java 8 internal class or as Runnable in Java 9: https://goo.gl/TfQWl6

The second thing was how to make a JEP proposal for solving the underlying 
problem in Java 10? As said before, on last FOSDEM we had some ideas how to 
make Hotspot able to unmap without the risk that the JVM SIGSEGV/SIGBUS or 
exposes private data. This would need some Hotspot changes ("volatile only 
during safepoints"), the idea was proposed by Andrew Haley.

Uwe

-
Uwe Schindler
uschind...@apache.org 
ASF Member, Apache Lucene PMC / Committer
Bremen, Germany
http://lucene.apache.org/

> -Original Message-
> From: Alan Bateman [mailto:alan.bate...@oracle.com]
> Sent: Saturday, December 10, 2016 6:14 AM
> To: Uwe Schindler <uschind...@apache.org>; jigsaw-...@openjdk.java.net;
> Core-Libs-Dev <core-libs-dev@openjdk.java.net>
> Subject: Re: Java 9 build 148 causes trouble in Apache
> Lucene/Solr/Elasticsearch
> 
> On 09/12/2016 22:32, Uwe Schindler wrote:
> 
> > Hi,
> >
> > I updated our Jenkins server for the JDK 9 preview testing to use build 148.
> Previously we had build 140 and build 147, which both worked without any
> issues. But after the update the following stuff goes wrong:
> >
> > (1) Unmapping of direct buffers no longer works, although this API was
> marked as critical because there is no replacement up to now, so code can
> unmap memory mapped files, which is one of the most important things
> Apache Lucene needs to use to access huge random access files while
> reading the index. Without memory mapping, the slowdown for Lucene
> users will be huge
> sun.misc.Cleaner was indeed on the original list of APIs for JEP 260 to
> identify as a "critical internal API". It turned out not to be useful
> because it would have required some way to get the Cleaner in the first
> place. That lead to the "new" hack that is reading the private "cleaner"
> field from DBB and treating it as a Runnable. That hack now breaks
> because setAccessible has changed in jdk-9+148 to align with the JSR 376
> proposal tracked as #AwkwardStrongEncapsulation.
> 
> No need to panic though, there is an update to JEP 260 coming soon for
> this specific need. Details TDB but it will probably be a method in
> jdk.unsupported module. It does mean that libraries using the old (or
> "new") hacks will need to change. I hope it will be seen as a reasonable
> compromise for this generally awkward issue.
> 
> -Alan

Re: Java 9 build 148 causes trouble in Apache Lucene/Solr/Elasticsearch

[Jochen Theodorou]

On 09.12.2016 23:32, Uwe Schindler wrote:

Hi,

I updated our Jenkins server for the JDK 9 preview testing to use build 148. 
Previously we had build 140 and build 147, which both worked without any 
issues. But after the update the following stuff goes wrong:

(1) Unmapping of direct buffers no longer works, although this API was marked 
as critical because there is no replacement up to now, so code can unmap memory 
mapped files, which is one of the most important things Apache Lucene needs to 
use to access huge random access files while reading the index. Without memory 
mapping, the slowdown for Lucene users will be huge

This is caused by the recent Jigsaw changes, published in build 148. 
Unfortunately we did not test the Jigsaw builds, so we would have noticed that 
earlier. Basically the following peace of code fails now (with or without 
doPrivileged and with/without security manager):

   final Class directBufferClass = 
Class.forName("java.nio.DirectByteBuffer");

   final Method m = directBufferClass.getMethod("cleaner");
   m.setAccessible(true);
   MethodHandle directBufferCleanerMethod = lookup.unreflect(m);
   Class cleanerClass = directBufferCleanerMethod.type().returnType();
   // build method handle for unmapping, full code is here: 
https://goo.gl/TfQWl6


I guess that is the effect of #AwkwardStrongEncapsulation. I would 
advise doing regular checks against the jigsaw builds to know about such 
problems in the future earlier... but seeing your code break without an 
obvious good solution sure is stressful. I feel with you.


[...]

(2) A second thing  we noticed is that Groovy no longer works and dies with 
strange error messages.


That is because versions including Groovy 2.4.7 are using
setAccessible(AccessibleObject[] array, true), and the array will also 
include private methods or fields. This worked till 
#AwkwardStrongEncapsulation because will then a class was either 
exported and its method can all be made accessible or not. For example 
on GAE or earlier versions of the module system. Now an exported class 
may break this, since its private methods can no longer be made 
accessible using setAccessible.


A fix for this is already committed, we are only waiting for release of 
Groovy 2.4.8. Of course even with the fix Groovy code can possibly 
break... for example if you did the direct buffer access in Groovy.


Btw, do not hesitate to ask about such problems on groovy-user, please.

bye Jochen

Re: Java 9 build 148 causes trouble in Apache Lucene/Solr/Elasticsearch

[Alan Bateman]

On 09/12/2016 23:06, Stephen Felts wrote:


I would highly recommend running with 
_JAVA_OPTIONS=-Dsun.reflect.debugModuleAccessChecks=true
It will tell you what add-options are required.
One minor downside is that it will produce the warning in cases where the 
software is already correctly handling the exception from setAccessible, so 
there can be noise.
Yes, this is a useful property to debug cases where setAccessible fails 
because the package is not exported or opened to the caller. There is 
more on this in JEP 261.


-Alan.

Re: Java 9 build 148 causes trouble in Apache Lucene/Solr/Elasticsearch

[Alan Bateman]

On 09/12/2016 22:32, Uwe Schindler wrote:


Hi,

I updated our Jenkins server for the JDK 9 preview testing to use build 148. 
Previously we had build 140 and build 147, which both worked without any 
issues. But after the update the following stuff goes wrong:

(1) Unmapping of direct buffers no longer works, although this API was marked 
as critical because there is no replacement up to now, so code can unmap memory 
mapped files, which is one of the most important things Apache Lucene needs to 
use to access huge random access files while reading the index. Without memory 
mapping, the slowdown for Lucene users will be huge
sun.misc.Cleaner was indeed on the original list of APIs for JEP 260 to 
identify as a "critical internal API". It turned out not to be useful 
because it would have required some way to get the Cleaner in the first 
place. That lead to the "new" hack that is reading the private "cleaner" 
field from DBB and treating it as a Runnable. That hack now breaks 
because setAccessible has changed in jdk-9+148 to align with the JSR 376 
proposal tracked as #AwkwardStrongEncapsulation.


No need to panic though, there is an update to JEP 260 coming soon for 
this specific need. Details TDB but it will probably be a method in 
jdk.unsupported module. It does mean that libraries using the old (or 
"new") hacks will need to change. I hope it will be seen as a reasonable 
compromise for this generally awkward issue.


-Alan

RE: Java 9 build 148 causes trouble in Apache Lucene/Solr/Elasticsearch

[Stephen Felts]

Unsafe is visible in JDK9.  See the list at http://openjdk.java.net/jeps/260

I agree that requiring command line options is a problem.
The experts don't want to merge module access into the security manager.

The link above says "Suggested additions to this list, justified by real-world 
use cases and estimates of developer and end-user impact, are welcome".  So you 
should make clear exactly API's that you want exposed.


-Original Message-
From: Uwe Schindler [mailto:uschind...@apache.org] 
Sent: Friday, December 09, 2016 6:22 PM
To: Stephen Felts; jigsaw-...@openjdk.java.net; Core-Libs-Dev
Subject: RE: Java 9 build 148 causes trouble in Apache Lucene/Solr/Elasticsearch

Hi,

Thanks for the hints to fix Groovy, although this is hard to do with ANT (which 
is our build system).

The -Dsun.reflect.debugModuleAccessChecks=true options help to debug, indeed, 
but it does not solve the underlying issue. Apache Solr/Lucene and 
Elasticsearch will no longer work with Java 9 unless you require users to add 
those strange options. Elasticsearch already runs with a SecurityManager by 
default, so the question is: why is this not handled by a security manager and 
a new permission like "crossModuleAccess/module/package"? Why must it be done 
on command line? This makes it impossible to ship something like Lucene that it 
work out of box together with correct policy files?

And as said in my previous mail: The direct bytebuffer unmapping has still no 
"official" way to do it, but it is critical to large scale database systems 
like Lucene/Solr/Elasticsearch. You have replacements in Java 9 for Unsafe 
(VarHandles,...), but still no way to allow unmapping of byte buffers that sit 
on huge resources or disallow deleting of files on windows. It was discussed on 
last FOSDEM to do something in Java 10 (I would like to get information how to 
propose the required change as Java 10 dev started now!), and in the meantime 
it was confirmed that some APIs in the JDK are "critical" and will be 
supported. But this is now 

So please re-add the special critical APIs back to the whitelist, so code like 
getting (legacy) Unsafe or unmapping direct buffers works without command line 
parameters that confuse people.

Uwe

-
Uwe Schindler
uschind...@apache.org
ASF Member, Apache Lucene PMC / Committer Bremen, Germany 
http://lucene.apache.org/
> -Original Message-
> From: Stephen Felts [mailto:stephen.fe...@oracle.com]
> Sent: Saturday, December 10, 2016 12:07 AM
> To: Uwe Schindler <uschind...@apache.org>; 
> jigsaw-...@openjdk.java.net; Core-Libs-Dev 
> <core-libs-dev@openjdk.java.net>
> Subject: RE: Java 9 build 148 causes trouble in Apache 
> Lucene/Solr/Elasticsearch
> 
> I would highly recommend running with _JAVA_OPTIONS=- 
> Dsun.reflect.debugModuleAccessChecks=true
> It will tell you what add-options are required.
> One minor downside is that it will produce the warning in cases where 
> the software is already correctly handling the exception from 
> setAccessible, so there can be noise.

Re: Java 9 build 148 causes trouble in Apache Lucene/Solr/Elasticsearch

[Kevin Rushforth]

I second the recommendation of using 
"-Dsun.reflect.debugModuleAccessChecks=true". We use gradle to build 
JavaFX and I ended up needing the following to allow our build to run 
with jdk-9+148:


export _JAVA_OPTIONS="-Dsun.reflect.debugModuleAccessChecks=true 
--add-opens=java.base/java.lang=ALL-UNNAMED 
--add-opens=java.base/java.util=ALL-UNNAMED 
--add-opens=java.base/java.lang.invoke=ALL-UNNAMED 
--add-opens=java.base/java.io=ALL-UNNAMED 
--add-opens=java.base/java.util.concurrent=ALL-UNNAMED 
--add-opens=java.base/java.text=ALL-UNNAMED"


-- Kevin


Stephen Felts wrote:

I would highly recommend running with 
_JAVA_OPTIONS=-Dsun.reflect.debugModuleAccessChecks=true
It will tell you what add-options are required.
One minor downside is that it will produce the warning in cases where the 
software is already correctly handling the exception from setAccessible, so 
there can be noise.
  

RE: Java 9 build 148 causes trouble in Apache Lucene/Solr/Elasticsearch

[Uwe Schindler]

Hi,

Thanks for the hints to fix Groovy, although this is hard to do with ANT (which 
is our build system).

The -Dsun.reflect.debugModuleAccessChecks=true options help to debug, indeed, 
but it does not solve the underlying issue. Apache Solr/Lucene and 
Elasticsearch will no longer work with Java 9 unless you require users to add 
those strange options. Elasticsearch already runs with a SecurityManager by 
default, so the question is: why is this not handled by a security manager and 
a new permission like "crossModuleAccess/module/package"? Why must it be done 
on command line? This makes it impossible to ship something like Lucene that it 
work out of box together with correct policy files?

And as said in my previous mail: The direct bytebuffer unmapping has still no 
"official" way to do it, but it is critical to large scale database systems 
like Lucene/Solr/Elasticsearch. You have replacements in Java 9 for Unsafe 
(VarHandles,...), but still no way to allow unmapping of byte buffers that sit 
on huge resources or disallow deleting of files on windows. It was discussed on 
last FOSDEM to do something in Java 10 (I would like to get information how to 
propose the required change as Java 10 dev started now!), and in the meantime 
it was confirmed that some APIs in the JDK are "critical" and will be 
supported. But this is now 

So please re-add the special critical APIs back to the whitelist, so code like 
getting (legacy) Unsafe or unmapping direct buffers works without command line 
parameters that confuse people.

Uwe

-
Uwe Schindler
uschind...@apache.org 
ASF Member, Apache Lucene PMC / Committer
Bremen, Germany
http://lucene.apache.org/
> -Original Message-
> From: Stephen Felts [mailto:stephen.fe...@oracle.com]
> Sent: Saturday, December 10, 2016 12:07 AM
> To: Uwe Schindler <uschind...@apache.org>; jigsaw-...@openjdk.java.net;
> Core-Libs-Dev <core-libs-dev@openjdk.java.net>
> Subject: RE: Java 9 build 148 causes trouble in Apache
> Lucene/Solr/Elasticsearch
> 
> I would highly recommend running with _JAVA_OPTIONS=-
> Dsun.reflect.debugModuleAccessChecks=true
> It will tell you what add-options are required.
> One minor downside is that it will produce the warning in cases where the
> software is already correctly handling the exception from setAccessible, so
> there can be noise.

RE: Java 9 build 148 causes trouble in Apache Lucene/Solr/Elasticsearch

[Stephen Felts]

I would highly recommend running with 
_JAVA_OPTIONS=-Dsun.reflect.debugModuleAccessChecks=true
It will tell you what add-options are required.
One minor downside is that it will produce the warning in cases where the 
software is already correctly handling the exception from setAccessible, so 
there can be noise.