Re: How to create a local build that targets Java 11, when building with 17?
Apologies for not replying yesterday and thank you, Greg, for answering the question. All I could add is that it's a pretty common semantic versioning scheme [1], although there is no rigorous method of policing it - very often it's just relying on common sense. Lucene often does add big code changes to what's a "minor" release in this scheme (9x) but only if they're backward compatible. Major releases are limited to breaking the API, bumping the minimum Java version, etc. Dawid [1] https://semver.org/ On Sat, May 6, 2023 at 1:53 AM Greg Miller wrote: > Hi Jonathan- > > The main branch is the tip of development, and what will eventually become > 10.0. It can use a later version of Java, make (some) > non-backwards-compatible API changes, etc. branch_9x tracks the latest 9.x > release, and must run on the version of Java supported by 9.x releases, > must be API backwards-compatible, etc. The general approach is to make > changes against main, and then backport those changes to branch_9x in a 9.x > friendly way if possible. Sometimes a change on main is complex enough that > backporting in a 9.x friendly manner isn't really feasible, in which case > the change will be released with 10.0. I'm sure I'm leaving out some > details, but hopefully this is helpful. You may also find this reference > useful: > https://cwiki.apache.org/confluence/display/LUCENE/BackwardsCompatibility > > Cheers, > -Greg > > On Fri, May 5, 2023 at 12:00 PM Jonathan Ellis wrote: > >> Thanks. What are the rules for what should go into main vs branch_9x? >> >> On Fri, May 5, 2023 at 1:54 PM Dawid Weiss wrote: >> >>> >>> The main branch is on Java 17, see build.gradle: >>> >>> // Minimum Java version required to compile and run Lucene. >>> minJavaVersion = JavaVersion.VERSION_17 >>> >>> Also, don't use the default gradle task created by convention; use this >>> one: >>> >>> ./gradlew mavenToLocal >>> >>> it's an alias but it publishes only a subset of relevant projects, not >>> all of them. >>> >>> Dawid >>> >>> On Fri, May 5, 2023 at 8:03 PM Jonathan Ellis wrote: >>> Actually my hack doesn't work, the manifest file changes but the .class files do not. On Fri, May 5, 2023 at 12:38 PM Jonathan Ellis wrote: > `./gradlew publishToMavenLocal` gives me Java 17 class files by > default, which surprises me since AFAIK 11 is still the minimum to run > Lucene. > > I hacked it to work by editing javac.gradle > sourceCompatibility = JavaVersion.VERSION_11 > targetCompatibility = JavaVersion.VERSION_11 > > Is there a cleaner way to do this? > > -- > Jonathan Ellis > co-founder, http://www.datastax.com > @spyced > -- Jonathan Ellis co-founder, http://www.datastax.com @spyced >>> >> >> -- >> Jonathan Ellis >> co-founder, http://www.datastax.com >> @spyced >> >
Re: How to create a local build that targets Java 11, when building with 17?
Hi Jonathan- The main branch is the tip of development, and what will eventually become 10.0. It can use a later version of Java, make (some) non-backwards-compatible API changes, etc. branch_9x tracks the latest 9.x release, and must run on the version of Java supported by 9.x releases, must be API backwards-compatible, etc. The general approach is to make changes against main, and then backport those changes to branch_9x in a 9.x friendly way if possible. Sometimes a change on main is complex enough that backporting in a 9.x friendly manner isn't really feasible, in which case the change will be released with 10.0. I'm sure I'm leaving out some details, but hopefully this is helpful. You may also find this reference useful: https://cwiki.apache.org/confluence/display/LUCENE/BackwardsCompatibility Cheers, -Greg On Fri, May 5, 2023 at 12:00 PM Jonathan Ellis wrote: > Thanks. What are the rules for what should go into main vs branch_9x? > > On Fri, May 5, 2023 at 1:54 PM Dawid Weiss wrote: > >> >> The main branch is on Java 17, see build.gradle: >> >> // Minimum Java version required to compile and run Lucene. >> minJavaVersion = JavaVersion.VERSION_17 >> >> Also, don't use the default gradle task created by convention; use this >> one: >> >> ./gradlew mavenToLocal >> >> it's an alias but it publishes only a subset of relevant projects, not >> all of them. >> >> Dawid >> >> On Fri, May 5, 2023 at 8:03 PM Jonathan Ellis wrote: >> >>> Actually my hack doesn't work, the manifest file changes but the .class >>> files do not. >>> >>> On Fri, May 5, 2023 at 12:38 PM Jonathan Ellis >>> wrote: >>> `./gradlew publishToMavenLocal` gives me Java 17 class files by default, which surprises me since AFAIK 11 is still the minimum to run Lucene. I hacked it to work by editing javac.gradle sourceCompatibility = JavaVersion.VERSION_11 targetCompatibility = JavaVersion.VERSION_11 Is there a cleaner way to do this? -- Jonathan Ellis co-founder, http://www.datastax.com @spyced >>> >>> >>> -- >>> Jonathan Ellis >>> co-founder, http://www.datastax.com >>> @spyced >>> >> > > -- > Jonathan Ellis > co-founder, http://www.datastax.com > @spyced >
Re: How to create a local build that targets Java 11, when building with 17?
Thanks. What are the rules for what should go into main vs branch_9x? On Fri, May 5, 2023 at 1:54 PM Dawid Weiss wrote: > > The main branch is on Java 17, see build.gradle: > > // Minimum Java version required to compile and run Lucene. > minJavaVersion = JavaVersion.VERSION_17 > > Also, don't use the default gradle task created by convention; use this > one: > > ./gradlew mavenToLocal > > it's an alias but it publishes only a subset of relevant projects, not all > of them. > > Dawid > > On Fri, May 5, 2023 at 8:03 PM Jonathan Ellis wrote: > >> Actually my hack doesn't work, the manifest file changes but the .class >> files do not. >> >> On Fri, May 5, 2023 at 12:38 PM Jonathan Ellis wrote: >> >>> `./gradlew publishToMavenLocal` gives me Java 17 class files by default, >>> which surprises me since AFAIK 11 is still the minimum to run Lucene. >>> >>> I hacked it to work by editing javac.gradle >>> sourceCompatibility = JavaVersion.VERSION_11 >>> targetCompatibility = JavaVersion.VERSION_11 >>> >>> Is there a cleaner way to do this? >>> >>> -- >>> Jonathan Ellis >>> co-founder, http://www.datastax.com >>> @spyced >>> >> >> >> -- >> Jonathan Ellis >> co-founder, http://www.datastax.com >> @spyced >> > -- Jonathan Ellis co-founder, http://www.datastax.com @spyced
Re: How to create a local build that targets Java 11, when building with 17?
The main branch is on Java 17, see build.gradle: // Minimum Java version required to compile and run Lucene. minJavaVersion = JavaVersion.VERSION_17 Also, don't use the default gradle task created by convention; use this one: ./gradlew mavenToLocal it's an alias but it publishes only a subset of relevant projects, not all of them. Dawid On Fri, May 5, 2023 at 8:03 PM Jonathan Ellis wrote: > Actually my hack doesn't work, the manifest file changes but the .class > files do not. > > On Fri, May 5, 2023 at 12:38 PM Jonathan Ellis wrote: > >> `./gradlew publishToMavenLocal` gives me Java 17 class files by default, >> which surprises me since AFAIK 11 is still the minimum to run Lucene. >> >> I hacked it to work by editing javac.gradle >> sourceCompatibility = JavaVersion.VERSION_11 >> targetCompatibility = JavaVersion.VERSION_11 >> >> Is there a cleaner way to do this? >> >> -- >> Jonathan Ellis >> co-founder, http://www.datastax.com >> @spyced >> > > > -- > Jonathan Ellis > co-founder, http://www.datastax.com > @spyced >
Re: How to create a local build that targets Java 11, when building with 17?
Actually my hack doesn't work, the manifest file changes but the .class files do not. On Fri, May 5, 2023 at 12:38 PM Jonathan Ellis wrote: > `./gradlew publishToMavenLocal` gives me Java 17 class files by default, > which surprises me since AFAIK 11 is still the minimum to run Lucene. > > I hacked it to work by editing javac.gradle > sourceCompatibility = JavaVersion.VERSION_11 > targetCompatibility = JavaVersion.VERSION_11 > > Is there a cleaner way to do this? > > -- > Jonathan Ellis > co-founder, http://www.datastax.com > @spyced > -- Jonathan Ellis co-founder, http://www.datastax.com @spyced
How to create a local build that targets Java 11, when building with 17?
`./gradlew publishToMavenLocal` gives me Java 17 class files by default, which surprises me since AFAIK 11 is still the minimum to run Lucene. I hacked it to work by editing javac.gradle sourceCompatibility = JavaVersion.VERSION_11 targetCompatibility = JavaVersion.VERSION_11 Is there a cleaner way to do this? -- Jonathan Ellis co-founder, http://www.datastax.com @spyced
Re: Java 11/17 Version Matrix
Let's wait for this functionality and see what happens. If the gain is significant then this provides an incentive to upgrade for everyone. MR-JARs will be a pain to keep consistent... Dawid On Tue, Sep 14, 2021 at 10:19 AM Adrien Grand wrote: > I think we should discuss options when Project Panama is released. Doing > frequent major releases forces users to reindex more often. If Project > Panama was released shortly and we decided to release Lucene 10 > immediately, this would force users to reindex their 8.x data to be able to > upgrade, I know of many Elasticsearch users for whom it would cause > headaches and I suspect that it's no different for Solr users and many > direct users of Lucene. I'd rather look into publishing a MR JAR of bumping > the minimum required Java version in a minor release than releasing Lucene > 10.0 less than 1 year after 9.0. > > On Mon, Sep 13, 2021 at 9:19 PM Uwe Schindler wrote: > >> In addition: we test for compatibility with Java 17 (both Lucene and >> Solr), so consumer is still able to use any version and has enough >> flexibility. >> >> Uwe >> >> Am 13. September 2021 18:52:53 UTC schrieb Dawid Weiss < >> dawid.we...@gmail.com>: >>> >>> >>> I agree with Uwe and Robert. JDK11, then the min bar should move if >>> there is something that brings value (be it performance, LTS or some other >>> attractive option). >>> >>> Dawid >>> >>> On Mon, Sep 13, 2021 at 8:15 PM Robert Muir wrote: >>> >>>> +1, I think the main thing to watch out for is project panama. If we >>>> get a 18.x/19.x release with non-incubating APIs, I think it makes >>>> sense to create a new major Lucene version. Even if it isn't an >>>> OpenJDK LTS release. It could really change a lot, especially >>>> regarding hotspots in the code such as postings/dv compression. So it >>>> would be great to allow a lot of folks to make a "take two" on these >>>> algorithms with vectorization in mind. This is just my opinion. >>>> >>>> On Mon, Sep 13, 2021 at 2:10 PM Uwe Schindler wrote: >>>> > >>>> > There are no good reasons to do Java 17 and it is way too early. >>>> > >>>> > >>>> > >>>> > Reagrding real optimizations, Lucene 17 is unfortunately not >>>> containing Project Panama or Vector API, so it looks more like Java 18/19 >>>> is a good candidate as a new minimum at a later stage. >>>> > >>>> > >>>> > >>>> > I’d release Lucene 9 with Java 11 (which is LTS) and then decide >>>> later if we update to some post-17 version to get the new vector and panama >>>> APIs (vector search, SIMD and also MMapDirectory v2). If we do this, we >>>> should simply release Lucene 10. >>>> > >>>> > >>>> > >>>> > Uwe >>>> > >>>> > >>>> > >>>> > - >>>> > >>>> > Uwe Schindler >>>> > >>>> > Achterdiek 19, D-28357 Bremen >>>> > >>>> > https://www.thetaphi.de >>>> > >>>> > eMail: u...@thetaphi.de >>>> > >>>> > >>>> > >>>> > From: Mike Drob >>>> > Sent: Monday, September 13, 2021 8:00 PM >>>> > To: Solr/Lucene Dev >>>> > Subject: Java 11/17 Version Matrix >>>> > >>>> > >>>> > >>>> > Hi Devs, >>>> > >>>> > >>>> > >>>> > What are our thoughts on Java 11 and 17 version compatibility going >>>> forward for Lucene 9? Will we support both? If so, would Java 11 support >>>> likely continue for the entire 9.x release line? >>>> > >>>> > >>>> > >>>> > Is there a JIRA tracking this? >>>> > >>>> > >>>> > >>>> > Thanks, >>>> > >>>> > Mike >>>> >>>> - >>>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >>>> For additional commands, e-mail: dev-h...@lucene.apache.org >>>> >>>> -- >> Uwe Schindler >> Achterdiek 19, 28357 Bremen >> https://www.thetaphi.de >> > > > -- > Adrien >
Re: Java 11/17 Version Matrix
Sorry, this is a bogus argument. Nobody is holding a gun to anyone's head and forcing them to upgrade. When we have new major functionality, we should be able to issue new major releases, to hell with elasticsearch users. On Tue, Sep 14, 2021 at 4:19 AM Adrien Grand wrote: > > I think we should discuss options when Project Panama is released. Doing > frequent major releases forces users to reindex more often. If Project Panama > was released shortly and we decided to release Lucene 10 immediately, this > would force users to reindex their 8.x data to be able to upgrade, I know of > many Elasticsearch users for whom it would cause headaches and I suspect that > it's no different for Solr users and many direct users of Lucene. I'd rather > look into publishing a MR JAR of bumping the minimum required Java version in > a minor release than releasing Lucene 10.0 less than 1 year after 9.0. > > On Mon, Sep 13, 2021 at 9:19 PM Uwe Schindler wrote: >> >> In addition: we test for compatibility with Java 17 (both Lucene and Solr), >> so consumer is still able to use any version and has enough flexibility. >> >> Uwe >> >> Am 13. September 2021 18:52:53 UTC schrieb Dawid Weiss >> : >>> >>> >>> I agree with Uwe and Robert. JDK11, then the min bar should move if there >>> is something that brings value (be it performance, LTS or some other >>> attractive option). >>> >>> Dawid >>> >>> On Mon, Sep 13, 2021 at 8:15 PM Robert Muir wrote: >>>> >>>> +1, I think the main thing to watch out for is project panama. If we >>>> get a 18.x/19.x release with non-incubating APIs, I think it makes >>>> sense to create a new major Lucene version. Even if it isn't an >>>> OpenJDK LTS release. It could really change a lot, especially >>>> regarding hotspots in the code such as postings/dv compression. So it >>>> would be great to allow a lot of folks to make a "take two" on these >>>> algorithms with vectorization in mind. This is just my opinion. >>>> >>>> On Mon, Sep 13, 2021 at 2:10 PM Uwe Schindler wrote: >>>> > >>>> > There are no good reasons to do Java 17 and it is way too early. >>>> > >>>> > >>>> > >>>> > Reagrding real optimizations, Lucene 17 is unfortunately not containing >>>> > Project Panama or Vector API, so it looks more like Java 18/19 is a good >>>> > candidate as a new minimum at a later stage. >>>> > >>>> > >>>> > >>>> > I’d release Lucene 9 with Java 11 (which is LTS) and then decide later >>>> > if we update to some post-17 version to get the new vector and panama >>>> > APIs (vector search, SIMD and also MMapDirectory v2). If we do this, we >>>> > should simply release Lucene 10. >>>> > >>>> > >>>> > >>>> > Uwe >>>> > >>>> > >>>> > >>>> > - >>>> > >>>> > Uwe Schindler >>>> > >>>> > Achterdiek 19, D-28357 Bremen >>>> > >>>> > https://www.thetaphi.de >>>> > >>>> > eMail: u...@thetaphi.de >>>> > >>>> > >>>> > >>>> > From: Mike Drob >>>> > Sent: Monday, September 13, 2021 8:00 PM >>>> > To: Solr/Lucene Dev >>>> > Subject: Java 11/17 Version Matrix >>>> > >>>> > >>>> > >>>> > Hi Devs, >>>> > >>>> > >>>> > >>>> > What are our thoughts on Java 11 and 17 version compatibility going >>>> > forward for Lucene 9? Will we support both? If so, would Java 11 support >>>> > likely continue for the entire 9.x release line? >>>> > >>>> > >>>> > >>>> > Is there a JIRA tracking this? >>>> > >>>> > >>>> > >>>> > Thanks, >>>> > >>>> > Mike >>>> >>>> - >>>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >>>> For additional commands, e-mail: dev-h...@lucene.apache.org >>>> >> -- >> Uwe Schindler >> Achterdiek 19, 28357 Bremen >> https://www.thetaphi.de > > > > -- > Adrien - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Java 11/17 Version Matrix
I think we should discuss options when Project Panama is released. Doing frequent major releases forces users to reindex more often. If Project Panama was released shortly and we decided to release Lucene 10 immediately, this would force users to reindex their 8.x data to be able to upgrade, I know of many Elasticsearch users for whom it would cause headaches and I suspect that it's no different for Solr users and many direct users of Lucene. I'd rather look into publishing a MR JAR of bumping the minimum required Java version in a minor release than releasing Lucene 10.0 less than 1 year after 9.0. On Mon, Sep 13, 2021 at 9:19 PM Uwe Schindler wrote: > In addition: we test for compatibility with Java 17 (both Lucene and > Solr), so consumer is still able to use any version and has enough > flexibility. > > Uwe > > Am 13. September 2021 18:52:53 UTC schrieb Dawid Weiss < > dawid.we...@gmail.com>: >> >> >> I agree with Uwe and Robert. JDK11, then the min bar should move if there >> is something that brings value (be it performance, LTS or some other >> attractive option). >> >> Dawid >> >> On Mon, Sep 13, 2021 at 8:15 PM Robert Muir wrote: >> >>> +1, I think the main thing to watch out for is project panama. If we >>> get a 18.x/19.x release with non-incubating APIs, I think it makes >>> sense to create a new major Lucene version. Even if it isn't an >>> OpenJDK LTS release. It could really change a lot, especially >>> regarding hotspots in the code such as postings/dv compression. So it >>> would be great to allow a lot of folks to make a "take two" on these >>> algorithms with vectorization in mind. This is just my opinion. >>> >>> On Mon, Sep 13, 2021 at 2:10 PM Uwe Schindler wrote: >>> > >>> > There are no good reasons to do Java 17 and it is way too early. >>> > >>> > >>> > >>> > Reagrding real optimizations, Lucene 17 is unfortunately not >>> containing Project Panama or Vector API, so it looks more like Java 18/19 >>> is a good candidate as a new minimum at a later stage. >>> > >>> > >>> > >>> > I’d release Lucene 9 with Java 11 (which is LTS) and then decide later >>> if we update to some post-17 version to get the new vector and panama APIs >>> (vector search, SIMD and also MMapDirectory v2). If we do this, we should >>> simply release Lucene 10. >>> > >>> > >>> > >>> > Uwe >>> > >>> > >>> > >>> > - >>> > >>> > Uwe Schindler >>> > >>> > Achterdiek 19, D-28357 Bremen >>> > >>> > https://www.thetaphi.de >>> > >>> > eMail: u...@thetaphi.de >>> > >>> > >>> > >>> > From: Mike Drob >>> > Sent: Monday, September 13, 2021 8:00 PM >>> > To: Solr/Lucene Dev >>> > Subject: Java 11/17 Version Matrix >>> > >>> > >>> > >>> > Hi Devs, >>> > >>> > >>> > >>> > What are our thoughts on Java 11 and 17 version compatibility going >>> forward for Lucene 9? Will we support both? If so, would Java 11 support >>> likely continue for the entire 9.x release line? >>> > >>> > >>> > >>> > Is there a JIRA tracking this? >>> > >>> > >>> > >>> > Thanks, >>> > >>> > Mike >>> >>> - >>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >>> For additional commands, e-mail: dev-h...@lucene.apache.org >>> >>> -- > Uwe Schindler > Achterdiek 19, 28357 Bremen > https://www.thetaphi.de > -- Adrien
Re: Java 11/17 Version Matrix
In addition: we test for compatibility with Java 17 (both Lucene and Solr), so consumer is still able to use any version and has enough flexibility. Uwe Am 13. September 2021 18:52:53 UTC schrieb Dawid Weiss : >I agree with Uwe and Robert. JDK11, then the min bar should move if there >is something that brings value (be it performance, LTS or some other >attractive option). > >Dawid > >On Mon, Sep 13, 2021 at 8:15 PM Robert Muir wrote: > >> +1, I think the main thing to watch out for is project panama. If we >> get a 18.x/19.x release with non-incubating APIs, I think it makes >> sense to create a new major Lucene version. Even if it isn't an >> OpenJDK LTS release. It could really change a lot, especially >> regarding hotspots in the code such as postings/dv compression. So it >> would be great to allow a lot of folks to make a "take two" on these >> algorithms with vectorization in mind. This is just my opinion. >> >> On Mon, Sep 13, 2021 at 2:10 PM Uwe Schindler wrote: >> > >> > There are no good reasons to do Java 17 and it is way too early. >> > >> > >> > >> > Reagrding real optimizations, Lucene 17 is unfortunately not containing >> Project Panama or Vector API, so it looks more like Java 18/19 is a good >> candidate as a new minimum at a later stage. >> > >> > >> > >> > I’d release Lucene 9 with Java 11 (which is LTS) and then decide later >> if we update to some post-17 version to get the new vector and panama APIs >> (vector search, SIMD and also MMapDirectory v2). If we do this, we should >> simply release Lucene 10. >> > >> > >> > >> > Uwe >> > >> > >> > >> > - >> > >> > Uwe Schindler >> > >> > Achterdiek 19, D-28357 Bremen >> > >> > https://www.thetaphi.de >> > >> > eMail: u...@thetaphi.de >> > >> > >> > >> > From: Mike Drob >> > Sent: Monday, September 13, 2021 8:00 PM >> > To: Solr/Lucene Dev >> > Subject: Java 11/17 Version Matrix >> > >> > >> > >> > Hi Devs, >> > >> > >> > >> > What are our thoughts on Java 11 and 17 version compatibility going >> forward for Lucene 9? Will we support both? If so, would Java 11 support >> likely continue for the entire 9.x release line? >> > >> > >> > >> > Is there a JIRA tracking this? >> > >> > >> > >> > Thanks, >> > >> > Mike >> >> - >> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >> For additional commands, e-mail: dev-h...@lucene.apache.org >> >> -- Uwe Schindler Achterdiek 19, 28357 Bremen https://www.thetaphi.de
Re: Java 11/17 Version Matrix
I agree with Uwe and Robert. JDK11, then the min bar should move if there is something that brings value (be it performance, LTS or some other attractive option). Dawid On Mon, Sep 13, 2021 at 8:15 PM Robert Muir wrote: > +1, I think the main thing to watch out for is project panama. If we > get a 18.x/19.x release with non-incubating APIs, I think it makes > sense to create a new major Lucene version. Even if it isn't an > OpenJDK LTS release. It could really change a lot, especially > regarding hotspots in the code such as postings/dv compression. So it > would be great to allow a lot of folks to make a "take two" on these > algorithms with vectorization in mind. This is just my opinion. > > On Mon, Sep 13, 2021 at 2:10 PM Uwe Schindler wrote: > > > > There are no good reasons to do Java 17 and it is way too early. > > > > > > > > Reagrding real optimizations, Lucene 17 is unfortunately not containing > Project Panama or Vector API, so it looks more like Java 18/19 is a good > candidate as a new minimum at a later stage. > > > > > > > > I’d release Lucene 9 with Java 11 (which is LTS) and then decide later > if we update to some post-17 version to get the new vector and panama APIs > (vector search, SIMD and also MMapDirectory v2). If we do this, we should > simply release Lucene 10. > > > > > > > > Uwe > > > > > > > > - > > > > Uwe Schindler > > > > Achterdiek 19, D-28357 Bremen > > > > https://www.thetaphi.de > > > > eMail: u...@thetaphi.de > > > > > > > > From: Mike Drob > > Sent: Monday, September 13, 2021 8:00 PM > > To: Solr/Lucene Dev > > Subject: Java 11/17 Version Matrix > > > > > > > > Hi Devs, > > > > > > > > What are our thoughts on Java 11 and 17 version compatibility going > forward for Lucene 9? Will we support both? If so, would Java 11 support > likely continue for the entire 9.x release line? > > > > > > > > Is there a JIRA tracking this? > > > > > > > > Thanks, > > > > Mike > > - > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org > >
Re: Java 11/17 Version Matrix
+1, I think the main thing to watch out for is project panama. If we get a 18.x/19.x release with non-incubating APIs, I think it makes sense to create a new major Lucene version. Even if it isn't an OpenJDK LTS release. It could really change a lot, especially regarding hotspots in the code such as postings/dv compression. So it would be great to allow a lot of folks to make a "take two" on these algorithms with vectorization in mind. This is just my opinion. On Mon, Sep 13, 2021 at 2:10 PM Uwe Schindler wrote: > > There are no good reasons to do Java 17 and it is way too early. > > > > Reagrding real optimizations, Lucene 17 is unfortunately not containing > Project Panama or Vector API, so it looks more like Java 18/19 is a good > candidate as a new minimum at a later stage. > > > > I’d release Lucene 9 with Java 11 (which is LTS) and then decide later if we > update to some post-17 version to get the new vector and panama APIs (vector > search, SIMD and also MMapDirectory v2). If we do this, we should simply > release Lucene 10. > > > > Uwe > > > > - > > Uwe Schindler > > Achterdiek 19, D-28357 Bremen > > https://www.thetaphi.de > > eMail: u...@thetaphi.de > > > > From: Mike Drob > Sent: Monday, September 13, 2021 8:00 PM > To: Solr/Lucene Dev > Subject: Java 11/17 Version Matrix > > > > Hi Devs, > > > > What are our thoughts on Java 11 and 17 version compatibility going forward > for Lucene 9? Will we support both? If so, would Java 11 support likely > continue for the entire 9.x release line? > > > > Is there a JIRA tracking this? > > > > Thanks, > > Mike - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
RE: Java 11/17 Version Matrix
There are no good reasons to do Java 17 and it is way too early. Reagrding real optimizations, Lucene 17 is unfortunately not containing Project Panama or Vector API, so it looks more like Java 18/19 is a good candidate as a new minimum at a later stage. I’d release Lucene 9 with Java 11 (which is LTS) and then decide later if we update to some post-17 version to get the new vector and panama APIs (vector search, SIMD and also MMapDirectory v2). If we do this, we should simply release Lucene 10. Uwe - Uwe Schindler Achterdiek 19, D-28357 Bremen <https://www.thetaphi.de> https://www.thetaphi.de eMail: u...@thetaphi.de From: Mike Drob Sent: Monday, September 13, 2021 8:00 PM To: Solr/Lucene Dev Subject: Java 11/17 Version Matrix Hi Devs, What are our thoughts on Java 11 and 17 version compatibility going forward for Lucene 9? Will we support both? If so, would Java 11 support likely continue for the entire 9.x release line? Is there a JIRA tracking this? Thanks, Mike
Java 11/17 Version Matrix
Hi Devs, What are our thoughts on Java 11 and 17 version compatibility going forward for Lucene 9? Will we support both? If so, would Java 11 support likely continue for the entire 9.x release line? Is there a JIRA tracking this? Thanks, Mike
Re: Solr 8.x and contribs requiring Java 11
Thanks Ishan, I had misunderstood! On Sat, Oct 31, 2020 at 2:12 AM Ishan Chattopadhyaya < ichattopadhy...@gmail.com> wrote: > Hi Adrien, > I don't suggest we do so in 8.7, unless there's CVE that need immediate > addressing. +1 to removing in branch_8x. > Thanks, > Ishan > > On Fri, 30 Oct, 2020, 11:22 pm Adrien Grand, wrote: > >> Ishan, why would this be a blocker for 8.7? Would it be good enough to >> remove in branch_8x? >> >> Le ven. 30 oct. 2020 à 18:33, Ishan Chattopadhyaya < >> ichattopadhy...@gmail.com> a écrit : >> >>> +1 to removing it (in 8.7 with a respin, if needed). If we can't support >>> it, there's no need to keep it. If someone wants, they can assume ownership >>> of a third party package. +1 to removing all non essential code and >>> trimming down the Solr distribution. There's no good reason for Solr, which >>> is in effect just a distributed layer on top of Lucene, to be a 200MB >>> download. >>> >>> On Fri, 30 Oct, 2020, 9:50 pm Uwe Schindler, wrote: >>> >>>> I fully agree with Erick, >>>> >>>> Please don't start and try to get 8.x on Java 11. Release Lucene/Solr 9! >>>> >>>> Uwe >>>> >>>> - >>>> Uwe Schindler >>>> Achterdiek 19, D-28357 Bremen >>>> https://www.thetaphi.de >>>> eMail: u...@thetaphi.de >>>> >>>> > -Original Message- >>>> > From: Erick Erickson >>>> > Sent: Friday, October 30, 2020 3:39 PM >>>> > To: dev@lucene.apache.org >>>> > Subject: Re: Solr 8.x and contribs requiring Java 11 >>>> > >>>> > I’m always reluctant to change something like this for a point >>>> release. >>>> > >>>> > I’ve been supposing that we’d release Solr 9 for a while, I always >>>> thought >>>> > that moving to Java 11 would be a driver for the 9.0 release but I >>>> wasn’t >>>> > correct in that. >>>> > >>>> > That expectation has been complicated by the whole reference impl >>>> thing. >>>> > >>>> > What I’d suggest is that we push forward with a 9.0 release and >>>> consider >>>> > it something of a placeholder while the reference impl either becomes >>>> > our main line or doesn’t. Then your problem disappears ;). >>>> Unfortunately, >>>> > a new major release always takes some time. >>>> > >>>> > FWIW, >>>> > Erick >>>> > >>>> > > On Oct 30, 2020, at 10:30 AM, Dawid Weiss >>>> > wrote: >>>> > > >>>> > > I've run into this in SOLR-14974. The dependency is on Java 11. >>>> > > Everything works if you build and run under Java 11 but of course it >>>> > > won't fly on Java 8 (won't even compile). >>>> > > >>>> > > I wonder what are your thoughts on keeping Java 8 as the minimum for >>>> > > Solr 8x. Is 8.x going to be on Java 8 forever? Or is minimum of Java >>>> > > 11 a possibility, only nobody got to it yet? >>>> > > >>>> > > Dawid >>>> > > >>>> > > >>>> - >>>> > > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >>>> > > For additional commands, e-mail: dev-h...@lucene.apache.org >>>> > > >>>> > >>>> > >>>> > - >>>> > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >>>> > For additional commands, e-mail: dev-h...@lucene.apache.org >>>> >>>> >>>> - >>>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >>>> For additional commands, e-mail: dev-h...@lucene.apache.org >>>> >>>> -- Adrien
Re: Solr 8.x and contribs requiring Java 11
> If we can't support it, there's no need to keep it. If someone wants, they > can assume ownership of a third party package. I am the author/ owner of that third party package, Ishan... and I've been working for two weeks to understand how Solr distributed processing works so that I could write that clustering extension's distributed version. And I did upgrade everything so that it works just fine (on master). Anyway, we initially helped with the clustering contrib but that was a long time ago and it's been neglected. There are some things that can't be done easily externally if you remove it but I don't know how many people will actually miss it if removed. One way to find out would be to just do it. Dawid - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Solr 8.x and contribs requiring Java 11
Hi Adrien, I don't suggest we do so in 8.7, unless there's CVE that need immediate addressing. +1 to removing in branch_8x. Thanks, Ishan On Fri, 30 Oct, 2020, 11:22 pm Adrien Grand, wrote: > Ishan, why would this be a blocker for 8.7? Would it be good enough to > remove in branch_8x? > > Le ven. 30 oct. 2020 à 18:33, Ishan Chattopadhyaya < > ichattopadhy...@gmail.com> a écrit : > >> +1 to removing it (in 8.7 with a respin, if needed). If we can't support >> it, there's no need to keep it. If someone wants, they can assume ownership >> of a third party package. +1 to removing all non essential code and >> trimming down the Solr distribution. There's no good reason for Solr, which >> is in effect just a distributed layer on top of Lucene, to be a 200MB >> download. >> >> On Fri, 30 Oct, 2020, 9:50 pm Uwe Schindler, wrote: >> >>> I fully agree with Erick, >>> >>> Please don't start and try to get 8.x on Java 11. Release Lucene/Solr 9! >>> >>> Uwe >>> >>> - >>> Uwe Schindler >>> Achterdiek 19, D-28357 Bremen >>> https://www.thetaphi.de >>> eMail: u...@thetaphi.de >>> >>> > -Original Message- >>> > From: Erick Erickson >>> > Sent: Friday, October 30, 2020 3:39 PM >>> > To: dev@lucene.apache.org >>> > Subject: Re: Solr 8.x and contribs requiring Java 11 >>> > >>> > I’m always reluctant to change something like this for a point release. >>> > >>> > I’ve been supposing that we’d release Solr 9 for a while, I always >>> thought >>> > that moving to Java 11 would be a driver for the 9.0 release but I >>> wasn’t >>> > correct in that. >>> > >>> > That expectation has been complicated by the whole reference impl >>> thing. >>> > >>> > What I’d suggest is that we push forward with a 9.0 release and >>> consider >>> > it something of a placeholder while the reference impl either becomes >>> > our main line or doesn’t. Then your problem disappears ;). >>> Unfortunately, >>> > a new major release always takes some time. >>> > >>> > FWIW, >>> > Erick >>> > >>> > > On Oct 30, 2020, at 10:30 AM, Dawid Weiss >>> > wrote: >>> > > >>> > > I've run into this in SOLR-14974. The dependency is on Java 11. >>> > > Everything works if you build and run under Java 11 but of course it >>> > > won't fly on Java 8 (won't even compile). >>> > > >>> > > I wonder what are your thoughts on keeping Java 8 as the minimum for >>> > > Solr 8x. Is 8.x going to be on Java 8 forever? Or is minimum of Java >>> > > 11 a possibility, only nobody got to it yet? >>> > > >>> > > Dawid >>> > > >>> > > - >>> > > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >>> > > For additional commands, e-mail: dev-h...@lucene.apache.org >>> > > >>> > >>> > >>> > - >>> > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >>> > For additional commands, e-mail: dev-h...@lucene.apache.org >>> >>> >>> - >>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >>> For additional commands, e-mail: dev-h...@lucene.apache.org >>> >>>
Re: Solr 8.x and contribs requiring Java 11
Ishan, why would this be a blocker for 8.7? Would it be good enough to remove in branch_8x? Le ven. 30 oct. 2020 à 18:33, Ishan Chattopadhyaya < ichattopadhy...@gmail.com> a écrit : > +1 to removing it (in 8.7 with a respin, if needed). If we can't support > it, there's no need to keep it. If someone wants, they can assume ownership > of a third party package. +1 to removing all non essential code and > trimming down the Solr distribution. There's no good reason for Solr, which > is in effect just a distributed layer on top of Lucene, to be a 200MB > download. > > On Fri, 30 Oct, 2020, 9:50 pm Uwe Schindler, wrote: > >> I fully agree with Erick, >> >> Please don't start and try to get 8.x on Java 11. Release Lucene/Solr 9! >> >> Uwe >> >> - >> Uwe Schindler >> Achterdiek 19, D-28357 Bremen >> https://www.thetaphi.de >> eMail: u...@thetaphi.de >> >> > -Original Message- >> > From: Erick Erickson >> > Sent: Friday, October 30, 2020 3:39 PM >> > To: dev@lucene.apache.org >> > Subject: Re: Solr 8.x and contribs requiring Java 11 >> > >> > I’m always reluctant to change something like this for a point release. >> > >> > I’ve been supposing that we’d release Solr 9 for a while, I always >> thought >> > that moving to Java 11 would be a driver for the 9.0 release but I >> wasn’t >> > correct in that. >> > >> > That expectation has been complicated by the whole reference impl thing. >> > >> > What I’d suggest is that we push forward with a 9.0 release and consider >> > it something of a placeholder while the reference impl either becomes >> > our main line or doesn’t. Then your problem disappears ;). >> Unfortunately, >> > a new major release always takes some time. >> > >> > FWIW, >> > Erick >> > >> > > On Oct 30, 2020, at 10:30 AM, Dawid Weiss >> > wrote: >> > > >> > > I've run into this in SOLR-14974. The dependency is on Java 11. >> > > Everything works if you build and run under Java 11 but of course it >> > > won't fly on Java 8 (won't even compile). >> > > >> > > I wonder what are your thoughts on keeping Java 8 as the minimum for >> > > Solr 8x. Is 8.x going to be on Java 8 forever? Or is minimum of Java >> > > 11 a possibility, only nobody got to it yet? >> > > >> > > Dawid >> > > >> > > - >> > > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >> > > For additional commands, e-mail: dev-h...@lucene.apache.org >> > > >> > >> > >> > - >> > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >> > For additional commands, e-mail: dev-h...@lucene.apache.org >> >> >> - >> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >> For additional commands, e-mail: dev-h...@lucene.apache.org >> >>
Re: Solr 8.x and contribs requiring Java 11
+1 to removing it (in 8.7 with a respin, if needed). If we can't support it, there's no need to keep it. If someone wants, they can assume ownership of a third party package. +1 to removing all non essential code and trimming down the Solr distribution. There's no good reason for Solr, which is in effect just a distributed layer on top of Lucene, to be a 200MB download. On Fri, 30 Oct, 2020, 9:50 pm Uwe Schindler, wrote: > I fully agree with Erick, > > Please don't start and try to get 8.x on Java 11. Release Lucene/Solr 9! > > Uwe > > - > Uwe Schindler > Achterdiek 19, D-28357 Bremen > https://www.thetaphi.de > eMail: u...@thetaphi.de > > > -Original Message- > > From: Erick Erickson > > Sent: Friday, October 30, 2020 3:39 PM > > To: dev@lucene.apache.org > > Subject: Re: Solr 8.x and contribs requiring Java 11 > > > > I’m always reluctant to change something like this for a point release. > > > > I’ve been supposing that we’d release Solr 9 for a while, I always > thought > > that moving to Java 11 would be a driver for the 9.0 release but I wasn’t > > correct in that. > > > > That expectation has been complicated by the whole reference impl thing. > > > > What I’d suggest is that we push forward with a 9.0 release and consider > > it something of a placeholder while the reference impl either becomes > > our main line or doesn’t. Then your problem disappears ;). Unfortunately, > > a new major release always takes some time. > > > > FWIW, > > Erick > > > > > On Oct 30, 2020, at 10:30 AM, Dawid Weiss > > wrote: > > > > > > I've run into this in SOLR-14974. The dependency is on Java 11. > > > Everything works if you build and run under Java 11 but of course it > > > won't fly on Java 8 (won't even compile). > > > > > > I wonder what are your thoughts on keeping Java 8 as the minimum for > > > Solr 8x. Is 8.x going to be on Java 8 forever? Or is minimum of Java > > > 11 a possibility, only nobody got to it yet? > > > > > > Dawid > > > > > > - > > > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > > > For additional commands, e-mail: dev-h...@lucene.apache.org > > > > > > > > > - > > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > > For additional commands, e-mail: dev-h...@lucene.apache.org > > > - > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org > >
RE: Solr 8.x and contribs requiring Java 11
I fully agree with Erick, Please don't start and try to get 8.x on Java 11. Release Lucene/Solr 9! Uwe - Uwe Schindler Achterdiek 19, D-28357 Bremen https://www.thetaphi.de eMail: u...@thetaphi.de > -Original Message- > From: Erick Erickson > Sent: Friday, October 30, 2020 3:39 PM > To: dev@lucene.apache.org > Subject: Re: Solr 8.x and contribs requiring Java 11 > > I’m always reluctant to change something like this for a point release. > > I’ve been supposing that we’d release Solr 9 for a while, I always thought > that moving to Java 11 would be a driver for the 9.0 release but I wasn’t > correct in that. > > That expectation has been complicated by the whole reference impl thing. > > What I’d suggest is that we push forward with a 9.0 release and consider > it something of a placeholder while the reference impl either becomes > our main line or doesn’t. Then your problem disappears ;). Unfortunately, > a new major release always takes some time. > > FWIW, > Erick > > > On Oct 30, 2020, at 10:30 AM, Dawid Weiss > wrote: > > > > I've run into this in SOLR-14974. The dependency is on Java 11. > > Everything works if you build and run under Java 11 but of course it > > won't fly on Java 8 (won't even compile). > > > > I wonder what are your thoughts on keeping Java 8 as the minimum for > > Solr 8x. Is 8.x going to be on Java 8 forever? Or is minimum of Java > > 11 a possibility, only nobody got to it yet? > > > > Dawid > > > > - > > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > > For additional commands, e-mail: dev-h...@lucene.apache.org > > > > > - > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Solr 8.x and contribs requiring Java 11
> The only option I see at the moment is to remove the clustering contrib entirely from 8x because it can't be upgraded. That makes sense to me. Maybe it shouldn't be a contrib henceforth? In other words, maybe it should be a 3rd-party package? Then our users could get updates to this contrib independently of the Lucene/Solr release cycle. It's one of the benefits of breaking up the Solr monolith. In a separate thread in this dev list, pertaining to the DIH, Gézapeti Cseh suggested that Solr might introduce something similar to "Apache Commons", but for Solr contribs. I think that's worth more investigation. It's obviously highly relevant to this thread. ~ David Smiley Apache Lucene/Solr Search Developer http://www.linkedin.com/in/davidwsmiley On Fri, Oct 30, 2020 at 10:52 AM Dawid Weiss wrote: > Yeah, right. Until 9x is not really satisfying - the problem is more > complex because people bring up CVEs for the clustering contrib that > is currently in 8x and, for better or worse, it affects this line for > the foreseeable future. I don't plan to invest time to make Java 8 > compatible backports of the clustering engine so I'm wondering what to > do with it now. The only option I see at the moment is to remove the > clustering contrib entirely from 8x because it can't be upgraded. > - > > Dawid > > On Fri, Oct 30, 2020 at 3:40 PM David Smiley wrote: > > > > SOLR-14974 is about a contrib, the clustering contrib in particular. > That contrib is a plugin, and it will eventually be "packaged" -- > https://issues.apache.org/jira/browse/SOLR-14688 which will ultimately > mean that someone running on Solr 8 that is also using Java 11 can install > that package when it's eventually released (when 9.0 is released, or > thereafter). So I don't think we should change Solr core / SolrJ 's > minimum Java requirements just because of the needs of a contrib. I think > that's solved by the package system. > > > > ~ David Smiley > > Apache Lucene/Solr Search Developer > > http://www.linkedin.com/in/davidwsmiley > > > > > > On Fri, Oct 30, 2020 at 10:31 AM Dawid Weiss > wrote: > >> > >> I've run into this in SOLR-14974. The dependency is on Java 11. > >> Everything works if you build and run under Java 11 but of course it > >> won't fly on Java 8 (won't even compile). > >> > >> I wonder what are your thoughts on keeping Java 8 as the minimum for > >> Solr 8x. Is 8.x going to be on Java 8 forever? Or is minimum of Java > >> 11 a possibility, only nobody got to it yet? > >> > >> Dawid > >> > >> - > >> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > >> For additional commands, e-mail: dev-h...@lucene.apache.org > >> > > - > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org > >
Re: Solr 8.x and contribs requiring Java 11
Yeah, right. Until 9x is not really satisfying - the problem is more complex because people bring up CVEs for the clustering contrib that is currently in 8x and, for better or worse, it affects this line for the foreseeable future. I don't plan to invest time to make Java 8 compatible backports of the clustering engine so I'm wondering what to do with it now. The only option I see at the moment is to remove the clustering contrib entirely from 8x because it can't be upgraded. - Dawid On Fri, Oct 30, 2020 at 3:40 PM David Smiley wrote: > > SOLR-14974 is about a contrib, the clustering contrib in particular. That > contrib is a plugin, and it will eventually be "packaged" -- > https://issues.apache.org/jira/browse/SOLR-14688 which will ultimately mean > that someone running on Solr 8 that is also using Java 11 can install that > package when it's eventually released (when 9.0 is released, or thereafter). > So I don't think we should change Solr core / SolrJ 's minimum Java > requirements just because of the needs of a contrib. I think that's solved > by the package system. > > ~ David Smiley > Apache Lucene/Solr Search Developer > http://www.linkedin.com/in/davidwsmiley > > > On Fri, Oct 30, 2020 at 10:31 AM Dawid Weiss wrote: >> >> I've run into this in SOLR-14974. The dependency is on Java 11. >> Everything works if you build and run under Java 11 but of course it >> won't fly on Java 8 (won't even compile). >> >> I wonder what are your thoughts on keeping Java 8 as the minimum for >> Solr 8x. Is 8.x going to be on Java 8 forever? Or is minimum of Java >> 11 a possibility, only nobody got to it yet? >> >> Dawid >> >> - >> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >> For additional commands, e-mail: dev-h...@lucene.apache.org >> - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Solr 8.x and contribs requiring Java 11
I’m always reluctant to change something like this for a point release. I’ve been supposing that we’d release Solr 9 for a while, I always thought that moving to Java 11 would be a driver for the 9.0 release but I wasn’t correct in that. That expectation has been complicated by the whole reference impl thing. What I’d suggest is that we push forward with a 9.0 release and consider it something of a placeholder while the reference impl either becomes our main line or doesn’t. Then your problem disappears ;). Unfortunately, a new major release always takes some time. FWIW, Erick > On Oct 30, 2020, at 10:30 AM, Dawid Weiss wrote: > > I've run into this in SOLR-14974. The dependency is on Java 11. > Everything works if you build and run under Java 11 but of course it > won't fly on Java 8 (won't even compile). > > I wonder what are your thoughts on keeping Java 8 as the minimum for > Solr 8x. Is 8.x going to be on Java 8 forever? Or is minimum of Java > 11 a possibility, only nobody got to it yet? > > Dawid > > - > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org > - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Solr 8.x and contribs requiring Java 11
SOLR-14974 is about a contrib, the clustering contrib in particular. That contrib is a plugin, and it will eventually be "packaged" -- https://issues.apache.org/jira/browse/SOLR-14688 which will ultimately mean that someone running on Solr 8 that is also using Java 11 can install that package when it's eventually released (when 9.0 is released, or thereafter). So I don't think we should change Solr core / SolrJ 's minimum Java requirements just because of the needs of a contrib. I think that's solved by the package system. ~ David Smiley Apache Lucene/Solr Search Developer http://www.linkedin.com/in/davidwsmiley On Fri, Oct 30, 2020 at 10:31 AM Dawid Weiss wrote: > I've run into this in SOLR-14974. The dependency is on Java 11. > Everything works if you build and run under Java 11 but of course it > won't fly on Java 8 (won't even compile). > > I wonder what are your thoughts on keeping Java 8 as the minimum for > Solr 8x. Is 8.x going to be on Java 8 forever? Or is minimum of Java > 11 a possibility, only nobody got to it yet? > > Dawid > > - > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org > >
Solr 8.x and contribs requiring Java 11
I've run into this in SOLR-14974. The dependency is on Java 11. Everything works if you build and run under Java 11 but of course it won't fly on Java 8 (won't even compile). I wonder what are your thoughts on keeping Java 8 as the minimum for Solr 8x. Is 8.x going to be on Java 8 forever? Or is minimum of Java 11 a possibility, only nobody got to it yet? Dawid - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
RE: [lucene-solr] branch master updated: master is on java 11!
Hi Robert,
please revert this and open an issue. There was a discussion about this and
commits like this should not be done without negotiation.
Uwe
-
Uwe Schindler
Achterdiek 19, D-28357 Bremen
https://www.thetaphi.de
eMail: u...@thetaphi.de
> -Original Message-
> From: rm...@apache.org
> Sent: Monday, August 31, 2020 10:52 AM
> To: comm...@lucene.apache.org
> Subject: [lucene-solr] branch master updated: master is on java 11!
>
> This is an automated email from the ASF dual-hosted git repository.
>
> rmuir pushed a commit to branch master
> in repository https://gitbox.apache.org/repos/asf/lucene-solr.git
>
>
> The following commit(s) were added to refs/heads/master by this push:
> new 4129daa master is on java 11!
> 4129daa is described below
>
> commit 4129daa6dd23f7f741cd38e6fcf2d4920d681ef6
> Author: Robert Muir
> AuthorDate: Mon Aug 31 04:53:44 2020 -0400
>
> master is on java 11!
> ---
> .../validate-source-patterns/check-source-patterns.groovy| 12
>
> 1 file changed, 12 deletions(-)
>
> diff --git a/gradle/validation/validate-source-patterns/check-source-
> patterns.groovy b/gradle/validation/validate-source-patterns/check-source-
> patterns.groovy
> index 774023d..aabcd27 100644
> --- a/gradle/validation/validate-source-patterns/check-source-patterns.groovy
> +++ b/gradle/validation/validate-source-patterns/check-source-patterns.groovy
> @@ -49,13 +49,6 @@ def invalidPatterns = [
>(~$/import java\.lang\.\w+;/$) : 'java.lang import is unnecessary'
> ]
>
> -// Python and others merrily use var declarations, this is a problem _only_
> in
> Java at least for 8x where we're forbidding var declarations
> -def invalidJavaOnlyPatterns = [
> - (~$/\n\s*var\s+.*=.*<>.*/$) : 'Diamond operators should not be used with
> var',
> - (~$/\n\s*var\s+/$) : 'var is not allowed in until we stop development on
> the
> 8x code line'
> -]
> -
> -
> def baseDir = properties['basedir'];
> def baseDirLen = baseDir.length() + 1;
>
> @@ -209,11 +202,6 @@ ant.fileScanner{
> reportViolation(f, "Solr test cases should extend SolrTestCase
> rather than
> LuceneTestCase");
>}
> }
> -invalidJavaOnlyPatterns.each { pattern,name ->
> - if (pattern.matcher(text).find()) {
> -reportViolation(f, name);
> - }
> -}
>}
>if (f.name.endsWith('.xml') || f.name.endsWith('.xml.template')) {
> checkLicenseHeaderPrecedes(f, '', xmlTagPattern,
> xmlCommentPattern, text, ratDocument);
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
RE: Performance in Solr 9 / Java 11
Hi, Yes, mr-jar build were always done in Lucene 8x (on Lucene’s part of the build). You might think: “How can that be as I am – as release manager – used JDK8 to build, and I am forced to do so?” This is a trick that works for some types of JDK9+ improvements, which rely on some simple “replacement” operations for static method that are highly optiized in later JDK versions using an compiler intrinsic. The trick is the following: We have (possibly slower) replacement implementations in Lucene’s codebase for the new method (see classes oal.utils.FutureArrays and FutureObjects) with identical static signature (this is why it only works for that case). After compilation of the code an additional Ant task is running that uses ASM to patch the produced JAR files of the Java 8 compiler to replace the usage of the FutureXxxx classes in the code: <https://github.com/apache/lucene-solr/blob/branch_8x/lucene/tools/src/groovy/patch-mrjar-classes.groovy> https://github.com/apache/lucene-solr/blob/branch_8x/lucene/tools/src/groovy/patch-mrjar-classes.groovy The patched classes are written to build/java9 and jarred into jar file below META-INF/versions/9 In master this was removed. If we have a similar thing in later JDK versions (static methods which got highly optimized versions with intrinsics in later Java versions), we can do the same, the above script just needs to be included into the Gradle build. But I have not figured out anything like this. The classical “MR-builds” are much more complex to setup, as it requires every developer to have the later version already installed and we need to do parallel compilation of different source trees which also has the risk of producing broken code that won’t work at runtime. So the “just replace method calls in Java 8 classes” is very convenient & safe, but limited. Very often it’s better for more complex cases, to add a factory pattern to your source code that uses a completely different implementation at runtime (like FSDirectory.open()). One example that might come later is a replacement for MMapDirectory. I am working on this, but it’s not yet useable (see my short talk at barcamp @ berlinbuzzwords this year). Here, we would have a separate impl of MMapDirectory like MemorySegmentDirectory that’s choosen at runtime depending on Java version. We would have a separate gradle module to implement that, which requires a later compiler. The reason for that approach is: The code is so different, that a classical MR JAR would complicate a lot. It’s better to declare this as a separate “implementation class”, so users get better stack traces on error as they actually see what implementation is used. In short: The release manager in 8.x does not need to care, it’s fully automatic and requires no later version of Java during build. Uwe - Uwe Schindler Achterdiek 19, D-28357 Bremen https://www.thetaphi.de eMail: u...@thetaphi.de From: Adrien Grand Sent: Sunday, August 30, 2020 9:00 AM To: Lucene Dev Subject: Re: Performance in Solr 9 / Java 11 Tomoko is correct, an MR JAR is created not only upon release but also every time you create a lucene-core JAR on branch_8x. On Sun, Aug 30, 2020 at 5:49 AM Tomoko Uchida mailto:tomoko.uchida.1...@gmail.com> > wrote: I believe mr-jar build is enabled in the 8x branch (LUCENE-7966), and the workaround was dropped on the master branch when the minimum java version was bumped up to java 11 (LUCENE-8738); if my understanding is correct. $ jar tf core/lucene-core-8.6.1.jar | grep META-INF/versions META-INF/versions/ META-INF/versions/9/ META-INF/versions/9/org/ META-INF/versions/9/org/apache/ ... $ jar tf core/build/libs/lucene-core-9.0.0-SNAPSHOT.jar | grep META-INF/versions // no outputs 2020年8月30日(日) 6:48 Mike Drob mailto:md...@apache.org> >: Do you know if these mr-jars are built by default as part of the release process? I definitely had no idea about them when doing 8.5.2 and did not even think to verify anything about it. On Sat, Aug 29, 2020 at 4:05 PM Adrien Grand mailto:jpou...@gmail.com> > wrote: It may only be indirectly related to your question, but there is support for vectorized operations of byte[] arrays that was added in JDK 13 (this blog https://richardstartin.github.io/posts/vectorised-byte-operations explains well what it is about) that we started leveraging for compressing terms dictionaries in Lucene 8.5: https://issues.apache.org/jira/browse/LUCENE-4702. I don't know how well this is known but our build also has logic to create multi-release JARs. We don't use it in master today but it's used on branch_8x, which requires Java 8, in order to use APIs that were introduced in Java 9 such as Arrays#mismatch. See the "patch-mr-jar" target in the branch_8x build: https://github.com/apache/lucene-solr/blob/branch_8x/lucene/common-build.xml#L602. So if
Re: Performance in Solr 9 / Java 11
Tomoko is correct, an MR JAR is created not only upon release but also every time you create a lucene-core JAR on branch_8x. On Sun, Aug 30, 2020 at 5:49 AM Tomoko Uchida wrote: > I believe mr-jar build is enabled in the 8x branch (LUCENE-7966), and the > workaround was dropped on the master branch when the minimum java version > was bumped up to java 11 (LUCENE-8738); if my understanding is correct. > > $ jar tf core/lucene-core-8.6.1.jar | grep META-INF/versions > META-INF/versions/ > META-INF/versions/9/ > META-INF/versions/9/org/ > META-INF/versions/9/org/apache/ > ... > > $ jar tf core/build/libs/lucene-core-9.0.0-SNAPSHOT.jar | grep > META-INF/versions > // no outputs > > > > > 2020年8月30日(日) 6:48 Mike Drob : > >> Do you know if these mr-jars are built by default as part of the release >> process? I definitely had no idea about them when doing 8.5.2 and did not >> even think to verify anything about it. >> >> On Sat, Aug 29, 2020 at 4:05 PM Adrien Grand wrote: >> >>> It may only be indirectly related to your question, but there is support >>> for vectorized operations of byte[] arrays that was added in JDK 13 (this >>> blog https://richardstartin.github.io/posts/vectorised-byte-operations >>> explains >>> well what it is about) that we started leveraging for compressing terms >>> dictionaries in Lucene 8.5: >>> https://issues.apache.org/jira/browse/LUCENE-4702. >>> >>> I don't know how well this is known but our build also has logic to >>> create multi-release JARs. We don't use it in master today but it's used on >>> branch_8x, which requires Java 8, in order to use APIs that were introduced >>> in Java 9 such as Arrays#mismatch. See the "patch-mr-jar" target in the >>> branch_8x build: >>> https://github.com/apache/lucene-solr/blob/branch_8x/lucene/common-build.xml#L602. >>> So if APIs that could help performance were introduced in say JDK 15, we >>> might still be able to leverage them in Lucene/Solr 9 using the same >>> mechanism. >>> >>> >>> >>> On Tue, Aug 11, 2020 at 1:12 AM Marcus Eagan >>> wrote: >>> >>>> In my IDE, I have a few profiling tools that I bounce between that I >>>> started using in my work at Lucidworks but I continue to use in my current >>>> work today. I have suspicions that there may be some performance >>>> improvements in Java 11 that we can exploit further. I'm curious as to if >>>> there has been any investigation, possibly Mark Miller or >>>> @u...@thetaphi.de , into performance improvements >>>> specific to the newer version of Java in Master? There are some obvious >>>> ones that we get for free, like a better GC, but curious as to prior work >>>> in this area before publishing anything that might be redundant or >>>> irrelevant. >>>> >>>> Best, >>>> >>>> -- >>>> Marcus Eagan >>>> >>>> >>>> >>>> >>> >>> -- >>> Adrien >>> >>> >>> -- Adrien
Re: Performance in Solr 9 / Java 11
I believe mr-jar build is enabled in the 8x branch (LUCENE-7966), and the workaround was dropped on the master branch when the minimum java version was bumped up to java 11 (LUCENE-8738); if my understanding is correct. $ jar tf core/lucene-core-8.6.1.jar | grep META-INF/versions META-INF/versions/ META-INF/versions/9/ META-INF/versions/9/org/ META-INF/versions/9/org/apache/ ... $ jar tf core/build/libs/lucene-core-9.0.0-SNAPSHOT.jar | grep META-INF/versions // no outputs 2020年8月30日(日) 6:48 Mike Drob : > Do you know if these mr-jars are built by default as part of the release > process? I definitely had no idea about them when doing 8.5.2 and did not > even think to verify anything about it. > > On Sat, Aug 29, 2020 at 4:05 PM Adrien Grand wrote: > >> It may only be indirectly related to your question, but there is support >> for vectorized operations of byte[] arrays that was added in JDK 13 (this >> blog https://richardstartin.github.io/posts/vectorised-byte-operations >> explains >> well what it is about) that we started leveraging for compressing terms >> dictionaries in Lucene 8.5: >> https://issues.apache.org/jira/browse/LUCENE-4702. >> >> I don't know how well this is known but our build also has logic to >> create multi-release JARs. We don't use it in master today but it's used on >> branch_8x, which requires Java 8, in order to use APIs that were introduced >> in Java 9 such as Arrays#mismatch. See the "patch-mr-jar" target in the >> branch_8x build: >> https://github.com/apache/lucene-solr/blob/branch_8x/lucene/common-build.xml#L602. >> So if APIs that could help performance were introduced in say JDK 15, we >> might still be able to leverage them in Lucene/Solr 9 using the same >> mechanism. >> >> >> >> On Tue, Aug 11, 2020 at 1:12 AM Marcus Eagan >> wrote: >> >>> In my IDE, I have a few profiling tools that I bounce between that I >>> started using in my work at Lucidworks but I continue to use in my current >>> work today. I have suspicions that there may be some performance >>> improvements in Java 11 that we can exploit further. I'm curious as to if >>> there has been any investigation, possibly Mark Miller or >>> @u...@thetaphi.de , into performance improvements >>> specific to the newer version of Java in Master? There are some obvious >>> ones that we get for free, like a better GC, but curious as to prior work >>> in this area before publishing anything that might be redundant or >>> irrelevant. >>> >>> Best, >>> >>> -- >>> Marcus Eagan >>> >>> >>> >>> >> >> -- >> Adrien >> >> >>
Re: Performance in Solr 9 / Java 11
Do you know if these mr-jars are built by default as part of the release process? I definitely had no idea about them when doing 8.5.2 and did not even think to verify anything about it. On Sat, Aug 29, 2020 at 4:05 PM Adrien Grand wrote: > It may only be indirectly related to your question, but there is support > for vectorized operations of byte[] arrays that was added in JDK 13 (this > blog https://richardstartin.github.io/posts/vectorised-byte-operations > explains > well what it is about) that we started leveraging for compressing terms > dictionaries in Lucene 8.5: > https://issues.apache.org/jira/browse/LUCENE-4702. > > I don't know how well this is known but our build also has logic to create > multi-release JARs. We don't use it in master today but it's used on > branch_8x, which requires Java 8, in order to use APIs that were introduced > in Java 9 such as Arrays#mismatch. See the "patch-mr-jar" target in the > branch_8x build: > https://github.com/apache/lucene-solr/blob/branch_8x/lucene/common-build.xml#L602. > So if APIs that could help performance were introduced in say JDK 15, we > might still be able to leverage them in Lucene/Solr 9 using the same > mechanism. > > > > On Tue, Aug 11, 2020 at 1:12 AM Marcus Eagan > wrote: > >> In my IDE, I have a few profiling tools that I bounce between that I >> started using in my work at Lucidworks but I continue to use in my current >> work today. I have suspicions that there may be some performance >> improvements in Java 11 that we can exploit further. I'm curious as to if >> there has been any investigation, possibly Mark Miller or >> @u...@thetaphi.de , into performance improvements >> specific to the newer version of Java in Master? There are some obvious >> ones that we get for free, like a better GC, but curious as to prior work >> in this area before publishing anything that might be redundant or >> irrelevant. >> >> Best, >> >> -- >> Marcus Eagan >> >> >> >> > > -- > Adrien > > >
Re: Performance in Solr 9 / Java 11
It may only be indirectly related to your question, but there is support for vectorized operations of byte[] arrays that was added in JDK 13 (this blog https://richardstartin.github.io/posts/vectorised-byte-operations explains well what it is about) that we started leveraging for compressing terms dictionaries in Lucene 8.5: https://issues.apache.org/jira/browse/LUCENE-4702. I don't know how well this is known but our build also has logic to create multi-release JARs. We don't use it in master today but it's used on branch_8x, which requires Java 8, in order to use APIs that were introduced in Java 9 such as Arrays#mismatch. See the "patch-mr-jar" target in the branch_8x build: https://github.com/apache/lucene-solr/blob/branch_8x/lucene/common-build.xml#L602. So if APIs that could help performance were introduced in say JDK 15, we might still be able to leverage them in Lucene/Solr 9 using the same mechanism. On Tue, Aug 11, 2020 at 1:12 AM Marcus Eagan wrote: > In my IDE, I have a few profiling tools that I bounce between that I > started using in my work at Lucidworks but I continue to use in my current > work today. I have suspicions that there may be some performance > improvements in Java 11 that we can exploit further. I'm curious as to if > there has been any investigation, possibly Mark Miller or @u...@thetaphi.de > , into performance improvements specific to the newer > version of Java in Master? There are some obvious ones that we get for > free, like a better GC, but curious as to prior work in this area before > publishing anything that might be redundant or irrelevant. > > Best, > > -- > Marcus Eagan > > -- Adrien
Performance in Solr 9 / Java 11
In my IDE, I have a few profiling tools that I bounce between that I started using in my work at Lucidworks but I continue to use in my current work today. I have suspicions that there may be some performance improvements in Java 11 that we can exploit further. I'm curious as to if there has been any investigation, possibly Mark Miller or @u...@thetaphi.de , into performance improvements specific to the newer version of Java in Master? There are some obvious ones that we get for free, like a better GC, but curious as to prior work in this area before publishing anything that might be redundant or irrelevant. Best, -- Marcus Eagan
[jira] [Closed] (SOLR-13631) Java 11 date parsing causing NPEs
[
https://issues.apache.org/jira/browse/SOLR-13631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Smiley closed SOLR-13631.
---
> Java 11 date parsing causing NPEs
> -
>
> Key: SOLR-13631
> URL: https://issues.apache.org/jira/browse/SOLR-13631
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Ishan Chattopadhyaya
>Assignee: Ishan Chattopadhyaya
>Priority: Major
>
> Across all my machines with Fedora 30, I encounter 20+ test failures on
> master with the following NPEs. Same when starting Solr (it doesn't start up).
> {code}
>[junit4] 2> Caused by: java.time.format.DateTimeParseException: Text
> '2019-07-14T07:09:13.341Z' could not be parsed: null
>[junit4] 2> at
> java.base/java.time.format.DateTimeFormatter.createError(DateTimeFormatter.java:2017)
>[junit4] 2> at
> java.base/java.time.format.DateTimeFormatter.parse(DateTimeFormatter.java:1920)
>[junit4] 2> at
> org.apache.solr.update.processor.ParseDateFieldUpdateProcessorFactory.parseInstant(ParseDateFieldUpdateProcessorFactory.java:230)
>[junit4] 2> at
> org.apache.solr.update.processor.ParseDateFieldUpdateProcessorFactory.validateFormatter(ParseDateFieldUpdateProcessorFactory.java:214)
>[junit4] 2> ... 44 more
> {code}
> Here are my environment details:
> OS: Fedora 30
> Java version:
> {code}
> [ishan@chromebox core] $ java -version
> openjdk version "11.0.3" 2019-04-16
> OpenJDK Runtime Environment 18.9 (build 11.0.3+7)
> OpenJDK 64-Bit Server VM 18.9 (build 11.0.3+7, mixed mode, sharing)
> {code}
> Kernel: 5.1.16-300.fc30.x86_64
> JDK was installed via "sudo dnf install java-11-openjdk-devel".
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (SOLR-13631) Java 11 date parsing causing NPEs
[
https://issues.apache.org/jira/browse/SOLR-13631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Smiley resolved SOLR-13631.
-
Resolution: Duplicate
I dug into this; see my first comment on : SOLR-13606
> Java 11 date parsing causing NPEs
> -
>
> Key: SOLR-13631
> URL: https://issues.apache.org/jira/browse/SOLR-13631
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Ishan Chattopadhyaya
>Assignee: Ishan Chattopadhyaya
>Priority: Major
>
> Across all my machines with Fedora 30, I encounter 20+ test failures on
> master with the following NPEs. Same when starting Solr (it doesn't start up).
> {code}
>[junit4] 2> Caused by: java.time.format.DateTimeParseException: Text
> '2019-07-14T07:09:13.341Z' could not be parsed: null
>[junit4] 2> at
> java.base/java.time.format.DateTimeFormatter.createError(DateTimeFormatter.java:2017)
>[junit4] 2> at
> java.base/java.time.format.DateTimeFormatter.parse(DateTimeFormatter.java:1920)
>[junit4] 2> at
> org.apache.solr.update.processor.ParseDateFieldUpdateProcessorFactory.parseInstant(ParseDateFieldUpdateProcessorFactory.java:230)
>[junit4] 2> at
> org.apache.solr.update.processor.ParseDateFieldUpdateProcessorFactory.validateFormatter(ParseDateFieldUpdateProcessorFactory.java:214)
>[junit4] 2> ... 44 more
> {code}
> Here are my environment details:
> OS: Fedora 30
> Java version:
> {code}
> [ishan@chromebox core] $ java -version
> openjdk version "11.0.3" 2019-04-16
> OpenJDK Runtime Environment 18.9 (build 11.0.3+7)
> OpenJDK 64-Bit Server VM 18.9 (build 11.0.3+7, mixed mode, sharing)
> {code}
> Kernel: 5.1.16-300.fc30.x86_64
> JDK was installed via "sudo dnf install java-11-openjdk-devel".
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13631) Java 11 date parsing causing NPEs
[
https://issues.apache.org/jira/browse/SOLR-13631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16884627#comment-16884627
]
Uwe Schindler commented on SOLR-13631:
--
This is for sure be caused by the fact that some Linux distributions using a
different type of timezone data file shared with other packages. That's a
antipattern but still done by some of them. I think Ubuntu gave up on this.
You'd open an issue at the distribution, showing this with a simple test case.
This is not a problem if Lucene.
> Java 11 date parsing causing NPEs
> -
>
> Key: SOLR-13631
> URL: https://issues.apache.org/jira/browse/SOLR-13631
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Ishan Chattopadhyaya
>Assignee: Ishan Chattopadhyaya
>Priority: Major
>
> Across all my machines with Fedora 30, I encounter 20+ test failures on
> master with the following NPEs. Same when starting Solr (it doesn't start up).
> {code}
>[junit4] 2> Caused by: java.time.format.DateTimeParseException: Text
> '2019-07-14T07:09:13.341Z' could not be parsed: null
>[junit4] 2> at
> java.base/java.time.format.DateTimeFormatter.createError(DateTimeFormatter.java:2017)
>[junit4] 2> at
> java.base/java.time.format.DateTimeFormatter.parse(DateTimeFormatter.java:1920)
>[junit4] 2> at
> org.apache.solr.update.processor.ParseDateFieldUpdateProcessorFactory.parseInstant(ParseDateFieldUpdateProcessorFactory.java:230)
>[junit4] 2> at
> org.apache.solr.update.processor.ParseDateFieldUpdateProcessorFactory.validateFormatter(ParseDateFieldUpdateProcessorFactory.java:214)
>[junit4] 2> ... 44 more
> {code}
> Here are my environment details:
> OS: Fedora 30
> Java version:
> {code}
> [ishan@chromebox core] $ java -version
> openjdk version "11.0.3" 2019-04-16
> OpenJDK Runtime Environment 18.9 (build 11.0.3+7)
> OpenJDK 64-Bit Server VM 18.9 (build 11.0.3+7, mixed mode, sharing)
> {code}
> Kernel: 5.1.16-300.fc30.x86_64
> JDK was installed via "sudo dnf install java-11-openjdk-devel".
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-13631) Java 11 date parsing causing NPEs
[
https://issues.apache.org/jira/browse/SOLR-13631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ishan Chattopadhyaya updated SOLR-13631:
Description:
Across all my machines with Fedora 30, I encounter 20+ test failures on master
with the following NPEs. Same when starting Solr (it doesn't start up).
{code}
[junit4] 2> Caused by: java.time.format.DateTimeParseException: Text
'2019-07-14T07:09:13.341Z' could not be parsed: null
[junit4] 2>at
java.base/java.time.format.DateTimeFormatter.createError(DateTimeFormatter.java:2017)
[junit4] 2>at
java.base/java.time.format.DateTimeFormatter.parse(DateTimeFormatter.java:1920)
[junit4] 2>at
org.apache.solr.update.processor.ParseDateFieldUpdateProcessorFactory.parseInstant(ParseDateFieldUpdateProcessorFactory.java:230)
[junit4] 2>at
org.apache.solr.update.processor.ParseDateFieldUpdateProcessorFactory.validateFormatter(ParseDateFieldUpdateProcessorFactory.java:214)
[junit4] 2>... 44 more
{code}
Here are my environment details:
OS: Fedora 30
Java version:
{code}
[ishan@chromebox core] $ java -version
openjdk version "11.0.3" 2019-04-16
OpenJDK Runtime Environment 18.9 (build 11.0.3+7)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.3+7, mixed mode, sharing)
{code}
Kernel: 5.1.16-300.fc30.x86_64
JDK was installed via "sudo dnf install java-11-openjdk-devel".
was:
Across all my machines with Fedora 30, I encounter 20+ test failures on master
with the following NPEs. Same when starting Solr (it doesn't start up).
{code}
[junit4] 2> Caused by: java.time.format.DateTimeParseException: Text
'2019-07-14T07:09:13.341Z' could not be parsed: null
[junit4] 2>at
java.base/java.time.format.DateTimeFormatter.createError(DateTimeFormatter.java:2017)
[junit4] 2>at
java.base/java.time.format.DateTimeFormatter.parse(DateTimeFormatter.java:1920)
[junit4] 2>at
org.apache.solr.update.processor.ParseDateFieldUpdateProcessorFactory.parseInstant(ParseDateFieldUpdateProcessorFactory.java:230)
[junit4] 2>at
org.apache.solr.update.processor.ParseDateFieldUpdateProcessorFactory.validateFormatter(ParseDateFieldUpdateProcessorFactory.java:214)
[junit4] 2>... 44 more
{code}
Here are my environment details:
OS: Fedora 30
Java version:
{code}
[ishan@chromebox core] $ java -version
openjdk version "11.0.3" 2019-04-16
OpenJDK Runtime Environment 18.9 (build 11.0.3+7)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.3+7, mixed mode, sharing)
{code}
Kernel: 5.1.16-300.fc30.x86_64
> Java 11 date parsing causing NPEs
> -
>
> Key: SOLR-13631
> URL: https://issues.apache.org/jira/browse/SOLR-13631
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Ishan Chattopadhyaya
>Assignee: Ishan Chattopadhyaya
>Priority: Major
>
> Across all my machines with Fedora 30, I encounter 20+ test failures on
> master with the following NPEs. Same when starting Solr (it doesn't start up).
> {code}
>[junit4] 2> Caused by: java.time.format.DateTimeParseException: Text
> '2019-07-14T07:09:13.341Z' could not be parsed: null
>[junit4] 2> at
> java.base/java.time.format.DateTimeFormatter.createError(DateTimeFormatter.java:2017)
>[junit4] 2> at
> java.base/java.time.format.DateTimeFormatter.parse(DateTimeFormatter.java:1920)
>[junit4] 2> at
> org.apache.solr.update.processor.ParseDateFieldUpdateProcessorFactory.parseInstant(ParseDateFieldUpdateProcessorFactory.java:230)
>[junit4] 2> at
> org.apache.solr.update.processor.ParseDateFieldUpdateProcessorFactory.validateFormatter(ParseDateFieldUpdateProcessorFactory.java:214)
>[junit4] 2> ... 44 more
> {code}
> Here are my environment details:
> OS: Fedora 30
> Java version:
> {code}
> [ishan@chromebox core] $ java -version
> openjdk version "11.0.3" 2019-04-16
> OpenJDK Runtime Environment 18.9 (build 11.0.3+7)
> OpenJDK 64-Bit Server VM 18.9 (build 11.0.3+7, mixed mode, sharing)
> {code}
> Kernel: 5.1.16-300.fc30.x86_64
> JDK was installed via "sudo dnf install java-11-openjdk-devel".
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-13631) Java 11 date parsing causing NPEs
[
https://issues.apache.org/jira/browse/SOLR-13631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ishan Chattopadhyaya updated SOLR-13631:
Description:
Across all my machines with Fedora 30, I encounter 20+ test failures on master
with the following NPEs. Same when starting Solr (it doesn't start up).
{code}
[junit4] 2> Caused by: java.time.format.DateTimeParseException: Text
'2019-07-14T07:09:13.341Z' could not be parsed: null
[junit4] 2>at
java.base/java.time.format.DateTimeFormatter.createError(DateTimeFormatter.java:2017)
[junit4] 2>at
java.base/java.time.format.DateTimeFormatter.parse(DateTimeFormatter.java:1920)
[junit4] 2>at
org.apache.solr.update.processor.ParseDateFieldUpdateProcessorFactory.parseInstant(ParseDateFieldUpdateProcessorFactory.java:230)
[junit4] 2>at
org.apache.solr.update.processor.ParseDateFieldUpdateProcessorFactory.validateFormatter(ParseDateFieldUpdateProcessorFactory.java:214)
[junit4] 2>... 44 more
{code}
Here are my environment details:
OS: Fedora 30
Java version:
{code}
[ishan@chromebox core] $ java -version
openjdk version "11.0.3" 2019-04-16
OpenJDK Runtime Environment 18.9 (build 11.0.3+7)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.3+7, mixed mode, sharing)
{code}
Kernel: 5.1.16-300.fc30.x86_64
was:
Across all my machines with Fedora 30, I encounter 20+ test failures with the
following NPEs. Same when starting Solr (it doesn't start up).
{code}
[junit4] 2> Caused by: java.time.format.DateTimeParseException: Text
'2019-07-14T07:09:13.341Z' could not be parsed: null
[junit4] 2>at
java.base/java.time.format.DateTimeFormatter.createError(DateTimeFormatter.java:2017)
[junit4] 2>at
java.base/java.time.format.DateTimeFormatter.parse(DateTimeFormatter.java:1920)
[junit4] 2>at
org.apache.solr.update.processor.ParseDateFieldUpdateProcessorFactory.parseInstant(ParseDateFieldUpdateProcessorFactory.java:230)
[junit4] 2>at
org.apache.solr.update.processor.ParseDateFieldUpdateProcessorFactory.validateFormatter(ParseDateFieldUpdateProcessorFactory.java:214)
[junit4] 2>... 44 more
{code}
Here are my environment details:
OS: Fedora 30
Java version:
{code}
[ishan@chromebox core] $ java -version
openjdk version "11.0.3" 2019-04-16
OpenJDK Runtime Environment 18.9 (build 11.0.3+7)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.3+7, mixed mode, sharing)
{code}
Kernel: 5.1.16-300.fc30.x86_64
> Java 11 date parsing causing NPEs
> -
>
> Key: SOLR-13631
> URL: https://issues.apache.org/jira/browse/SOLR-13631
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Ishan Chattopadhyaya
>Assignee: Ishan Chattopadhyaya
>Priority: Major
>
> Across all my machines with Fedora 30, I encounter 20+ test failures on
> master with the following NPEs. Same when starting Solr (it doesn't start up).
> {code}
>[junit4] 2> Caused by: java.time.format.DateTimeParseException: Text
> '2019-07-14T07:09:13.341Z' could not be parsed: null
>[junit4] 2> at
> java.base/java.time.format.DateTimeFormatter.createError(DateTimeFormatter.java:2017)
>[junit4] 2> at
> java.base/java.time.format.DateTimeFormatter.parse(DateTimeFormatter.java:1920)
>[junit4] 2> at
> org.apache.solr.update.processor.ParseDateFieldUpdateProcessorFactory.parseInstant(ParseDateFieldUpdateProcessorFactory.java:230)
>[junit4] 2> at
> org.apache.solr.update.processor.ParseDateFieldUpdateProcessorFactory.validateFormatter(ParseDateFieldUpdateProcessorFactory.java:214)
>[junit4] 2> ... 44 more
> {code}
> Here are my environment details:
> OS: Fedora 30
> Java version:
> {code}
> [ishan@chromebox core] $ java -version
> openjdk version "11.0.3" 2019-04-16
> OpenJDK Runtime Environment 18.9 (build 11.0.3+7)
> OpenJDK 64-Bit Server VM 18.9 (build 11.0.3+7, mixed mode, sharing)
> {code}
> Kernel: 5.1.16-300.fc30.x86_64
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-13631) Java 11 date parsing causing NPEs
Ishan Chattopadhyaya created SOLR-13631:
---
Summary: Java 11 date parsing causing NPEs
Key: SOLR-13631
URL: https://issues.apache.org/jira/browse/SOLR-13631
Project: Solr
Issue Type: Improvement
Security Level: Public (Default Security Level. Issues are Public)
Reporter: Ishan Chattopadhyaya
Assignee: Ishan Chattopadhyaya
Across all my machines with Fedora 30, I encounter 20+ test failures with the
following NPEs. Same when starting Solr (it doesn't start up).
{code}
[junit4] 2> Caused by: java.time.format.DateTimeParseException: Text
'2019-07-14T07:09:13.341Z' could not be parsed: null
[junit4] 2>at
java.base/java.time.format.DateTimeFormatter.createError(DateTimeFormatter.java:2017)
[junit4] 2>at
java.base/java.time.format.DateTimeFormatter.parse(DateTimeFormatter.java:1920)
[junit4] 2>at
org.apache.solr.update.processor.ParseDateFieldUpdateProcessorFactory.parseInstant(ParseDateFieldUpdateProcessorFactory.java:230)
[junit4] 2>at
org.apache.solr.update.processor.ParseDateFieldUpdateProcessorFactory.validateFormatter(ParseDateFieldUpdateProcessorFactory.java:214)
[junit4] 2>... 44 more
{code}
Here are my environment details:
OS: Fedora 30
Java version:
{code}
[ishan@chromebox core] $ java -version
openjdk version "11.0.3" 2019-04-16
OpenJDK Runtime Environment 18.9 (build 11.0.3+7)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.3+7, mixed mode, sharing)
{code}
Kernel: 5.1.16-300.fc30.x86_64
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-12988) Skip running tests with SSL on Java 11 to 11.0.2
[
https://issues.apache.org/jira/browse/SOLR-12988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hoss Man updated SOLR-12988:
Attachment: SOLR-12988.patch
Status: Open (was: Open)
I'm attaching a patch I've worked up which:
* Reinstates SSL testing, both explicit and randomized
* modifies SSLTestConfig constructor to {{assumeSslIsSafeToTest()}}
* implements {{assumeSslIsSafeToTest()}} using a (safe to backport) regex
based helper method that is called on the JVM_VERSION
** the regex is based on my research as to the SSL bugs we've definitely seen
in past jenkins builds, and which JDK builds they exist in
* includes unit of the regex helper method to try and ensure no false
negatives/positives
...I think this is good to go once we get the JDKs upgraded on our jenkins
machines, but I'm still testing & I'd appreciate sanity checking eyeballs from
other folks.
(in particular does anyone know how to find a list of all "official" ea builds
from OpenJDK? I _believe_ "13-ea+26" was the first "13-ea" release, which
should means only the pre-ea "testing" versions of 13 are affected by
JDK-8224829, but I don't actually know how to verify that)
> Skip running tests with SSL on Java 11 to 11.0.2
>
>
> Key: SOLR-12988
> URL: https://issues.apache.org/jira/browse/SOLR-12988
> Project: Solr
> Issue Type: Test
>Reporter: Hoss Man
>Assignee: Cao Manh Dat
>Priority: Major
> Labels: Java11, Java12
> Attachments: SOLR-12988.patch, SOLR-12988.patch, SOLR-13413.patch
>
>
> HTTPCLIENT-1967 indicates that HttpClient can't be used properly with
> TLSv1.3. It caused some test failures below, therefore we should skip running
> tests with SSL on Java 11 to 11.0.2.
> TestMiniSolrCloudClusterSSL.testSslWithCheckPeerName seems to fail 100% of
> the time when run with java11 (or java12), regardless of seed, on both master
> & 7x.
> The nature of the problem and the way our htp stack works suggests it *may*
> ultimately be a jetty bug (perhaps related to [jetty
> issue#2711|https://github.com/eclipse/jetty.project/issues/2711]?)
> *HOWEVER* ... as far as i can tell, whatever the root cause is, seems to have
> been fixed on the {{jira/http2}} branch (as of
> 52bc163dc1804c31af09c1fba99647005da415ad) which should hopefully be getting
> merged to master soon.
> Filing this issue largely for tracking purpose, although we may also want to
> use it for discussions/considerations of other backports/fixes to 7x
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12988) Skip running tests with SSL on Java 11 to 11.0.2
[
https://issues.apache.org/jira/browse/SOLR-12988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16869251#comment-16869251
]
Cao Manh Dat commented on SOLR-12988:
-
Kinda shame, thanks Hoss!
> Skip running tests with SSL on Java 11 to 11.0.2
>
>
> Key: SOLR-12988
> URL: https://issues.apache.org/jira/browse/SOLR-12988
> Project: Solr
> Issue Type: Test
>Reporter: Hoss Man
>Assignee: Cao Manh Dat
>Priority: Major
> Labels: Java11, Java12
> Attachments: SOLR-12988.patch, SOLR-13413.patch
>
>
> HTTPCLIENT-1967 indicates that HttpClient can't be used properly with
> TLSv1.3. It caused some test failures below, therefore we should skip running
> tests with SSL on Java 11 to 11.0.2.
> TestMiniSolrCloudClusterSSL.testSslWithCheckPeerName seems to fail 100% of
> the time when run with java11 (or java12), regardless of seed, on both master
> & 7x.
> The nature of the problem and the way our htp stack works suggests it *may*
> ultimately be a jetty bug (perhaps related to [jetty
> issue#2711|https://github.com/eclipse/jetty.project/issues/2711]?)
> *HOWEVER* ... as far as i can tell, whatever the root cause is, seems to have
> been fixed on the {{jira/http2}} branch (as of
> 52bc163dc1804c31af09c1fba99647005da415ad) which should hopefully be getting
> merged to master soon.
> Filing this issue largely for tracking purpose, although we may also want to
> use it for discussions/considerations of other backports/fixes to 7x
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12988) Skip running tests with SSL on Java 11 to 11.0.2
[
https://issues.apache.org/jira/browse/SOLR-12988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16869012#comment-16869012
]
ASF subversion and git services commented on SOLR-12988:
Commit 456bc4dbf978f9a7c2bc16fc009057683bd7b2ca in lucene-solr's branch
refs/heads/branch_8x from Chris M. Hostetter
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=456bc4d ]
SOLR-12988: Fix Revert
an (erroneous) assigment to useSsl was left in the constructor after the
(intended) conditional assigment, rendering the conditional logic useless
(cherry picked from commit c8c2f2f25b28da694fae88868b12347bc5a2393c)
> Skip running tests with SSL on Java 11 to 11.0.2
>
>
> Key: SOLR-12988
> URL: https://issues.apache.org/jira/browse/SOLR-12988
> Project: Solr
> Issue Type: Test
>Reporter: Hoss Man
>Assignee: Cao Manh Dat
>Priority: Major
> Labels: Java11, Java12
> Attachments: SOLR-12988.patch, SOLR-13413.patch
>
>
> HTTPCLIENT-1967 indicates that HttpClient can't be used properly with
> TLSv1.3. It caused some test failures below, therefore we should skip running
> tests with SSL on Java 11 to 11.0.2.
> TestMiniSolrCloudClusterSSL.testSslWithCheckPeerName seems to fail 100% of
> the time when run with java11 (or java12), regardless of seed, on both master
> & 7x.
> The nature of the problem and the way our htp stack works suggests it *may*
> ultimately be a jetty bug (perhaps related to [jetty
> issue#2711|https://github.com/eclipse/jetty.project/issues/2711]?)
> *HOWEVER* ... as far as i can tell, whatever the root cause is, seems to have
> been fixed on the {{jira/http2}} branch (as of
> 52bc163dc1804c31af09c1fba99647005da415ad) which should hopefully be getting
> merged to master soon.
> Filing this issue largely for tracking purpose, although we may also want to
> use it for discussions/considerations of other backports/fixes to 7x
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12988) Skip running tests with SSL on Java 11 to 11.0.2
[
https://issues.apache.org/jira/browse/SOLR-12988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16869013#comment-16869013
]
ASF subversion and git services commented on SOLR-12988:
Commit c8c2f2f25b28da694fae88868b12347bc5a2393c in lucene-solr's branch
refs/heads/master from Chris M. Hostetter
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=c8c2f2f ]
SOLR-12988: Fix Revert
an (erroneous) assigment to useSsl was left in the constructor after the
(intended) conditional assigment, rendering the conditional logic useless
> Skip running tests with SSL on Java 11 to 11.0.2
>
>
> Key: SOLR-12988
> URL: https://issues.apache.org/jira/browse/SOLR-12988
> Project: Solr
> Issue Type: Test
>Reporter: Hoss Man
>Assignee: Cao Manh Dat
>Priority: Major
> Labels: Java11, Java12
> Attachments: SOLR-12988.patch, SOLR-13413.patch
>
>
> HTTPCLIENT-1967 indicates that HttpClient can't be used properly with
> TLSv1.3. It caused some test failures below, therefore we should skip running
> tests with SSL on Java 11 to 11.0.2.
> TestMiniSolrCloudClusterSSL.testSslWithCheckPeerName seems to fail 100% of
> the time when run with java11 (or java12), regardless of seed, on both master
> & 7x.
> The nature of the problem and the way our htp stack works suggests it *may*
> ultimately be a jetty bug (perhaps related to [jetty
> issue#2711|https://github.com/eclipse/jetty.project/issues/2711]?)
> *HOWEVER* ... as far as i can tell, whatever the root cause is, seems to have
> been fixed on the {{jira/http2}} branch (as of
> 52bc163dc1804c31af09c1fba99647005da415ad) which should hopefully be getting
> merged to master soon.
> Filing this issue largely for tracking purpose, although we may also want to
> use it for discussions/considerations of other backports/fixes to 7x
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12988) Skip running tests with SSL on Java 11 to 11.0.2
[
https://issues.apache.org/jira/browse/SOLR-12988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16867963#comment-16867963
]
ASF subversion and git services commented on SOLR-12988:
Commit 150e4f9863147354a489bbd62519bf2e586f41b9 in lucene-solr's branch
refs/heads/branch_8x from Cao Manh Dat
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=150e4f9 ]
SOLR-12988: Revert changes
> Skip running tests with SSL on Java 11 to 11.0.2
>
>
> Key: SOLR-12988
> URL: https://issues.apache.org/jira/browse/SOLR-12988
> Project: Solr
> Issue Type: Test
>Reporter: Hoss Man
>Assignee: Cao Manh Dat
>Priority: Major
> Labels: Java11, Java12
> Attachments: SOLR-12988.patch, SOLR-13413.patch
>
>
> HTTPCLIENT-1967 indicates that HttpClient can't be used properly with
> TLSv1.3. It caused some test failures below, therefore we should skip running
> tests with SSL on Java 11 to 11.0.2.
> TestMiniSolrCloudClusterSSL.testSslWithCheckPeerName seems to fail 100% of
> the time when run with java11 (or java12), regardless of seed, on both master
> & 7x.
> The nature of the problem and the way our htp stack works suggests it *may*
> ultimately be a jetty bug (perhaps related to [jetty
> issue#2711|https://github.com/eclipse/jetty.project/issues/2711]?)
> *HOWEVER* ... as far as i can tell, whatever the root cause is, seems to have
> been fixed on the {{jira/http2}} branch (as of
> 52bc163dc1804c31af09c1fba99647005da415ad) which should hopefully be getting
> merged to master soon.
> Filing this issue largely for tracking purpose, although we may also want to
> use it for discussions/considerations of other backports/fixes to 7x
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12988) Skip running tests with SSL on Java 11 to 11.0.2
[
https://issues.apache.org/jira/browse/SOLR-12988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16867962#comment-16867962
]
ASF subversion and git services commented on SOLR-12988:
Commit e3752e87d005d407108f95e72b14c82d60f1c082 in lucene-solr's branch
refs/heads/master from Cao Manh Dat
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=e3752e8 ]
SOLR-12988: Revert changes
> Skip running tests with SSL on Java 11 to 11.0.2
>
>
> Key: SOLR-12988
> URL: https://issues.apache.org/jira/browse/SOLR-12988
> Project: Solr
> Issue Type: Test
>Reporter: Hoss Man
>Assignee: Cao Manh Dat
>Priority: Major
> Labels: Java11, Java12
> Attachments: SOLR-12988.patch, SOLR-13413.patch
>
>
> HTTPCLIENT-1967 indicates that HttpClient can't be used properly with
> TLSv1.3. It caused some test failures below, therefore we should skip running
> tests with SSL on Java 11 to 11.0.2.
> TestMiniSolrCloudClusterSSL.testSslWithCheckPeerName seems to fail 100% of
> the time when run with java11 (or java12), regardless of seed, on both master
> & 7x.
> The nature of the problem and the way our htp stack works suggests it *may*
> ultimately be a jetty bug (perhaps related to [jetty
> issue#2711|https://github.com/eclipse/jetty.project/issues/2711]?)
> *HOWEVER* ... as far as i can tell, whatever the root cause is, seems to have
> been fixed on the {{jira/http2}} branch (as of
> 52bc163dc1804c31af09c1fba99647005da415ad) which should hopefully be getting
> merged to master soon.
> Filing this issue largely for tracking purpose, although we may also want to
> use it for discussions/considerations of other backports/fixes to 7x
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-12988) Skip running tests with SSL on Java 11 to 11.0.2
[
https://issues.apache.org/jira/browse/SOLR-12988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cao Manh Dat updated SOLR-12988:
Attachment: (was: SOLR-12988.patch)
> Skip running tests with SSL on Java 11 to 11.0.2
>
>
> Key: SOLR-12988
> URL: https://issues.apache.org/jira/browse/SOLR-12988
> Project: Solr
> Issue Type: Test
>Reporter: Hoss Man
>Assignee: Cao Manh Dat
>Priority: Major
> Labels: Java11, Java12
> Attachments: SOLR-12988.patch, SOLR-13413.patch
>
>
> HTTPCLIENT-1967 indicates that HttpClient can't be used properly with
> TLSv1.3. It caused some test failures below, therefore we should skip running
> tests with SSL on Java 11 to 11.0.2.
> TestMiniSolrCloudClusterSSL.testSslWithCheckPeerName seems to fail 100% of
> the time when run with java11 (or java12), regardless of seed, on both master
> & 7x.
> The nature of the problem and the way our htp stack works suggests it *may*
> ultimately be a jetty bug (perhaps related to [jetty
> issue#2711|https://github.com/eclipse/jetty.project/issues/2711]?)
> *HOWEVER* ... as far as i can tell, whatever the root cause is, seems to have
> been fixed on the {{jira/http2}} branch (as of
> 52bc163dc1804c31af09c1fba99647005da415ad) which should hopefully be getting
> merged to master soon.
> Filing this issue largely for tracking purpose, although we may also want to
> use it for discussions/considerations of other backports/fixes to 7x
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-12988) Skip running tests with SSL on Java 11 to 11.0.2
[
https://issues.apache.org/jira/browse/SOLR-12988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cao Manh Dat updated SOLR-12988:
Attachment: SOLR-12988.patch
> Skip running tests with SSL on Java 11 to 11.0.2
>
>
> Key: SOLR-12988
> URL: https://issues.apache.org/jira/browse/SOLR-12988
> Project: Solr
> Issue Type: Test
>Reporter: Hoss Man
>Assignee: Cao Manh Dat
>Priority: Major
> Labels: Java11, Java12
> Attachments: SOLR-12988.patch, SOLR-12988.patch, SOLR-13413.patch
>
>
> HTTPCLIENT-1967 indicates that HttpClient can't be used properly with
> TLSv1.3. It caused some test failures below, therefore we should skip running
> tests with SSL on Java 11 to 11.0.2.
> TestMiniSolrCloudClusterSSL.testSslWithCheckPeerName seems to fail 100% of
> the time when run with java11 (or java12), regardless of seed, on both master
> & 7x.
> The nature of the problem and the way our htp stack works suggests it *may*
> ultimately be a jetty bug (perhaps related to [jetty
> issue#2711|https://github.com/eclipse/jetty.project/issues/2711]?)
> *HOWEVER* ... as far as i can tell, whatever the root cause is, seems to have
> been fixed on the {{jira/http2}} branch (as of
> 52bc163dc1804c31af09c1fba99647005da415ad) which should hopefully be getting
> merged to master soon.
> Filing this issue largely for tracking purpose, although we may also want to
> use it for discussions/considerations of other backports/fixes to 7x
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12988) Skip running tests with SSL on Java 11 to 11.0.2
[
https://issues.apache.org/jira/browse/SOLR-12988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16867957#comment-16867957
]
Cao Manh Dat commented on SOLR-12988:
-
{quote}
That's very suprising since java.lang.Runtime.Version didn't exist until
java9...
{quote}
Hmm, now you mention, it seems weird too me as well. But the {{ant compile}}
success at my machine with java 1.8.0_191.
{quote}
But to re-iterate: even if/once we have a "good" patch (that can be backported
to 8x) i STILL think we need to revert all of these changes, and not move
forward with any patch (or existing changes to allow SSL testing on java11)
until the jenkins boxes are running 11.0.3.
{quote}
Ok that makes sense to me, I will revert changes and let you handle this issue,
since both of us understand the problem, we just need a way to handle the test
now which it seems you will handle it properly and more stable than me.
Just a note here, since the awaitsFix of Mark 6 months ago + the enforcment of
using Java 11 on master. None tests on master were run with HTTPS. Hoping that
this issue can move fast.
> Skip running tests with SSL on Java 11 to 11.0.2
>
>
> Key: SOLR-12988
> URL: https://issues.apache.org/jira/browse/SOLR-12988
> Project: Solr
> Issue Type: Test
>Reporter: Hoss Man
>Assignee: Cao Manh Dat
>Priority: Major
> Labels: Java11, Java12
> Attachments: SOLR-12988.patch, SOLR-13413.patch
>
>
> HTTPCLIENT-1967 indicates that HttpClient can't be used properly with
> TLSv1.3. It caused some test failures below, therefore we should skip running
> tests with SSL on Java 11 to 11.0.2.
> TestMiniSolrCloudClusterSSL.testSslWithCheckPeerName seems to fail 100% of
> the time when run with java11 (or java12), regardless of seed, on both master
> & 7x.
> The nature of the problem and the way our htp stack works suggests it *may*
> ultimately be a jetty bug (perhaps related to [jetty
> issue#2711|https://github.com/eclipse/jetty.project/issues/2711]?)
> *HOWEVER* ... as far as i can tell, whatever the root cause is, seems to have
> been fixed on the {{jira/http2}} branch (as of
> 52bc163dc1804c31af09c1fba99647005da415ad) which should hopefully be getting
> merged to master soon.
> Filing this issue largely for tracking purpose, although we may also want to
> use it for discussions/considerations of other backports/fixes to 7x
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12988) Skip running tests with SSL on Java 11 to 11.0.2
[
https://issues.apache.org/jira/browse/SOLR-12988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16867933#comment-16867933
]
Hoss Man commented on SOLR-12988:
-
bq. Firstly, I don't see any problem with compiling branch_8x on java8 locally.
That's very suprising since java.lang.Runtime.Version didn't exist until
java9...
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-8.x-Linux/732/
(and locally for me running w/ java8, java11, etc...)
{noformat}
[javac] Compiling 55 source files to
/home/jenkins/workspace/Lucene-Solr-8.x-Linux/solr/build/solr-test-framework/classes/java
[javac]
/home/jenkins/workspace/Lucene-Solr-8.x-Linux/solr/test-framework/src/java/org/apache/solr/util/SSLTestConfig.java:108:
error: cannot
find symbol
[javac] if (Constants.JRE_IS_MINIMUM_JAVA11 &&
Runtime.version().compareTo(Runtime.Version.parse("11.0.3")) < 0) {
[javac] ^
[javac] symbol: method version()
[javac] location: class Runtime
[javac]
/home/jenkins/workspace/Lucene-Solr-8.x-Linux/solr/test-framework/src/java/org/apache/solr/util/SSLTestConfig.java:108:
error: cannot
find symbol
[javac] if (Constants.JRE_IS_MINIMUM_JAVA11 &&
Runtime.version().compareTo(Runtime.Version.parse("11.0.3")) < 0) {
[javac]
^
[javac] symbol: variable Version
[javac] location: class Runtime
{noformat}
bq. Secondly, I rushed the commit to revert changes back to it used to be +
minimal checks so Jenkins will become happy. ...
No, you didn't "revert" anything back to "minimal checks" ... you "fixed
forward" to *different* checks. That's not the same as "reverting" to what we
had before: which was a known and relatively stable state.
We're talking about very fundemental changes to test infrastructure that impact
over a thousand tests -- we should be hammering on these locally for a *LONG*
time, with differnet JVMs, on both branches, before pushing forward any changes
-- and as i've suggested repeatedly: we should not push *ANY* serious changes
-- particularly jvm version dependent behavior cahnages -- until the jenkins
boxes are running 11.0.3, so _they_ are ready to hammer on whatever changes we
commit, in case there are *OTHER* SSL bugs we haven't found or considered yet.
Even if we have rock solid "don't use ssl unless 11.0.3" logic in our tests,
that's a potential time bomb that might go off w/some unknown bug once jenkins
is upgraded -- we shouldn't just leave that in our code waiting to see what
happens if/when you or I are offline for an extended period of time when
jenkins gets upgraded.
bq. ...does that most recent patch matched to your idea?
Well, besides the fact that it still uses Runtime.Version (so won't compile on
branch_8x) i guess it's ok? but i'm still a little worried about having it only
be conditional on the java versoin and not on the JVM impl...
bq. ...that could still lead to weird situations if/when people use non-OpenJDK
based JVMs, or potentially use their own builds of Open-JDK that they've
patched, etc...
Ideally i'd much rather have have a quick and dirty helper method we could use
to ask "does this jvm manifest this bug", that actually uses the SSLEngine to
see if these bugs get triggered -- but that's fairly non trivial, so i guess
just checking the version info is fine (although i think we should almost
certainly be checking the java vendor in addition to the java version)
But to re-iterate: even if/once we have a "good" patch (that can be backported
to 8x) i *STILL* think we need to revert all of these changes, and not move
forward with any patch (or existing changes to allow SSL testing on java11)
until the jenkins boxes are running 11.0.3.
> Skip running tests with SSL on Java 11 to 11.0.2
>
>
> Key: SOLR-12988
> URL: https://issues.apache.org/jira/browse/SOLR-12988
> Project: Solr
> Issue Type: Test
>Reporter: Hoss Man
>Assignee: Cao Manh Dat
>Priority: Major
> Labels: Java11, Java12
> Attachments: SOLR-12988.patch, SOLR-13413.patch
>
>
> HTTPCLIENT-1967 indicates that HttpClient can't be used properly with
> TLSv1.3. It caused some test failures below, therefore we should skip running
> tests with SSL on Java 11 to 11.0.2.
> TestMiniSolrCloudClusterSSL.testSslWithCheckPeerName seems to fail 100% of
> the time when run with java11 (or java12), regardless of seed, on both master
> & 7x.
> The nature of the problem and the way our htp stack works suggests it
[jira] [Commented] (SOLR-12988) Skip running tests with SSL on Java 11 to 11.0.2
[
https://issues.apache.org/jira/browse/SOLR-12988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16867897#comment-16867897
]
Cao Manh Dat commented on SOLR-12988:
-
[~hossman] does that most recent patch matched to your idea?
> Skip running tests with SSL on Java 11 to 11.0.2
>
>
> Key: SOLR-12988
> URL: https://issues.apache.org/jira/browse/SOLR-12988
> Project: Solr
> Issue Type: Test
>Reporter: Hoss Man
>Assignee: Cao Manh Dat
>Priority: Major
> Labels: Java11, Java12
> Attachments: SOLR-12988.patch, SOLR-13413.patch
>
>
> HTTPCLIENT-1967 indicates that HttpClient can't be used properly with
> TLSv1.3. It caused some test failures below, therefore we should skip running
> tests with SSL on Java 11 to 11.0.2.
> TestMiniSolrCloudClusterSSL.testSslWithCheckPeerName seems to fail 100% of
> the time when run with java11 (or java12), regardless of seed, on both master
> & 7x.
> The nature of the problem and the way our htp stack works suggests it *may*
> ultimately be a jetty bug (perhaps related to [jetty
> issue#2711|https://github.com/eclipse/jetty.project/issues/2711]?)
> *HOWEVER* ... as far as i can tell, whatever the root cause is, seems to have
> been fixed on the {{jira/http2}} branch (as of
> 52bc163dc1804c31af09c1fba99647005da415ad) which should hopefully be getting
> merged to master soon.
> Filing this issue largely for tracking purpose, although we may also want to
> use it for discussions/considerations of other backports/fixes to 7x
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-12988) Skip running tests with SSL on Java 11 to 11.0.2
[
https://issues.apache.org/jira/browse/SOLR-12988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cao Manh Dat updated SOLR-12988:
Attachment: SOLR-12988.patch
> Skip running tests with SSL on Java 11 to 11.0.2
>
>
> Key: SOLR-12988
> URL: https://issues.apache.org/jira/browse/SOLR-12988
> Project: Solr
> Issue Type: Test
>Reporter: Hoss Man
>Assignee: Cao Manh Dat
>Priority: Major
> Labels: Java11, Java12
> Attachments: SOLR-12988.patch, SOLR-13413.patch
>
>
> HTTPCLIENT-1967 indicates that HttpClient can't be used properly with
> TLSv1.3. It caused some test failures below, therefore we should skip running
> tests with SSL on Java 11 to 11.0.2.
> TestMiniSolrCloudClusterSSL.testSslWithCheckPeerName seems to fail 100% of
> the time when run with java11 (or java12), regardless of seed, on both master
> & 7x.
> The nature of the problem and the way our htp stack works suggests it *may*
> ultimately be a jetty bug (perhaps related to [jetty
> issue#2711|https://github.com/eclipse/jetty.project/issues/2711]?)
> *HOWEVER* ... as far as i can tell, whatever the root cause is, seems to have
> been fixed on the {{jira/http2}} branch (as of
> 52bc163dc1804c31af09c1fba99647005da415ad) which should hopefully be getting
> merged to master soon.
> Filing this issue largely for tracking purpose, although we may also want to
> use it for discussions/considerations of other backports/fixes to 7x
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12988) Skip running tests with SSL on Java 11 to 11.0.2
[
https://issues.apache.org/jira/browse/SOLR-12988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16867891#comment-16867891
]
Cao Manh Dat commented on SOLR-12988:
-
Hi [~hossman] I kinda think we are misunderstanding each others here.
{quote}
that doesn't even fucking compile with java8
{quote}
Firstly, I don't see any problem with compiling branch_8x on java8 locally.
Secondly, I rushed the commit to revert changes back to it used to be + minimal
checks so Jenkins will become happy. THEN we can continue doing what we should
do after discussion.
Thirdly, I quite understand your idea now, so we should call {{assumFalse()}}
in {{SSLTestConfig}} whenever SSL is enabled on Java 11 -> 11.0.2. Luckily that
it still matches with issue's name and we do not need to change issue name
(again).
> Skip running tests with SSL on Java 11 to 11.0.2
>
>
> Key: SOLR-12988
> URL: https://issues.apache.org/jira/browse/SOLR-12988
> Project: Solr
> Issue Type: Test
>Reporter: Hoss Man
>Assignee: Cao Manh Dat
>Priority: Major
> Labels: Java11, Java12
> Attachments: SOLR-13413.patch
>
>
> HTTPCLIENT-1967 indicates that HttpClient can't be used properly with
> TLSv1.3. It caused some test failures below, therefore we should skip running
> tests with SSL on Java 11 to 11.0.2.
> TestMiniSolrCloudClusterSSL.testSslWithCheckPeerName seems to fail 100% of
> the time when run with java11 (or java12), regardless of seed, on both master
> & 7x.
> The nature of the problem and the way our htp stack works suggests it *may*
> ultimately be a jetty bug (perhaps related to [jetty
> issue#2711|https://github.com/eclipse/jetty.project/issues/2711]?)
> *HOWEVER* ... as far as i can tell, whatever the root cause is, seems to have
> been fixed on the {{jira/http2}} branch (as of
> 52bc163dc1804c31af09c1fba99647005da415ad) which should hopefully be getting
> merged to master soon.
> Filing this issue largely for tracking purpose, although we may also want to
> use it for discussions/considerations of other backports/fixes to 7x
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12988) Skip running tests with SSL on Java 11 to 11.0.2
[
https://issues.apache.org/jira/browse/SOLR-12988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16867866#comment-16867866
]
Hoss Man commented on SOLR-12988:
-
Here's what i think...
# You should revert *EVERYTHING* you've commited to this issue – including
your most recent commit to branch_8x *that doesn't even fucking compile with
java8 !*
# We should wait for Uwe to have a chance to see & respond to my email
requesting that jenkins be upgrade to 11.0.3 (it's only been 1 day)
# You should slow WAAAY down on things, and stop rushing to commit things.
Twice now you've asked me my opinion (w/o even posting a patch showing exactly
what your suggestion was) and then rushed to commit changes (the first time
only 2 hours later, the second 10 minutes later ... both while i was asleep)
... apparently w/o even checking to see if it compiled first! if you want
opinions on your ideas give people time to (wake up and) think about it, other
wise why bother asking?
# Once Jenkins is running 11.0.3, *THEN* we should do the bare minimum to
re-enable SSL testing, and wait until we see lots and lots of jenkins builds,
running 11.0.3, to confirm there aren't any *other* SSL problems, before we
make any other decisions about changes to either the Runtime behavior of solr,
or to the tests.
{quote}But I kinda wondering what we should do with the tests? Should we
enforce tests to run in TLSv1.2 for 11 to 11.02? Since not all developers -
jenkins do know this and do the upgrade their sdk?
{quote}
I already shared my thoughts on this...
{quote}Fundementally i think it's a very bad idea to have Solr's behavior
radically change based on introspection of the JVM details – it makes it very
hard to test/reproduce problems. I think it makes a lot more sense for solr to
simply log "Your JVM is known to have some problems, see URL for details" and
let the failures happen if they are going to happen. ...
{quote}
if devs are running tests with a broken JVM, then the tests can & should fail
... that's the job of the tests. it's a bad idea to make the tests "hide" the
failure by "faking" that things work using a degraded cipher, or skipping SSL
completely (yes, i also think mark's changes to SSLTestConfig in December as
part of his commit on this issue was a terrible idea as well) ... the *ONLY*
thing we should _consider_ allowing tests to change about their behavior if
they see a JVM is "broken" is to {{SKIP}} ie:
assume(SomethingThatIsFalseForTheBrokenJVM) ...
{quote}(on the Junit tests side, having assumes around JVM version is fine –
because even then it's not a "silent" behavior change, it's an explicitly "test
ignored because XYZ")
{quote}
...fundementally, this psuedo-code (which mark added to SSLTestConfig in
december and still exists in a slightly diff form in your latest commit) should
*NOT* exist anywhere in our test scaffolding...
{code:java}
if (testOrRandomizationWantsSSL && isCurrentJVMBrokenWithSSL()) {
testOrRandomizationWantsSSL = false;
}
{code}
instead, it should be...
{code:java}
if (testOrRandomizationWantsSSL) {
assumeFalse("Test (or randomization) wants to use SSL for this seed, " +
"but SSL is known to fail on your JVM",
isCurrentJVMBrokenWithSSL());
}
{code}
Saying "we're going to have our this silently use TLSv1.2 (or skip SSL) on your
JVM because it's broken even though that's not what would happen if you tried
to actually run solr on ths broken JVM" is the absolute worst possible thing we
can do.
> Skip running tests with SSL on Java 11 to 11.0.2
>
>
> Key: SOLR-12988
> URL: https://issues.apache.org/jira/browse/SOLR-12988
> Project: Solr
> Issue Type: Test
>Reporter: Hoss Man
>Assignee: Cao Manh Dat
>Priority: Major
> Labels: Java11, Java12
> Attachments: SOLR-13413.patch
>
>
> HTTPCLIENT-1967 indicates that HttpClient can't be used properly with
> TLSv1.3. It caused some test failures below, therefore we should skip running
> tests with SSL on Java 11 to 11.0.2.
> TestMiniSolrCloudClusterSSL.testSslWithCheckPeerName seems to fail 100% of
> the time when run with java11 (or java12), regardless of seed, on both master
> & 7x.
> The nature of the problem and the way our htp stack works suggests it *may*
> ultimately be a jetty bug (perhaps related to [jetty
> issue#2711|https://github.com/eclipse/jetty.project/issues/2711]?)
> *HOWEVER* ... as far as i can tell, whatever the root cause is, seems to have
> been fixed on the {{jira/http2}} branch (as of
> 52bc163dc1804c31af09c1fba99647005da415ad) which should hopefully be getting
> m
[jira] [Commented] (SOLR-12988) Skip running tests with SSL on Java 11 to 11.0.2
[
https://issues.apache.org/jira/browse/SOLR-12988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16867451#comment-16867451
]
ASF subversion and git services commented on SOLR-12988:
Commit 64e3cc1789bfd69c2f11caeec6c2c268239f409e in lucene-solr's branch
refs/heads/branch_8x from Cao Manh Dat
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=64e3cc1 ]
SOLR-12988: Skip running tests with SSL on Java 11 to 11.0.2
> Skip running tests with SSL on Java 11 to 11.0.2
>
>
> Key: SOLR-12988
> URL: https://issues.apache.org/jira/browse/SOLR-12988
> Project: Solr
> Issue Type: Test
>Reporter: Hoss Man
>Assignee: Cao Manh Dat
>Priority: Major
> Labels: Java11, Java12
> Attachments: SOLR-13413.patch
>
>
> HTTPCLIENT-1967 indicates that HttpClient can't be used properly with
> TLSv1.3. It caused some test failures below, therefore we should skip running
> tests with SSL on Java 11 to 11.0.2.
> TestMiniSolrCloudClusterSSL.testSslWithCheckPeerName seems to fail 100% of
> the time when run with java11 (or java12), regardless of seed, on both master
> & 7x.
> The nature of the problem and the way our htp stack works suggests it *may*
> ultimately be a jetty bug (perhaps related to [jetty
> issue#2711|https://github.com/eclipse/jetty.project/issues/2711]?)
> *HOWEVER* ... as far as i can tell, whatever the root cause is, seems to have
> been fixed on the {{jira/http2}} branch (as of
> 52bc163dc1804c31af09c1fba99647005da415ad) which should hopefully be getting
> merged to master soon.
> Filing this issue largely for tracking purpose, although we may also want to
> use it for discussions/considerations of other backports/fixes to 7x
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12988) Skip running tests with SSL on Java 11 to 11.0.2
[
https://issues.apache.org/jira/browse/SOLR-12988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16867449#comment-16867449
]
ASF subversion and git services commented on SOLR-12988:
Commit 91944a468e6bf68bb46f1dad986533c9728c0690 in lucene-solr's branch
refs/heads/master from Cao Manh Dat
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=91944a4 ]
SOLR-12988: Skip running tests with SSL on Java 11 to 11.0.2
> Skip running tests with SSL on Java 11 to 11.0.2
>
>
> Key: SOLR-12988
> URL: https://issues.apache.org/jira/browse/SOLR-12988
> Project: Solr
> Issue Type: Test
>Reporter: Hoss Man
>Assignee: Cao Manh Dat
>Priority: Major
> Labels: Java11, Java12
> Attachments: SOLR-13413.patch
>
>
> HTTPCLIENT-1967 indicates that HttpClient can't be used properly with
> TLSv1.3. It caused some test failures below, therefore we should skip running
> tests with SSL on Java 11 to 11.0.2.
> TestMiniSolrCloudClusterSSL.testSslWithCheckPeerName seems to fail 100% of
> the time when run with java11 (or java12), regardless of seed, on both master
> & 7x.
> The nature of the problem and the way our htp stack works suggests it *may*
> ultimately be a jetty bug (perhaps related to [jetty
> issue#2711|https://github.com/eclipse/jetty.project/issues/2711]?)
> *HOWEVER* ... as far as i can tell, whatever the root cause is, seems to have
> been fixed on the {{jira/http2}} branch (as of
> 52bc163dc1804c31af09c1fba99647005da415ad) which should hopefully be getting
> merged to master soon.
> Filing this issue largely for tracking purpose, although we may also want to
> use it for discussions/considerations of other backports/fixes to 7x
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-12988) Skip running tests with SSL on Java 11 to 11.0.2
[
https://issues.apache.org/jira/browse/SOLR-12988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cao Manh Dat updated SOLR-12988:
Description:
HTTPCLIENT-1967 indicates that HttpClient can't be used properly with TLSv1.3.
It caused some test failures below, therefore we should skip running tests with
SSL on Java 11 to 11.0.2.
TestMiniSolrCloudClusterSSL.testSslWithCheckPeerName seems to fail 100% of the
time when run with java11 (or java12), regardless of seed, on both master & 7x.
The nature of the problem and the way our htp stack works suggests it *may*
ultimately be a jetty bug (perhaps related to [jetty
issue#2711|https://github.com/eclipse/jetty.project/issues/2711]?)
*HOWEVER* ... as far as i can tell, whatever the root cause is, seems to have
been fixed on the {{jira/http2}} branch (as of
52bc163dc1804c31af09c1fba99647005da415ad) which should hopefully be getting
merged to master soon.
Filing this issue largely for tracking purpose, although we may also want to
use it for discussions/considerations of other backports/fixes to 7x
was:
HTTPCLIENT-1967 indicates that HttpClient can't be used properly with TLSv1.3.
It caused some test failures below, therefore we should enforce HttpClient to
uses TLSv1.2 or lower versions.
TestMiniSolrCloudClusterSSL.testSslWithCheckPeerName seems to fail 100% of the
time when run with java11 (or java12), regardless of seed, on both master & 7x.
The nature of the problem and the way our htp stack works suggests it *may*
ultimately be a jetty bug (perhaps related to [jetty
issue#2711|https://github.com/eclipse/jetty.project/issues/2711]?)
*HOWEVER* ... as far as i can tell, whatever the root cause is, seems to have
been fixed on the {{jira/http2}} branch (as of
52bc163dc1804c31af09c1fba99647005da415ad) which should hopefully be getting
merged to master soon.
Filing this issue largely for tracking purpose, although we may also want to
use it for discussions/considerations of other backports/fixes to 7x
> Skip running tests with SSL on Java 11 to 11.0.2
>
>
> Key: SOLR-12988
> URL: https://issues.apache.org/jira/browse/SOLR-12988
> Project: Solr
> Issue Type: Test
>Reporter: Hoss Man
>Assignee: Cao Manh Dat
>Priority: Major
> Labels: Java11, Java12
> Attachments: SOLR-13413.patch
>
>
> HTTPCLIENT-1967 indicates that HttpClient can't be used properly with
> TLSv1.3. It caused some test failures below, therefore we should skip running
> tests with SSL on Java 11 to 11.0.2.
> TestMiniSolrCloudClusterSSL.testSslWithCheckPeerName seems to fail 100% of
> the time when run with java11 (or java12), regardless of seed, on both master
> & 7x.
> The nature of the problem and the way our htp stack works suggests it *may*
> ultimately be a jetty bug (perhaps related to [jetty
> issue#2711|https://github.com/eclipse/jetty.project/issues/2711]?)
> *HOWEVER* ... as far as i can tell, whatever the root cause is, seems to have
> been fixed on the {{jira/http2}} branch (as of
> 52bc163dc1804c31af09c1fba99647005da415ad) which should hopefully be getting
> merged to master soon.
> Filing this issue largely for tracking purpose, although we may also want to
> use it for discussions/considerations of other backports/fixes to 7x
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-12988) Skip running tests with SSL on Java 11 to 11.0.2
[
https://issues.apache.org/jira/browse/SOLR-12988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cao Manh Dat updated SOLR-12988:
Summary: Skip running tests with SSL on Java 11 to 11.0.2 (was: Avoid
using TLSv1.3 for HttpClient)
> Skip running tests with SSL on Java 11 to 11.0.2
>
>
> Key: SOLR-12988
> URL: https://issues.apache.org/jira/browse/SOLR-12988
> Project: Solr
> Issue Type: Test
>Reporter: Hoss Man
>Assignee: Cao Manh Dat
>Priority: Major
> Labels: Java11, Java12
> Attachments: SOLR-13413.patch
>
>
> HTTPCLIENT-1967 indicates that HttpClient can't be used properly with
> TLSv1.3. It caused some test failures below, therefore we should enforce
> HttpClient to uses TLSv1.2 or lower versions.
> TestMiniSolrCloudClusterSSL.testSslWithCheckPeerName seems to fail 100% of
> the time when run with java11 (or java12), regardless of seed, on both master
> & 7x.
> The nature of the problem and the way our htp stack works suggests it *may*
> ultimately be a jetty bug (perhaps related to [jetty
> issue#2711|https://github.com/eclipse/jetty.project/issues/2711]?)
> *HOWEVER* ... as far as i can tell, whatever the root cause is, seems to have
> been fixed on the {{jira/http2}} branch (as of
> 52bc163dc1804c31af09c1fba99647005da415ad) which should hopefully be getting
> merged to master soon.
> Filing this issue largely for tracking purpose, although we may also want to
> use it for discussions/considerations of other backports/fixes to 7x
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13424) Ref Guide PDF build throws Java warnings after requiring Java 11 on master
[
https://issues.apache.org/jira/browse/SOLR-13424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16835025#comment-16835025
]
Hoss Man commented on SOLR-13424:
-
both sets of warnings come using {{org.asciidoctor}} code (either directly
execing when building the barebones HTML, or programatically to parse metadata
out of the files for building the nav data) which delegates to jruby where the
reflection lives...
https://github.com/jruby/jruby/issues/4834
https://github.com/jruby/jruby/issues/5559
...i'm not sure why the {{\-build-raw-pdf}} target doesn't generate the same
warnings, but i don't think there is much we can do about it until there is a
jruby fix
> Ref Guide PDF build throws Java warnings after requiring Java 11 on master
> --
>
> Key: SOLR-13424
> URL: https://issues.apache.org/jira/browse/SOLR-13424
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: documentation
>Reporter: Cassandra Targett
>Priority: Major
>
> Since the {{master}} branch requires Java 11, the Ref Guide build has started
> throwing WARNINGs that we should look into. Despite the WARNINGs, the PDF
> build finishes successfully and appears the same as usual.
> There are a couple moving parts here - first some classes [~hossman] wrote to
> validate links in the PDF, and second a .jar from the {{asciidoctor-ant}}
> project. I suspect some of these are related to Hoss' code and some are from
> the project that's doing the PDF conversion.
> I'll put the output from a Jenkins job in the comment to show the errors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8780) Improve ByteBufferGuard in Java 11
[
https://issues.apache.org/jira/browse/LUCENE-8780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16829739#comment-16829739
]
Uwe Schindler commented on LUCENE-8780:
---
I did a second patch that uses AtomicBoolean instead of VarHandles. The
underlying code is the same: {{AtomicBoolean.getOpaque()}} calls a VarHandle
behind the scenes (BTW, AtomicInteger, AtomicBoolean,... all were rewritten and
use the VarHandle mechanism, see e.g.,
[https://hg.openjdk.java.net/jdk/jdk11/file/1ddf9a99e4ad/src/java.base/share/classes/java/util/concurrent/atomic/AtomicBoolean.java]).
Here is this version #2:
https://github.com/apache/lucene-solr/compare/master...uschindler:jira/LUCENE-8780-v2
The effect was worse, so it's not an option. But this brings me to the
conclusion: The actual calls using the other memory models are not actually the
problem. Also the VarHandles and AtomicBooleans are correctly optimized away,
but it looks like because of the complexity of the optimizations on the lowest
level, it takes much longer until it gets faster and some optimizations are not
applied at all (you cannot remove opaque reads, because the memory model
eventually makes changes from other threads visible).
Here are the results from the AtomicBoolean:
{noformat}
Report after iter 10:
TaskQPS orig StdDev QPS patch StdDev
Pct diff
IntNRQ 22.48 (12.8%) 16.86 (13.0%)
-25.0% ( -45% -0%)
PKLookup 112.32 (14.5%) 92.37 (7.6%)
-17.8% ( -34% -5%)
OrHighLow 207.87 (16.6%) 185.52 (4.1%)
-10.8% ( -26% - 11%)
OrNotHighMed 992.06 (9.3%) 914.79 (1.5%)
-7.8% ( -16% -3%)
HighSpanNear5.22 (10.2%)4.83 (5.4%)
-7.5% ( -20% -9%)
Fuzzy1 44.66 (9.7%) 41.46 (2.1%)
-7.2% ( -17% -5%)
MedSpanNear8.24 (18.2%)7.67 (12.4%)
-6.9% ( -31% - 28%)
LowSloppyPhrase6.91 (19.1%)6.46 (13.4%)
-6.6% ( -32% - 31%)
Wildcard 43.23 (13.9%) 40.47 (6.0%)
-6.4% ( -23% - 15%)
LowPhrase 11.89 (11.4%) 11.18 (3.7%)
-6.0% ( -18% - 10%)
OrHighNotMed 1188.55 (6.3%) 1118.58 (1.3%)
-5.9% ( -12% -1%)
Fuzzy2 66.58 (1.4%) 62.85 (1.7%)
-5.6% ( -8% - -2%)
HighPhrase 32.87 (11.8%) 31.15 (8.8%)
-5.2% ( -23% - 17%)
OrNotHighLow 537.79 (2.2%) 511.56 (8.8%)
-4.9% ( -15% -6%)
MedSloppyPhrase 44.16 (10.0%) 42.08 (2.3%)
-4.7% ( -15% -8%)
OrNotHighHigh 984.54 (2.2%) 942.20 (1.7%)
-4.3% ( -8% -0%)
AndHighHigh6.40 (12.1%)6.15 (12.8%)
-3.9% ( -25% - 23%)
AndHighMed 57.29 (9.9%) 55.19 (3.4%)
-3.7% ( -15% - 10%)
HighSloppyPhrase4.60 (14.3%)4.44 (6.0%)
-3.5% ( -20% - 19%)
OrHighNotHigh 853.88 (2.7%) 824.51 (3.7%)
-3.4% ( -9% -3%)
MedPhrase 73.25 (2.6%) 70.85 (3.4%)
-3.3% ( -9% -2%)
LowTerm 1130.00 (5.4%) 1093.38 (2.5%)
-3.2% ( -10% -4%)
OrHighMed 34.61 (2.3%) 33.58 (3.1%)
-3.0% ( -8% -2%)
MedTerm 994.47 (7.9%) 975.29 (7.9%)
-1.9% ( -16% - 15%)
OrHighNotLow 762.68 (3.0%) 749.09 (5.2%)
-1.8% ( -9% -6%)
Respell 53.06 (6.9%) 52.44 (11.5%)
-1.2% ( -18% - 18%)
LowSpanNear8.29 (5.3%)8.30 (10.0%)
0.1% ( -14% - 16%)
HighTermDayOfYearSort 22.78 (6.4%) 22.88 (7.1%)
0.4% ( -12% - 14%)
HighTerm 822.80 (3.2%) 827.84 (8.2%)
0.6% ( -10% - 12%)
OrHighHigh8.85 (10.0%)8.92 (14.5%)
0.8% ( -21% - 28%)
AndHighLow 258.08 (4.7%) 261.18 (10.0%)
1.2% ( -12% - 16%)
HighTermMonthSort 13.86 (9.2%) 14.63 (22.9%)
5.6% ( -24% - 41%)
Prefix3 27.01 (10.1%) 28.72 (25.0%)
6.3% ( -26% - 46%)
{noformat}
As said before, removing the null check does not matter at all, it just makes
the variance on short running tests less evident, but the average is identical.
> Improve ByteBufferGuard in Java 11
> --
>
> Key:
[jira] [Commented] (LUCENE-8780) Improve ByteBufferGuard in Java 11
[
https://issues.apache.org/jira/browse/LUCENE-8780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16829444#comment-16829444
]
Robert Muir commented on LUCENE-8780:
-
i imagine any slowdown only impacts stuff doing lots of tiny reads vs reading
big byte[].
The current code in most cases will deliver ACE in someones unit test rather
than a crash. Maybe if you are lucky you even get an ACE rather than sigsegv in
production too. To me it seems like the promise of this patch is that you stand
a better chance, even after code is running for a while (10k times or
whatever). our best effort check will no longer be optimized away? but thats
also its downside: it wont get optimized away even if you have no bugs in your
code.
seems like we just decide on the correct tradeoff. personally i lean towards
more aggressive safety: the user is using java, they dont expect sigsegv. just
like they dont expect to run out of mmaps because its tied to gc, and they dont
expect all their searches bottlenecked on one file handle, due to stupid
synchronizatiom on a relative file pointer we dont even care about.
> Improve ByteBufferGuard in Java 11
> --
>
> Key: LUCENE-8780
> URL: https://issues.apache.org/jira/browse/LUCENE-8780
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/store
>Affects Versions: master (9.0)
>Reporter: Uwe Schindler
>Assignee: Uwe Schindler
>Priority: Major
> Labels: Java11
> Attachments: LUCENE-8780.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In LUCENE-7409 we added {{ByteBufferGuard}} to protect MMapDirectory from
> crushing the JVM with SIGSEGV when you close and unmap the mmapped buffers of
> an IndexInput, while another thread is accessing it.
> The idea was to do a volatile write access to flush the caches (to trigger a
> full fence) and set a non-volatile boolean to true. All accesses would check
> the boolean and stop the caller from accessing the underlying ByteBuffer.
> This worked most of the time, until the JVM optimized away the plain read
> access to the boolean (you can easily see this after some runtime of our
> by-default ignored testcase).
> With master on Java 11, we can improve the whole thing. Using VarHandles you
> can use any access type when reading or writing the boolean. After reading
> Doug Lea's expanation <http://gee.cs.oswego.edu/dl/html/j9mm.html> and some
> testing, I was no longer able to crush my JDK (even after running for minutes
> unmapping bytebuffers).
> The apraoch is the same, we do a full-fenced write (standard volatile write)
> when we unmap, then we yield the thread (to finish in-flight reads in other
> threads) and then unmap all byte buffers.
> On the test side (read access), instead of using a plain read, we use the new
> "opaque read". Opaque reads are the same as plain reads, there are only
> different order requirements. Actually the main difference is explained by
> Doug like this: "For example in constructions in which the only modification
> of some variable x is for one thread to write in Opaque (or stronger) mode,
> X.setOpaque(this, 1), any other thread spinning in
> while(X.getOpaque(this)!=1){} will eventually terminate. Note that this
> guarantee does NOT hold in Plain mode, in which spin loops may (and usually
> do) infinitely loop -- they are not required to notice that a write ever
> occurred in another thread if it was not seen on first encounter." - And
> that's waht we want to have: We don't want to do volatile reads, but we want
> to prevent the compiler from optimizing away our read to the boolean. So we
> want it to "eventually" see the change. By the much stronger volatile write,
> the cache effects should be visible even faster (like in our Java 8 approach,
> just now we improved our read side).
> The new code is much slimmer (theoretically we could also use a AtomicBoolean
> for that and use the new method {{getOpaque()}}, but I wanted to prevent
> extra method calls, so I used a VarHandle directly).
> It's setup like this:
> - The underlying boolean field is a private member (with unused
> SuppressWarnings, as its unused by the java compiler), marked as volatile
> (that's the recommendation, but in reality it does not matter at all).
> - We create a VarHandle to access this boolean, we never do this directly
> (this is why the volatile marking does not affect us).
> - We use VarHandle.setVolatile() to change our "invalidated" boolean to
> "true", so enforcing a full fence
> - On the read s
[jira] [Comment Edited] (LUCENE-8780) Improve ByteBufferGuard in Java 11
re is no way to make
ByteBufferGuard "safe" unless you use a volatile or with an aquire/release
fence (that can be implemented also with the varhandles). When using it, I was
unable to crush my VM, but slowdown is definitely there. Also manually putting
in fences does not help.
The main difference between plain read and opaque (as used here) is not that
the produced CPU instructions are different, the difference is only how the
optimizer may optimize code. With plain reads, a simple spin-loop will never
end, as the hotspot compiler clearly sees that the value can't change, and
because it's not volatile, opaque or whatever, he will remove it for this
thread. This does not only happen in spin loops, it also happens in our code if
it's executed often enough. This is not a bug, it's wanted.
I tuned the testcase to crush always: Just insert a loop of 10.000 reads BEFORE
you unmap, then it fails always (in Java 8). With opaque it was also doing
this, but not reproducible, so there is an improvement. IMHO, the reason why we
see a slowdown for some queries could be coming from that: Hotspot is removing
the "if (invalidated)" check, as it cannot change in the current thread. With
opaque it can't do it, so the code is slower on the long term. I will try to
get some assembly output later, just have to install hsdis.dll.
To make it bulletproof, we have to wait for official support by the ByteBuffer
API to officially unmap (https://bugs.openjdk.java.net/browse/JDK-4724038),
with our checks we cannot make it safe. Another approach would be to expand the
SIGSEGV signal handler checks in the JVM to throw an InternalError (they do it
now if the operating system truncates the file and so the mapped are changes).
I don't know why they not generally do a SIGSEGV check and if it happens inside
DirectBuffer they just throw an Exception (possibly delayed as in the
truncated file case).
So we have to decide what we want to do as a workaround:
- We can't be bulletproof
- We can catch with our current code most cases where you open an index, start
a query and then close the reader. This only works shortly after program
started and everything is not yet optimized. As soon as there were multiple
queries already running and you close the JVM later, it SIGSEGV almost always.
Not even "opaque" semantics help here.
- With the current code we can go a bit further, but we have some slowdown, but
it's still not save.
> Improve ByteBufferGuard in Java 11
> --
>
> Key: LUCENE-8780
> URL: https://issues.apache.org/jira/browse/LUCENE-8780
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/store
>Affects Versions: master (9.0)
>Reporter: Uwe Schindler
>Assignee: Uwe Schindler
>Priority: Major
> Labels: Java11
> Attachments: LUCENE-8780.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In LUCENE-7409 we added {{ByteBufferGuard}} to protect MMapDirectory from
> crushing the JVM with SIGSEGV when you close and unmap the mmapped buffers of
> an IndexInput, while another thread is accessing it.
> The idea was to do a volatile write access to flush the caches (to trigger a
> full fence) and set a non-volatile boolean to true. All accesses would check
> the boolean and stop the caller from accessing the underlying ByteBuffer.
> This worked most of the time, until the JVM optimized away the plain read
> access to the boolean (you can easily see this after some runtime of our
> by-default ignored testcase).
> With master on Java 11, we can improve the whole thing. Using VarHandles you
> can use any access type when reading or writing the boolean. After reading
> Doug Lea's expanation <http://gee.cs.oswego.edu/dl/html/j9mm.html> and some
> testing, I was no longer able to crush my JDK (even after running for minutes
> unmapping bytebuffers).
> The apraoch is the same, we do a full-fenced write (standard volatile write)
> when we unmap, then we yield the thread (to finish in-flight reads in other
> threads) and then unmap all byte buffers.
> On the test side (read access), instead of using a plain read, we use the new
> "opaque read". Opaque reads are the same as plain reads, there are only
> different order requirements. Actually the main difference is explained by
> Doug like this: "For example in constructions in which the only modification
> of some variable x is for one thread to write in Opaque (or stronger) mode,
> X.setOpaque(this, 1), any other thread spinning in
> while(X.getOpaque(this)!=1){} will eventually terminate. Note that this
> guarantee does NOT hold in Plain mode, in which spin loops may (and usually
> do) inf
[jira] [Comment Edited] (LUCENE-8780) Improve ByteBufferGuard in Java 11
n is definitely there. Also manually putting
in fences does not help.
The main difference between plain read and opaque (as used here) is not that
the produced CPU instructions are different, the difference is only how the
optimizer may optimize code. With plain reads, a simple spin-loop will never
end, as the hotspot compiler clearly sees that the value can't change, and
because it's not volatile, opaque or whatever, he will remove it for this
thread. This does not only happen in spin loops, it also happens in our code if
it's executed often enough. This is not a bug, it's wanted.
I tuned the testcase to crush always: Just insert a loop of 10.000 reads BEFORE
you unmap, then it fails always (in Java 8). With opaque it was also doing
this, but not reproducible, so there is an improvement. IMHO, the reason why we
see a slowdown for some queries could be coming from that: Hotspot is removing
the "if (invalidated)" check, as it cannot change in the current thread. With
opaque it can't do it, so the code is slower on the long term. I will try to
get some assembly output later, just have to install hsdis.dll.
To make it bulletproof, we have to wait for official support by the ByteBuffer
API to officially unmap (https://bugs.openjdk.java.net/browse/JDK-4724038),
with our checks we cannot make it safe. Another approach would be to expand the
SIGSEGV signal handler checks in the JVM to throw an InternalError (they do it
now if the operating system truncates the file and so the mapped are changes).
I don't know why they not generally do a SIGSEGV check and if it happens inside
DirectBuffer they just throw an Exception (possibly delayed as in the
truncated file case).
So we have to decide what we want to do as a workaround:
- We can't be bulletproof
- We can catch with our current code most cases where you open an index, start
a query and then close the reader. This only works shortly after program
started and everything is not yet optimized. As soon as there were multiple
queries already running and you close the JVM later, it SIGSEGV almost always.
Not even "opaque" semantics help here.
- With the current code we can go a bit further, but we have some slowdown, but
it's still not save.
> Improve ByteBufferGuard in Java 11
> --
>
> Key: LUCENE-8780
> URL: https://issues.apache.org/jira/browse/LUCENE-8780
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/store
>Affects Versions: master (9.0)
>Reporter: Uwe Schindler
>Assignee: Uwe Schindler
>Priority: Major
> Labels: Java11
> Attachments: LUCENE-8780.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In LUCENE-7409 we added {{ByteBufferGuard}} to protect MMapDirectory from
> crushing the JVM with SIGSEGV when you close and unmap the mmapped buffers of
> an IndexInput, while another thread is accessing it.
> The idea was to do a volatile write access to flush the caches (to trigger a
> full fence) and set a non-volatile boolean to true. All accesses would check
> the boolean and stop the caller from accessing the underlying ByteBuffer.
> This worked most of the time, until the JVM optimized away the plain read
> access to the boolean (you can easily see this after some runtime of our
> by-default ignored testcase).
> With master on Java 11, we can improve the whole thing. Using VarHandles you
> can use any access type when reading or writing the boolean. After reading
> Doug Lea's expanation <http://gee.cs.oswego.edu/dl/html/j9mm.html> and some
> testing, I was no longer able to crush my JDK (even after running for minutes
> unmapping bytebuffers).
> The apraoch is the same, we do a full-fenced write (standard volatile write)
> when we unmap, then we yield the thread (to finish in-flight reads in other
> threads) and then unmap all byte buffers.
> On the test side (read access), instead of using a plain read, we use the new
> "opaque read". Opaque reads are the same as plain reads, there are only
> different order requirements. Actually the main difference is explained by
> Doug like this: "For example in constructions in which the only modification
> of some variable x is for one thread to write in Opaque (or stronger) mode,
> X.setOpaque(this, 1), any other thread spinning in
> while(X.getOpaque(this)!=1){} will eventually terminate. Note that this
> guarantee does NOT hold in Plain mode, in which spin loops may (and usually
> do) infinitely loop -- they are not required to notice that a write ever
> occurred in another thread if it was not seen on first encounter." - And
> that's waht we want to have: We don't want to do volatile reads, but we
[jira] [Commented] (LUCENE-8780) Improve ByteBufferGuard in Java 11
[
https://issues.apache.org/jira/browse/LUCENE-8780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16829428#comment-16829428
]
Uwe Schindler commented on LUCENE-8780:
---
Hi,
I did some tests over the last night. The fluctuating results can be improved
by removing the null-check on the "cleaner". Normally this one should be
optimized away, but that seems to be too hard. Without it, you still have some
variance in the results, but it's better.
In general the new code is a bit slower, which is more visible if you have
shorter runtime, so it looks like optimizing aways the VarHandle simply takes
longer, and this degrades oaverage performance. I don't know how long the
benchmark runs a warmup.
bq. Every time I read about those memory ordering models I seem to get it and
two weeks later I'm again confused...
That's the ssme for me. But our use-case seems to be pretty good explained in
the above quotes by Doug Lea. It clearly says that a volatile write is harder
than a opaque read, but the opaque read is only visible eventually.
I was already talking with Hotspot guys: In short, there is no way to make
ByteBufferGuard "safe" unless you use a volatile or with an aquire/release
fence (that can be implemented also with the varhandles). When using it, I was
unable to crush my VM, but slowdown is definitely there. Also manually putting
in fences does not help.
The main difference between plain read and opaque (as used here) is not that
the produced CPU instructions are different, the difference is only how the
optimizer may optimize code. With plain reads, a simple spin-loop will never
end, as the hotspot compiler clearly sees that the value can't change, and
because it's not volatile, opaque or whatever, he will remove it for this
thread. This does not only happen in spin loops, it also happens in our code if
it's executed often enough. This is not a bug, it's wanted.
I tuned the testcase to crush always: Just insert a loop of 10.000 reads BEFORE
you unmap, then it fails always (in Java 8). With opaque it was also doing
this, but not reproducible, so there is an improvement. IMHO, the reason why we
see a slowdown for some queries could be coming from that: Hotspot is removing
the "if (invalidated)" check, as it cannot change in the current thread. With
opaque it can't do it, so the code is slower on the long term. I will try to
get some assembly output later, just have to install hsdis.dll.
To make it bulletproof, we have to wait for official support by the ByteBuffer
API to officially unmap (https://bugs.openjdk.java.net/browse/JDK-4724038),
with our checks we cannot make it safe. Another approach would be to expand the
SIGSEGV signal handler checks in the JVM to throw an InternalError (they do it
now if the operating system truncates the file and so the mapped are changes).
I don't know why they not generally do a SIGSEGV check and if it happens inside
DirectBuffer they just throw an Exception (possibly delayed as in the
truncated file case).
So we have to decide what we want to do as a workaround:
- We can't be bulletproof
- We can catch with our current code most cases where you open an index, start
a query and then close the reader. This only works shortly after program
started and everything is not yet optimized. As soon as there were multiple
queries already running and you close the JVM later, it SIGSEGV almost always.
Not even "opaque" semantics help here.
- With the current code we can go a bit further, but we have some slowdown, but
it's still not save.
> Improve ByteBufferGuard in Java 11
> --
>
> Key: LUCENE-8780
> URL: https://issues.apache.org/jira/browse/LUCENE-8780
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/store
>Affects Versions: master (9.0)
>Reporter: Uwe Schindler
>Assignee: Uwe Schindler
>Priority: Major
> Labels: Java11
> Attachments: LUCENE-8780.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In LUCENE-7409 we added {{ByteBufferGuard}} to protect MMapDirectory from
> crushing the JVM with SIGSEGV when you close and unmap the mmapped buffers of
> an IndexInput, while another thread is accessing it.
> The idea was to do a volatile write access to flush the caches (to trigger a
> full fence) and set a non-volatile boolean to true. All accesses would check
> the boolean and stop the caller from accessing the underlying ByteBuffer.
> This worked most of the time, until the JVM optimized away the plain read
> access to the boolean (you can easily see this after some runtime of our
> by-default ignored testcase).
> With master on Java 11, we can improve
[jira] [Commented] (LUCENE-8780) Improve ByteBufferGuard in Java 11
[
https://issues.apache.org/jira/browse/LUCENE-8780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16829031#comment-16829031
]
Dawid Weiss commented on LUCENE-8780:
-
Every time I read about those memory ordering models I seem to get it and two
weeks later I'm again confused... So I don't know if what you propose is right
or wrong. The volatile mode always seemed to me to be regulating relationships
among other volatile reads/ writes (and not specifying what happens to other
variable reads/ writes not covered by volatiles)... perhaps you can piggyback
fences on top of that, but maybe you can't. We could fire an e-mail to
concurrency dev list so that more knowledgeable folks take a peek and let us
know? Just saying...
Also: "until the JVM optimized away the plain read access to the boolean" --
when did that happen, do you have a reference openjdk Jira issue?
> Improve ByteBufferGuard in Java 11
> --
>
> Key: LUCENE-8780
> URL: https://issues.apache.org/jira/browse/LUCENE-8780
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/store
>Affects Versions: master (9.0)
>Reporter: Uwe Schindler
>Assignee: Uwe Schindler
>Priority: Major
> Labels: Java11
> Attachments: LUCENE-8780.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In LUCENE-7409 we added {{ByteBufferGuard}} to protect MMapDirectory from
> crushing the JVM with SIGSEGV when you close and unmap the mmapped buffers of
> an IndexInput, while another thread is accessing it.
> The idea was to do a volatile write access to flush the caches (to trigger a
> full fence) and set a non-volatile boolean to true. All accesses would check
> the boolean and stop the caller from accessing the underlying ByteBuffer.
> This worked most of the time, until the JVM optimized away the plain read
> access to the boolean (you can easily see this after some runtime of our
> by-default ignored testcase).
> With master on Java 11, we can improve the whole thing. Using VarHandles you
> can use any access type when reading or writing the boolean. After reading
> Doug Lea's expanation <http://gee.cs.oswego.edu/dl/html/j9mm.html> and some
> testing, I was no longer able to crush my JDK (even after running for minutes
> unmapping bytebuffers).
> The apraoch is the same, we do a full-fenced write (standard volatile write)
> when we unmap, then we yield the thread (to finish in-flight reads in other
> threads) and then unmap all byte buffers.
> On the test side (read access), instead of using a plain read, we use the new
> "opaque read". Opaque reads are the same as plain reads, there are only
> different order requirements. Actually the main difference is explained by
> Doug like this: "For example in constructions in which the only modification
> of some variable x is for one thread to write in Opaque (or stronger) mode,
> X.setOpaque(this, 1), any other thread spinning in
> while(X.getOpaque(this)!=1){} will eventually terminate. Note that this
> guarantee does NOT hold in Plain mode, in which spin loops may (and usually
> do) infinitely loop -- they are not required to notice that a write ever
> occurred in another thread if it was not seen on first encounter." - And
> that's waht we want to have: We don't want to do volatile reads, but we want
> to prevent the compiler from optimizing away our read to the boolean. So we
> want it to "eventually" see the change. By the much stronger volatile write,
> the cache effects should be visible even faster (like in our Java 8 approach,
> just now we improved our read side).
> The new code is much slimmer (theoretically we could also use a AtomicBoolean
> for that and use the new method {{getOpaque()}}, but I wanted to prevent
> extra method calls, so I used a VarHandle directly).
> It's setup like this:
> - The underlying boolean field is a private member (with unused
> SuppressWarnings, as its unused by the java compiler), marked as volatile
> (that's the recommendation, but in reality it does not matter at all).
> - We create a VarHandle to access this boolean, we never do this directly
> (this is why the volatile marking does not affect us).
> - We use VarHandle.setVolatile() to change our "invalidated" boolean to
> "true", so enforcing a full fence
> - On the read side we use VarHandle.getOpaque() instead of VarHandle.get()
> (like in our old code for Java 8).
> I had to tune our test a bit, as the VarHandles make it take longer until it
> actually crushes (as optimizations jum
[jira] [Commented] (LUCENE-8780) Improve ByteBufferGuard in Java 11
[
https://issues.apache.org/jira/browse/LUCENE-8780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16828826#comment-16828826
]
Mike Sokolov commented on LUCENE-8780:
--
I don't have a good theory, but I was curious so I ran a few tests, and one
thing I saw is that if you limit to a single searcher thread, you see only the
negative side of this distribution, or at least it becomes more negative.
> Improve ByteBufferGuard in Java 11
> --
>
> Key: LUCENE-8780
> URL: https://issues.apache.org/jira/browse/LUCENE-8780
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/store
>Affects Versions: master (9.0)
>Reporter: Uwe Schindler
>Assignee: Uwe Schindler
>Priority: Major
> Labels: Java11
> Attachments: LUCENE-8780.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In LUCENE-7409 we added {{ByteBufferGuard}} to protect MMapDirectory from
> crushing the JVM with SIGSEGV when you close and unmap the mmapped buffers of
> an IndexInput, while another thread is accessing it.
> The idea was to do a volatile write access to flush the caches (to trigger a
> full fence) and set a non-volatile boolean to true. All accesses would check
> the boolean and stop the caller from accessing the underlying ByteBuffer.
> This worked most of the time, until the JVM optimized away the plain read
> access to the boolean (you can easily see this after some runtime of our
> by-default ignored testcase).
> With master on Java 11, we can improve the whole thing. Using VarHandles you
> can use any access type when reading or writing the boolean. After reading
> Doug Lea's expanation <http://gee.cs.oswego.edu/dl/html/j9mm.html> and some
> testing, I was no longer able to crush my JDK (even after running for minutes
> unmapping bytebuffers).
> The apraoch is the same, we do a full-fenced write (standard volatile write)
> when we unmap, then we yield the thread (to finish in-flight reads in other
> threads) and then unmap all byte buffers.
> On the test side (read access), instead of using a plain read, we use the new
> "opaque read". Opaque reads are the same as plain reads, there are only
> different order requirements. Actually the main difference is explained by
> Doug like this: "For example in constructions in which the only modification
> of some variable x is for one thread to write in Opaque (or stronger) mode,
> X.setOpaque(this, 1), any other thread spinning in
> while(X.getOpaque(this)!=1){} will eventually terminate. Note that this
> guarantee does NOT hold in Plain mode, in which spin loops may (and usually
> do) infinitely loop -- they are not required to notice that a write ever
> occurred in another thread if it was not seen on first encounter." - And
> that's waht we want to have: We don't want to do volatile reads, but we want
> to prevent the compiler from optimizing away our read to the boolean. So we
> want it to "eventually" see the change. By the much stronger volatile write,
> the cache effects should be visible even faster (like in our Java 8 approach,
> just now we improved our read side).
> The new code is much slimmer (theoretically we could also use a AtomicBoolean
> for that and use the new method {{getOpaque()}}, but I wanted to prevent
> extra method calls, so I used a VarHandle directly).
> It's setup like this:
> - The underlying boolean field is a private member (with unused
> SuppressWarnings, as its unused by the java compiler), marked as volatile
> (that's the recommendation, but in reality it does not matter at all).
> - We create a VarHandle to access this boolean, we never do this directly
> (this is why the volatile marking does not affect us).
> - We use VarHandle.setVolatile() to change our "invalidated" boolean to
> "true", so enforcing a full fence
> - On the read side we use VarHandle.getOpaque() instead of VarHandle.get()
> (like in our old code for Java 8).
> I had to tune our test a bit, as the VarHandles make it take longer until it
> actually crushes (as optimizations jump in later). I also used a random for
> the reads to prevent the optimizer from removing all the bytebuffer reads.
> When we commit this, we can disable the test again (it takes approx 50 secs
> on my machine).
> I'd still like to see the differences between the plain read and the opaque
> read in production, so maybe [~mikemccand] or [~rcmuir] can do a comparison
> with nightly benchmarker?
> Have fun, maybe [~dweiss] has some ideas, too.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (LUCENE-8780) Improve ByteBufferGuard in Java 11
Pct diff
IntNRQ 30.88 (0.6%) 26.33 (0.8%)
-14.7% ( -16% - -13%)
PKLookup 107.70 (2.7%) 94.31 (2.9%)
-12.4% ( -17% - -7%)
AndHighHigh 10.76 (11.5%) 10.17 (3.3%)
-5.4% ( -18% - 10%)
Fuzzy2 45.10 (7.7%) 43.21 (9.0%)
-4.2% ( -19% - 13%)
LowSloppyPhrase7.28 (16.8%)6.98 (6.3%)
-4.2% ( -23% - 22%)
OrHighNotLow 783.24 (7.1%) 751.37 (2.5%)
-4.1% ( -12% -5%)
OrHighNotHigh 934.39 (6.5%) 896.38 (2.1%)
-4.1% ( -11% -4%)
Respell 45.36 (10.6%) 43.65 (7.0%)
-3.8% ( -19% - 15%)
OrNotHighHigh 779.95 (3.8%) 752.28 (1.8%)
-3.5% ( -8% -2%)
HighSloppyPhrase 10.37 (12.8%) 10.03 (3.5%)
-3.3% ( -17% - 14%)
LowPhrase 11.60 (8.9%) 11.23 (1.7%)
-3.2% ( -12% -8%)
LowTerm 1694.00 (8.9%) 1642.34 (5.5%)
-3.0% ( -16% - 12%)
MedTerm 1292.82 (9.3%) 1253.69 (8.2%)
-3.0% ( -18% - 15%)
AndHighMed 71.41 (9.9%) 69.77 (7.5%)
-2.3% ( -17% - 16%)
OrNotHighMed 634.32 (7.2%) 620.67 (7.5%)
-2.2% ( -15% - 13%)
Prefix3 110.65 (14.9%) 108.55 (8.7%)
-1.9% ( -22% - 25%)
OrHighLow 347.02 (4.3%) 340.51 (9.9%)
-1.9% ( -15% - 12%)
OrNotHighLow 591.61 (5.5%) 580.60 (9.0%)
-1.9% ( -15% - 13%)
OrHighNotMed 1258.21 (1.8%) 1237.28 (5.0%)
-1.7% ( -8% -5%)
Fuzzy1 91.79 (4.3%) 90.77 (11.1%)
-1.1% ( -15% - 14%)
OrHighMed 10.29 (7.9%) 10.25 (11.8%)
-0.4% ( -18% - 20%)
Wildcard 52.28 (6.3%) 52.21 (6.8%)
-0.1% ( -12% - 13%)
OrHighHigh8.16 (6.9%)8.22 (9.3%)
0.8% ( -14% - 18%)
AndHighLow 563.89 (9.1%) 569.31 (15.3%)
1.0% ( -21% - 27%)
HighPhrase 15.88 (9.3%) 16.04 (13.0%)
1.0% ( -19% - 25%)
MedPhrase 14.84 (9.0%) 15.15 (12.8%)
2.1% ( -18% - 26%)
HighSpanNear2.16 (9.8%)2.21 (10.1%)
2.3% ( -16% - 24%)
MedSloppyPhrase 18.48 (15.4%) 18.96 (18.9%)
2.6% ( -27% - 43%)
MedSpanNear 17.75 (3.8%) 18.31 (10.0%)
3.1% ( -10% - 17%)
HighTerm 1031.00 (9.9%) 1068.12 (17.1%)
3.6% ( -21% - 33%)
LowSpanNear8.22 (5.5%)8.53 (13.3%)
3.7% ( -14% - 23%)
HighTermDayOfYearSort9.78 (11.0%) 10.25 (18.2%)
4.8% ( -21% - 38%)
HighTermMonthSort 23.40 (26.5%) 27.11 (32.1%)
15.9% ( -33% - 101%)
{noformat}
The total runtime of each run did not change, always approx 280s per run
patched and unpatched. Not sure how to interpret this.
> Improve ByteBufferGuard in Java 11
> --
>
> Key: LUCENE-8780
> URL: https://issues.apache.org/jira/browse/LUCENE-8780
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/store
>Affects Versions: master (9.0)
>Reporter: Uwe Schindler
>Assignee: Uwe Schindler
>Priority: Major
> Labels: Java11
> Attachments: LUCENE-8780.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In LUCENE-7409 we added {{ByteBufferGuard}} to protect MMapDirectory from
> crushing the JVM with SIGSEGV when you close and unmap the mmapped buffers of
> an IndexInput, while another thread is accessing it.
> The idea was to do a volatile write access to flush the caches (to trigger a
> full fence) and set a non-volatile boolean to true. All accesses would check
> the boolean and stop the caller from accessing the underlying ByteBuffer.
> This worked most of the time, until the JVM optimized away the plain read
> access to the boolean (you can easily see this after some runtime of our
> by-default ignored testcase).
> With master on Java 11, we can improve the whole thing. Using VarHandles you
> can use any access type when reading or writing the boolean. After reading
> Doug Lea's expanation <http://gee.cs.oswego.edu/dl/html/j9mm.html> and some
> testing, I was no longer able to crush my JDK (even after running for mi
[jira] [Comment Edited] (LUCENE-8780) Improve ByteBufferGuard in Java 11
Pct diff
IntNRQ 30.88 (0.6%) 26.33 (0.8%)
-14.7% ( -16% - -13%)
PKLookup 107.70 (2.7%) 94.31 (2.9%)
-12.4% ( -17% - -7%)
AndHighHigh 10.76 (11.5%) 10.17 (3.3%)
-5.4% ( -18% - 10%)
Fuzzy2 45.10 (7.7%) 43.21 (9.0%)
-4.2% ( -19% - 13%)
LowSloppyPhrase7.28 (16.8%)6.98 (6.3%)
-4.2% ( -23% - 22%)
OrHighNotLow 783.24 (7.1%) 751.37 (2.5%)
-4.1% ( -12% -5%)
OrHighNotHigh 934.39 (6.5%) 896.38 (2.1%)
-4.1% ( -11% -4%)
Respell 45.36 (10.6%) 43.65 (7.0%)
-3.8% ( -19% - 15%)
OrNotHighHigh 779.95 (3.8%) 752.28 (1.8%)
-3.5% ( -8% -2%)
HighSloppyPhrase 10.37 (12.8%) 10.03 (3.5%)
-3.3% ( -17% - 14%)
LowPhrase 11.60 (8.9%) 11.23 (1.7%)
-3.2% ( -12% -8%)
LowTerm 1694.00 (8.9%) 1642.34 (5.5%)
-3.0% ( -16% - 12%)
MedTerm 1292.82 (9.3%) 1253.69 (8.2%)
-3.0% ( -18% - 15%)
AndHighMed 71.41 (9.9%) 69.77 (7.5%)
-2.3% ( -17% - 16%)
OrNotHighMed 634.32 (7.2%) 620.67 (7.5%)
-2.2% ( -15% - 13%)
Prefix3 110.65 (14.9%) 108.55 (8.7%)
-1.9% ( -22% - 25%)
OrHighLow 347.02 (4.3%) 340.51 (9.9%)
-1.9% ( -15% - 12%)
OrNotHighLow 591.61 (5.5%) 580.60 (9.0%)
-1.9% ( -15% - 13%)
OrHighNotMed 1258.21 (1.8%) 1237.28 (5.0%)
-1.7% ( -8% -5%)
Fuzzy1 91.79 (4.3%) 90.77 (11.1%)
-1.1% ( -15% - 14%)
OrHighMed 10.29 (7.9%) 10.25 (11.8%)
-0.4% ( -18% - 20%)
Wildcard 52.28 (6.3%) 52.21 (6.8%)
-0.1% ( -12% - 13%)
OrHighHigh8.16 (6.9%)8.22 (9.3%)
0.8% ( -14% - 18%)
AndHighLow 563.89 (9.1%) 569.31 (15.3%)
1.0% ( -21% - 27%)
HighPhrase 15.88 (9.3%) 16.04 (13.0%)
1.0% ( -19% - 25%)
MedPhrase 14.84 (9.0%) 15.15 (12.8%)
2.1% ( -18% - 26%)
HighSpanNear2.16 (9.8%)2.21 (10.1%)
2.3% ( -16% - 24%)
MedSloppyPhrase 18.48 (15.4%) 18.96 (18.9%)
2.6% ( -27% - 43%)
MedSpanNear 17.75 (3.8%) 18.31 (10.0%)
3.1% ( -10% - 17%)
HighTerm 1031.00 (9.9%) 1068.12 (17.1%)
3.6% ( -21% - 33%)
LowSpanNear8.22 (5.5%)8.53 (13.3%)
3.7% ( -14% - 23%)
HighTermDayOfYearSort9.78 (11.0%) 10.25 (18.2%)
4.8% ( -21% - 38%)
HighTermMonthSort 23.40 (26.5%) 27.11 (32.1%)
15.9% ( -33% - 101%)
{noformat}
The total runtime of each run did not change. Not sure how to interpret this.
> Improve ByteBufferGuard in Java 11
> --
>
> Key: LUCENE-8780
> URL: https://issues.apache.org/jira/browse/LUCENE-8780
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/store
>Affects Versions: master (9.0)
>Reporter: Uwe Schindler
>Assignee: Uwe Schindler
>Priority: Major
> Labels: Java11
> Attachments: LUCENE-8780.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In LUCENE-7409 we added {{ByteBufferGuard}} to protect MMapDirectory from
> crushing the JVM with SIGSEGV when you close and unmap the mmapped buffers of
> an IndexInput, while another thread is accessing it.
> The idea was to do a volatile write access to flush the caches (to trigger a
> full fence) and set a non-volatile boolean to true. All accesses would check
> the boolean and stop the caller from accessing the underlying ByteBuffer.
> This worked most of the time, until the JVM optimized away the plain read
> access to the boolean (you can easily see this after some runtime of our
> by-default ignored testcase).
> With master on Java 11, we can improve the whole thing. Using VarHandles you
> can use any access type when reading or writing the boolean. After reading
> Doug Lea's expanation <http://gee.cs.oswego.edu/dl/html/j9mm.html> and some
> testing, I was no longer able to crush my JDK (even after running for minutes
> unmapping bytebuffers).
> The apraoch is the sa
[jira] [Commented] (LUCENE-8780) Improve ByteBufferGuard in Java 11
[
https://issues.apache.org/jira/browse/LUCENE-8780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16828039#comment-16828039
]
Uwe Schindler commented on LUCENE-8780:
---
Thats the result after 20 runs with 6 searcher threads (with ParallelGC) on
Mike's lucenebench:
{noformat}
use java command /home/jenkins/tools/java/64bit/jdk-11.0.2/bin/java -server
-Xms2g -Xmx2g -XX:+UseParallelGC -Xbatch
JAVA:
openjdk version "11.0.2" 2019-01-15
OpenJDK Runtime Environment 18.9 (build 11.0.2+9)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.2+9, mixed mode)
OS:
Linux serv1.sd-datasolutions.de 4.18.0-17-generic #18~18.04.1-Ubuntu SMP Fri
Mar 15 15:27:12 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
[...]
Report after iter 19:
TaskQPS orig StdDev QPS patch StdDev
Pct diff
IntNRQ 30.88 (0.6%) 26.33 (0.8%)
-14.7% ( -16% - -13%)
PKLookup 107.70 (2.7%) 94.31 (2.9%)
-12.4% ( -17% - -7%)
AndHighHigh 10.76 (11.5%) 10.17 (3.3%)
-5.4% ( -18% - 10%)
Fuzzy2 45.10 (7.7%) 43.21 (9.0%)
-4.2% ( -19% - 13%)
LowSloppyPhrase7.28 (16.8%)6.98 (6.3%)
-4.2% ( -23% - 22%)
OrHighNotLow 783.24 (7.1%) 751.37 (2.5%)
-4.1% ( -12% -5%)
OrHighNotHigh 934.39 (6.5%) 896.38 (2.1%)
-4.1% ( -11% -4%)
Respell 45.36 (10.6%) 43.65 (7.0%)
-3.8% ( -19% - 15%)
OrNotHighHigh 779.95 (3.8%) 752.28 (1.8%)
-3.5% ( -8% -2%)
HighSloppyPhrase 10.37 (12.8%) 10.03 (3.5%)
-3.3% ( -17% - 14%)
LowPhrase 11.60 (8.9%) 11.23 (1.7%)
-3.2% ( -12% -8%)
LowTerm 1694.00 (8.9%) 1642.34 (5.5%)
-3.0% ( -16% - 12%)
MedTerm 1292.82 (9.3%) 1253.69 (8.2%)
-3.0% ( -18% - 15%)
AndHighMed 71.41 (9.9%) 69.77 (7.5%)
-2.3% ( -17% - 16%)
OrNotHighMed 634.32 (7.2%) 620.67 (7.5%)
-2.2% ( -15% - 13%)
Prefix3 110.65 (14.9%) 108.55 (8.7%)
-1.9% ( -22% - 25%)
OrHighLow 347.02 (4.3%) 340.51 (9.9%)
-1.9% ( -15% - 12%)
OrNotHighLow 591.61 (5.5%) 580.60 (9.0%)
-1.9% ( -15% - 13%)
OrHighNotMed 1258.21 (1.8%) 1237.28 (5.0%)
-1.7% ( -8% -5%)
Fuzzy1 91.79 (4.3%) 90.77 (11.1%)
-1.1% ( -15% - 14%)
OrHighMed 10.29 (7.9%) 10.25 (11.8%)
-0.4% ( -18% - 20%)
Wildcard 52.28 (6.3%) 52.21 (6.8%)
-0.1% ( -12% - 13%)
OrHighHigh8.16 (6.9%)8.22 (9.3%)
0.8% ( -14% - 18%)
AndHighLow 563.89 (9.1%) 569.31 (15.3%)
1.0% ( -21% - 27%)
HighPhrase 15.88 (9.3%) 16.04 (13.0%)
1.0% ( -19% - 25%)
MedPhrase 14.84 (9.0%) 15.15 (12.8%)
2.1% ( -18% - 26%)
HighSpanNear2.16 (9.8%)2.21 (10.1%)
2.3% ( -16% - 24%)
MedSloppyPhrase 18.48 (15.4%) 18.96 (18.9%)
2.6% ( -27% - 43%)
MedSpanNear 17.75 (3.8%) 18.31 (10.0%)
3.1% ( -10% - 17%)
HighTerm 1031.00 (9.9%) 1068.12 (17.1%)
3.6% ( -21% - 33%)
LowSpanNear8.22 (5.5%)8.53 (13.3%)
3.7% ( -14% - 23%)
HighTermDayOfYearSort9.78 (11.0%) 10.25 (18.2%)
4.8% ( -21% - 38%)
HighTermMonthSort 23.40 (26.5%) 27.11 (32.1%)
15.9% ( -33% - 101%)
{noformat}
The total runtime of each run did not change. Not sure how to interpret this.
> Improve ByteBufferGuard in Java 11
> --
>
> Key: LUCENE-8780
> URL: https://issues.apache.org/jira/browse/LUCENE-8780
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/store
>Affects Versions: master (9.0)
>Reporter: Uwe Schindler
>Assignee: Uwe Schindler
>Priority: Major
> Labels: Java11
> Attachments: LUCENE-8780.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In LUCENE-7409 we added {{ByteBufferGuard}} to protect MMapDirectory from
> crushing the JVM with SIGSEGV when you close and unmap the mmapped buffers
[GitHub] [lucene-solr] uschindler opened a new pull request #658: LUCENE-8780: Improve ByteBufferGuard with Java 11 VarHandles and new …
uschindler opened a new pull request #658: LUCENE-8780: Improve ByteBufferGuard with Java 11 VarHandles and new … URL: https://github.com/apache/lucene-solr/pull/658 …memory model This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8780) Improve ByteBufferGuard in Java 11
[
https://issues.apache.org/jira/browse/LUCENE-8780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16827872#comment-16827872
]
Uwe Schindler commented on LUCENE-8780:
---
I created a pull request for easier review and perf testing (easy chackout of
branch from my github repo): https://github.com/apache/lucene-solr/pull/658
> Improve ByteBufferGuard in Java 11
> --
>
> Key: LUCENE-8780
> URL: https://issues.apache.org/jira/browse/LUCENE-8780
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/store
>Affects Versions: master (9.0)
>Reporter: Uwe Schindler
>Assignee: Uwe Schindler
>Priority: Major
> Labels: Java11
> Attachments: LUCENE-8780.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In LUCENE-7409 we added {{ByteBufferGuard}} to protect MMapDirectory from
> crushing the JVM with SIGSEGV when you close and unmap the mmapped buffers of
> an IndexInput, while another thread is accessing it.
> The idea was to do a volatile write access to flush the caches (to trigger a
> full fence) and set a non-volatile boolean to true. All accesses would check
> the boolean and stop the caller from accessing the underlying ByteBuffer.
> This worked most of the time, until the JVM optimized away the plain read
> access to the boolean (you can easily see this after some runtime of our
> by-default ignored testcase).
> With master on Java 11, we can improve the whole thing. Using VarHandles you
> can use any access type when reading or writing the boolean. After reading
> Doug Lea's expanation <http://gee.cs.oswego.edu/dl/html/j9mm.html> and some
> testing, I was no longer able to crush my JDK (even after running for minutes
> unmapping bytebuffers).
> The apraoch is the same, we do a full-fenced write (standard volatile write)
> when we unmap, then we yield the thread (to finish in-flight reads in other
> threads) and then unmap all byte buffers.
> On the test side (read access), instead of using a plain read, we use the new
> "opaque read". Opaque reads are the same as plain reads, there are only
> different order requirements. Actually the main difference is explained by
> Doug like this: "For example in constructions in which the only modification
> of some variable x is for one thread to write in Opaque (or stronger) mode,
> X.setOpaque(this, 1), any other thread spinning in
> while(X.getOpaque(this)!=1){} will eventually terminate. Note that this
> guarantee does NOT hold in Plain mode, in which spin loops may (and usually
> do) infinitely loop -- they are not required to notice that a write ever
> occurred in another thread if it was not seen on first encounter." - And
> that's waht we want to have: We don't want to do volatile reads, but we want
> to prevent the compiler from optimizing away our read to the boolean. So we
> want it to "eventually" see the change. By the much stronger volatile write,
> the cache effects should be visible even faster (like in our Java 8 approach,
> just now we improved our read side).
> The new code is much slimmer (theoretically we could also use a AtomicBoolean
> for that and use the new method {{getOpaque()}}, but I wanted to prevent
> extra method calls, so I used a VarHandle directly).
> It's setup like this:
> - The underlying boolean field is a private member (with unused
> SuppressWarnings, as its unused by the java compiler), marked as volatile
> (that's the recommendation, but in reality it does not matter at all).
> - We create a VarHandle to access this boolean, we never do this directly
> (this is why the volatile marking does not affect us).
> - We use VarHandle.setVolatile() to change our "invalidated" boolean to
> "true", so enforcing a full fence
> - On the read side we use VarHandle.getOpaque() instead of VarHandle.get()
> (like in our old code for Java 8).
> I had to tune our test a bit, as the VarHandles make it take longer until it
> actually crushes (as optimizations jump in later). I also used a random for
> the reads to prevent the optimizer from removing all the bytebuffer reads.
> When we commit this, we can disable the test again (it takes approx 50 secs
> on my machine).
> I'd still like to see the differences between the plain read and the opaque
> read in production, so maybe [~mikemccand] or [~rcmuir] can do a comparison
> with nightly benchmarker?
> Have fun, maybe [~dweiss] has some ideas, too.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-8780) Improve ByteBufferGuard in Java 11
[
https://issues.apache.org/jira/browse/LUCENE-8780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uwe Schindler updated LUCENE-8780:
--
Labels: Java11 (was: )
> Improve ByteBufferGuard in Java 11
> --
>
> Key: LUCENE-8780
> URL: https://issues.apache.org/jira/browse/LUCENE-8780
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/store
>Affects Versions: master (9.0)
>Reporter: Uwe Schindler
>Assignee: Uwe Schindler
>Priority: Major
> Labels: Java11
> Attachments: LUCENE-8780.patch
>
>
> In LUCENE-7409 we added {{ByteBufferGuard}} to protect MMapDirectory from
> crushing the JVM with SIGSEGV when you close and unmap the mmapped buffers of
> an IndexInput, while another thread is accessing it.
> The idea was to do a volatile write access to flush the caches (to trigger a
> full fence) and set a non-volatile boolean to true. All accesses would check
> the boolean and stop the caller from accessing the underlying ByteBuffer.
> This worked most of the time, until the JVM optimized away the plain read
> access to the boolean (you can easily see this after some runtime of our
> by-default ignored testcase).
> With master on Java 11, we can improve the whole thing. Using VarHandles you
> can use any access type when reading or writing the boolean. After reading
> Doug Lea's expanation <http://gee.cs.oswego.edu/dl/html/j9mm.html> and some
> testing, I was no longer able to crush my JDK (even after running for minutes
> unmapping bytebuffers).
> The apraoch is the same, we do a full-fenced write (standard volatile write)
> when we unmap, then we yield the thread (to finish in-flight reads in other
> threads) and then unmap all byte buffers.
> On the test side (read access), instead of using a plain read, we use the new
> "opaque read". Opaque reads are the same as plain reads, there are only
> different order requirements. Actually the main difference is explained by
> Doug like this: "For example in constructions in which the only modification
> of some variable x is for one thread to write in Opaque (or stronger) mode,
> X.setOpaque(this, 1), any other thread spinning in
> while(X.getOpaque(this)!=1){} will eventually terminate. Note that this
> guarantee does NOT hold in Plain mode, in which spin loops may (and usually
> do) infinitely loop -- they are not required to notice that a write ever
> occurred in another thread if it was not seen on first encounter." - And
> that's waht we want to have: We don't want to do volatile reads, but we want
> to prevent the compiler from optimizing away our read to the boolean. So we
> want it to "eventually" see the change. By the much stronger volatile write,
> the cache effects should be visible even faster (like in our Java 8 approach,
> just now we improved our read side).
> The new code is much slimmer (theoretically we could also use a AtomicBoolean
> for that and use the new method {{getOpaque()}}, but I wanted to prevent
> extra method calls, so I used a VarHandle directly).
> It's setup like this:
> - The underlying boolean field is a private member (with unused
> SuppressWarnings, as its unused by the java compiler), marked as volatile
> (that's the recommendation, but in reality it does not matter at all).
> - We create a VarHandle to access this boolean, we never do this directly
> (this is why the volatile marking does not affect us).
> - We use VarHandle.setVolatile() to change our "invalidated" boolean to
> "true", so enforcing a full fence
> - On the read side we use VarHandle.getOpaque() instead of VarHandle.get()
> (like in our old code for Java 8).
> I had to tune our test a bit, as the VarHandles make it take longer until it
> actually crushes (as optimizations jump in later). I also used a random for
> the reads to prevent the optimizer from removing all the bytebuffer reads.
> When we commit this, we can disable the test again (it takes approx 50 secs
> on my machine).
> I'd still like to see the differences between the plain read and the opaque
> read in production, so maybe [~mikemccand] or [~rcmuir] can do a comparison
> with nightly benchmarker?
> Have fun, maybe [~dweiss] has some ideas, too.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-8780) Improve ByteBufferGuard in Java 11
[
https://issues.apache.org/jira/browse/LUCENE-8780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uwe Schindler updated LUCENE-8780:
--
Description:
In LUCENE-7409 we added {{ByteBufferGuard}} to protect MMapDirectory from
crushing the JVM with SIGSEGV when you close and unmap the mmapped buffers of
an IndexInput, while another thread is accessing it.
The idea was to do a volatile write access to flush the caches (to trigger a
full fence) and set a non-volatile boolean to true. All accesses would check
the boolean and stop the caller from accessing the underlying ByteBuffer. This
worked most of the time, until the JVM optimized away the plain read access to
the boolean (you can easily see this after some runtime of our by-default
ignored testcase).
With master on Java 11, we can improve the whole thing. Using VarHandles you
can use any access type when reading or writing the boolean. After reading Doug
Lea's expanation <http://gee.cs.oswego.edu/dl/html/j9mm.html> and some testing,
I was no longer able to crush my JDK (even after running for minutes unmapping
bytebuffers).
The apraoch is the same, we do a full-fenced write (standard volatile write)
when we unmap, then we yield the thread (to finish in-flight reads in other
threads) and then unmap all byte buffers.
On the test side (read access), instead of using a plain read, we use the new
"opaque read". Opaque reads are the same as plain reads, there are only
different order requirements. Actually the main difference is explained by Doug
like this: "For example in constructions in which the only modification of some
variable x is for one thread to write in Opaque (or stronger) mode,
X.setOpaque(this, 1), any other thread spinning in
while(X.getOpaque(this)!=1){} will eventually terminate. Note that this
guarantee does NOT hold in Plain mode, in which spin loops may (and usually do)
infinitely loop -- they are not required to notice that a write ever occurred
in another thread if it was not seen on first encounter." - And that's waht we
want to have: We don't want to do volatile reads, but we want to prevent the
compiler from optimizing away our read to the boolean. So we want it to
"eventually" see the change. By the much stronger volatile write, the cache
effects should be visible even faster (like in our Java 8 approach, just now we
improved our read side).
The new code is much slimmer (theoretically we could also use a AtomicBoolean
for that and use the new method {{getOpaque()}}, but I wanted to prevent extra
method calls, so I used a VarHandle directly).
It's setup like this:
- The underlying boolean field is a private member (with unused
SuppressWarnings, as its unused by the java compiler), marked as volatile
(that's the recommendation, but in reality it does not matter at all).
- We create a VarHandle to access this boolean, we never do this directly (this
is why the volatile marking does not affect us).
- We use VarHandle.setVolatile() to change our "invalidated" boolean to "true",
so enforcing a full fence
- On the read side we use VarHandle.getOpaque() instead of VarHandle.get()
(like in our old code for Java 8).
I had to tune our test a bit, as the VarHandles make it take longer until it
actually crushes (as optimizations jump in later). I also used a random for the
reads to prevent the optimizer from removing all the bytebuffer reads. When we
commit this, we can disable the test again (it takes approx 50 secs on my
machine).
I'd still like to see the differences between the plain read and the opaque
read in production, so maybe [~mikemccand] or [~rcmuir] can do a comparison
with nightly benchmarker?
Have fun, maybe [~dweiss] has some ideas, too.
was:
In LUCENE-7409 we added {{ByteBufferGuard}} to protect MMapDirectory from
crushing the JVM with SIGSEGV when you close and unmap the mmapped buffers of
an IndexInput, while another thread is accessing it.
The idea was to do a volatile write access to flush the caches (to trigger a
full fence) and set a non-volatile boolean to true. All accesses would check
the boolean and stop the caller from accessing the underlying ByteBuffer. This
worked most of the time, until the JVM optimized away the plain read access to
the boolean (you can easily see this after some runtime of our by-default
ignored testcase).
With master on Java 11, we can improve the whole thing. Using VarHandles you
can use any access type when reading or writing the boolean. After reading Doug
Lea's expanation <http://gee.cs.oswego.edu/dl/html/j9mm.html> and some testing,
I was no longer able to crush my JDK (even after running for minutes unmapping
bytebuffers).
The apraoch is the same, we do a full-fenced write (standard volatile write)
when we unmap, then we yield the thread (to finish in-flight reads in other
threads) and then un
[jira] [Updated] (LUCENE-8780) Improve ByteBufferGuard in Java 11
[
https://issues.apache.org/jira/browse/LUCENE-8780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uwe Schindler updated LUCENE-8780:
--
Attachment: LUCENE-8780.patch
> Improve ByteBufferGuard in Java 11
> --
>
> Key: LUCENE-8780
> URL: https://issues.apache.org/jira/browse/LUCENE-8780
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/store
>Affects Versions: master (9.0)
>Reporter: Uwe Schindler
>Assignee: Uwe Schindler
>Priority: Major
> Attachments: LUCENE-8780.patch
>
>
> In LUCENE-7409 we added {{ByteBufferGuard}} to protect MMapDirectory from
> crushing the JVM with SIGSEGV when you close and unmap the mmapped buffers of
> an IndexInput, while another thread is accessing it.
> The idea was to do a volatile write access to flush the caches (to trigger a
> full fence) and set a non-volatile boolean to true. All accesses would check
> the boolean and stop the caller from accessing the underlying ByteBuffer.
> This worked most of the time, until the JVM optimized away the plain read
> access to the boolean (you can easily see this after some runtime of our
> by-default ignored testcase).
> With master on Java 11, we can improve the whole thing. Using VarHandles you
> can use any access type when reading or writing the boolean. After reading
> Doug Lea's expanation and some testing, I was no longer able to crush my JDK
> (even after running for minutes unmapping bytebuffers).
> The apraoch is the same, we do a full-fenced write (standard volatile write)
> when we unmap, then we yield the thread (to finish in-flight reads in other
> threads) and then unmap all byte buffers.
> On the test side (read access), instead of using a plain read, we use the new
> "opaque read". Opaque reads are the same as plain reads, there are only
> different order requirements. Actually the main difference is explained by
> Doug like this: "For example in constructions in which the only modification
> of some variable x is for one thread to write in Opaque (or stronger) mode,
> X.setOpaque(this, 1), any other thread spinning in
> while(X.getOpaque(this)!=1){} will eventually terminate. Note that this
> guarantee does NOT hold in Plain mode, in which spin loops may (and usually
> do) infinitely loop -- they are not required to notice that a write ever
> occurred in another thread if it was not seen on first encounter." - And
> that's waht we want to have: We don't want to do volatile reads, but we want
> to prevent the compiler from optimizing away our read to the boolean. So we
> want it to "eventually" see the change. By the much stronger volatile write,
> the cache effects should be visible even faster (like in our Java 8 approach,
> jsut now we improved our read side).
> The new code is much slimmer (theoretically we could also use a AtomicBoolean
> for that and use the new method {{getOpaque()}}, but I wanted to prevent
> extra metod calls, so I used a VarHandle directly.
> It's setup like this:
> - The underlying boolean is a private member (with unused suppress wanrings,
> as its unused by the java compiler), marked as volatile (that's the
> recommendation, but in reality it does not matter at all).
> - We create a VarHandle to access this boolean, we never do this directly
> (this is why the volatile marking does not affect us).
> - We use VarHandle.setVolatile() to change out "invalidated" boolean to
> "true", so enforcing a full fence
> - On the read side we use VarHandle.getOpaque() instead of VarHandle.get()
> (as it would be the effect of our old code in Java 8).
> I had to tune our test a bit, as the VarHandles make it take longer until it
> actually crushes (as optimizations jump in later). I also used a random for
> the reads to prevent the optimizer from removing all the bytebuffer reads.
> When we commit this, we can disable the test again (it takes approx 50 secs
> on my machine).
> I'd still like to see the differences between the plain read and the opaque
> read in production, so maybe [~mikemccand] or [~rcmuir] can do a comparison
> with nightly benchmarker?
> Have fun, maybe [~dweiss] has some ideas, too.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-8780) Improve ByteBufferGuard in Java 11
[
https://issues.apache.org/jira/browse/LUCENE-8780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uwe Schindler updated LUCENE-8780:
--
Description:
In LUCENE-7409 we added {{ByteBufferGuard}} to protect MMapDirectory from
crushing the JVM with SIGSEGV when you close and unmap the mmapped buffers of
an IndexInput, while another thread is accessing it.
The idea was to do a volatile write access to flush the caches (to trigger a
full fence) and set a non-volatile boolean to true. All accesses would check
the boolean and stop the caller from accessing the underlying ByteBuffer. This
worked most of the time, until the JVM optimized away the plain read access to
the boolean (you can easily see this after some runtime of our by-default
ignored testcase).
With master on Java 11, we can improve the whole thing. Using VarHandles you
can use any access type when reading or writing the boolean. After reading Doug
Lea's expanation <http://gee.cs.oswego.edu/dl/html/j9mm.html> and some testing,
I was no longer able to crush my JDK (even after running for minutes unmapping
bytebuffers).
The apraoch is the same, we do a full-fenced write (standard volatile write)
when we unmap, then we yield the thread (to finish in-flight reads in other
threads) and then unmap all byte buffers.
On the test side (read access), instead of using a plain read, we use the new
"opaque read". Opaque reads are the same as plain reads, there are only
different order requirements. Actually the main difference is explained by Doug
like this: "For example in constructions in which the only modification of some
variable x is for one thread to write in Opaque (or stronger) mode,
X.setOpaque(this, 1), any other thread spinning in
while(X.getOpaque(this)!=1){} will eventually terminate. Note that this
guarantee does NOT hold in Plain mode, in which spin loops may (and usually do)
infinitely loop -- they are not required to notice that a write ever occurred
in another thread if it was not seen on first encounter." - And that's waht we
want to have: We don't want to do volatile reads, but we want to prevent the
compiler from optimizing away our read to the boolean. So we want it to
"eventually" see the change. By the much stronger volatile write, the cache
effects should be visible even faster (like in our Java 8 approach, jsut now we
improved our read side).
The new code is much slimmer (theoretically we could also use a AtomicBoolean
for that and use the new method {{getOpaque()}}, but I wanted to prevent extra
metod calls, so I used a VarHandle directly.
It's setup like this:
- The underlying boolean is a private member (with unused suppress wanrings, as
its unused by the java compiler), marked as volatile (that's the
recommendation, but in reality it does not matter at all).
- We create a VarHandle to access this boolean, we never do this directly (this
is why the volatile marking does not affect us).
- We use VarHandle.setVolatile() to change out "invalidated" boolean to "true",
so enforcing a full fence
- On the read side we use VarHandle.getOpaque() instead of VarHandle.get() (as
it would be the effect of our old code in Java 8).
I had to tune our test a bit, as the VarHandles make it take longer until it
actually crushes (as optimizations jump in later). I also used a random for the
reads to prevent the optimizer from removing all the bytebuffer reads. When we
commit this, we can disable the test again (it takes approx 50 secs on my
machine).
I'd still like to see the differences between the plain read and the opaque
read in production, so maybe [~mikemccand] or [~rcmuir] can do a comparison
with nightly benchmarker?
Have fun, maybe [~dweiss] has some ideas, too.
was:
In LUCENE-7409 we added {{ByteBufferGuard}} to protect MMapDirectory from
crushing the JVM with SIGSEGV when you close and unmap the mmapped buffers of
an IndexInput, while another thread is accessing it.
The idea was to do a volatile write access to flush the caches (to trigger a
full fence) and set a non-volatile boolean to true. All accesses would check
the boolean and stop the caller from accessing the underlying ByteBuffer. This
worked most of the time, until the JVM optimized away the plain read access to
the boolean (you can easily see this after some runtime of our by-default
ignored testcase).
With master on Java 11, we can improve the whole thing. Using VarHandles you
can use any access type when reading or writing the boolean. After reading Doug
Lea's expanation and some testing, I was no longer able to crush my JDK (even
after running for minutes unmapping bytebuffers).
The apraoch is the same, we do a full-fenced write (standard volatile write)
when we unmap, then we yield the thread (to finish in-flight reads in other
threads) and then unmap all byte buffers.
On the test s
[jira] [Commented] (LUCENE-8780) Improve ByteBufferGuard in Java 11
[
https://issues.apache.org/jira/browse/LUCENE-8780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16827607#comment-16827607
]
Uwe Schindler commented on LUCENE-8780:
---
Patch: [^LUCENE-8780.patch]
> Improve ByteBufferGuard in Java 11
> --
>
> Key: LUCENE-8780
> URL: https://issues.apache.org/jira/browse/LUCENE-8780
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/store
>Affects Versions: master (9.0)
>Reporter: Uwe Schindler
>Assignee: Uwe Schindler
>Priority: Major
> Attachments: LUCENE-8780.patch
>
>
> In LUCENE-7409 we added {{ByteBufferGuard}} to protect MMapDirectory from
> crushing the JVM with SIGSEGV when you close and unmap the mmapped buffers of
> an IndexInput, while another thread is accessing it.
> The idea was to do a volatile write access to flush the caches (to trigger a
> full fence) and set a non-volatile boolean to true. All accesses would check
> the boolean and stop the caller from accessing the underlying ByteBuffer.
> This worked most of the time, until the JVM optimized away the plain read
> access to the boolean (you can easily see this after some runtime of our
> by-default ignored testcase).
> With master on Java 11, we can improve the whole thing. Using VarHandles you
> can use any access type when reading or writing the boolean. After reading
> Doug Lea's expanation and some testing, I was no longer able to crush my JDK
> (even after running for minutes unmapping bytebuffers).
> The apraoch is the same, we do a full-fenced write (standard volatile write)
> when we unmap, then we yield the thread (to finish in-flight reads in other
> threads) and then unmap all byte buffers.
> On the test side (read access), instead of using a plain read, we use the new
> "opaque read". Opaque reads are the same as plain reads, there are only
> different order requirements. Actually the main difference is explained by
> Doug like this: "For example in constructions in which the only modification
> of some variable x is for one thread to write in Opaque (or stronger) mode,
> X.setOpaque(this, 1), any other thread spinning in
> while(X.getOpaque(this)!=1){} will eventually terminate. Note that this
> guarantee does NOT hold in Plain mode, in which spin loops may (and usually
> do) infinitely loop -- they are not required to notice that a write ever
> occurred in another thread if it was not seen on first encounter." - And
> that's waht we want to have: We don't want to do volatile reads, but we want
> to prevent the compiler from optimizing away our read to the boolean. So we
> want it to "eventually" see the change. By the much stronger volatile write,
> the cache effects should be visible even faster (like in our Java 8 approach,
> jsut now we improved our read side).
> The new code is much slimmer (theoretically we could also use a AtomicBoolean
> for that and use the new method {{getOpaque()}}, but I wanted to prevent
> extra metod calls, so I used a VarHandle directly.
> It's setup like this:
> - The underlying boolean is a private member (with unused suppress wanrings,
> as its unused by the java compiler), marked as volatile (that's the
> recommendation, but in reality it does not matter at all).
> - We create a VarHandle to access this boolean, we never do this directly
> (this is why the volatile marking does not affect us).
> - We use VarHandle.setVolatile() to change out "invalidated" boolean to
> "true", so enforcing a full fence
> - On the read side we use VarHandle.getOpaque() instead of VarHandle.get()
> (as it would be the effect of our old code in Java 8).
> I had to tune our test a bit, as the VarHandles make it take longer until it
> actually crushes (as optimizations jump in later). I also used a random for
> the reads to prevent the optimizer from removing all the bytebuffer reads.
> When we commit this, we can disable the test again (it takes approx 50 secs
> on my machine).
> I'd still like to see the differences between the plain read and the opaque
> read in production, so maybe [~mikemccand] or [~rcmuir] can do a comparison
> with nightly benchmarker?
> Have fun, maybe [~dweiss] has some ideas, too.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (LUCENE-8780) Improve ByteBufferGuard in Java 11
Uwe Schindler created LUCENE-8780:
-
Summary: Improve ByteBufferGuard in Java 11
Key: LUCENE-8780
URL: https://issues.apache.org/jira/browse/LUCENE-8780
Project: Lucene - Core
Issue Type: Improvement
Components: core/store
Affects Versions: master (9.0)
Reporter: Uwe Schindler
Assignee: Uwe Schindler
In LUCENE-7409 we added {{ByteBufferGuard}} to protect MMapDirectory from
crushing the JVM with SIGSEGV when you close and unmap the mmapped buffers of
an IndexInput, while another thread is accessing it.
The idea was to do a volatile write access to flush the caches (to trigger a
full fence) and set a non-volatile boolean to true. All accesses would check
the boolean and stop the caller from accessing the underlying ByteBuffer. This
worked most of the time, until the JVM optimized away the plain read access to
the boolean (you can easily see this after some runtime of our by-default
ignored testcase).
With master on Java 11, we can improve the whole thing. Using VarHandles you
can use any access type when reading or writing the boolean. After reading Doug
Lea's expanation and some testing, I was no longer able to crush my JDK (even
after running for minutes unmapping bytebuffers).
The apraoch is the same, we do a full-fenced write (standard volatile write)
when we unmap, then we yield the thread (to finish in-flight reads in other
threads) and then unmap all byte buffers.
On the test side (read access), instead of using a plain read, we use the new
"opaque read". Opaque reads are the same as plain reads, there are only
different order requirements. Actually the main difference is explained by Doug
like this: "For example in constructions in which the only modification of some
variable x is for one thread to write in Opaque (or stronger) mode,
X.setOpaque(this, 1), any other thread spinning in
while(X.getOpaque(this)!=1){} will eventually terminate. Note that this
guarantee does NOT hold in Plain mode, in which spin loops may (and usually do)
infinitely loop -- they are not required to notice that a write ever occurred
in another thread if it was not seen on first encounter." - And that's waht we
want to have: We don't want to do volatile reads, but we want to prevent the
compiler from optimizing away our read to the boolean. So we want it to
"eventually" see the change. By the much stronger volatile write, the cache
effects should be visible even faster (like in our Java 8 approach, jsut now we
improved our read side).
The new code is much slimmer (theoretically we could also use a AtomicBoolean
for that and use the new method {{getOpaque()}}, but I wanted to prevent extra
metod calls, so I used a VarHandle directly.
It's setup like this:
- The underlying boolean is a private member (with unused suppress wanrings, as
its unused by the java compiler), marked as volatile (that's the
recommendation, but in reality it does not matter at all).
- We create a VarHandle to access this boolean, we never do this directly (this
is why the volatile marking does not affect us).
- We use VarHandle.setVolatile() to change out "invalidated" boolean to "true",
so enforcing a full fence
- On the read side we use VarHandle.getOpaque() instead of VarHandle.get() (as
it would be the effect of our old code in Java 8).
I had to tune our test a bit, as the VarHandles make it take longer until it
actually crushes (as optimizations jump in later). I also used a random for the
reads to prevent the optimizer from removing all the bytebuffer reads. When we
commit this, we can disable the test again (it takes approx 50 secs on my
machine).
I'd still like to see the differences between the plain read and the opaque
read in production, so maybe [~mikemccand] or [~rcmuir] can do a comparison
with nightly benchmarker?
Have fun, maybe [~dweiss] has some ideas, too.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11579) Building Solr (master) using Java 11 with Maven has some old plugins
[ https://issues.apache.org/jira/browse/SOLR-11579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16825819#comment-16825819 ] Daniel Collins commented on SOLR-11579: --- The only remaining item here is patching the last few Maven modules (maven-surefire-plugin, and maven-bundle-plugin), and switching gmaven plugin to groovy-maven-plugin. Other than that, all the old issues are now resolved. [~erickerickson] should I just withdraw this and create a new one with the small patch (this ticket has kind of gone full circle!) or should we still keep it as part of this? > Building Solr (master) using Java 11 with Maven has some old plugins > > > Key: SOLR-11579 > URL: https://issues.apache.org/jira/browse/SOLR-11579 > Project: Solr > Issue Type: New Feature > Security Level: Public(Default Security Level. Issues are Public) > Components: Build >Affects Versions: 8.0 > Environment: Apache Maven 3.6.0 > (97c98ec64a1fdfee7767ce5ffb20918da4f719f3; 2018-10-24T19:41:47+01:00) > Maven home: C:\Users\CollinsDaniel\Utils\apache-maven-3.6.0\bin\.. > Java version: 11.0.2, vendor: Oracle Corporation, runtime: > C:\Users\CollinsDaniel\Utils\jdk-11.0.2 > Default locale: en_GB, platform encoding: Cp1252 > OS name: "windows 10", version: "10.0", arch: "amd64", family: "windows" > openjdk version "11.0.2" 2019-01-15 > OpenJDK Runtime Environment 18.9 (build 11.0.2+9) > OpenJDK 64-Bit Server VM 18.9 (build 11.0.2+9, mixed mode) >Reporter: Daniel Collins >Priority: Minor > Labels: maven, maven-enforcer-plugin > Attachments: SOLR-11579.patch > > > I'm calling this minor as I don't believe we officially support Java 11 (but > correct me if I'm wrong), and I've only tried on master > Several issues/notes: > # -enforcer plugin breaks if you try to do mvn install- > # -javadoc plugin breaks if you try to do mvn javadoc:jar (on Windows)- > # warnings from groovy-maven-plugin (TODO) > # -a lot of the maven plugins are quite old- > # forbiddenapis has some issues with Solr (I assume due to module split-up, > javax is no longer part of core) (TODO) > There is a patch for 3 now -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-11579) Building Solr (master) using Java 11 and Maven has some issues
[ https://issues.apache.org/jira/browse/SOLR-11579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Collins updated SOLR-11579: -- Attachment: (was: SOLR-11579_4) > Building Solr (master) using Java 11 and Maven has some issues > -- > > Key: SOLR-11579 > URL: https://issues.apache.org/jira/browse/SOLR-11579 > Project: Solr > Issue Type: New Feature > Security Level: Public(Default Security Level. Issues are Public) > Components: Build >Affects Versions: 8.0 > Environment: Apache Maven 3.6.0 > (97c98ec64a1fdfee7767ce5ffb20918da4f719f3; 2018-10-24T19:41:47+01:00) > Maven home: C:\Users\CollinsDaniel\Utils\apache-maven-3.6.0\bin\.. > Java version: 11.0.2, vendor: Oracle Corporation, runtime: > C:\Users\CollinsDaniel\Utils\jdk-11.0.2 > Default locale: en_GB, platform encoding: Cp1252 > OS name: "windows 10", version: "10.0", arch: "amd64", family: "windows" > openjdk version "11.0.2" 2019-01-15 > OpenJDK Runtime Environment 18.9 (build 11.0.2+9) > OpenJDK 64-Bit Server VM 18.9 (build 11.0.2+9, mixed mode) >Reporter: Daniel Collins >Priority: Minor > Labels: maven, maven-enforcer-plugin > Attachments: SOLR-11579.patch > > > I'm calling this minor as I don't believe we officially support Java 11 (but > correct me if I'm wrong), and I've only tried on master > Several issues/notes: > # -enforcer plugin breaks if you try to do mvn install- > # -javadoc plugin breaks if you try to do mvn javadoc:jar (on Windows)- > # warnings from groovy-maven-plugin (TODO) > # -a lot of the maven plugins are quite old- > # forbiddenapis has some issues with Solr (I assume due to module split-up, > javax is no longer part of core) (TODO) > I have patches for 1, 2, and 4. Still working on 3 and 5. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-11579) Building Solr (master) using Java 11 with Maven has some old plugins
[ https://issues.apache.org/jira/browse/SOLR-11579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Collins updated SOLR-11579: -- Summary: Building Solr (master) using Java 11 with Maven has some old plugins (was: Building Solr (master) using Java 11 and Maven has some issues) > Building Solr (master) using Java 11 with Maven has some old plugins > > > Key: SOLR-11579 > URL: https://issues.apache.org/jira/browse/SOLR-11579 > Project: Solr > Issue Type: New Feature > Security Level: Public(Default Security Level. Issues are Public) > Components: Build >Affects Versions: 8.0 > Environment: Apache Maven 3.6.0 > (97c98ec64a1fdfee7767ce5ffb20918da4f719f3; 2018-10-24T19:41:47+01:00) > Maven home: C:\Users\CollinsDaniel\Utils\apache-maven-3.6.0\bin\.. > Java version: 11.0.2, vendor: Oracle Corporation, runtime: > C:\Users\CollinsDaniel\Utils\jdk-11.0.2 > Default locale: en_GB, platform encoding: Cp1252 > OS name: "windows 10", version: "10.0", arch: "amd64", family: "windows" > openjdk version "11.0.2" 2019-01-15 > OpenJDK Runtime Environment 18.9 (build 11.0.2+9) > OpenJDK 64-Bit Server VM 18.9 (build 11.0.2+9, mixed mode) >Reporter: Daniel Collins >Priority: Minor > Labels: maven, maven-enforcer-plugin > Attachments: SOLR-11579.patch > > > I'm calling this minor as I don't believe we officially support Java 11 (but > correct me if I'm wrong), and I've only tried on master > Several issues/notes: > # -enforcer plugin breaks if you try to do mvn install- > # -javadoc plugin breaks if you try to do mvn javadoc:jar (on Windows)- > # warnings from groovy-maven-plugin (TODO) > # -a lot of the maven plugins are quite old- > # forbiddenapis has some issues with Solr (I assume due to module split-up, > javax is no longer part of core) (TODO) > I have patches for 1, 2, and 4. Still working on 3 and 5. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-11579) Building Solr (master) using Java 11 and Maven has some issues
[ https://issues.apache.org/jira/browse/SOLR-11579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Collins updated SOLR-11579: -- Attachment: SOLR-11579.patch > Building Solr (master) using Java 11 and Maven has some issues > -- > > Key: SOLR-11579 > URL: https://issues.apache.org/jira/browse/SOLR-11579 > Project: Solr > Issue Type: New Feature > Security Level: Public(Default Security Level. Issues are Public) > Components: Build >Affects Versions: 8.0 > Environment: Apache Maven 3.6.0 > (97c98ec64a1fdfee7767ce5ffb20918da4f719f3; 2018-10-24T19:41:47+01:00) > Maven home: C:\Users\CollinsDaniel\Utils\apache-maven-3.6.0\bin\.. > Java version: 11.0.2, vendor: Oracle Corporation, runtime: > C:\Users\CollinsDaniel\Utils\jdk-11.0.2 > Default locale: en_GB, platform encoding: Cp1252 > OS name: "windows 10", version: "10.0", arch: "amd64", family: "windows" > openjdk version "11.0.2" 2019-01-15 > OpenJDK Runtime Environment 18.9 (build 11.0.2+9) > OpenJDK 64-Bit Server VM 18.9 (build 11.0.2+9, mixed mode) >Reporter: Daniel Collins >Priority: Minor > Labels: maven, maven-enforcer-plugin > Attachments: SOLR-11579.patch > > > I'm calling this minor as I don't believe we officially support Java 11 (but > correct me if I'm wrong), and I've only tried on master > Several issues/notes: > # -enforcer plugin breaks if you try to do mvn install- > # -javadoc plugin breaks if you try to do mvn javadoc:jar (on Windows)- > # warnings from groovy-maven-plugin (TODO) > # -a lot of the maven plugins are quite old- > # forbiddenapis has some issues with Solr (I assume due to module split-up, > javax is no longer part of core) (TODO) > I have patches for 1, 2, and 4. Still working on 3 and 5. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-11579) Building Solr (master) using Java 11 and Maven has some issues
[ https://issues.apache.org/jira/browse/SOLR-11579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Collins updated SOLR-11579: -- Attachment: (was: SOLR-11579_2) > Building Solr (master) using Java 11 and Maven has some issues > -- > > Key: SOLR-11579 > URL: https://issues.apache.org/jira/browse/SOLR-11579 > Project: Solr > Issue Type: New Feature > Security Level: Public(Default Security Level. Issues are Public) > Components: Build >Affects Versions: 8.0 > Environment: Apache Maven 3.6.0 > (97c98ec64a1fdfee7767ce5ffb20918da4f719f3; 2018-10-24T19:41:47+01:00) > Maven home: C:\Users\CollinsDaniel\Utils\apache-maven-3.6.0\bin\.. > Java version: 11.0.2, vendor: Oracle Corporation, runtime: > C:\Users\CollinsDaniel\Utils\jdk-11.0.2 > Default locale: en_GB, platform encoding: Cp1252 > OS name: "windows 10", version: "10.0", arch: "amd64", family: "windows" > openjdk version "11.0.2" 2019-01-15 > OpenJDK Runtime Environment 18.9 (build 11.0.2+9) > OpenJDK 64-Bit Server VM 18.9 (build 11.0.2+9, mixed mode) >Reporter: Daniel Collins >Priority: Minor > Labels: maven, maven-enforcer-plugin > Attachments: SOLR-11579.patch > > > I'm calling this minor as I don't believe we officially support Java 11 (but > correct me if I'm wrong), and I've only tried on master > Several issues/notes: > # -enforcer plugin breaks if you try to do mvn install- > # -javadoc plugin breaks if you try to do mvn javadoc:jar (on Windows)- > # warnings from groovy-maven-plugin (TODO) > # -a lot of the maven plugins are quite old- > # forbiddenapis has some issues with Solr (I assume due to module split-up, > javax is no longer part of core) (TODO) > I have patches for 1, 2, and 4. Still working on 3 and 5. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-11579) Building Solr (master) using Java 11 with Maven has some old plugins
[ https://issues.apache.org/jira/browse/SOLR-11579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Collins updated SOLR-11579: -- Description: I'm calling this minor as I don't believe we officially support Java 11 (but correct me if I'm wrong), and I've only tried on master Several issues/notes: # -enforcer plugin breaks if you try to do mvn install- # -javadoc plugin breaks if you try to do mvn javadoc:jar (on Windows)- # warnings from groovy-maven-plugin (TODO) # -a lot of the maven plugins are quite old- # forbiddenapis has some issues with Solr (I assume due to module split-up, javax is no longer part of core) (TODO) There is a patch for 3 now was: I'm calling this minor as I don't believe we officially support Java 11 (but correct me if I'm wrong), and I've only tried on master Several issues/notes: # -enforcer plugin breaks if you try to do mvn install- # -javadoc plugin breaks if you try to do mvn javadoc:jar (on Windows)- # warnings from groovy-maven-plugin (TODO) # -a lot of the maven plugins are quite old- # forbiddenapis has some issues with Solr (I assume due to module split-up, javax is no longer part of core) (TODO) I have patches for 1, 2, and 4. Still working on 3 and 5. > Building Solr (master) using Java 11 with Maven has some old plugins > > > Key: SOLR-11579 > URL: https://issues.apache.org/jira/browse/SOLR-11579 > Project: Solr > Issue Type: New Feature > Security Level: Public(Default Security Level. Issues are Public) > Components: Build >Affects Versions: 8.0 > Environment: Apache Maven 3.6.0 > (97c98ec64a1fdfee7767ce5ffb20918da4f719f3; 2018-10-24T19:41:47+01:00) > Maven home: C:\Users\CollinsDaniel\Utils\apache-maven-3.6.0\bin\.. > Java version: 11.0.2, vendor: Oracle Corporation, runtime: > C:\Users\CollinsDaniel\Utils\jdk-11.0.2 > Default locale: en_GB, platform encoding: Cp1252 > OS name: "windows 10", version: "10.0", arch: "amd64", family: "windows" > openjdk version "11.0.2" 2019-01-15 > OpenJDK Runtime Environment 18.9 (build 11.0.2+9) > OpenJDK 64-Bit Server VM 18.9 (build 11.0.2+9, mixed mode) >Reporter: Daniel Collins >Priority: Minor > Labels: maven, maven-enforcer-plugin > Attachments: SOLR-11579.patch > > > I'm calling this minor as I don't believe we officially support Java 11 (but > correct me if I'm wrong), and I've only tried on master > Several issues/notes: > # -enforcer plugin breaks if you try to do mvn install- > # -javadoc plugin breaks if you try to do mvn javadoc:jar (on Windows)- > # warnings from groovy-maven-plugin (TODO) > # -a lot of the maven plugins are quite old- > # forbiddenapis has some issues with Solr (I assume due to module split-up, > javax is no longer part of core) (TODO) > There is a patch for 3 now -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11579) Building Solr (master) using Java 11 and Maven has some issues
[ https://issues.apache.org/jira/browse/SOLR-11579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16824914#comment-16824914 ] Daniel Collins commented on SOLR-11579: --- I now have openjdk 11 on both Linux and Windows. The javadoc plugin issue has gone away (mvn javadoc:jar fails on Windows for some other reason, still trying to work out what that is, but it works on Linux and ant documentation works on Windows). Most of the plugins are now updated, there are a few left, so I'll put a small patch together for that. > Building Solr (master) using Java 11 and Maven has some issues > -- > > Key: SOLR-11579 > URL: https://issues.apache.org/jira/browse/SOLR-11579 > Project: Solr > Issue Type: New Feature > Security Level: Public(Default Security Level. Issues are Public) > Components: Build >Affects Versions: 8.0 > Environment: Apache Maven 3.6.0 > (97c98ec64a1fdfee7767ce5ffb20918da4f719f3; 2018-10-24T19:41:47+01:00) > Maven home: C:\Users\CollinsDaniel\Utils\apache-maven-3.6.0\bin\.. > Java version: 11.0.2, vendor: Oracle Corporation, runtime: > C:\Users\CollinsDaniel\Utils\jdk-11.0.2 > Default locale: en_GB, platform encoding: Cp1252 > OS name: "windows 10", version: "10.0", arch: "amd64", family: "windows" > openjdk version "11.0.2" 2019-01-15 > OpenJDK Runtime Environment 18.9 (build 11.0.2+9) > OpenJDK 64-Bit Server VM 18.9 (build 11.0.2+9, mixed mode) >Reporter: Daniel Collins >Priority: Minor > Labels: maven, maven-enforcer-plugin > Attachments: SOLR-11579_2, SOLR-11579_4 > > > I'm calling this minor as I don't believe we officially support Java 11 (but > correct me if I'm wrong), and I've only tried on master > Several issues/notes: > # -enforcer plugin breaks if you try to do mvn install- > # -javadoc plugin breaks if you try to do mvn javadoc:jar (on Windows)- > # warnings from groovy-maven-plugin (TODO) > # -a lot of the maven plugins are quite old- > # forbiddenapis has some issues with Solr (I assume due to module split-up, > javax is no longer part of core) (TODO) > I have patches for 1, 2, and 4. Still working on 3 and 5. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-11579) Building Solr (master) using Java 11 and Maven has some issues
[ https://issues.apache.org/jira/browse/SOLR-11579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Collins updated SOLR-11579: -- Description: I'm calling this minor as I don't believe we officially support Java 11 (but correct me if I'm wrong), and I've only tried on master Several issues/notes: # -enforcer plugin breaks if you try to do mvn install- # -javadoc plugin breaks if you try to do mvn javadoc:jar (on Windows)- # warnings from groovy-maven-plugin (TODO) # -a lot of the maven plugins are quite old- # forbiddenapis has some issues with Solr (I assume due to module split-up, javax is no longer part of core) (TODO) I have patches for 1, 2, and 4. Still working on 3 and 5. was: I'm calling this minor as I don't believe we officially support Java 11 (but correct me if I'm wrong), and I've only tried on master Several issues/notes: # -enforcer plugin breaks if you try to do mvn install- # -javadoc plugin breaks if you try to do mvn javadoc:jar (on Windows)- # warnings from groovy-maven-plugin (TODO) # a lot of the maven plugins are quite old # forbiddenapis has some issues with Solr (I assume due to module split-up, javax is no longer part of core) (TODO) I have patches for 1, 2, and 4. Still working on 3 and 5. > Building Solr (master) using Java 11 and Maven has some issues > -- > > Key: SOLR-11579 > URL: https://issues.apache.org/jira/browse/SOLR-11579 > Project: Solr > Issue Type: New Feature > Security Level: Public(Default Security Level. Issues are Public) > Components: Build >Affects Versions: 8.0 > Environment: Apache Maven 3.6.0 > (97c98ec64a1fdfee7767ce5ffb20918da4f719f3; 2018-10-24T19:41:47+01:00) > Maven home: C:\Users\CollinsDaniel\Utils\apache-maven-3.6.0\bin\.. > Java version: 11.0.2, vendor: Oracle Corporation, runtime: > C:\Users\CollinsDaniel\Utils\jdk-11.0.2 > Default locale: en_GB, platform encoding: Cp1252 > OS name: "windows 10", version: "10.0", arch: "amd64", family: "windows" > openjdk version "11.0.2" 2019-01-15 > OpenJDK Runtime Environment 18.9 (build 11.0.2+9) > OpenJDK 64-Bit Server VM 18.9 (build 11.0.2+9, mixed mode) >Reporter: Daniel Collins >Priority: Minor > Labels: maven, maven-enforcer-plugin > Attachments: SOLR-11579_2, SOLR-11579_4 > > > I'm calling this minor as I don't believe we officially support Java 11 (but > correct me if I'm wrong), and I've only tried on master > Several issues/notes: > # -enforcer plugin breaks if you try to do mvn install- > # -javadoc plugin breaks if you try to do mvn javadoc:jar (on Windows)- > # warnings from groovy-maven-plugin (TODO) > # -a lot of the maven plugins are quite old- > # forbiddenapis has some issues with Solr (I assume due to module split-up, > javax is no longer part of core) (TODO) > I have patches for 1, 2, and 4. Still working on 3 and 5. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-11579) Building Solr (master) using Java 11 and Maven has some issues
[ https://issues.apache.org/jira/browse/SOLR-11579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Collins updated SOLR-11579: -- Description: I'm calling this minor as I don't believe we officially support Java 11 (but correct me if I'm wrong), and I've only tried on master Several issues/notes: # -enforcer plugin breaks if you try to do mvn install- # -javadoc plugin breaks if you try to do mvn javadoc:jar (on Windows)- # warnings from groovy-maven-plugin (TODO) # a lot of the maven plugins are quite old # forbiddenapis has some issues with Solr (I assume due to module split-up, javax is no longer part of core) (TODO) I have patches for 1, 2, and 4. Still working on 3 and 5. was: I'm calling this minor as I don't believe we officially support Java 11 (but correct me if I'm wrong), and I've only tried on master Several issues/notes: # -enforcer plugin breaks if you try to do mvn install- # javadoc plugin breaks if you try to do mvn javadoc:jar (on Windows) # warnings from groovy-maven-plugin (TODO) # a lot of the maven plugins are quite old # forbiddenapis has some issues with Solr (I assume due to module split-up, javax is no longer part of core) (TODO) I have patches for 1, 2, and 4. Still working on 3 and 5. > Building Solr (master) using Java 11 and Maven has some issues > -- > > Key: SOLR-11579 > URL: https://issues.apache.org/jira/browse/SOLR-11579 > Project: Solr > Issue Type: New Feature > Security Level: Public(Default Security Level. Issues are Public) > Components: Build >Affects Versions: 8.0 > Environment: Apache Maven 3.6.0 > (97c98ec64a1fdfee7767ce5ffb20918da4f719f3; 2018-10-24T19:41:47+01:00) > Maven home: C:\Users\CollinsDaniel\Utils\apache-maven-3.6.0\bin\.. > Java version: 11.0.2, vendor: Oracle Corporation, runtime: > C:\Users\CollinsDaniel\Utils\jdk-11.0.2 > Default locale: en_GB, platform encoding: Cp1252 > OS name: "windows 10", version: "10.0", arch: "amd64", family: "windows" > openjdk version "11.0.2" 2019-01-15 > OpenJDK Runtime Environment 18.9 (build 11.0.2+9) > OpenJDK 64-Bit Server VM 18.9 (build 11.0.2+9, mixed mode) >Reporter: Daniel Collins >Priority: Minor > Labels: maven, maven-enforcer-plugin > Attachments: SOLR-11579_2, SOLR-11579_4 > > > I'm calling this minor as I don't believe we officially support Java 11 (but > correct me if I'm wrong), and I've only tried on master > Several issues/notes: > # -enforcer plugin breaks if you try to do mvn install- > # -javadoc plugin breaks if you try to do mvn javadoc:jar (on Windows)- > # warnings from groovy-maven-plugin (TODO) > # a lot of the maven plugins are quite old > # forbiddenapis has some issues with Solr (I assume due to module split-up, > javax is no longer part of core) (TODO) > I have patches for 1, 2, and 4. Still working on 3 and 5. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-13424) Ref Guide PDF build throws Java warnings after requiring Java 11 on master
[
https://issues.apache.org/jira/browse/SOLR-13424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cassandra Targett updated SOLR-13424:
-
Summary: Ref Guide PDF build throws Java warnings after requiring Java 11
on master (was: Ref Guide PDF build throws Java errors after requiring Java 11
on master)
> Ref Guide PDF build throws Java warnings after requiring Java 11 on master
> --
>
> Key: SOLR-13424
> URL: https://issues.apache.org/jira/browse/SOLR-13424
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: documentation
>Reporter: Cassandra Targett
>Priority: Major
>
> Since the {{master}} branch requires Java 11, the Ref Guide build has started
> throwing WARNINGs that we should look into. Despite the WARNINGs, the PDF
> build finishes successfully and appears the same as usual.
> There are a couple moving parts here - first some classes [~hossman] wrote to
> validate links in the PDF, and second a .jar from the {{asciidoctor-ant}}
> project. I suspect some of these are related to Hoss' code and some are from
> the project that's doing the PDF conversion.
> I'll put the output from a Jenkins job in the comment to show the errors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13424) Ref Guide PDF build throws Java errors after requiring Java 11 on master
[
https://issues.apache.org/jira/browse/SOLR-13424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16824216#comment-16824216
]
Cassandra Targett commented on SOLR-13424:
--
This is from
https://builds.apache.org/view/L/view/Lucene/job/Solr-reference-guide-master/15365/
(23 April 2019).
The PDF target has several dependent targets, so runs in stages.
The {{build-nav-data-files}} dependent target produces the following. This is
from Hoss' classes, and builds the hierarchical page navigation:
{code}
build-nav-data-files:
[java] Building up tree of all known pages
[java] unsupported Java version "11", defaulting to 1.7
[java] WARNING: An illegal reflective access operation has occurred
[java] WARNING: Illegal reflective access by org.jruby.util.io.FilenoUtil
(file:/home/jenkins/.ivy2/cache/org.asciidoctor/asciidoctor-ant/jars/asciidoctor-ant-1.6.0-alpha.5.jar)
to method sun.nio.ch.SelChImpl.getFD()
[java] WARNING: Please consider reporting this to the maintainers of
org.jruby.util.io.FilenoUtil
[java] WARNING: Use --illegal-access=warn to enable warnings of further
illegal reflective access operations
[java] WARNING: All illegal access operations will be denied in a future
release
[java] Creating
/home/jenkins/jenkins-slave/workspace/Solr-reference-guide-master/solr/build/solr-ref-guide/content/_data/pdf-main-body.adoc
[java] Creating
/home/jenkins/jenkins-slave/workspace/Solr-reference-guide-master/solr/build/solr-ref-guide/content/_data/scrollnav.json
[java] Creating
/home/jenkins/jenkins-slave/workspace/Solr-reference-guide-master/solr/build/solr-ref-guide/content/_data/sidebar.json
{code}
The {{bare-bones-html-validation}} creates a stripped-down HTML for checking
that inter-document links work properly, and is also from Hoss' classes, but
uses asciidoctor-ant to do the conversion to HTML (some unrelated messages at
the end have been stripped):
{code}
bare-bones-html-validation:
[mkdir] Created dir:
/home/jenkins/jenkins-slave/workspace/Solr-reference-guide-master/solr/build/solr-ref-guide/bare-bones-html
[asciidoctor:convert] unsupported Java version "11", defaulting to 1.7
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.jruby.util.io.FilenoUtil
(file:/home/jenkins/jenkins-slave/workspace/Solr-reference-guide-master/solr/solr-ref-guide/lib/asciidoctor-ant-1.6.0-alpha.5.jar)
to method sun.nio.ch.SelChImpl.getFD()
WARNING: Please consider reporting this to the maintainers of
org.jruby.util.io.FilenoUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal
reflective access operations
WARNING: All illegal access operations will be denied in a future release
[asciidoctor:convert] Render asciidoc files from
/home/jenkins/jenkins-slave/workspace/Solr-reference-guide-master/solr/build/solr-ref-guide/content
to
/home/jenkins/jenkins-slave/workspace/Solr-reference-guide-master/solr/build/solr-ref-guide/bare-bones-html
with backend=html5
...
[java] Processed 2526 links (1839 relative) to 3355 anchors in 253 files
[echo] Validated Links & Anchors via:
/home/jenkins/jenkins-slave/workspace/Solr-reference-guide-master/solr/build/solr-ref-guide/bare-bones-html/
{code}
> Ref Guide PDF build throws Java errors after requiring Java 11 on master
>
>
> Key: SOLR-13424
> URL: https://issues.apache.org/jira/browse/SOLR-13424
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: documentation
>Reporter: Cassandra Targett
>Priority: Major
>
> Since the {{master}} branch requires Java 11, the Ref Guide build has started
> throwing WARNINGs that we should look into. Despite the WARNINGs, the PDF
> build finishes successfully and appears the same as usual.
> There are a couple moving parts here - first some classes [~hossman] wrote to
> validate links in the PDF, and second a .jar from the {{asciidoctor-ant}}
> project. I suspect some of these are related to Hoss' code and some are from
> the project that's doing the PDF conversion.
> I'll put the output from a Jenkins job in the comment to show the errors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-13424) Ref Guide PDF build throws Java errors after requiring Java 11 on master
Cassandra Targett created SOLR-13424:
Summary: Ref Guide PDF build throws Java errors after requiring
Java 11 on master
Key: SOLR-13424
URL: https://issues.apache.org/jira/browse/SOLR-13424
Project: Solr
Issue Type: Bug
Security Level: Public (Default Security Level. Issues are Public)
Components: documentation
Reporter: Cassandra Targett
Since the {{master}} branch requires Java 11, the Ref Guide build has started
throwing WARNINGs that we should look into. Despite the WARNINGs, the PDF build
finishes successfully and appears the same as usual.
There are a couple moving parts here - first some classes [~hossman] wrote to
validate links in the PDF, and second a .jar from the {{asciidoctor-ant}}
project. I suspect some of these are related to Hoss' code and some are from
the project that's doing the PDF conversion.
I'll put the output from a Jenkins job in the comment to show the errors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
RE: Change of master to Java 11 tomorrow, 2019-04-16, 12:00 UTC
That groovy code is unfixable, it's a known issue. Warning existed since Java 9... We may try to update to later groovy, but it's a known issue and can be done later. One thing after the other! Just ignore it. For those who follow Jenkins jobs during the last years, it's well known. Uwe Am April 16, 2019 6:22:09 PM UTC schrieb Chris Hostetter : > >: The master branch was updated to use Java 11 - all Jenkins Jobs are >back online! Let's see if additional Jenkins failures occur. > >FYI: first oddity i noticed is coming from some groovy code that seems >to run as part of "common.test" ? > >$ git clean -fx && cd solr/core/ && ant test >-Dtestcase=TestCloudConsistency >... > [junit4] Tests summary: 1 suite, 2 tests > [echo] 5 slowest tests: >[junit4:tophints] 315.23s | org.apache.solr.cloud.ShardSplitTest >[junit4:tophints] 260.44s | org.apache.solr.search.facet.TestJsonFacets >[junit4:tophints] 208.15s | org.apache.solr.cloud.TestHossSanity >[junit4:tophints] 186.80s | >org.apache.solr.cloud.api.collections.ShardSplitTest >[junit4:tophints] 125.64s | org.apache.solr.cloud.OnlyLeaderIndexesTest > >-check-totals: >WARNING: An illegal reflective access operation has occurred >WARNING: Illegal reflective access by >org.codehaus.groovy.reflection.CachedClass >(file:/home/hossman/.ivy2/cache/org.codehaus.groovy/groovy-all/jars/groovy-all-2.4.15.jar) >to method java.lang.Object.finalize() >WARNING: Please consider reporting this to the maintainers of >org.codehaus.groovy.reflection.CachedClass >WARNING: Use --illegal-access=warn to enable warnings of further >illegal reflective access operations >WARNING: All illegal access operations will be denied in a future >release > >common.test: > >BUILD SUCCESSFUL >Total time: 1 minute 31 seconds > >I'm using... > >hossman@tray:~/lucene/dev/solr/core [master] $ java -version >openjdk version "11.0.2" 2019-01-15 >OpenJDK Runtime Environment 18.9 (build 11.0.2+9) >OpenJDK 64-Bit Server VM 18.9 (build 11.0.2+9, mixed mode) > >...which i believe is the latest? > >-Hoss >http://www.lucidworks.com/ > >- >To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >For additional commands, e-mail: dev-h...@lucene.apache.org -- Uwe Schindler Achterdiek 19, 28357 Bremen https://www.thetaphi.de
RE: Change of master to Java 11 tomorrow, 2019-04-16, 12:00 UTC
Hi again, I was on my mobile only, so here is the "lengthy" explanation: > FYI: first oddity i noticed is coming from some groovy code that seems > to run as part of "common.test" ? > -check-totals: > WARNING: An illegal reflective access operation has occurred > WARNING: Illegal reflective access by > org.codehaus.groovy.reflection.CachedClass > (file:/home/hossman/.ivy2/cache/org.codehaus.groovy/groovy- > all/jars/groovy-all-2.4.15.jar) to method java.lang.Object.finalize() > WARNING: Please consider reporting this to the maintainers of > org.codehaus.groovy.reflection.CachedClass > WARNING: Use --illegal-access=warn to enable warnings of further illegal > reflective access operations > WARNING: All illegal access operations will be denied in a future release This is a warning that's triggered by the default behavior of Groovy to call "setAccessible" on all and every field/method of every class it sees. If it does this on a JDK one, the warning is printed. But for Groovy it's harmless, as a failing "setAccessible" is just ignored. The method is then just not available. As our Groovy code does not need to access private stuff, it does not matter is these are accessible or not. Gradle is also full of those warnings, they are just hidden by the startup scripts that passes a special flag to the JVM. For Ant we can't do this, as it's not our startup script. If Java forbids in a later version that we access internal classes - who cares?! The whole thing is a stupidity in early Groovy versions when they decided that Groovy has access to everything by default. Stupid decision, but we have to live with it. Groovy is still arguing against the default in Java to print the warning, but there are no movements. As far as I know in Groovy 3 they may have removed that legacy behaviour, but I am not sure. Let's try it out. For now, just ignore the warning, it's harmless! Asciidoctor has a similar problem BTW, but there it might be serious as a failure to make something accessible may block it from running. Uwe - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
RE: Change of master to Java 11 tomorrow, 2019-04-16, 12:00 UTC
: > - The HTML linter based on HTML Tidy was disabled until we found a : > replacement (no HTML 5 support - Java 11 produces HTML5 Javadocs). So it : > might happen that bad javadocs are not detected in master but after merge : > to 8.x branch they might suddenly appear. : : Maybe we can use the infrastriucture using ASCIIdoctor of the Solr Ref guide here? It seems to also do link / HTML checks. The code in the ref-guide's CheckLinksAndAnchors.java is *very* specialized around dealing with the specific problems that exist in asciidoctor when building merged/monolithic docs (like our PDF) from multiple source documents -- ie: only a small bit of it is about checking links, most of it is about checking for duplicate anchors; and even the link checking makes a lot of assumptions about the structure/conventions of generated asciidoctor output. If someone wanted to build a quick and dirty javadoc link checker, the jsoup parsing code in CheckLinksAndAnchors.java might be a good starting place, but it would probably be eaiser to build a new one then trying to generalize what's there to work on arbitrary html. -Hoss http://www.lucidworks.com/ - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
RE: Change of master to Java 11 tomorrow, 2019-04-16, 12:00 UTC
: The master branch was updated to use Java 11 - all Jenkins Jobs are back online! Let's see if additional Jenkins failures occur. FYI: first oddity i noticed is coming from some groovy code that seems to run as part of "common.test" ? $ git clean -fx && cd solr/core/ && ant test -Dtestcase=TestCloudConsistency ... [junit4] Tests summary: 1 suite, 2 tests [echo] 5 slowest tests: [junit4:tophints] 315.23s | org.apache.solr.cloud.ShardSplitTest [junit4:tophints] 260.44s | org.apache.solr.search.facet.TestJsonFacets [junit4:tophints] 208.15s | org.apache.solr.cloud.TestHossSanity [junit4:tophints] 186.80s | org.apache.solr.cloud.api.collections.ShardSplitTest [junit4:tophints] 125.64s | org.apache.solr.cloud.OnlyLeaderIndexesTest -check-totals: WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by org.codehaus.groovy.reflection.CachedClass (file:/home/hossman/.ivy2/cache/org.codehaus.groovy/groovy-all/jars/groovy-all-2.4.15.jar) to method java.lang.Object.finalize() WARNING: Please consider reporting this to the maintainers of org.codehaus.groovy.reflection.CachedClass WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release common.test: BUILD SUCCESSFUL Total time: 1 minute 31 seconds I'm using... hossman@tray:~/lucene/dev/solr/core [master] $ java -version openjdk version "11.0.2" 2019-01-15 OpenJDK Runtime Environment 18.9 (build 11.0.2+9) OpenJDK 64-Bit Server VM 18.9 (build 11.0.2+9, mixed mode) ...which i believe is the latest? -Hoss http://www.lucidworks.com/ - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Change of master to Java 11 tomorrow, 2019-04-16, 12:00 UTC
Big THANKS to all who worked on this. I just took about 10 minutes to have everything up and running in IntelliJ, building from the command shell, etc. Most all of that just waiting for compilations to finish and tests to run…. That was about as painless as I could hope for. Erick > On Apr 16, 2019, at 5:13 AM, Uwe Schindler wrote: > > Hi, > > The master branch was updated to use Java 11 - all Jenkins Jobs are back > online! Let's see if additional Jenkins failures occur. > > Go ahead and update your local dev environment: > >> As discussed on https://issues.apache.org/jira/browse/LUCENE-8738, we are >> planning to switch master branch to use Java 11 tomorrow. The Apache and >> Policeman Jenkins servers are prepared to handle that change, but now it's >> your turn: >> >> - Install a JDK 11 (remark: no Oracle JDK is needed, just use OpenJDK, e.g. >> from AdoptOpenJDK) >> - If you are working on master, you have to use Java 11 also in your IDE (run >> ant eclipse, netbeans,...). We tested IDE integrations for the common cases. >> If there are bugs, just fix them if you stumble on them. >> - If you are also maintaining Jenkins servers, update their configs! >> - If your merge/cherry pick changes from master to 8.x, don't forget to run >> precommit with Java 8 (!) and adopt needed changes. >> >> We will merge the branch with those changes tomorrow, 2019-04-16 >> afternoon UTC time, around 12 pm. As always, some hickups might happen, >> so ignore failing Jenkins jobs! I hope it settles by the time. > > Here is the TODO list if you have an idea: > >> What's not working: >> >> - The HTML linter based on HTML Tidy was disabled until we found a >> replacement (no HTML 5 support - Java 11 produces HTML5 Javadocs). So it >> might happen that bad javadocs are not detected in master but after merge >> to 8.x branch they might suddenly appear. > > Maybe we can use the infrastriucture using ASCIIdoctor of the Solr Ref guide > here? It seems to also do link / HTML checks. > >> - OpenClover code coverage Jenkins Jobs were disabled (no support yet for >> Java 11) >> >> TODOs: >> >> - Use Java 11 features, e.g. replace many unmodifiable collections with >> Map.of/Set.of/List.of constructs >> - Cleanup new warnings (we have to do this anyways, as javac is not even >> happy in Java 8) > > Thanks, especially to Adrien for doing the initial work! > Uwe > > > - > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org > - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
RE: Change of master to Java 11 tomorrow, 2019-04-16, 12:00 UTC
Hi, The master branch was updated to use Java 11 - all Jenkins Jobs are back online! Let's see if additional Jenkins failures occur. Go ahead and update your local dev environment: > As discussed on https://issues.apache.org/jira/browse/LUCENE-8738, we are > planning to switch master branch to use Java 11 tomorrow. The Apache and > Policeman Jenkins servers are prepared to handle that change, but now it's > your turn: > > - Install a JDK 11 (remark: no Oracle JDK is needed, just use OpenJDK, e.g. > from AdoptOpenJDK) > - If you are working on master, you have to use Java 11 also in your IDE (run > ant eclipse, netbeans,...). We tested IDE integrations for the common cases. > If there are bugs, just fix them if you stumble on them. > - If you are also maintaining Jenkins servers, update their configs! > - If your merge/cherry pick changes from master to 8.x, don't forget to run > precommit with Java 8 (!) and adopt needed changes. > > We will merge the branch with those changes tomorrow, 2019-04-16 > afternoon UTC time, around 12 pm. As always, some hickups might happen, > so ignore failing Jenkins jobs! I hope it settles by the time. Here is the TODO list if you have an idea: > What's not working: > > - The HTML linter based on HTML Tidy was disabled until we found a > replacement (no HTML 5 support - Java 11 produces HTML5 Javadocs). So it > might happen that bad javadocs are not detected in master but after merge > to 8.x branch they might suddenly appear. Maybe we can use the infrastriucture using ASCIIdoctor of the Solr Ref guide here? It seems to also do link / HTML checks. > - OpenClover code coverage Jenkins Jobs were disabled (no support yet for > Java 11) > > TODOs: > > - Use Java 11 features, e.g. replace many unmodifiable collections with > Map.of/Set.of/List.of constructs > - Cleanup new warnings (we have to do this anyways, as javac is not even > happy in Java 8) Thanks, especially to Adrien for doing the initial work! Uwe - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Change of master to Java 11 tomorrow, 2019-04-16, 12:00 UTC
Hi committers, As discussed on https://issues.apache.org/jira/browse/LUCENE-8738, we are planning to switch master branch to use Java 11 tomorrow. The Apache and Policeman Jenkins servers are prepared to handle that change, but now it's your turn: - Install a JDK 11 (remark: no Oracle JDK is needed, just use OpenJDK, e.g. from AdoptOpenJDK) - If you are working on master, you have to use Java 11 also in your IDE (run ant eclipse, netbeans,...). We tested IDE integrations for the common cases. If there are bugs, just fix them if you stumble on them. - If you are also maintaining Jenkins servers, update their configs! - If your merge/cherry pick changes from master to 8.x, don't forget to run precommit with Java 8 (!) and adopt needed changes. We will merge the branch with those changes tomorrow, 2019-04-16 afternoon UTC time, around 12 pm. As always, some hickups might happen, so ignore failing Jenkins jobs! I hope it settles by the time. What's not working: - The HTML linter based on HTML Tidy was disabled until we found a replacement (no HTML 5 support - Java 11 produces HTML5 Javadocs). So it might happen that bad javadocs are not detected in master but after merge to 8.x branch they might suddenly appear. - OpenClover code coverage Jenkins Jobs were disabled (no support yet for Java 11) TODOs: - Use Java 11 features, e.g. replace many unmodifiable collections with Map.of/Set.of/List.of constructs - Cleanup new warnings (we have to do this anyways, as javac is not even happy in Java 8) Uwe - Uwe Schindler Achterdiek 19, D-28357 Bremen https://www.thetaphi.de eMail: u...@thetaphi.de - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-8763) Update to OpenClover that works with Java 11
[ https://issues.apache.org/jira/browse/LUCENE-8763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uwe Schindler updated LUCENE-8763: -- Labels: Java11 (was: ) > Update to OpenClover that works with Java 11 > > > Key: LUCENE-8763 > URL: https://issues.apache.org/jira/browse/LUCENE-8763 > Project: Lucene - Core > Issue Type: Bug > Components: general/build >Affects Versions: master (9.0) >Reporter: Uwe Schindler >Assignee: Uwe Schindler >Priority: Major > Labels: Java11 > > OpenClover does not yet have a release that works with Java 11, so we have to > disable the task and add a TODO for updating it. > I will commit this for now to the LUCENE-8738 branch. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8763) Update to OpenClover that works with Java 11
[ https://issues.apache.org/jira/browse/LUCENE-8763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16817061#comment-16817061 ] ASF subversion and git services commented on LUCENE-8763: - Commit 831e940e02305eef6d99799bc1ba52e62d7190f5 in lucene-solr's branch refs/heads/jira/LUCENE-8738 from Uwe Schindler [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=831e940 ] LUCENE-8738, LUCENE-8763: Disable OpenClover until a release with Java 11 support is available > Update to OpenClover that works with Java 11 > > > Key: LUCENE-8763 > URL: https://issues.apache.org/jira/browse/LUCENE-8763 > Project: Lucene - Core > Issue Type: Bug > Components: general/build >Affects Versions: master (9.0) >Reporter: Uwe Schindler >Assignee: Uwe Schindler >Priority: Major > > OpenClover does not yet have a release that works with Java 11, so we have to > disable the task and add a TODO for updating it. > I will commit this for now to the LUCENE-8738 branch. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (LUCENE-8763) Update to OpenClover that works with Java 11
Uwe Schindler created LUCENE-8763: - Summary: Update to OpenClover that works with Java 11 Key: LUCENE-8763 URL: https://issues.apache.org/jira/browse/LUCENE-8763 Project: Lucene - Core Issue Type: Bug Components: general/build Affects Versions: master (9.0) Reporter: Uwe Schindler Assignee: Uwe Schindler OpenClover does not yet have a release that works with Java 11, so we have to disable the task and add a TODO for updating it. I will commit this for now to the LUCENE-8738 branch. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-11579) Building Solr (master) using Java 11 and Maven has some issues
[ https://issues.apache.org/jira/browse/SOLR-11579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Collins updated SOLR-11579: -- Summary: Building Solr (master) using Java 11 and Maven has some issues (was: Building Solr using Java 9 and Maven doesn't work) > Building Solr (master) using Java 11 and Maven has some issues > -- > > Key: SOLR-11579 > URL: https://issues.apache.org/jira/browse/SOLR-11579 > Project: Solr > Issue Type: New Feature > Security Level: Public(Default Security Level. Issues are Public) > Components: Build >Affects Versions: 8.0 > Environment: Apache Maven 3.6.0 > (97c98ec64a1fdfee7767ce5ffb20918da4f719f3; 2018-10-24T19:41:47+01:00) > Maven home: C:\Users\CollinsDaniel\Utils\apache-maven-3.6.0\bin\.. > Java version: 11.0.2, vendor: Oracle Corporation, runtime: > C:\Users\CollinsDaniel\Utils\jdk-11.0.2 > Default locale: en_GB, platform encoding: Cp1252 > OS name: "windows 10", version: "10.0", arch: "amd64", family: "windows" > openjdk version "11.0.2" 2019-01-15 > OpenJDK Runtime Environment 18.9 (build 11.0.2+9) > OpenJDK 64-Bit Server VM 18.9 (build 11.0.2+9, mixed mode) >Reporter: Daniel Collins >Priority: Minor > Labels: maven, maven-enforcer-plugin > Attachments: SOLR-11579_2, SOLR-11579_4 > > > I'm calling this minor as I don't believe we officially support Java 11 (but > correct me if I'm wrong), and I've only tried on master > Several issues/notes: > # -enforcer plugin breaks if you try to do mvn install- > # javadoc plugin breaks if you try to do mvn javadoc:jar (on Windows) > # warnings from groovy-maven-plugin (TODO) > # a lot of the maven plugins are quite old > # forbiddenapis has some issues with Solr (I assume due to module split-up, > javax is no longer part of core) (TODO) > I have patches for 1, 2, and 4. Still working on 3 and 5. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: [VOTE] Master/9.0 to require Java 11
This vote has passed. I opened https://issues.apache.org/jira/browse/LUCENE-8738. Thanks all. On Mon, Mar 25, 2019 at 1:27 AM Varun Thacker wrote: > > +1 > > On Wed, Mar 20, 2019 at 9:47 AM David Smiley wrote: >> >> +1 -- great point about Lucene/Solr's next major release being a ways off; >> this makes the decision pretty easy. >> >> ~ David Smiley >> Apache Lucene/Solr Search Developer >> http://www.linkedin.com/in/davidwsmiley >> >> >> On Tue, Mar 19, 2019 at 2:23 PM Adrien Grand wrote: >>> >>> Hello, >>> >>> Now that Lucene/Solr 8.0 has shipped I'd like us to consider requiring >>> Java 11 for 9.0, currently the master branch. We had 18 months between >>> 7.0 and 8.0, so if we assume a similar interval between 8.0 and 9.0 >>> that would mean releasing 9.0 about 2 years after Java 11, which >>> sounds like a conservative requirement to me. >>> >>> What do you think? >>> >>> Here is my +1. >>> >>> -- >>> Adrien >>> >>> - >>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >>> For additional commands, e-mail: dev-h...@lucene.apache.org >>> -- Adrien - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: [VOTE] Master/9.0 to require Java 11
+1 On Wed, Mar 20, 2019 at 9:47 AM David Smiley wrote: > +1 -- great point about Lucene/Solr's next major release being a ways off; > this makes the decision pretty easy. > > ~ David Smiley > Apache Lucene/Solr Search Developer > http://www.linkedin.com/in/davidwsmiley > > > On Tue, Mar 19, 2019 at 2:23 PM Adrien Grand wrote: > >> Hello, >> >> Now that Lucene/Solr 8.0 has shipped I'd like us to consider requiring >> Java 11 for 9.0, currently the master branch. We had 18 months between >> 7.0 and 8.0, so if we assume a similar interval between 8.0 and 9.0 >> that would mean releasing 9.0 about 2 years after Java 11, which >> sounds like a conservative requirement to me. >> >> What do you think? >> >> Here is my +1. >> >> -- >> Adrien >> >> - >> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >> For additional commands, e-mail: dev-h...@lucene.apache.org >> >>
