[jira] [Comment Edited] (LUCENE-7966) build mr-jar and use some java 9 methods if available
[
https://issues.apache.org/jira/browse/LUCENE-7966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356766#comment-16356766
]
Uwe Schindler edited comment on LUCENE-7966 at 2/8/18 10:30 AM:
FYI, after this patch:
{noformat}
bash-4.3$ pwd
/cygdrive/c/Users/Uwe
Schindler/Projects/lucene/trunk-lusolr1/lucene/backward-codecs
bash-4.3$ fgrep -R StringHelper *
src/java/org/apache/lucene/codecs/lucene54/LegacyStringHelper.java:abstract
class LegacyStringHelper {
src/java/org/apache/lucene/codecs/lucene54/LegacyStringHelper.java: private
LegacyStringHelper() {
src/java/org/apache/lucene/codecs/lucene54/Lucene54DocValuesConsumer.java:
prefixSum += LegacyStringHelper.bytesDifference(previousValue.get(), v);
src/java/org/apache/lucene/codecs/lucene54/Lucene54DocValuesConsumer.java:
int sharedPrefix = Math.min(255,
LegacyStringHelper.bytesDifference(lastTerm.get(), v));
src/java/org/apache/lucene/codecs/lucene54/Lucene54DocValuesConsumer.java:
int len = LegacyStringHelper.sortKeyLength(priorTerm.get(), b);
{noformat}
was (Author: thetaphi):
FYI, after this patch:
{code:noformat}
bash-4.3$ pwd
/cygdrive/c/Users/Uwe
Schindler/Projects/lucene/trunk-lusolr1/lucene/backward-codecs
bash-4.3$ fgrep -R StringHelper *
src/java/org/apache/lucene/codecs/lucene54/LegacyStringHelper.java:abstract
class LegacyStringHelper {
src/java/org/apache/lucene/codecs/lucene54/LegacyStringHelper.java: private
LegacyStringHelper() {
src/java/org/apache/lucene/codecs/lucene54/Lucene54DocValuesConsumer.java:
prefixSum += LegacyStringHelper.bytesDifference(previousValue.get(), v);
src/java/org/apache/lucene/codecs/lucene54/Lucene54DocValuesConsumer.java:
int sharedPrefix = Math.min(255,
LegacyStringHelper.bytesDifference(lastTerm.get(), v));
src/java/org/apache/lucene/codecs/lucene54/Lucene54DocValuesConsumer.java:
int len = LegacyStringHelper.sortKeyLength(priorTerm.get(), b);
{noformat}
> build mr-jar and use some java 9 methods if available
> -
>
> Key: LUCENE-7966
> URL: https://issues.apache.org/jira/browse/LUCENE-7966
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/other, general/build
>Reporter: Robert Muir
>Assignee: Uwe Schindler
>Priority: Major
> Labels: Java9
> Fix For: master (8.0)
>
> Attachments: LUCENE-7966-7x-backwards.patch,
> LUCENE-7966-7x-backwards.patch, LUCENE-7966-v2.patch, LUCENE-7966.patch,
> LUCENE-7966.patch, LUCENE-7966.patch, LUCENE-7966.patch, LUCENE-7966.patch
>
>
> See background: http://openjdk.java.net/jeps/238
> It would be nice to use some of the newer array methods and range checking
> methods in java 9 for example, without waiting for lucene 10 or something. If
> we build an MR-jar, we can start migrating our code to use java 9 methods
> right now, it will use optimized methods from java 9 when thats available,
> otherwise fall back to java 8 code.
> This patch adds:
> {code}
> Objects.checkIndex(int,int)
> Objects.checkFromToIndex(int,int,int)
> Objects.checkFromIndexSize(int,int,int)
> Arrays.mismatch(byte[],int,int,byte[],int,int)
> Arrays.compareUnsigned(byte[],int,int,byte[],int,int)
> Arrays.equal(byte[],int,int,byte[],int,int)
> // did not add char/int/long/short/etc but of course its possible if needed
> {code}
> It sets these up in {{org.apache.lucene.future}} as 1-1 mappings to java
> methods. This way, we can simply directly replace call sites with java 9
> methods when java 9 is a minimum. Simple 1-1 mappings mean also that we only
> have to worry about testing that our java 8 fallback methods work.
> I found that many of the current byte array methods today are willy-nilly and
> very lenient for example, passing invalid offsets at times and relying on
> compare methods not throwing exceptions, etc. I fixed all the instances in
> core/codecs but have not looked at the problems with AnalyzingSuggester. Also
> SimpleText still uses a silly method in ArrayUtil in similar crazy way, have
> not removed that one yet.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (LUCENE-7966) build mr-jar and use some java 9 methods if available
[
https://issues.apache.org/jira/browse/LUCENE-7966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356740#comment-16356740
]
Uwe Schindler edited comment on LUCENE-7966 at 2/8/18 9:55 AM:
---
Here is the patch: [^LUCENE-7966-7x-backwards.patch]
First cherry pick the main MR-JAR commit and then apply this patch.
This failing test works after that:
{noformat}
ant test -Dtestcase=TestLucene54DocValuesFormat -Dtests.method=testThreads2
-Dtests.seed=61E04EB48E682E0C -Dtests.multiplier=3 -Dtests.slow=true
-Dtests.locale=ar-LY -Dtests.timezone=Asia/Choibalsan -Dtests.asserts=true
-Dtests.file.encoding=ISO-8859-1
{noformat}
(this was the failing test from jenkins - it reproduces, but it depends on the
seed)
was (Author: thetaphi):
Here is the patch: [^LUCENE-7966-7x-backwards.patch]
> build mr-jar and use some java 9 methods if available
> -
>
> Key: LUCENE-7966
> URL: https://issues.apache.org/jira/browse/LUCENE-7966
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/other, general/build
>Reporter: Robert Muir
>Assignee: Uwe Schindler
>Priority: Major
> Labels: Java9
> Fix For: master (8.0)
>
> Attachments: LUCENE-7966-7x-backwards.patch, LUCENE-7966-v2.patch,
> LUCENE-7966.patch, LUCENE-7966.patch, LUCENE-7966.patch, LUCENE-7966.patch,
> LUCENE-7966.patch
>
>
> See background: http://openjdk.java.net/jeps/238
> It would be nice to use some of the newer array methods and range checking
> methods in java 9 for example, without waiting for lucene 10 or something. If
> we build an MR-jar, we can start migrating our code to use java 9 methods
> right now, it will use optimized methods from java 9 when thats available,
> otherwise fall back to java 8 code.
> This patch adds:
> {code}
> Objects.checkIndex(int,int)
> Objects.checkFromToIndex(int,int,int)
> Objects.checkFromIndexSize(int,int,int)
> Arrays.mismatch(byte[],int,int,byte[],int,int)
> Arrays.compareUnsigned(byte[],int,int,byte[],int,int)
> Arrays.equal(byte[],int,int,byte[],int,int)
> // did not add char/int/long/short/etc but of course its possible if needed
> {code}
> It sets these up in {{org.apache.lucene.future}} as 1-1 mappings to java
> methods. This way, we can simply directly replace call sites with java 9
> methods when java 9 is a minimum. Simple 1-1 mappings mean also that we only
> have to worry about testing that our java 8 fallback methods work.
> I found that many of the current byte array methods today are willy-nilly and
> very lenient for example, passing invalid offsets at times and relying on
> compare methods not throwing exceptions, etc. I fixed all the instances in
> core/codecs but have not looked at the problems with AnalyzingSuggester. Also
> SimpleText still uses a silly method in ArrayUtil in similar crazy way, have
> not removed that one yet.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (LUCENE-7966) build mr-jar and use some java 9 methods if available
[
https://issues.apache.org/jira/browse/LUCENE-7966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356707#comment-16356707
]
Uwe Schindler edited comment on LUCENE-7966 at 2/8/18 9:35 AM:
---
Hi Robert,
I agree. My proposal that I am working on: Let's clone the old StringHelper
methods to backwards-codecs (something like
{{LegacyStringHelper.bytesDifference}} and {{sortKeyLength}}) and change
backwards codecs to use them. So it might be slower when reading/writing old
indexes, but we are safe!
New indexes (that use same codecs like master) are fast and do better checks
(no failures yet on master).
was (Author: thetaphi):
Hi Robert,
I agree. My proposal that I am working on: Let's clone the old StringHelper
method to backwards-codecs (something like LegacyStringHelper.bytesDifference)
and change backwards codecs to use it). So it might be slower when
reading/writing old indexes, but we are safe.
New indexes (that use same codecs like master) are fast and do better checks
(no failures yet on master).
> build mr-jar and use some java 9 methods if available
> -
>
> Key: LUCENE-7966
> URL: https://issues.apache.org/jira/browse/LUCENE-7966
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/other, general/build
>Reporter: Robert Muir
>Assignee: Uwe Schindler
>Priority: Major
> Labels: Java9
> Fix For: master (8.0)
>
> Attachments: LUCENE-7966-v2.patch, LUCENE-7966.patch,
> LUCENE-7966.patch, LUCENE-7966.patch, LUCENE-7966.patch, LUCENE-7966.patch
>
>
> See background: http://openjdk.java.net/jeps/238
> It would be nice to use some of the newer array methods and range checking
> methods in java 9 for example, without waiting for lucene 10 or something. If
> we build an MR-jar, we can start migrating our code to use java 9 methods
> right now, it will use optimized methods from java 9 when thats available,
> otherwise fall back to java 8 code.
> This patch adds:
> {code}
> Objects.checkIndex(int,int)
> Objects.checkFromToIndex(int,int,int)
> Objects.checkFromIndexSize(int,int,int)
> Arrays.mismatch(byte[],int,int,byte[],int,int)
> Arrays.compareUnsigned(byte[],int,int,byte[],int,int)
> Arrays.equal(byte[],int,int,byte[],int,int)
> // did not add char/int/long/short/etc but of course its possible if needed
> {code}
> It sets these up in {{org.apache.lucene.future}} as 1-1 mappings to java
> methods. This way, we can simply directly replace call sites with java 9
> methods when java 9 is a minimum. Simple 1-1 mappings mean also that we only
> have to worry about testing that our java 8 fallback methods work.
> I found that many of the current byte array methods today are willy-nilly and
> very lenient for example, passing invalid offsets at times and relying on
> compare methods not throwing exceptions, etc. I fixed all the instances in
> core/codecs but have not looked at the problems with AnalyzingSuggester. Also
> SimpleText still uses a silly method in ArrayUtil in similar crazy way, have
> not removed that one yet.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (LUCENE-7966) build mr-jar and use some java 9 methods if available
[
https://issues.apache.org/jira/browse/LUCENE-7966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351776#comment-16351776
]
Uwe Schindler edited comment on LUCENE-7966 at 2/4/18 1:31 PM:
---
The change in branch_7x broke anonther codec, because
{{StringHelper.bytesDifference()}} changed its requirements on the terms passed
in.
I revert...
We should fix the issue and then merge the branch also to 7.x.
was (Author: thetaphi):
The change in branch_7x broke a nonther codec, because
StringHelper.bytesDifference() changed its requirements on the terms passed in.
In revert...
We should fix the issue and then merge the branch also to 7.x.
> build mr-jar and use some java 9 methods if available
> -
>
> Key: LUCENE-7966
> URL: https://issues.apache.org/jira/browse/LUCENE-7966
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/other, general/build
>Reporter: Robert Muir
>Assignee: Uwe Schindler
>Priority: Major
> Labels: Java9
> Attachments: LUCENE-7966-v2.patch, LUCENE-7966.patch,
> LUCENE-7966.patch, LUCENE-7966.patch, LUCENE-7966.patch, LUCENE-7966.patch
>
>
> See background: http://openjdk.java.net/jeps/238
> It would be nice to use some of the newer array methods and range checking
> methods in java 9 for example, without waiting for lucene 10 or something. If
> we build an MR-jar, we can start migrating our code to use java 9 methods
> right now, it will use optimized methods from java 9 when thats available,
> otherwise fall back to java 8 code.
> This patch adds:
> {code}
> Objects.checkIndex(int,int)
> Objects.checkFromToIndex(int,int,int)
> Objects.checkFromIndexSize(int,int,int)
> Arrays.mismatch(byte[],int,int,byte[],int,int)
> Arrays.compareUnsigned(byte[],int,int,byte[],int,int)
> Arrays.equal(byte[],int,int,byte[],int,int)
> // did not add char/int/long/short/etc but of course its possible if needed
> {code}
> It sets these up in {{org.apache.lucene.future}} as 1-1 mappings to java
> methods. This way, we can simply directly replace call sites with java 9
> methods when java 9 is a minimum. Simple 1-1 mappings mean also that we only
> have to worry about testing that our java 8 fallback methods work.
> I found that many of the current byte array methods today are willy-nilly and
> very lenient for example, passing invalid offsets at times and relying on
> compare methods not throwing exceptions, etc. I fixed all the instances in
> core/codecs but have not looked at the problems with AnalyzingSuggester. Also
> SimpleText still uses a silly method in ArrayUtil in similar crazy way, have
> not removed that one yet.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (LUCENE-7966) build mr-jar and use some java 9 methods if available
[
https://issues.apache.org/jira/browse/LUCENE-7966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347613#comment-16347613
]
Uwe Schindler edited comment on LUCENE-7966 at 1/31/18 9:38 PM:
FYI, here are my changes to [~mikemccand]'s lucenebench to use JAR files
instead of class files from source checkout. It's not complete, it just works.
Please neglect my ignorance for Python: https://paste.apache.org/V5LA
Without it, you would see a slowdown with Java 9, added by the additional
checks! (I tested it)
was (Author: thetaphi):
FYI, here are my changes to [~mikemccand]'s lucenebench to use JAR files
instead of class files from source checkout. It's not complete, it just works.
Please neglect my ignorance for Python: https://paste.apache.org/V5LA
Without it, you would see a slowdown with Java added by the additional checks!
(I tested it)
> build mr-jar and use some java 9 methods if available
> -
>
> Key: LUCENE-7966
> URL: https://issues.apache.org/jira/browse/LUCENE-7966
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/other, general/build
>Reporter: Robert Muir
>Priority: Major
> Labels: Java9
> Attachments: LUCENE-7966.patch, LUCENE-7966.patch, LUCENE-7966.patch,
> LUCENE-7966.patch, LUCENE-7966.patch
>
>
> See background: http://openjdk.java.net/jeps/238
> It would be nice to use some of the newer array methods and range checking
> methods in java 9 for example, without waiting for lucene 10 or something. If
> we build an MR-jar, we can start migrating our code to use java 9 methods
> right now, it will use optimized methods from java 9 when thats available,
> otherwise fall back to java 8 code.
> This patch adds:
> {code}
> Objects.checkIndex(int,int)
> Objects.checkFromToIndex(int,int,int)
> Objects.checkFromIndexSize(int,int,int)
> Arrays.mismatch(byte[],int,int,byte[],int,int)
> Arrays.compareUnsigned(byte[],int,int,byte[],int,int)
> Arrays.equal(byte[],int,int,byte[],int,int)
> // did not add char/int/long/short/etc but of course its possible if needed
> {code}
> It sets these up in {{org.apache.lucene.future}} as 1-1 mappings to java
> methods. This way, we can simply directly replace call sites with java 9
> methods when java 9 is a minimum. Simple 1-1 mappings mean also that we only
> have to worry about testing that our java 8 fallback methods work.
> I found that many of the current byte array methods today are willy-nilly and
> very lenient for example, passing invalid offsets at times and relying on
> compare methods not throwing exceptions, etc. I fixed all the instances in
> core/codecs but have not looked at the problems with AnalyzingSuggester. Also
> SimpleText still uses a silly method in ArrayUtil in similar crazy way, have
> not removed that one yet.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (LUCENE-7966) build mr-jar and use some java 9 methods if available
[
https://issues.apache.org/jira/browse/LUCENE-7966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16345192#comment-16345192
]
Uwe Schindler edited comment on LUCENE-7966 at 1/31/18 10:16 AM:
-
I worked today and made the branch on github up to date:
* It now applies the classfile patching to all modules, Solr was disabled (not
needed there)
* It runs tests in Java 9 with a modified classpath (it puts the patched
classes before the main classes), so we can actually also test the patched
stuff on Java 9. This "hack" just emulates a MR JAR file. It is a really dirty
hack, because it only applies this to the main classes. Of course, the tests
for our own implementations of {{Objects/Arrays}} classes is done in Java 8 and
Java 9 using Lucene's code, the hack just enables our unit tests that use the
Lucene API to use the native Java 9 classes behind the scenes when executed in
Java 9, so our production code is thoroughly tested. There is a new system
property of the test framework to ({{tests.withJava9Patches}}, defaults to
{{true}}), so you can disable this and only test our own {{Objects/Arrays}}
implementations, also on Java 9. To fully fix the classpath complexity we
should think about using the JAR files for testing and not build a classpath
with raw class files. But that's time for another issue.
I will now do some tests with Mike's luceneutils benchmarking tool. I just have
to fix it to use the JAR files instead of using plain class files.
was (Author: thetaphi):
I worked today and made the branch on github up to date:
* It now applies the classfile patching to all modules, Solr was disabled (not
needed there)
* It runs tests in Java 9 with a modified classpath (it puts the patched
classes before the main classes), so we can actually also test the patched
stuff on Java 9. This "hack" just emulates a MR JAR file. It is a really dirty
hack, because it only applies this to the main classes. Of course, the tests
for our own implementations of Objects/Arrays classes is done in Java 8 and
Java 9 using Lucene's code, the hack just enables our uint tests using the
Lucene API to use the native Java 9 classes when executed in Java 9. To fully
fix the classpath complexity we should think about using the JAR files for
testing and not build a classpath with raw class files. But that's time for
another issue.
I will now do some tests with Mike's luceneutils benchmarking tool. I just have
to fix it to use the JAR files instead of using plain class files.
> build mr-jar and use some java 9 methods if available
> -
>
> Key: LUCENE-7966
> URL: https://issues.apache.org/jira/browse/LUCENE-7966
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/other, general/build
>Reporter: Robert Muir
>Priority: Major
> Labels: Java9
> Attachments: LUCENE-7966.patch, LUCENE-7966.patch, LUCENE-7966.patch,
> LUCENE-7966.patch, LUCENE-7966.patch
>
>
> See background: http://openjdk.java.net/jeps/238
> It would be nice to use some of the newer array methods and range checking
> methods in java 9 for example, without waiting for lucene 10 or something. If
> we build an MR-jar, we can start migrating our code to use java 9 methods
> right now, it will use optimized methods from java 9 when thats available,
> otherwise fall back to java 8 code.
> This patch adds:
> {code}
> Objects.checkIndex(int,int)
> Objects.checkFromToIndex(int,int,int)
> Objects.checkFromIndexSize(int,int,int)
> Arrays.mismatch(byte[],int,int,byte[],int,int)
> Arrays.compareUnsigned(byte[],int,int,byte[],int,int)
> Arrays.equal(byte[],int,int,byte[],int,int)
> // did not add char/int/long/short/etc but of course its possible if needed
> {code}
> It sets these up in {{org.apache.lucene.future}} as 1-1 mappings to java
> methods. This way, we can simply directly replace call sites with java 9
> methods when java 9 is a minimum. Simple 1-1 mappings mean also that we only
> have to worry about testing that our java 8 fallback methods work.
> I found that many of the current byte array methods today are willy-nilly and
> very lenient for example, passing invalid offsets at times and relying on
> compare methods not throwing exceptions, etc. I fixed all the instances in
> core/codecs but have not looked at the problems with AnalyzingSuggester. Also
> SimpleText still uses a silly method in ArrayUtil in similar crazy way, have
> not removed that one yet.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (LUCENE-7966) build mr-jar and use some java 9 methods if available
[
https://issues.apache.org/jira/browse/LUCENE-7966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16345192#comment-16345192
]
Uwe Schindler edited comment on LUCENE-7966 at 1/31/18 10:10 AM:
-
I worked today and made the branch on github up to date:
* It now applies the classfile patching to all modules, Solr was disabled (not
needed there)
* It runs tests in Java 9 with a modified classpath (it puts the patched
classes before the main classes), so we can actually also test the patched
stuff on Java 9. This "hack" just emulates a MR JAR file. It is a really dirty
hack, because it only applies this to the main classes. Of course, the tests
for our own implementations of Objects/Arrays classes is done in Java 8 and
Java 9 using Lucene's code, the hack just enables our uint tests using the
Lucene API to use the native Java 9 classes when executed in Java 9. To fully
fix the classpath complexity we should think about using the JAR files for
testing and not build a classpath with raw class files. But that's time for
another issue.
I will now do some tests with Mike's luceneutils benchmarking tool. I just have
to fix it to use the JAR files instead of using plain class files.
was (Author: thetaphi):
I worked today and made the branch on github up to date:
* It now applies the classfile patching to all modules, Solr was disabled (not
needed there)
* It runs tests in Java 9 with a modified classpath (it puts the patched
classes before the main classes), so we can actually also test the pathed
stuff. This "hack" just emulates a MR JAR file. It is really dirty, because it
only applies this to the main classes. To fully fix this issue we should think
about using the JAR files for testing and not build a classpath with raw class
files. But that's time for another issue.
I will now do some tests with Mike's luceneutils benchmarking tool. I just have
to fix it to use the JAR files instead of using plain class files.
> build mr-jar and use some java 9 methods if available
> -
>
> Key: LUCENE-7966
> URL: https://issues.apache.org/jira/browse/LUCENE-7966
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/other, general/build
>Reporter: Robert Muir
>Priority: Major
> Labels: Java9
> Attachments: LUCENE-7966.patch, LUCENE-7966.patch, LUCENE-7966.patch,
> LUCENE-7966.patch, LUCENE-7966.patch
>
>
> See background: http://openjdk.java.net/jeps/238
> It would be nice to use some of the newer array methods and range checking
> methods in java 9 for example, without waiting for lucene 10 or something. If
> we build an MR-jar, we can start migrating our code to use java 9 methods
> right now, it will use optimized methods from java 9 when thats available,
> otherwise fall back to java 8 code.
> This patch adds:
> {code}
> Objects.checkIndex(int,int)
> Objects.checkFromToIndex(int,int,int)
> Objects.checkFromIndexSize(int,int,int)
> Arrays.mismatch(byte[],int,int,byte[],int,int)
> Arrays.compareUnsigned(byte[],int,int,byte[],int,int)
> Arrays.equal(byte[],int,int,byte[],int,int)
> // did not add char/int/long/short/etc but of course its possible if needed
> {code}
> It sets these up in {{org.apache.lucene.future}} as 1-1 mappings to java
> methods. This way, we can simply directly replace call sites with java 9
> methods when java 9 is a minimum. Simple 1-1 mappings mean also that we only
> have to worry about testing that our java 8 fallback methods work.
> I found that many of the current byte array methods today are willy-nilly and
> very lenient for example, passing invalid offsets at times and relying on
> compare methods not throwing exceptions, etc. I fixed all the instances in
> core/codecs but have not looked at the problems with AnalyzingSuggester. Also
> SimpleText still uses a silly method in ArrayUtil in similar crazy way, have
> not removed that one yet.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (LUCENE-7966) build mr-jar and use some java 9 methods if available
[
https://issues.apache.org/jira/browse/LUCENE-7966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16345192#comment-16345192
]
Uwe Schindler edited comment on LUCENE-7966 at 1/31/18 10:06 AM:
-
I worked today and made the branch on github up to date:
* It now applies the classfile patching to all modules, Solr was disabled (not
needed there)
* It runs tests in Java 9 with a modified classpath (it puts the patched
classes before the main classes), so we can actually also test the pathed
stuff. This "hack" just emulates a MR JAR file. It is really dirty, because it
only applies this to the main classes. To fully fix this issue we should think
about using the JAR files for testing and not build a classpath with raw class
files. But that's time for another issue.
I will now do some tests with Mike's luceneutils benchmarking tool. I just have
to fix it to use the JAR files instead of using plain class files.
was (Author: thetaphi):
I worked today and made the branch on github up to date:
* It now applies the classfile patching to all modules, Solr was disabled (not
needed there)
* It runs tests in Java 9 with a modified classpath (it puts the patched
classes before the main classes), so we can actually also test the pathed
stuff. This "hack" just emulates a MR JAR file. It is really dirty, because it
only applies this to the main classes. To fully fix this issue we should think
about using the JAR files for testing and not build a classpath with raw class
files. But that's time for another issue.
I will now do some tests with the lucene benchmarking. I just have to fix it to
use the JAR files instead of using plain jar files.
> build mr-jar and use some java 9 methods if available
> -
>
> Key: LUCENE-7966
> URL: https://issues.apache.org/jira/browse/LUCENE-7966
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/other, general/build
>Reporter: Robert Muir
>Priority: Major
> Labels: Java9
> Attachments: LUCENE-7966.patch, LUCENE-7966.patch, LUCENE-7966.patch,
> LUCENE-7966.patch, LUCENE-7966.patch
>
>
> See background: http://openjdk.java.net/jeps/238
> It would be nice to use some of the newer array methods and range checking
> methods in java 9 for example, without waiting for lucene 10 or something. If
> we build an MR-jar, we can start migrating our code to use java 9 methods
> right now, it will use optimized methods from java 9 when thats available,
> otherwise fall back to java 8 code.
> This patch adds:
> {code}
> Objects.checkIndex(int,int)
> Objects.checkFromToIndex(int,int,int)
> Objects.checkFromIndexSize(int,int,int)
> Arrays.mismatch(byte[],int,int,byte[],int,int)
> Arrays.compareUnsigned(byte[],int,int,byte[],int,int)
> Arrays.equal(byte[],int,int,byte[],int,int)
> // did not add char/int/long/short/etc but of course its possible if needed
> {code}
> It sets these up in {{org.apache.lucene.future}} as 1-1 mappings to java
> methods. This way, we can simply directly replace call sites with java 9
> methods when java 9 is a minimum. Simple 1-1 mappings mean also that we only
> have to worry about testing that our java 8 fallback methods work.
> I found that many of the current byte array methods today are willy-nilly and
> very lenient for example, passing invalid offsets at times and relying on
> compare methods not throwing exceptions, etc. I fixed all the instances in
> core/codecs but have not looked at the problems with AnalyzingSuggester. Also
> SimpleText still uses a silly method in ArrayUtil in similar crazy way, have
> not removed that one yet.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (LUCENE-7966) build mr-jar and use some java 9 methods if available
[
https://issues.apache.org/jira/browse/LUCENE-7966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308486#comment-16308486
]
Uwe Schindler edited comment on LUCENE-7966 at 1/2/18 6:46 PM:
---
Sorry for silence on this. I wanted to improve the patch tomorrow.
[~mikemccand]:
bq. I retested w/ Uwe's patch:
By reading the Java mailing lists, I have an idea what could cause Mike's
differences: Are you sure that you used the same garbage collector in Java 8
and Java 9? By default Java 9 uses G1GC, which Java 8 uses the good old
ParallelGC by default. On the mailinglist jdk-dev there was a discussion that
suddenly computing-intensive stuff was significantly slower with Java 9. The
reason was the Garbage Collector! So be sure to use the same one (give it
explicit on command line). G1GC adds additional barriers and because of them
the number of free CPU registers is lowered by 1. This causes some algorithms
to behave worse as missing CPU registers don't allow to do everything in the
CPU ([~dweiss] has seen a sigicficant slowdown for some of his code). More than
10% slowdown is possible! This also affects code that has no garbage collection
and barriers because of G1. It's just limited resources causing this.
Interestingly the person complaining was talking about LZ4 compression:
http://openjdk.5641.n7.nabble.com/Reduced-performance-in-Java-9-0-1-vs-8u152-td322825.html
Could you compare Java 8 and Java 9 with same garbage collector? (a) G1GC on
both and (b) ParallelGC and/or CMS ?
Uwe
was (Author: thetaphi):
Sorry for silence on this. I wanted to improve the patch tomorrow.
[~mikemccand]:
bq. I retested w/ Uwe's patch:
By reading the Java mailing lists, I have an idea what could cause Mike's
differences: Are you sure that you used the same garbage collector in Java 8
and Java 9? By default Java 9 uses G1GC, which Java 8 uses the good old
ParallelGC by default. On the mailinglist jdk-dev there was a discussion that
suddenly computing-intensive stuff was significantly slower with Java 9. The
reason was the Garbage Collector! So be sure to use the same one (give it
explicit on command line). G1GC adds additional barriers and because of them
the number of free CPU registers is lowered by 1. This causes some algorithms
to behave worse as missing CPU registers don't allow to do everything in the
CPU ([~dweiss] has seen a sigicficant slowdown for some of his code). More than
10% slowdown is possible! This also affects code that has no garbage collection
and barriers because of G1. It's just limited resources causing this.
Could you compare Java 8 and Java 9 with same garbage collector? (a) G1GC on
both and (b) ParallelGC and/or CMS ?
Uwe
> build mr-jar and use some java 9 methods if available
> -
>
> Key: LUCENE-7966
> URL: https://issues.apache.org/jira/browse/LUCENE-7966
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/other, general/build
>Reporter: Robert Muir
> Labels: Java9
> Attachments: LUCENE-7966.patch, LUCENE-7966.patch, LUCENE-7966.patch,
> LUCENE-7966.patch, LUCENE-7966.patch
>
>
> See background: http://openjdk.java.net/jeps/238
> It would be nice to use some of the newer array methods and range checking
> methods in java 9 for example, without waiting for lucene 10 or something. If
> we build an MR-jar, we can start migrating our code to use java 9 methods
> right now, it will use optimized methods from java 9 when thats available,
> otherwise fall back to java 8 code.
> This patch adds:
> {code}
> Objects.checkIndex(int,int)
> Objects.checkFromToIndex(int,int,int)
> Objects.checkFromIndexSize(int,int,int)
> Arrays.mismatch(byte[],int,int,byte[],int,int)
> Arrays.compareUnsigned(byte[],int,int,byte[],int,int)
> Arrays.equal(byte[],int,int,byte[],int,int)
> // did not add char/int/long/short/etc but of course its possible if needed
> {code}
> It sets these up in {{org.apache.lucene.future}} as 1-1 mappings to java
> methods. This way, we can simply directly replace call sites with java 9
> methods when java 9 is a minimum. Simple 1-1 mappings mean also that we only
> have to worry about testing that our java 8 fallback methods work.
> I found that many of the current byte array methods today are willy-nilly and
> very lenient for example, passing invalid offsets at times and relying on
> compare methods not throwing exceptions, etc. I fixed all the instances in
> core/codecs but have not looked at the problems with AnalyzingSuggester. Also
> SimpleText still uses a silly method in ArrayUtil in similar crazy way, have
> not removed that one yet.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail:
[jira] [Comment Edited] (LUCENE-7966) build mr-jar and use some java 9 methods if available
[
https://issues.apache.org/jira/browse/LUCENE-7966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308486#comment-16308486
]
Uwe Schindler edited comment on LUCENE-7966 at 1/2/18 6:42 PM:
---
Sorry for silence on this. I wanted to improve the patch tomorrow.
[~mikemccand]:
bq. I retested w/ Uwe's patch:
By reading the Java mailing lists, I have an idea what could cause Mike's
differences: Are you sure that you used the same garbage collector in Java 8
and Java 9? By default Java 9 uses G1GC, which Java 8 uses the good old
ParallelGC by default. On the mailinglist jdk-dev there was a discussion that
suddenly computing-intensive stuff was significantly slower with Java 9. The
reason was the Garbage Collector! So be sure to use the same one (give it
explicit on command line). G1GC adds additional barriers and because of them
the number of free CPU registers is lowered by 1. This causes some algorithms
to behave worse as missing CPU registers don't allow to do everything in the
CPU ([~dweiss] has seen a sigicficant slowdown for some of his code). More than
10% slowdown is possible! This also affects code that has no garbage collection
and barriers because of G1. It's just limited resources causing this.
Could you compare Java 8 and Java 9 with same garbage collector? (a) G1GC on
both and (b) ParallelGC and/or CMS ?
Uwe
was (Author: thetaphi):
Sorry for silence on this. I wanted to improve the patch tomorrow.
[~mikemccand]:
bq. I retested w/ Uwe's patch:
By reading the Java mailing lists, I have an idea what could cause Mike's
differences: Are you sure that you used the same garbage collector in Java 8
and Java 9. By default Java 9 uses G1GC, which Java 8 uses the good old
ParallelGC by default. On the mailinglist jdk-dev there was a discussion that
suddenly computing-intensive stuff was significantly slower with Java 9. The
reason was the Garbage Collector! So be sure to use the same one (give it
explicit on command line). G1GC adds additional barriers and because of them
the number of free CPU registers is lowered by 1. This causes some algorithms
to behave worse as missing CPU registers don't allow to do everything in the
CPU ([~dweiss] has seen a sigicficant slowdown for some of his code). More than
10% slowdown is possible! This also affects code that has no garbage collection
and barriers because of G1. It's just limited resources causing this.
Could you compare Java 8 and Java 9 with same garbage collector? (a) G1GC on
both and (b) ParallelGC and/or CMS ?
Uwe
> build mr-jar and use some java 9 methods if available
> -
>
> Key: LUCENE-7966
> URL: https://issues.apache.org/jira/browse/LUCENE-7966
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/other, general/build
>Reporter: Robert Muir
> Labels: Java9
> Attachments: LUCENE-7966.patch, LUCENE-7966.patch, LUCENE-7966.patch,
> LUCENE-7966.patch, LUCENE-7966.patch
>
>
> See background: http://openjdk.java.net/jeps/238
> It would be nice to use some of the newer array methods and range checking
> methods in java 9 for example, without waiting for lucene 10 or something. If
> we build an MR-jar, we can start migrating our code to use java 9 methods
> right now, it will use optimized methods from java 9 when thats available,
> otherwise fall back to java 8 code.
> This patch adds:
> {code}
> Objects.checkIndex(int,int)
> Objects.checkFromToIndex(int,int,int)
> Objects.checkFromIndexSize(int,int,int)
> Arrays.mismatch(byte[],int,int,byte[],int,int)
> Arrays.compareUnsigned(byte[],int,int,byte[],int,int)
> Arrays.equal(byte[],int,int,byte[],int,int)
> // did not add char/int/long/short/etc but of course its possible if needed
> {code}
> It sets these up in {{org.apache.lucene.future}} as 1-1 mappings to java
> methods. This way, we can simply directly replace call sites with java 9
> methods when java 9 is a minimum. Simple 1-1 mappings mean also that we only
> have to worry about testing that our java 8 fallback methods work.
> I found that many of the current byte array methods today are willy-nilly and
> very lenient for example, passing invalid offsets at times and relying on
> compare methods not throwing exceptions, etc. I fixed all the instances in
> core/codecs but have not looked at the problems with AnalyzingSuggester. Also
> SimpleText still uses a silly method in ArrayUtil in similar crazy way, have
> not removed that one yet.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (LUCENE-7966) build mr-jar and use some java 9 methods if available
[
https://issues.apache.org/jira/browse/LUCENE-7966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183010#comment-16183010
]
Uwe Schindler edited comment on LUCENE-7966 at 9/27/17 6:15 PM:
[~dweiss]: The patched class files are actually easier to maintain, as we do
not need Java 9 to compile, no duplicate class files in source folder, or some
fake Java 9 signature files (with questionable license) on bootclasspath (see
my previous branch). This was the main reason to rewrite the class files
instead of maintaining multiple source files. It's just a nice side-effect to
no longer need the delegation methods. So I personally like the patching
approach much more, as it's well tested by the ASM maintainers (we just use
their code, no custom impl). It would be horrible if we'd instead require all
committers to have both Java 8 and Java 9 installed!
The question here was just for confirmation and comparison of both approaches,
if they have some side effects.
bq. The slowdown on pic (the most compressible file) is reproducible
[~jpountz]: The one with biggest slowdown on Java 8 is the one with biggest
speedup in Java 9. The reason is quite clear: The Java 8 implementation by
Robert does more checks than the "old" LZ4 implementation (for safety and to be
compatible with new Java 9 impl). But on Java 9 the new method used is an
intrinsic, so we have a huge perf win!
was (Author: thetaphi):
[~dweiss]: The patched class files are actually easier to maintain, as we do
not need Java 9 to compile, no duplicate class files in source folder, or some
fake Java 9 signature files (with questionable license) on bootclasspath (see
my previous branch). This was the main reason to rewrite the class files
instead of maintaining multiple source files. It's just a nice side-effect to
no longer need the delegation methods. So I personally like the patching
approach much more. It would be horrible if we'd require all committers to have
both Java 8 and Java 9 installed!
The question here was just for confirmation and comparison of both approaches,
if they have some side effects.
bq. The slowdown on pic (the most compressible file) is reproducible
[~jpountz]: The one with biggest slowdown on Java 8 is the one with biggest
speedup in Java 9. The reason is quite clear: The Java 8 implementation by
Robert does more checks than the "old" LZ4 implementation (for safety and to be
compatible with new Java 9 impl). But on Java 9 the new method used is an
intrinsic, so we have a huge perf win!
> build mr-jar and use some java 9 methods if available
> -
>
> Key: LUCENE-7966
> URL: https://issues.apache.org/jira/browse/LUCENE-7966
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/other, general/build
>Reporter: Robert Muir
> Labels: Java9
> Attachments: LUCENE-7966.patch, LUCENE-7966.patch, LUCENE-7966.patch,
> LUCENE-7966.patch, LUCENE-7966.patch
>
>
> See background: http://openjdk.java.net/jeps/238
> It would be nice to use some of the newer array methods and range checking
> methods in java 9 for example, without waiting for lucene 10 or something. If
> we build an MR-jar, we can start migrating our code to use java 9 methods
> right now, it will use optimized methods from java 9 when thats available,
> otherwise fall back to java 8 code.
> This patch adds:
> {code}
> Objects.checkIndex(int,int)
> Objects.checkFromToIndex(int,int,int)
> Objects.checkFromIndexSize(int,int,int)
> Arrays.mismatch(byte[],int,int,byte[],int,int)
> Arrays.compareUnsigned(byte[],int,int,byte[],int,int)
> Arrays.equal(byte[],int,int,byte[],int,int)
> // did not add char/int/long/short/etc but of course its possible if needed
> {code}
> It sets these up in {{org.apache.lucene.future}} as 1-1 mappings to java
> methods. This way, we can simply directly replace call sites with java 9
> methods when java 9 is a minimum. Simple 1-1 mappings mean also that we only
> have to worry about testing that our java 8 fallback methods work.
> I found that many of the current byte array methods today are willy-nilly and
> very lenient for example, passing invalid offsets at times and relying on
> compare methods not throwing exceptions, etc. I fixed all the instances in
> core/codecs but have not looked at the problems with AnalyzingSuggester. Also
> SimpleText still uses a silly method in ArrayUtil in similar crazy way, have
> not removed that one yet.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (LUCENE-7966) build mr-jar and use some java 9 methods if available
[
https://issues.apache.org/jira/browse/LUCENE-7966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183010#comment-16183010
]
Uwe Schindler edited comment on LUCENE-7966 at 9/27/17 6:14 PM:
[~dweiss]: The patched class files are actually easier to maintain, as we do
not need Java 9 to compile, no duplicate class files in source folder, or some
fake Java 9 signature files (with questionable license) on bootclasspath (see
my previous branch). This was the main reason to rewrite the class files
instead of maintaining multiple source files. It's just a nice side-effect to
no longer need the delegation methods. So I personally like the patching
approach much more. It would be horrible if we'd require all committers to have
both Java 8 and Java 9 installed!
The question here was just for confirmation and comparison of both approaches,
if they have some side effects.
bq. The slowdown on pic (the most compressible file) is reproducible
[~jpountz]: The one with biggest slowdown on Java 8 is the one with biggest
speedup in Java 9. The reason is quite clear: The Java 8 implementation by
Robert does more checks than the "old" LZ4 implementation (for safety and to be
compatible with new Java 9 impl). But on Java 9 the new method used is an
intrinsic, so we have a huge perf win!
was (Author: thetaphi):
[~dweiss]: The patched class files are actually easier to maintain, as we do
not need Java 9 to compile, no duplicate class files in source folder, or some
fake Java 9 signature files (with questionable license) on bootclasspath (see
my previous branch). This was the main reason to rewrite the class files
instead of maintaining multiple source files. It's just a nice side-effect to
no longer need the delegation methods. So I perosnally like the approach much
more. It would be horrible if we'd require all committers to have both Java 8
and Java 9 installed!
The question here was just for confirmation and comparison of both approaches,
if they have some side effects.
bq. The slowdown on pic (the most compressible file) is reproducible
[~jpountz]: The one with biggest slowdown on Java 8 is the one with biggest
speedup in Java 9. The reason is quite clear: The Java 8 implementation by
Robert does more checks than the "old" LZ4 implementation (for safety and to be
compatible with new Java 9 impl). But on Java 9 the new method used is an
intrinsic, so we have a huge perf win!
> build mr-jar and use some java 9 methods if available
> -
>
> Key: LUCENE-7966
> URL: https://issues.apache.org/jira/browse/LUCENE-7966
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/other, general/build
>Reporter: Robert Muir
> Labels: Java9
> Attachments: LUCENE-7966.patch, LUCENE-7966.patch, LUCENE-7966.patch,
> LUCENE-7966.patch, LUCENE-7966.patch
>
>
> See background: http://openjdk.java.net/jeps/238
> It would be nice to use some of the newer array methods and range checking
> methods in java 9 for example, without waiting for lucene 10 or something. If
> we build an MR-jar, we can start migrating our code to use java 9 methods
> right now, it will use optimized methods from java 9 when thats available,
> otherwise fall back to java 8 code.
> This patch adds:
> {code}
> Objects.checkIndex(int,int)
> Objects.checkFromToIndex(int,int,int)
> Objects.checkFromIndexSize(int,int,int)
> Arrays.mismatch(byte[],int,int,byte[],int,int)
> Arrays.compareUnsigned(byte[],int,int,byte[],int,int)
> Arrays.equal(byte[],int,int,byte[],int,int)
> // did not add char/int/long/short/etc but of course its possible if needed
> {code}
> It sets these up in {{org.apache.lucene.future}} as 1-1 mappings to java
> methods. This way, we can simply directly replace call sites with java 9
> methods when java 9 is a minimum. Simple 1-1 mappings mean also that we only
> have to worry about testing that our java 8 fallback methods work.
> I found that many of the current byte array methods today are willy-nilly and
> very lenient for example, passing invalid offsets at times and relying on
> compare methods not throwing exceptions, etc. I fixed all the instances in
> core/codecs but have not looked at the problems with AnalyzingSuggester. Also
> SimpleText still uses a silly method in ArrayUtil in similar crazy way, have
> not removed that one yet.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (LUCENE-7966) build mr-jar and use some java 9 methods if available
[
https://issues.apache.org/jira/browse/LUCENE-7966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16182509#comment-16182509
]
Adrien Grand edited comment on LUCENE-7966 at 9/27/17 12:58 PM:
Now with Java 8
|| File || Time to compress without patch || Time to compress with the patch
||Difference ||
| bib | 1033357 | 1027901 | -0.5% |
| book1 | 7906551 | 7961422 | +0.7% |
| book2 | 5335458 | 5348388 | +0.2% |
| geo | 1629253 | 1644770 | +1.0% |
| news | 3569115 | 3609734 | +1.1% |
| obj1 | 176601 | 178634 | +1.2% |
| obj2 | 1970078 | 1993204 | +1.2% |
| paper1 | 412884 | 409665 | -0.8% |
| paper2 | 772086 | 772052 | -0.0% |
| paper3 | 401949 | 397526 | -1.1% |
| paper4 | 118664 | 117747 | -0.8% |
| paper5 | 104995 | 104855 | -0.1% |
| paper6 | 296315 | 295813 | -0.2% |
| pic | 1619648 | 1698345 | +4.9% |
| progc | 299270 | 298341 | -0.3% |
| progl | 440063 | 443714 | +0.8% |
| progp | 265607 | 266388 | +0.3% |
| trans | 554452 | 555335 | +0.2% |
The slowdown on pic (the most compressible file) is reproducible but other
files show very similar performance with and without the patch.
was (Author: jpountz):
Now with Java 8
|| File || Time to compress without patch || Time to compress with the patch
||Difference ||
| bib | 1033357 | 1027901 | -0.5% |
| book1 | 7906551 | 7961422 | +0.7% |
| book2 | 5335458 | 5348388 | +0.2% |
| geo | 1629253 | 1644770 | +1.0% |
| news | 3569115 | 3609734 | +1.1% |
| obj1 | 176601 | 178634 | +1.2% |
| obj2 | 1970078 | 1993204 | +1.2% |
| paper1 | 412884 | 409665 | -0.8% |
| paper2 | 772086 | 772052 | -0.0% |
| paper3 | 401949 | 397526 | -1.1% |
| paper4 | 118664 | 117747 | -0.8% |
| paper5 | 104995 | 104855 | -0.1% |
| paper6 | 296315 | 295813 | -0.2% |
| pic | 1619648 | 1698345 | +4.9% |
| progc | 299270 | 298341 | -0.3% |
| progl | 440063 | 443714 | +0.8% |
| progp | 265607 | 266388 | 0.3% |
| trans | 554452 | 555335 | 0.2% |
The slowdown on pic (the most compressible file) is reproducible but other
files show very similar performance with and without the patch.
> build mr-jar and use some java 9 methods if available
> -
>
> Key: LUCENE-7966
> URL: https://issues.apache.org/jira/browse/LUCENE-7966
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/other, general/build
>Reporter: Robert Muir
> Labels: Java9
> Attachments: LUCENE-7966.patch, LUCENE-7966.patch, LUCENE-7966.patch,
> LUCENE-7966.patch, LUCENE-7966.patch
>
>
> See background: http://openjdk.java.net/jeps/238
> It would be nice to use some of the newer array methods and range checking
> methods in java 9 for example, without waiting for lucene 10 or something. If
> we build an MR-jar, we can start migrating our code to use java 9 methods
> right now, it will use optimized methods from java 9 when thats available,
> otherwise fall back to java 8 code.
> This patch adds:
> {code}
> Objects.checkIndex(int,int)
> Objects.checkFromToIndex(int,int,int)
> Objects.checkFromIndexSize(int,int,int)
> Arrays.mismatch(byte[],int,int,byte[],int,int)
> Arrays.compareUnsigned(byte[],int,int,byte[],int,int)
> Arrays.equal(byte[],int,int,byte[],int,int)
> // did not add char/int/long/short/etc but of course its possible if needed
> {code}
> It sets these up in {{org.apache.lucene.future}} as 1-1 mappings to java
> methods. This way, we can simply directly replace call sites with java 9
> methods when java 9 is a minimum. Simple 1-1 mappings mean also that we only
> have to worry about testing that our java 8 fallback methods work.
> I found that many of the current byte array methods today are willy-nilly and
> very lenient for example, passing invalid offsets at times and relying on
> compare methods not throwing exceptions, etc. I fixed all the instances in
> core/codecs but have not looked at the problems with AnalyzingSuggester. Also
> SimpleText still uses a silly method in ArrayUtil in similar crazy way, have
> not removed that one yet.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (LUCENE-7966) build mr-jar and use some java 9 methods if available
[
https://issues.apache.org/jira/browse/LUCENE-7966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16182509#comment-16182509
]
Adrien Grand edited comment on LUCENE-7966 at 9/27/17 12:57 PM:
Now with Java 8
|| File || Time to compress without patch || Time to compress with the patch
||Difference ||
| bib | 1033357 | 1027901 | -0.5% |
| book1 | 7906551 | 7961422 | +0.7% |
| book2 | 5335458 | 5348388 | +0.2% |
| geo | 1629253 | 1644770 | +1.0% |
| news | 3569115 | 3609734 | +1.1% |
| obj1 | 176601 | 178634 | +1.2% |
| obj2 | 1970078 | 1993204 | +1.2% |
| paper1 | 412884 | 409665 | -0.8% |
| paper2 | 772086 | 772052 | -0.0% |
| paper3 | 401949 | 397526 | -1.1% |
| paper4 | 118664 | 117747 | -0.8% |
| paper5 | 104995 | 104855 | -0.1% |
| paper6 | 296315 | 295813 | -0.2% |
| pic | 1619648 | 1698345 | +4.9% |
| progc | 299270 | 298341 | -0.3% |
| progl | 440063 | 443714 | +0.8% |
| progp | 265607 | 266388 | 0.3% |
| trans | 554452 | 555335 | 0.2% |
The slowdown on pic (the most compressible file) is reproducible but other
files show very similar performance with and without the patch.
was (Author: jpountz):
Now with Java 8
|| File || Time to compress without patch || Time to compress with the patch
||Difference ||
| bib | 1033357 | 1027901 | -0.5 |
| book1 | 7906551 | 7961422 | 0.7 |
| book2 | 5335458 | 5348388 | 0.2 |
| geo | 1629253 | 1644770 | 1.0 |
| news | 3569115 | 3609734 | 1.1 |
| obj1 | 176601 | 178634 | 1.2 |
| obj2 | 1970078 | 1993204 | 1.2 |
| paper1 | 412884 | 409665 | -0.8 |
| paper2 | 772086 | 772052 | -0.0 |
| paper3 | 401949 | 397526 | -1.1 |
| paper4 | 118664 | 117747 | -0.8 |
| paper5 | 104995 | 104855 | -0.1 |
| paper6 | 296315 | 295813 | -0.2 |
| pic | 1619648 | 1698345 | 4.9 |
| progc | 299270 | 298341 | -0.3 |
| progl | 440063 | 443714 | 0.8 |
| progp | 265607 | 266388 | 0.3 |
| trans | 554452 | 555335 | 0.2 |
The slowdown on pic (the most compressible file) is reproducible but other
files show very similar performance with and without the patch.
> build mr-jar and use some java 9 methods if available
> -
>
> Key: LUCENE-7966
> URL: https://issues.apache.org/jira/browse/LUCENE-7966
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/other, general/build
>Reporter: Robert Muir
> Labels: Java9
> Attachments: LUCENE-7966.patch, LUCENE-7966.patch, LUCENE-7966.patch,
> LUCENE-7966.patch, LUCENE-7966.patch
>
>
> See background: http://openjdk.java.net/jeps/238
> It would be nice to use some of the newer array methods and range checking
> methods in java 9 for example, without waiting for lucene 10 or something. If
> we build an MR-jar, we can start migrating our code to use java 9 methods
> right now, it will use optimized methods from java 9 when thats available,
> otherwise fall back to java 8 code.
> This patch adds:
> {code}
> Objects.checkIndex(int,int)
> Objects.checkFromToIndex(int,int,int)
> Objects.checkFromIndexSize(int,int,int)
> Arrays.mismatch(byte[],int,int,byte[],int,int)
> Arrays.compareUnsigned(byte[],int,int,byte[],int,int)
> Arrays.equal(byte[],int,int,byte[],int,int)
> // did not add char/int/long/short/etc but of course its possible if needed
> {code}
> It sets these up in {{org.apache.lucene.future}} as 1-1 mappings to java
> methods. This way, we can simply directly replace call sites with java 9
> methods when java 9 is a minimum. Simple 1-1 mappings mean also that we only
> have to worry about testing that our java 8 fallback methods work.
> I found that many of the current byte array methods today are willy-nilly and
> very lenient for example, passing invalid offsets at times and relying on
> compare methods not throwing exceptions, etc. I fixed all the instances in
> core/codecs but have not looked at the problems with AnalyzingSuggester. Also
> SimpleText still uses a silly method in ArrayUtil in similar crazy way, have
> not removed that one yet.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (LUCENE-7966) build mr-jar and use some java 9 methods if available
[
https://issues.apache.org/jira/browse/LUCENE-7966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16182499#comment-16182499
]
Uwe Schindler edited comment on LUCENE-7966 at 9/27/17 12:43 PM:
-
Thanks Adrien!
bq. Results are slightly less good than previously but it could well be noise.
This can only be noise, because the code we are running is the same as
Robert's. I just removed the additional indirection in the Java 9 code path by
patching the class files directly.
I hope you are still ready: Can you also run this comparison with Java 8? I
assume the current numbers are with Java 9. Just to be sure that our "emulation
layer" does not slowdown Java 8.
was (Author: thetaphi):
Thanks Adrien!
bq. Results are slightly less good than previously but it could well be noise.
This can only be noise, because the code we are running is the same as
Robert's. I just reoved the additional indirection in the Java 9 code path.
I hope you are still ready: Can you also run this comparison with Java 8? I
assume the current numbers are with Java 9. Just to be sure that our "emulation
layer" does not slowdown Java 8.
> build mr-jar and use some java 9 methods if available
> -
>
> Key: LUCENE-7966
> URL: https://issues.apache.org/jira/browse/LUCENE-7966
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/other, general/build
>Reporter: Robert Muir
> Labels: Java9
> Attachments: LUCENE-7966.patch, LUCENE-7966.patch, LUCENE-7966.patch,
> LUCENE-7966.patch, LUCENE-7966.patch
>
>
> See background: http://openjdk.java.net/jeps/238
> It would be nice to use some of the newer array methods and range checking
> methods in java 9 for example, without waiting for lucene 10 or something. If
> we build an MR-jar, we can start migrating our code to use java 9 methods
> right now, it will use optimized methods from java 9 when thats available,
> otherwise fall back to java 8 code.
> This patch adds:
> {code}
> Objects.checkIndex(int,int)
> Objects.checkFromToIndex(int,int,int)
> Objects.checkFromIndexSize(int,int,int)
> Arrays.mismatch(byte[],int,int,byte[],int,int)
> Arrays.compareUnsigned(byte[],int,int,byte[],int,int)
> Arrays.equal(byte[],int,int,byte[],int,int)
> // did not add char/int/long/short/etc but of course its possible if needed
> {code}
> It sets these up in {{org.apache.lucene.future}} as 1-1 mappings to java
> methods. This way, we can simply directly replace call sites with java 9
> methods when java 9 is a minimum. Simple 1-1 mappings mean also that we only
> have to worry about testing that our java 8 fallback methods work.
> I found that many of the current byte array methods today are willy-nilly and
> very lenient for example, passing invalid offsets at times and relying on
> compare methods not throwing exceptions, etc. I fixed all the instances in
> core/codecs but have not looked at the problems with AnalyzingSuggester. Also
> SimpleText still uses a silly method in ArrayUtil in similar crazy way, have
> not removed that one yet.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (LUCENE-7966) build mr-jar and use some java 9 methods if available
[
https://issues.apache.org/jira/browse/LUCENE-7966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170377#comment-16170377
]
Uwe Schindler edited comment on LUCENE-7966 at 9/18/17 5:53 PM:
I implemented by latest idea based again on Robert's patch:
https://github.com/apache/lucene-solr/compare/master...uschindler:jira/LUCENE-7966-v2
This approach is much more clean: We compile against Robert's replacement
classes {{FutureObjects}} and {{FutureArrays}} (that have to contain the same
method signatures as the Java 9 original, but we can add a test for this later
with smoketester) as usual with Java 8. Before packaging the JAR file we read
all class files and patch all {{FutureObjects/FutureArrays}} references to
refer to the Java 9 class. The patched output is sent to a separate folder
{{build/classes/java9}}. The JAR file is then packaged to take both variants,
placing the patched ones in the Java 9 MultiRelease part.
Currently only the lucene-core.jar file uses the patched stuff, so stuff
outside lucene-core (e.g., codecs) does not yet automatically add Java 9
variants, instead it will use Robert's classes. If this is the way to go, I
will move the patcher to the global tools directory and we can apply patching
to all JAR files of the distribution. WARNING: We cannot support Maven builds
here, Maven always builds a Java8-only JAR file!
[~mikemccand], [~jpountz]: Could you build a lucene-core.jar file with the
above branch on Github and do your tests again? The main difference here is
that the JAR file no longer contains a delegator class. Instead all class files
that were originally compiled with FutureObjects/FutureArrays (for Java 8
support) are patched to directly use the Java 9 Arrays/Objects methods, without
using a delegator class. Keep in mind: This currently only support
lucene-core.jar, the codecs JAR file is not yet Multirelease with this patch.
When building with {{ant jar}} inside {{lucene/core}} you should see output
like this:
{noformat}
[compile shit...]
[copy] Copying 3 files to C:\Users\Uwe
Schindler\Projects\lucene\trunk-lusolr1\lucene\build\core\classes\java
-mrjar-classes-uptodate:
resolve-groovy:
[ivy:cachepath] :: resolving dependencies ::
org.codehaus.groovy#groovy-all-caller;working
[ivy:cachepath] confs: [default]
[ivy:cachepath] found org.codehaus.groovy#groovy-all;2.4.8 in public
[ivy:cachepath] :: resolution report :: resolve 170ms :: artifacts dl 5ms
-
| |modules|| artifacts |
| conf | number| search|dwnlded|evicted|| number|dwnlded|
-
| default | 1 | 0 | 0 | 0 || 1 | 0 |
-
patch-mrjar-classes:
[ivy:cachepath] :: resolving dependencies ::
org.ow2.asm#asm-commons-caller;working
[ivy:cachepath] confs: [default]
[ivy:cachepath] found org.ow2.asm#asm-commons;5.1 in public
[ivy:cachepath] found org.ow2.asm#asm-tree;5.1 in public
[ivy:cachepath] found org.ow2.asm#asm;5.1 in public
[ivy:cachepath] :: resolution report :: resolve 701ms :: artifacts dl 8ms
-
| |modules|| artifacts |
| conf | number| search|dwnlded|evicted|| number|dwnlded|
-
| default | 3 | 0 | 0 | 0 || 3 | 0 |
-
[groovy] Remapped:
org/apache/lucene/analysis/tokenattributes/CharTermAttributeImpl
[groovy] Remapped: org/apache/lucene/codecs/compressing/LZ4
[groovy] Remapped: org/apache/lucene/document/BinaryPoint$2
[groovy] Remapped: org/apache/lucene/document/DoubleRange
[groovy] Remapped: org/apache/lucene/document/FloatRange
[groovy] Remapped: org/apache/lucene/document/IntRange
[groovy] Remapped: org/apache/lucene/document/LongRange
[groovy] Remapped: org/apache/lucene/index/BitsSlice
[groovy] Remapped: org/apache/lucene/index/CodecReader
[groovy] Remapped: org/apache/lucene/index/MergeReaderWrapper
[groovy] Remapped: org/apache/lucene/search/BooleanScorer$TailPriorityQueue
[groovy] Remapped: org/apache/lucene/util/BytesRef
[groovy] Remapped: org/apache/lucene/util/BytesRefArray
[groovy] Remapped:
org/apache/lucene/util/CharsRef$UTF16SortedAsUTF8Comparator
[groovy] Remapped: org/apache/lucene/util/CharsRef
[groovy] Remapped: org/apache/lucene/util/IntsRef
[groovy] Remapped:
[jira] [Comment Edited] (LUCENE-7966) build mr-jar and use some java 9 methods if available
[
https://issues.apache.org/jira/browse/LUCENE-7966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170377#comment-16170377
]
Uwe Schindler edited comment on LUCENE-7966 at 9/18/17 5:52 PM:
I implemented by latest idea based again on Robert's patch:
https://github.com/apache/lucene-solr/compare/master...uschindler:jira/LUCENE-7966-v2
This approach is much more clean: We compile against Robert's replacement
classes {{FutureObjects}} and {{FutureArrays}} (that have to contain the same
method signatures as the Java 9 original, but we can add a test for this later
with smoketester) as usual with Java 8. Before packaging the JAR file we read
all class files and patch all {{FutureObjects/FutureArrays}} references to
refer to the Java 9 class. The patched output is sent to a separate folder
{{build/classes/java9}}. The JAR file is then packaged to take both variants,
placing the patched ones in the Java 9 MultiRelease part.
Currently only the lucene-core.jar file uses the patched stuff, so stuff
outside lucene-core (e.g., codecs) does not yet automatically add Java 9
variants, instead it will use Robert's classes. If this is the way to go, I
will move the patcher to the global tools directory and we can apply patching
to all JAR files of the distribution. WARNING: We cannot support Maven builds
here, Maven always builds a Java8-only JAR file!
[~mikemccand], [~jpountz]: Could you build a lucene-core.jar file with the
above branch on Github and do your tests again? The main difference here is
that the JAR file no longer contains a delegator class. Instead all class files
that were originally compiled with FutureObjects/FutureArrays (for Java 8
support) are patched to directly use the Java 9 Arrays/Objects methods, without
using a delegator class. Keep in mind: This currently only support
lucene-core.jar, the codecs JAR file is not yet Multirelease with this patch.
When building with {{ant jar}} inside {{lucene/core}} you should see output
like this:
{noformat}
[compile shit...]
[copy] Copying 3 files to C:\Users\Uwe
Schindler\Projects\lucene\trunk-lusolr1\lucene\build\core\classes\java
-mrjar-classes-uptodate:
resolve-groovy:
[ivy:cachepath] :: resolving dependencies ::
org.codehaus.groovy#groovy-all-caller;working
[ivy:cachepath] confs: [default]
[ivy:cachepath] found org.codehaus.groovy#groovy-all;2.4.8 in public
[ivy:cachepath] :: resolution report :: resolve 170ms :: artifacts dl 5ms
-
| |modules|| artifacts |
| conf | number| search|dwnlded|evicted|| number|dwnlded|
-
| default | 1 | 0 | 0 | 0 || 1 | 0 |
-
patch-mrjar-classes:
[ivy:cachepath] :: resolving dependencies ::
org.ow2.asm#asm-commons-caller;working
[ivy:cachepath] confs: [default]
[ivy:cachepath] found org.ow2.asm#asm-commons;5.1 in public
[ivy:cachepath] found org.ow2.asm#asm-tree;5.1 in public
[ivy:cachepath] found org.ow2.asm#asm;5.1 in public
[ivy:cachepath] :: resolution report :: resolve 701ms :: artifacts dl 8ms
-
| |modules|| artifacts |
| conf | number| search|dwnlded|evicted|| number|dwnlded|
-
| default | 3 | 0 | 0 | 0 || 3 | 0 |
-
[groovy] Remapped:
org/apache/lucene/analysis/tokenattributes/CharTermAttributeImpl
[groovy] Remapped: org/apache/lucene/codecs/compressing/LZ4
[groovy] Remapped: org/apache/lucene/document/BinaryPoint$2
[groovy] Remapped: org/apache/lucene/document/DoubleRange
[groovy] Remapped: org/apache/lucene/document/FloatRange
[groovy] Remapped: org/apache/lucene/document/IntRange
[groovy] Remapped: org/apache/lucene/document/LongRange
[groovy] Remapped: org/apache/lucene/index/BitsSlice
[groovy] Remapped: org/apache/lucene/index/CodecReader
[groovy] Remapped: org/apache/lucene/index/MergeReaderWrapper
[groovy] Remapped: org/apache/lucene/search/BooleanScorer$TailPriorityQueue
[groovy] Remapped: org/apache/lucene/util/BytesRef
[groovy] Remapped: org/apache/lucene/util/BytesRefArray
[groovy] Remapped:
org/apache/lucene/util/CharsRef$UTF16SortedAsUTF8Comparator
[groovy] Remapped: org/apache/lucene/util/CharsRef
[groovy] Remapped: org/apache/lucene/util/IntsRef
[groovy] Remapped:
[jira] [Comment Edited] (LUCENE-7966) build mr-jar and use some java 9 methods if available
[
https://issues.apache.org/jira/browse/LUCENE-7966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170388#comment-16170388
]
Uwe Schindler edited comment on LUCENE-7966 at 9/18/17 5:52 PM:
bq. Why exactly is this the case? with my patch maven should work. Maven just
uses the jars produced by ant. The smoketester validates they are exactly the
same.
Maven works when you build the JAR files for Maven with our ANT targets. What
does not work is the pom.xml build generated by sarowes maven build. That one
will build JAR files only with the FutureXxx stuff, but no multirelease stuff.
Sorry for being imprecise.
was (Author: thetaphi):
bq. Why exactly is this the case? with my patch maven should work. Maven just
uses the jars produced by ant. The smoketester validates they are exactly the
same.
Maven works when you build the JAR files for Maven. What does not work is the
pom.xml build generated by sarowes maven build. That one will build JAR files
only with the FutureXxx stuff, but no multirelease stuff. Sorry for being
imprecise.
> build mr-jar and use some java 9 methods if available
> -
>
> Key: LUCENE-7966
> URL: https://issues.apache.org/jira/browse/LUCENE-7966
> Project: Lucene - Core
> Issue Type: Improvement
> Components: general/build
>Reporter: Robert Muir
> Attachments: LUCENE-7966.patch, LUCENE-7966.patch, LUCENE-7966.patch,
> LUCENE-7966.patch, LUCENE-7966.patch
>
>
> See background: http://openjdk.java.net/jeps/238
> It would be nice to use some of the newer array methods and range checking
> methods in java 9 for example, without waiting for lucene 10 or something. If
> we build an MR-jar, we can start migrating our code to use java 9 methods
> right now, it will use optimized methods from java 9 when thats available,
> otherwise fall back to java 8 code.
> This patch adds:
> {code}
> Objects.checkIndex(int,int)
> Objects.checkFromToIndex(int,int,int)
> Objects.checkFromIndexSize(int,int,int)
> Arrays.mismatch(byte[],int,int,byte[],int,int)
> Arrays.compareUnsigned(byte[],int,int,byte[],int,int)
> Arrays.equal(byte[],int,int,byte[],int,int)
> // did not add char/int/long/short/etc but of course its possible if needed
> {code}
> It sets these up in {{org.apache.lucene.future}} as 1-1 mappings to java
> methods. This way, we can simply directly replace call sites with java 9
> methods when java 9 is a minimum. Simple 1-1 mappings mean also that we only
> have to worry about testing that our java 8 fallback methods work.
> I found that many of the current byte array methods today are willy-nilly and
> very lenient for example, passing invalid offsets at times and relying on
> compare methods not throwing exceptions, etc. I fixed all the instances in
> core/codecs but have not looked at the problems with AnalyzingSuggester. Also
> SimpleText still uses a silly method in ArrayUtil in similar crazy way, have
> not removed that one yet.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (LUCENE-7966) build mr-jar and use some java 9 methods if available
[
https://issues.apache.org/jira/browse/LUCENE-7966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170377#comment-16170377
]
Uwe Schindler edited comment on LUCENE-7966 at 9/18/17 5:50 PM:
I implemented by latest idea based again on Robert's patch:
https://github.com/apache/lucene-solr/compare/master...uschindler:jira/LUCENE-7966-v2
This approach is much more clean: We compile against Robert's replacement
classes {{FutureObjects}} and {{FutureArrays}} (that have to contain the same
method signatures as the Java 9 original, but we can add a test for this later
with smoketester) as usual with Java 8. Before packaging the JAR file we read
all class files and patch all {{FutureObjects/FutureArrays}} references to
refer to the Java 9 class. The patched output is sent to a separate folder
{{build/classes/java9}}. The JAR file is then packaged to take both variants,
placing the patched ones in the Java 9 MultiRelease part.
Currently only the lucene-core.jar file uses the patched stuff, so stuff
outside lucene-core (e.g., codecs) does not yet automatically add Java 9
variants, instead it will use Robert's classes. If this is the way to go, I
will move the patcher to the global tools directory and we can apply patching
to all JAR files of the distribution. WARNING: We cannot support Maven builds
here, Maven always builds a Java8-only JAR file!
[~mikemccand], [~jpountz]: Could you build a lucene-core.jar file with the
above branch on Github and do your tests again? The main difference here is
that the JAR file no longer contains a delegator class. Instead all class files
that were originally compiled with FutureObjects/FutureArrays (for Java 8
support) are patched to directly use the Java 9 Arrays/Objects methods, without
using a delegator class. Keep in mind: This currently only support
lucene-core.jar, the codecs JAR file is not yet Multirelease with this patch.
When building with {{ant jar}} inside {{lucene/core}} you should see output
like this:
{{noformat}}
[compile shit...]
[copy] Copying 3 files to C:\Users\Uwe
Schindler\Projects\lucene\trunk-lusolr1\lucene\build\core\classes\java
-mrjar-classes-uptodate:
resolve-groovy:
[ivy:cachepath] :: resolving dependencies ::
org.codehaus.groovy#groovy-all-caller;working
[ivy:cachepath] confs: [default]
[ivy:cachepath] found org.codehaus.groovy#groovy-all;2.4.8 in public
[ivy:cachepath] :: resolution report :: resolve 170ms :: artifacts dl 5ms
-
| |modules|| artifacts |
| conf | number| search|dwnlded|evicted|| number|dwnlded|
-
| default | 1 | 0 | 0 | 0 || 1 | 0 |
-
patch-mrjar-classes:
[ivy:cachepath] :: resolving dependencies ::
org.ow2.asm#asm-commons-caller;working
[ivy:cachepath] confs: [default]
[ivy:cachepath] found org.ow2.asm#asm-commons;5.1 in public
[ivy:cachepath] found org.ow2.asm#asm-tree;5.1 in public
[ivy:cachepath] found org.ow2.asm#asm;5.1 in public
[ivy:cachepath] :: resolution report :: resolve 701ms :: artifacts dl 8ms
-
| |modules|| artifacts |
| conf | number| search|dwnlded|evicted|| number|dwnlded|
-
| default | 3 | 0 | 0 | 0 || 3 | 0 |
-
[groovy] Remapped:
org/apache/lucene/analysis/tokenattributes/CharTermAttributeImpl
[groovy] Remapped: org/apache/lucene/codecs/compressing/LZ4
[groovy] Remapped: org/apache/lucene/document/BinaryPoint$2
[groovy] Remapped: org/apache/lucene/document/DoubleRange
[groovy] Remapped: org/apache/lucene/document/FloatRange
[groovy] Remapped: org/apache/lucene/document/IntRange
[groovy] Remapped: org/apache/lucene/document/LongRange
[groovy] Remapped: org/apache/lucene/index/BitsSlice
[groovy] Remapped: org/apache/lucene/index/CodecReader
[groovy] Remapped: org/apache/lucene/index/MergeReaderWrapper
[groovy] Remapped: org/apache/lucene/search/BooleanScorer$TailPriorityQueue
[groovy] Remapped: org/apache/lucene/util/BytesRef
[groovy] Remapped: org/apache/lucene/util/BytesRefArray
[groovy] Remapped:
org/apache/lucene/util/CharsRef$UTF16SortedAsUTF8Comparator
[groovy] Remapped: org/apache/lucene/util/CharsRef
[groovy] Remapped: org/apache/lucene/util/IntsRef
[groovy] Remapped:
[jira] [Comment Edited] (LUCENE-7966) build mr-jar and use some java 9 methods if available
[
https://issues.apache.org/jira/browse/LUCENE-7966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170027#comment-16170027
]
Uwe Schindler edited comment on LUCENE-7966 at 9/18/17 3:21 PM:
Here is the class remapper: https://paste.apache.org/bAzx
Basically it rewrites all references to oal.future.FutureXxxx to the Java 9
type java.util.Xxxx. All files that contain in the Java 8 code in
{{build/classes/java}} references to our own FutureXxx classes (the remapper
sets remapped=true for those) are saved to in rewritten formto a separate
directory {{build/classes/java9}} in parallel to the original and are packaged
into the multirelease part of the JAR. All classes that have no references to
our FutureXxx backports are kept out.
This can be done as a general task and may be applied to all Lucene/Solr
modules.
I will update my branch later.
was (Author: thetaphi):
Here is the class remapper: https://paste.apache.org/NUOp
Basically it rewrites all references to oal.future.FutureXxxx to the Java 9
type java.util.Xxxx. All files that contain in the Java 8 code in
{{build/classes/java}} references to our own FutureXxx classes (the remapper
sets remapped=true for those) are saved to in rewritten formto a separate
directory {{build/classes/java9}} in parallel to the original and are packaged
into the multirelease part of the JAR. All classes that have no references to
our FutureXxx backports are kept out.
This can be done as a general task and may be applied to all Lucene/Solr
modules.
I will update my branch later.
> build mr-jar and use some java 9 methods if available
> -
>
> Key: LUCENE-7966
> URL: https://issues.apache.org/jira/browse/LUCENE-7966
> Project: Lucene - Core
> Issue Type: Improvement
> Components: general/build
>Reporter: Robert Muir
> Attachments: LUCENE-7966.patch, LUCENE-7966.patch, LUCENE-7966.patch,
> LUCENE-7966.patch, LUCENE-7966.patch
>
>
> See background: http://openjdk.java.net/jeps/238
> It would be nice to use some of the newer array methods and range checking
> methods in java 9 for example, without waiting for lucene 10 or something. If
> we build an MR-jar, we can start migrating our code to use java 9 methods
> right now, it will use optimized methods from java 9 when thats available,
> otherwise fall back to java 8 code.
> This patch adds:
> {code}
> Objects.checkIndex(int,int)
> Objects.checkFromToIndex(int,int,int)
> Objects.checkFromIndexSize(int,int,int)
> Arrays.mismatch(byte[],int,int,byte[],int,int)
> Arrays.compareUnsigned(byte[],int,int,byte[],int,int)
> Arrays.equal(byte[],int,int,byte[],int,int)
> // did not add char/int/long/short/etc but of course its possible if needed
> {code}
> It sets these up in {{org.apache.lucene.future}} as 1-1 mappings to java
> methods. This way, we can simply directly replace call sites with java 9
> methods when java 9 is a minimum. Simple 1-1 mappings mean also that we only
> have to worry about testing that our java 8 fallback methods work.
> I found that many of the current byte array methods today are willy-nilly and
> very lenient for example, passing invalid offsets at times and relying on
> compare methods not throwing exceptions, etc. I fixed all the instances in
> core/codecs but have not looked at the problems with AnalyzingSuggester. Also
> SimpleText still uses a silly method in ArrayUtil in similar crazy way, have
> not removed that one yet.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (LUCENE-7966) build mr-jar and use some java 9 methods if available
[
https://issues.apache.org/jira/browse/LUCENE-7966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16169838#comment-16169838
]
Uwe Schindler edited comment on LUCENE-7966 at 9/18/17 1:43 PM:
When thinking last night about the whole thing a bit more, I had a cool idea:
Currently we use ASM to generate the stub files to compile against (see my
Github repo). On top of these stubs we use a "wrapper class" that just
delegates all methods to the Java 9 one. IMHO, this is not nice for the
optimizer (although it can handle that). But the
oal.future.FutureObjects/FutureArrays classes just contain the same signatures
as their Java 9 variants would contain. So my idea is to use ASM to patch all
classes:
- Use a groovy script that runs on the compiler output, before building the JAR
file
- Load class with ASM and use ASM's rewriter functionality to change the
classname of all occurences of oal.future.FutureObjects/FutureArrays and
replace them by java.util.Objects/Arrays. We can use this utility out of ASM to
do this:
[http://asm.ow2.org/asm50/javadoc/user/org/objectweb/asm/commons/ClassRemapper.html].
Whenever a class file contaisn references to FutureXXX classes, we patch it
using asm and write it out to META-INF folder as Java 9 variant.
- Then package MR jar.
The good thing:
- we don't need stub files to compile with Java 8. We just need the smoke
tester to verify the patched class files actually resolves against Java 9
during the Java 9 checks
- we have no license issues, because we don't need to generate and commit the
stubs. In our source files we solely use oal.future.Objects/Arrays. Adapting to
Java 9 is done by constant pool renaming :-)
What do you think? I will try this variant a bit later today. We can use the
same approach for other Java 9 classes, too! Maybe this also helps with the
issues Mike has seen (I am not happy to have the degelator class).
was (Author: thetaphi):
When thinking last night about the whole thing a bit more, I had a cool idea:
Currently we use ASM to generate the stub files to compile against (see my
Github repo). On top of these stubs we use a "wrapper class" that just
delegates all methods to the Java 9 one. IMHO, this is not nice for the
optimizer (although it can handle that). But the
oal.util.FutureObjects/FutureArrays classes just contain the same signatures as
their Java 9 variants would contain. So my idea is to use ASM to patch all
classes:
- Use a groovy script that runs on the compiler output, before building the JAR
file
- Load class with ASM and use ASM's rewriter functionality to change the
classname of all occurences of oal.util.FutureObjects/FutureArrays and replace
them by java.util.Objects/Arrays. We can use this utility out of ASM to do
this:
[http://asm.ow2.org/asm50/javadoc/user/org/objectweb/asm/commons/ClassRemapper.html].
Whenever a class file contaisn references to FutureXXX classes, we patch it
using asm and write it out to META-INF folder as Java 9 variant.
- Then package MR jar.
The good thing:
- we don't need stub files to compile with Java 8. We just need the smoke
tester to verify the patched class files actually resolves against Java 9
during the Java 9 checks
- we have no license issues, because we don't need to generate and commit the
stubs. In our source files we solely use oal.util.Objects/Arrays. Adapting to
Java 9 is done by constant pool renaming :-)
What do you think? I will try this variant a bit later today. We can use the
same approach for other Java 9 classes, too! Maybe this also helps with the
issues Mike has seen (I am not happy to have the degelator class).
> build mr-jar and use some java 9 methods if available
> -
>
> Key: LUCENE-7966
> URL: https://issues.apache.org/jira/browse/LUCENE-7966
> Project: Lucene - Core
> Issue Type: Improvement
> Components: general/build
>Reporter: Robert Muir
> Attachments: LUCENE-7966.patch, LUCENE-7966.patch, LUCENE-7966.patch,
> LUCENE-7966.patch, LUCENE-7966.patch
>
>
> See background: http://openjdk.java.net/jeps/238
> It would be nice to use some of the newer array methods and range checking
> methods in java 9 for example, without waiting for lucene 10 or something. If
> we build an MR-jar, we can start migrating our code to use java 9 methods
> right now, it will use optimized methods from java 9 when thats available,
> otherwise fall back to java 8 code.
> This patch adds:
> {code}
> Objects.checkIndex(int,int)
> Objects.checkFromToIndex(int,int,int)
> Objects.checkFromIndexSize(int,int,int)
> Arrays.mismatch(byte[],int,int,byte[],int,int)
> Arrays.compareUnsigned(byte[],int,int,byte[],int,int)
> Arrays.equal(byte[],int,int,byte[],int,int)
> // did not add char/int/long/short/etc but of course its possible if
[jira] [Comment Edited] (LUCENE-7966) build mr-jar and use some java 9 methods if available
[
https://issues.apache.org/jira/browse/LUCENE-7966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16169838#comment-16169838
]
Uwe Schindler edited comment on LUCENE-7966 at 9/18/17 10:27 AM:
-
When thinking last night about the whole thing a bit more, I had a cool idea:
Currently we use ASM to generate the stub files to compile against (see my
Github repo). On top of these stubs we use a "wrapper class" that just
delegates all methods to the Java 9 one. IMHO, this is not nice for the
optimizer (although it can handle that). But the
oal.util.FutureObjects/FutureArrays classes just contain the same signatures as
their Java 9 variants would contain. So my idea is to use ASM to patch all
classes:
- Use a groovy script that runs on the compiler output, before building the JAR
file
- Load class with ASM and use ASM's rewriter functionality to change the
classname of all occurences of oal.util.FutureObjects/FutureArrays and replace
them by java.util.Objects/Arrays. We can use this utility out of ASM to do
this:
[http://asm.ow2.org/asm50/javadoc/user/org/objectweb/asm/commons/ClassRemapper.html].
Whenever a class file contaisn references to FutureXXX classes, we patch it
using asm and write it out to META-INF folder as Java 9 variant.
- Then package MR jar.
The good thing:
- we don't need stub files to compile with Java 8. We just need the smoke
tester to verify the patched class files actually resolves against Java 9
during the Java 9 checks
- we have no license issues, because we don't need to generate and commit the
stubs. In our source files we solely use oal.util.Objects/Arrays. Adapting to
Java 9 is done by constant pool renaming :-)
What do you think? I will try this variant a bit later today. We can use the
same approach for other Java 9 classes, too! Maybe this also helps with the
issues Mike has seen (I am not happy to have the degelator class).
was (Author: thetaphi):
When thinking last night about the whole thing a bit more, I had a cool idea:
Currently we use ASM to generate the stub files to compile against (see my
Github repo). On top of these stubs we use a "wrapper class" that just
delegates all methods to the Java 9 one. IMHO, this is not nice for the
optimizer (although it can handle that). But the
oal.util.FutureObjects/FutureArrays classes just contain the same signatures as
their Java 9 variants would contain. So my idea is to use ASM to patch all
classes:
- Use a groovy script that runs on the compiler output, before building the JAR
file
- Load class with ASM and use ASM's rewriter functionality to change the
classname of all occurences of oal.util.FutureObjects/FutureArrays and replace
them by java.util.Objects/Arrays. We can use this utility out of ASM to do
this:
[http://asm.ow2.org/asm50/javadoc/user/org/objectweb/asm/commons/ClassRemapper.html]
Whenever a class file contaisn references to FutureXXX classes, we patch it
using asm and write it out to META-INF folder as Java 9 variant.
- Then package MR jar.
The good thing:
- we don't need stub files to compile with Java 8. We just need the smoke
tester to verify the patched class files actually resolves against Java 9
during the Java 9 checks
- we have no license issues, because we don't need to generate and commit the
stubs. In our source files we solely use oal.util.Objects/Arrays. Adapting to
Java 9 is done by constant pool renaming :-)
What do you think? I will try this variant a bit later today. We can use the
same approach for other Java 9 classes, too! Maybe this also helps with the
issues Mike has seen (I am not happy to have the degelator class).
> build mr-jar and use some java 9 methods if available
> -
>
> Key: LUCENE-7966
> URL: https://issues.apache.org/jira/browse/LUCENE-7966
> Project: Lucene - Core
> Issue Type: Improvement
> Components: general/build
>Reporter: Robert Muir
> Attachments: LUCENE-7966.patch, LUCENE-7966.patch, LUCENE-7966.patch,
> LUCENE-7966.patch, LUCENE-7966.patch
>
>
> See background: http://openjdk.java.net/jeps/238
> It would be nice to use some of the newer array methods and range checking
> methods in java 9 for example, without waiting for lucene 10 or something. If
> we build an MR-jar, we can start migrating our code to use java 9 methods
> right now, it will use optimized methods from java 9 when thats available,
> otherwise fall back to java 8 code.
> This patch adds:
> {code}
> Objects.checkIndex(int,int)
> Objects.checkFromToIndex(int,int,int)
> Objects.checkFromIndexSize(int,int,int)
> Arrays.mismatch(byte[],int,int,byte[],int,int)
> Arrays.compareUnsigned(byte[],int,int,byte[],int,int)
> Arrays.equal(byte[],int,int,byte[],int,int)
> // did not add char/int/long/short/etc but of course its possible if
[jira] [Comment Edited] (LUCENE-7966) build mr-jar and use some java 9 methods if available
[
https://issues.apache.org/jira/browse/LUCENE-7966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16169838#comment-16169838
]
Uwe Schindler edited comment on LUCENE-7966 at 9/18/17 10:27 AM:
-
When thinking last night about the whole thing a bit more, I had a cool idea:
Currently we use ASM to generate the stub files to compile against (see my
Github repo). On top of these stubs we use a "wrapper class" that just
delegates all methods to the Java 9 one. IMHO, this is not nice for the
optimizer (although it can handle that). But the
oal.util.FutureObjects/FutureArrays classes just contain the same signatures as
their Java 9 variants would contain. So my idea is to use ASM to patch all
classes:
- Use a groovy script that runs on the compiler output, before building the JAR
file
- Load class with ASM and use ASM's rewriter functionality to change the
classname of all occurences of oal.util.FutureObjects/FutureArrays and replace
them by java.util.Objects/Arrays. We can use this utility out of ASM to do
this:
[http://asm.ow2.org/asm50/javadoc/user/org/objectweb/asm/commons/ClassRemapper.html]
Whenever a class file contaisn references to FutureXXX classes, we patch it
using asm and write it out to META-INF folder as Java 9 variant.
- Then package MR jar.
The good thing:
- we don't need stub files to compile with Java 8. We just need the smoke
tester to verify the patched class files actually resolves against Java 9
during the Java 9 checks
- we have no license issues, because we don't need to generate and commit the
stubs. In our source files we solely use oal.util.Objects/Arrays. Adapting to
Java 9 is done by constant pool renaming :-)
What do you think? I will try this variant a bit later today. We can use the
same approach for other Java 9 classes, too! Maybe this also helps with the
issues Mike has seen (I am not happy to have the degelator class).
was (Author: thetaphi):
When thinking last night about the whole thing a bit more, I had a cool idea:
Currently we use ASM to generate the stub files to compile against (see my
Github repo). On top of these stubs we use a "wrapper class" that just
delegates all methods to the Java 9 one. IMHO, this is not nice for the
optimizer (although it can handle that). But the
oal.util.FutureObjects/FutureArrays classes just contain the same signatures as
their Java 9 variants would contain. So my idea is to use ASM to patch all
classes:
- Use a groovy script that runs on the compiler output, before building the JAR
file
- Load class with ASM and use ASM's rewriter functionality to change the
classname of all occurences of oal.util.FutureObjects/FutureArrays and replace
them by java.util.Objects/Arrays. Whenever a class file matches this pattern,
we patch it using asm and write it out to META-INF folder as Java 9 variant.
- Then package MR jar.
The good thing:
- we don't need stub files to compile with Java 8. We just need the smoke
tester to verify the patched class files actually resolves against Java 9
during the Java 9 checks
- we have no license issues, because we don't need to generate and commit the
stubs. In our source files we solely use oal.util.Objects/Arrays. Adapting to
Java 9 is done by constant pool renaming :-)
What do you think? I will try this variant a bit later today. We can use the
same approach for other Java 9 classes, too! Maybe this also helps with the
issues Mike has seen (I am not happy to have the degelator class).
> build mr-jar and use some java 9 methods if available
> -
>
> Key: LUCENE-7966
> URL: https://issues.apache.org/jira/browse/LUCENE-7966
> Project: Lucene - Core
> Issue Type: Improvement
> Components: general/build
>Reporter: Robert Muir
> Attachments: LUCENE-7966.patch, LUCENE-7966.patch, LUCENE-7966.patch,
> LUCENE-7966.patch, LUCENE-7966.patch
>
>
> See background: http://openjdk.java.net/jeps/238
> It would be nice to use some of the newer array methods and range checking
> methods in java 9 for example, without waiting for lucene 10 or something. If
> we build an MR-jar, we can start migrating our code to use java 9 methods
> right now, it will use optimized methods from java 9 when thats available,
> otherwise fall back to java 8 code.
> This patch adds:
> {code}
> Objects.checkIndex(int,int)
> Objects.checkFromToIndex(int,int,int)
> Objects.checkFromIndexSize(int,int,int)
> Arrays.mismatch(byte[],int,int,byte[],int,int)
> Arrays.compareUnsigned(byte[],int,int,byte[],int,int)
> Arrays.equal(byte[],int,int,byte[],int,int)
> // did not add char/int/long/short/etc but of course its possible if needed
> {code}
> It sets these up in {{org.apache.lucene.future}} as 1-1 mappings to java
> methods. This way, we can simply directly replace call sites with
[jira] [Comment Edited] (LUCENE-7966) build mr-jar and use some java 9 methods if available
[
https://issues.apache.org/jira/browse/LUCENE-7966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16169480#comment-16169480
]
Uwe Schindler edited comment on LUCENE-7966 at 9/18/17 12:20 AM:
-
This is just my patch on top of Robert's:
https://github.com/apache/lucene-solr/commit/e69d77023fbfc60bb0baf16dc5102ab76419cf03
was (Author: thetaphi):
This is just my patch on top of Robert's:
https://github.com/uschindler/lucene-solr/commit/94fc2ff8eba29168e36cef360df6f929a8bb18a2
> build mr-jar and use some java 9 methods if available
> -
>
> Key: LUCENE-7966
> URL: https://issues.apache.org/jira/browse/LUCENE-7966
> Project: Lucene - Core
> Issue Type: Improvement
> Components: general/build
>Reporter: Robert Muir
> Attachments: LUCENE-7966.patch, LUCENE-7966.patch, LUCENE-7966.patch,
> LUCENE-7966.patch, LUCENE-7966.patch
>
>
> See background: http://openjdk.java.net/jeps/238
> It would be nice to use some of the newer array methods and range checking
> methods in java 9 for example, without waiting for lucene 10 or something. If
> we build an MR-jar, we can start migrating our code to use java 9 methods
> right now, it will use optimized methods from java 9 when thats available,
> otherwise fall back to java 8 code.
> This patch adds:
> {code}
> Objects.checkIndex(int,int)
> Objects.checkFromToIndex(int,int,int)
> Objects.checkFromIndexSize(int,int,int)
> Arrays.mismatch(byte[],int,int,byte[],int,int)
> Arrays.compareUnsigned(byte[],int,int,byte[],int,int)
> Arrays.equal(byte[],int,int,byte[],int,int)
> // did not add char/int/long/short/etc but of course its possible if needed
> {code}
> It sets these up in {{org.apache.lucene.future}} as 1-1 mappings to java
> methods. This way, we can simply directly replace call sites with java 9
> methods when java 9 is a minimum. Simple 1-1 mappings mean also that we only
> have to worry about testing that our java 8 fallback methods work.
> I found that many of the current byte array methods today are willy-nilly and
> very lenient for example, passing invalid offsets at times and relying on
> compare methods not throwing exceptions, etc. I fixed all the instances in
> core/codecs but have not looked at the problems with AnalyzingSuggester. Also
> SimpleText still uses a silly method in ArrayUtil in similar crazy way, have
> not removed that one yet.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
