[jira] [Commented] (LUCENE-9500) Did we hit a DEFLATE bug?
[ https://issues.apache.org/jira/browse/LUCENE-9500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17558427#comment-17558427 ] Uwe Schindler commented on LUCENE-9500: --- I opened a PR to remove the fix on main branch: https://github.com/apache/lucene/pull/977 > Did we hit a DEFLATE bug? > - > > Key: LUCENE-9500 > URL: https://issues.apache.org/jira/browse/LUCENE-9500 > Project: Lucene - Core > Issue Type: Bug > Components: core/index >Affects Versions: 8.x, 9.0, 8.7 >Reporter: Adrien Grand >Assignee: Uwe Schindler >Priority: Critical > Labels: Java13, Java14, Java15, java11, jdk11, jdk13, jdk14, > jdk15 > Fix For: 8.x, 9.0, 8.7 > > Attachments: PresetDictTest.java, test_data.txt > > Time Spent: 2h 20m > Remaining Estimate: 0h > > I've been digging > [https://ci-builds.apache.org/job/Lucene/job/Lucene-Solr-NightlyTests-master/23/] > all day and managed to isolate a simple reproduction that shows the problem. > I've been starring at it all day and can't find what we are doing wrong, > which makes me wonder whether we're calling DEFLATE the wrong way or whether > we hit a DEFLATE bug. I've looked at it so much that I may be missing the > most obvious stuff. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9500) Did we hit a DEFLATE bug?
[
https://issues.apache.org/jira/browse/LUCENE-9500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17206678#comment-17206678
]
Uwe Schindler commented on LUCENE-9500:
---
The bug is fixed in JDK 16 build 18:
{noformat}
$ gradlew :lucene:core:test -Ptests.verbose=true --tests
"TestLucene87StoredFieldsFormatHighCompression" | grep buggy
2> JDK is buggy (JDK-8252739): false
{noformat}
> Did we hit a DEFLATE bug?
> -
>
> Key: LUCENE-9500
> URL: https://issues.apache.org/jira/browse/LUCENE-9500
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/index
>Affects Versions: 8.x, master (9.0), 8.7
>Reporter: Adrien Grand
>Assignee: Uwe Schindler
>Priority: Critical
> Labels: Java13, Java14, Java15, java11, jdk11, jdk13, jdk14,
> jdk15
> Fix For: 8.x, master (9.0), 8.7
>
> Attachments: PresetDictTest.java, test_data.txt
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> I've been digging
> [https://ci-builds.apache.org/job/Lucene/job/Lucene-Solr-NightlyTests-master/23/]
> all day and managed to isolate a simple reproduction that shows the problem.
> I've been starring at it all day and can't find what we are doing wrong,
> which makes me wonder whether we're calling DEFLATE the wrong way or whether
> we hit a DEFLATE bug. I've looked at it so much that I may be missing the
> most obvious stuff.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9500) Did we hit a DEFLATE bug?
[ https://issues.apache.org/jira/browse/LUCENE-9500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17190659#comment-17190659 ] Dawid Weiss commented on LUCENE-9500: - bq. I missed to run precommit after cherry-picking. No, not you, Uwe! :) [yellow card emoji] > Did we hit a DEFLATE bug? > - > > Key: LUCENE-9500 > URL: https://issues.apache.org/jira/browse/LUCENE-9500 > Project: Lucene - Core > Issue Type: Bug > Components: core/index >Affects Versions: 8.x, master (9.0), 8.7 >Reporter: Adrien Grand >Assignee: Uwe Schindler >Priority: Critical > Labels: Java13, Java14, Java15, java11, jdk11, jdk13, jdk14, > jdk15 > Fix For: 8.x, master (9.0), 8.7 > > Attachments: PresetDictTest.java, test_data.txt > > Time Spent: 2h 20m > Remaining Estimate: 0h > > I've been digging > [https://ci-builds.apache.org/job/Lucene/job/Lucene-Solr-NightlyTests-master/23/] > all day and managed to isolate a simple reproduction that shows the problem. > I've been starring at it all day and can't find what we are doing wrong, > which makes me wonder whether we're calling DEFLATE the wrong way or whether > we hit a DEFLATE bug. I've looked at it so much that I may be missing the > most obvious stuff. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9500) Did we hit a DEFLATE bug?
[ https://issues.apache.org/jira/browse/LUCENE-9500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17190658#comment-17190658 ] Uwe Schindler commented on LUCENE-9500: --- Ah, ECJ was complaining :-) Tahnks for fixing. I missed to run precommit after cherry-picking. > Did we hit a DEFLATE bug? > - > > Key: LUCENE-9500 > URL: https://issues.apache.org/jira/browse/LUCENE-9500 > Project: Lucene - Core > Issue Type: Bug > Components: core/index >Affects Versions: 8.x, master (9.0), 8.7 >Reporter: Adrien Grand >Assignee: Uwe Schindler >Priority: Critical > Labels: Java13, Java14, Java15, java11, jdk11, jdk13, jdk14, > jdk15 > Fix For: 8.x, master (9.0), 8.7 > > Attachments: PresetDictTest.java, test_data.txt > > Time Spent: 2h 20m > Remaining Estimate: 0h > > I've been digging > [https://ci-builds.apache.org/job/Lucene/job/Lucene-Solr-NightlyTests-master/23/] > all day and managed to isolate a simple reproduction that shows the problem. > I've been starring at it all day and can't find what we are doing wrong, > which makes me wonder whether we're calling DEFLATE the wrong way or whether > we hit a DEFLATE bug. I've looked at it so much that I may be missing the > most obvious stuff. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9500) Did we hit a DEFLATE bug?
[ https://issues.apache.org/jira/browse/LUCENE-9500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17190653#comment-17190653 ] Uwe Schindler commented on LUCENE-9500: --- Thanks Adrien! Was this causing javadocs build failure? > Did we hit a DEFLATE bug? > - > > Key: LUCENE-9500 > URL: https://issues.apache.org/jira/browse/LUCENE-9500 > Project: Lucene - Core > Issue Type: Bug > Components: core/index >Affects Versions: 8.x, master (9.0), 8.7 >Reporter: Adrien Grand >Assignee: Uwe Schindler >Priority: Critical > Labels: Java13, Java14, Java15, java11, jdk11, jdk13, jdk14, > jdk15 > Fix For: 8.x, master (9.0), 8.7 > > Attachments: PresetDictTest.java, test_data.txt > > Time Spent: 2h 20m > Remaining Estimate: 0h > > I've been digging > [https://ci-builds.apache.org/job/Lucene/job/Lucene-Solr-NightlyTests-master/23/] > all day and managed to isolate a simple reproduction that shows the problem. > I've been starring at it all day and can't find what we are doing wrong, > which makes me wonder whether we're calling DEFLATE the wrong way or whether > we hit a DEFLATE bug. I've looked at it so much that I may be missing the > most obvious stuff. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9500) Did we hit a DEFLATE bug?
[ https://issues.apache.org/jira/browse/LUCENE-9500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17190652#comment-17190652 ] ASF subversion and git services commented on LUCENE-9500: - Commit d7299890c75bfe403f14390a0dfb70e2689fdf3c in lucene-solr's branch refs/heads/branch_8x from Adrien Grand [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=d729989 ] LUCENE-9500: There is no setDictionary(ByteBuffer) in JDK8. > Did we hit a DEFLATE bug? > - > > Key: LUCENE-9500 > URL: https://issues.apache.org/jira/browse/LUCENE-9500 > Project: Lucene - Core > Issue Type: Bug > Components: core/index >Affects Versions: 8.x, master (9.0), 8.7 >Reporter: Adrien Grand >Assignee: Uwe Schindler >Priority: Critical > Labels: Java13, Java14, Java15, java11, jdk11, jdk13, jdk14, > jdk15 > Fix For: 8.x, master (9.0), 8.7 > > Attachments: PresetDictTest.java, test_data.txt > > Time Spent: 2h 20m > Remaining Estimate: 0h > > I've been digging > [https://ci-builds.apache.org/job/Lucene/job/Lucene-Solr-NightlyTests-master/23/] > all day and managed to isolate a simple reproduction that shows the problem. > I've been starring at it all day and can't find what we are doing wrong, > which makes me wonder whether we're calling DEFLATE the wrong way or whether > we hit a DEFLATE bug. I've looked at it so much that I may be missing the > most obvious stuff. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9500) Did we hit a DEFLATE bug?
[ https://issues.apache.org/jira/browse/LUCENE-9500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17190207#comment-17190207 ] Uwe Schindler commented on LUCENE-9500: --- Thanks to all for help! > Did we hit a DEFLATE bug? > - > > Key: LUCENE-9500 > URL: https://issues.apache.org/jira/browse/LUCENE-9500 > Project: Lucene - Core > Issue Type: Bug > Components: core/index >Affects Versions: master (9.0) >Reporter: Adrien Grand >Assignee: Uwe Schindler >Priority: Minor > Labels: Java13, Java14, Java15, java11, jdk11, jdk13, jdk14, > jdk15 > Attachments: PresetDictTest.java, test_data.txt > > Time Spent: 2h 20m > Remaining Estimate: 0h > > I've been digging > [https://ci-builds.apache.org/job/Lucene/job/Lucene-Solr-NightlyTests-master/23/] > all day and managed to isolate a simple reproduction that shows the problem. > I've been starring at it all day and can't find what we are doing wrong, > which makes me wonder whether we're calling DEFLATE the wrong way or whether > we hit a DEFLATE bug. I've looked at it so much that I may be missing the > most obvious stuff. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9500) Did we hit a DEFLATE bug?
[ https://issues.apache.org/jira/browse/LUCENE-9500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17190203#comment-17190203 ] ASF subversion and git services commented on LUCENE-9500: - Commit d031f8e7bd62eb3a894cf436f81d398e3d7db493 in lucene-solr's branch refs/heads/branch_8x from Uwe Schindler [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=d031f8e ] LUCENE-9500: Separate the Deflater hack from the Lucene code to a subclass of java.util.zip.Deflater (#1824) > Did we hit a DEFLATE bug? > - > > Key: LUCENE-9500 > URL: https://issues.apache.org/jira/browse/LUCENE-9500 > Project: Lucene - Core > Issue Type: Bug > Components: core/index >Affects Versions: master (9.0) >Reporter: Adrien Grand >Assignee: Uwe Schindler >Priority: Minor > Labels: Java13, Java14, Java15, java11, jdk11, jdk13, jdk14, > jdk15 > Attachments: PresetDictTest.java, test_data.txt > > Time Spent: 2h 10m > Remaining Estimate: 0h > > I've been digging > [https://ci-builds.apache.org/job/Lucene/job/Lucene-Solr-NightlyTests-master/23/] > all day and managed to isolate a simple reproduction that shows the problem. > I've been starring at it all day and can't find what we are doing wrong, > which makes me wonder whether we're calling DEFLATE the wrong way or whether > we hit a DEFLATE bug. I've looked at it so much that I may be missing the > most obvious stuff. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9500) Did we hit a DEFLATE bug?
[ https://issues.apache.org/jira/browse/LUCENE-9500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17190200#comment-17190200 ] ASF subversion and git services commented on LUCENE-9500: - Commit 99df3814abff7c40e80530edc90a2f008a3b92b5 in lucene-solr's branch refs/heads/master from Uwe Schindler [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=99df381 ] LUCENE-9500: Separate the Deflater hack from the Lucene code to a subclass of java.util.zip.Deflater (#1824) > Did we hit a DEFLATE bug? > - > > Key: LUCENE-9500 > URL: https://issues.apache.org/jira/browse/LUCENE-9500 > Project: Lucene - Core > Issue Type: Bug > Components: core/index >Affects Versions: master (9.0) >Reporter: Adrien Grand >Assignee: Uwe Schindler >Priority: Minor > Labels: Java13, Java14, Java15, java11, jdk11, jdk13, jdk14, > jdk15 > Attachments: PresetDictTest.java, test_data.txt > > Time Spent: 2h 10m > Remaining Estimate: 0h > > I've been digging > [https://ci-builds.apache.org/job/Lucene/job/Lucene-Solr-NightlyTests-master/23/] > all day and managed to isolate a simple reproduction that shows the problem. > I've been starring at it all day and can't find what we are doing wrong, > which makes me wonder whether we're calling DEFLATE the wrong way or whether > we hit a DEFLATE bug. I've looked at it so much that I may be missing the > most obvious stuff. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9500) Did we hit a DEFLATE bug?
[ https://issues.apache.org/jira/browse/LUCENE-9500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17190063#comment-17190063 ] Uwe Schindler commented on LUCENE-9500: --- Here is a pull request that separates the hack from the lucene code, this makes maintenance easier: https://github.com/apache/lucene-solr/pull/1824 We can now add conditions about Java versions to the factory method returning the Deflater instance. Interestingly in master we still have Constants.JDK_8 or JDK_7 (we need to remove, haha). > Did we hit a DEFLATE bug? > - > > Key: LUCENE-9500 > URL: https://issues.apache.org/jira/browse/LUCENE-9500 > Project: Lucene - Core > Issue Type: Bug > Components: core/index >Affects Versions: master (9.0) >Reporter: Adrien Grand >Priority: Minor > Labels: Java13, Java14, Java15, java11, jdk11, jdk13, jdk14, > jdk15 > Attachments: PresetDictTest.java, test_data.txt > > Time Spent: 10m > Remaining Estimate: 0h > > I've been digging > [https://ci-builds.apache.org/job/Lucene/job/Lucene-Solr-NightlyTests-master/23/] > all day and managed to isolate a simple reproduction that shows the problem. > I've been starring at it all day and can't find what we are doing wrong, > which makes me wonder whether we're calling DEFLATE the wrong way or whether > we hit a DEFLATE bug. I've looked at it so much that I may be missing the > most obvious stuff. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9500) Did we hit a DEFLATE bug?
[ https://issues.apache.org/jira/browse/LUCENE-9500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17190030#comment-17190030 ] Adrien Grand commented on LUCENE-9500: -- [~uschindler] This code was backported already but not released yet indeed, which is why I didn't add a CHANGES entry for it. > Did we hit a DEFLATE bug? > - > > Key: LUCENE-9500 > URL: https://issues.apache.org/jira/browse/LUCENE-9500 > Project: Lucene - Core > Issue Type: Bug > Components: core/index >Affects Versions: master (9.0) >Reporter: Adrien Grand >Priority: Minor > Labels: Java13, Java14, Java15, java11, jdk11, jdk13, jdk14, > jdk15 > Attachments: PresetDictTest.java, test_data.txt > > > I've been digging > [https://ci-builds.apache.org/job/Lucene/job/Lucene-Solr-NightlyTests-master/23/] > all day and managed to isolate a simple reproduction that shows the problem. > I've been starring at it all day and can't find what we are doing wrong, > which makes me wonder whether we're calling DEFLATE the wrong way or whether > we hit a DEFLATE bug. I've looked at it so much that I may be missing the > most obvious stuff. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9500) Did we hit a DEFLATE bug?
[ https://issues.apache.org/jira/browse/LUCENE-9500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17190018#comment-17190018 ] Uwe Schindler commented on LUCENE-9500: --- I have seen your fix also in branch_8x. But this was not code that was released yet? In any case, if somebody runs with Java 8, we know for sure that it works correct. > Did we hit a DEFLATE bug? > - > > Key: LUCENE-9500 > URL: https://issues.apache.org/jira/browse/LUCENE-9500 > Project: Lucene - Core > Issue Type: Bug > Components: core/index >Affects Versions: master (9.0) >Reporter: Adrien Grand >Priority: Minor > Labels: Java13, Java14, Java15, java11, jdk11, jdk13, jdk14, > jdk15 > Attachments: PresetDictTest.java, test_data.txt > > > I've been digging > [https://ci-builds.apache.org/job/Lucene/job/Lucene-Solr-NightlyTests-master/23/] > all day and managed to isolate a simple reproduction that shows the problem. > I've been starring at it all day and can't find what we are doing wrong, > which makes me wonder whether we're calling DEFLATE the wrong way or whether > we hit a DEFLATE bug. I've looked at it so much that I may be missing the > most obvious stuff. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9500) Did we hit a DEFLATE bug?
[ https://issues.apache.org/jira/browse/LUCENE-9500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17189996#comment-17189996 ] Adrien Grand commented on LUCENE-9500: -- I'm seeing only a 1-2% slowdown when copying the content and indexing stored fields only, this is totally fine IMO. Thanks for reporting that bug Uwe, and thanks Robert for finding it. I had been looking for this kind of bug on this very file, but on a version that was not affected by this bug! > Did we hit a DEFLATE bug? > - > > Key: LUCENE-9500 > URL: https://issues.apache.org/jira/browse/LUCENE-9500 > Project: Lucene - Core > Issue Type: Bug > Components: core/index >Affects Versions: master (9.0) >Reporter: Adrien Grand >Priority: Minor > Labels: Java13, Java14, Java15, java11, jdk11, jdk13, jdk14, > jdk15 > Attachments: PresetDictTest.java, test_data.txt > > > I've been digging > [https://ci-builds.apache.org/job/Lucene/job/Lucene-Solr-NightlyTests-master/23/] > all day and managed to isolate a simple reproduction that shows the problem. > I've been starring at it all day and can't find what we are doing wrong, > which makes me wonder whether we're calling DEFLATE the wrong way or whether > we hit a DEFLATE bug. I've looked at it so much that I may be missing the > most obvious stuff. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9500) Did we hit a DEFLATE bug?
[ https://issues.apache.org/jira/browse/LUCENE-9500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17189977#comment-17189977 ] Uwe Schindler commented on LUCENE-9500: --- [~jpountz]: I am not sure how to handle this. If we can detect the bug somehow very fast, we may add some if/else condition for broken implementations (maybe do some quick test in static initializer). Alternatively we use the Arrays.copyOf for the time beeing. Is this performance critical? > Did we hit a DEFLATE bug? > - > > Key: LUCENE-9500 > URL: https://issues.apache.org/jira/browse/LUCENE-9500 > Project: Lucene - Core > Issue Type: Bug > Components: core/index >Affects Versions: master (9.0) >Reporter: Adrien Grand >Priority: Minor > Labels: Java13, Java14, Java15, java11, jdk11, jdk13, jdk14, > jdk15 > Attachments: PresetDictTest.java, test_data.txt > > > I've been digging > [https://ci-builds.apache.org/job/Lucene/job/Lucene-Solr-NightlyTests-master/23/] > all day and managed to isolate a simple reproduction that shows the problem. > I've been starring at it all day and can't find what we are doing wrong, > which makes me wonder whether we're calling DEFLATE the wrong way or whether > we hit a DEFLATE bug. I've looked at it so much that I may be missing the > most obvious stuff. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9500) Did we hit a DEFLATE bug?
[ https://issues.apache.org/jira/browse/LUCENE-9500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17189963#comment-17189963 ] Uwe Schindler commented on LUCENE-9500: --- The whole thing was introduced in JDK 11.0.5, versions before that work fine (I tested all starting with 11.0.2). JDK-12 seems unaffected. JDK-13 and later all have this bug. This was side-effect of a fix with critical regions leading to Hotspot crushes in Deflater/Inflater. > Did we hit a DEFLATE bug? > - > > Key: LUCENE-9500 > URL: https://issues.apache.org/jira/browse/LUCENE-9500 > Project: Lucene - Core > Issue Type: Bug >Reporter: Adrien Grand >Priority: Minor > Attachments: PresetDictTest.java, test_data.txt > > > I've been digging > [https://ci-builds.apache.org/job/Lucene/job/Lucene-Solr-NightlyTests-master/23/] > all day and managed to isolate a simple reproduction that shows the problem. > I've been starring at it all day and can't find what we are doing wrong, > which makes me wonder whether we're calling DEFLATE the wrong way or whether > we hit a DEFLATE bug. I've looked at it so much that I may be missing the > most obvious stuff. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9500) Did we hit a DEFLATE bug?
[ https://issues.apache.org/jira/browse/LUCENE-9500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17189945#comment-17189945 ] Uwe Schindler commented on LUCENE-9500: --- JDK Issue is here: https://bugs.openjdk.java.net/browse/JDK-8252739 > Did we hit a DEFLATE bug? > - > > Key: LUCENE-9500 > URL: https://issues.apache.org/jira/browse/LUCENE-9500 > Project: Lucene - Core > Issue Type: Bug >Reporter: Adrien Grand >Priority: Minor > Attachments: PresetDictTest.java, test_data.txt > > > I've been digging > [https://ci-builds.apache.org/job/Lucene/job/Lucene-Solr-NightlyTests-master/23/] > all day and managed to isolate a simple reproduction that shows the problem. > I've been starring at it all day and can't find what we are doing wrong, > which makes me wonder whether we're calling DEFLATE the wrong way or whether > we hit a DEFLATE bug. I've looked at it so much that I may be missing the > most obvious stuff. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9500) Did we hit a DEFLATE bug?
[ https://issues.apache.org/jira/browse/LUCENE-9500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17189941#comment-17189941 ] Uwe Schindler commented on LUCENE-9500: --- Looks like this was introduced here: https://github.com/openjdk/jdk/commit/9115f920d2405d889f5a1176d3488aa77a0322d4#diff-681a63e0502608f0fae34c9859d6fec7 > Did we hit a DEFLATE bug? > - > > Key: LUCENE-9500 > URL: https://issues.apache.org/jira/browse/LUCENE-9500 > Project: Lucene - Core > Issue Type: Bug >Reporter: Adrien Grand >Priority: Minor > Attachments: PresetDictTest.java, test_data.txt > > > I've been digging > [https://ci-builds.apache.org/job/Lucene/job/Lucene-Solr-NightlyTests-master/23/] > all day and managed to isolate a simple reproduction that shows the problem. > I've been starring at it all day and can't find what we are doing wrong, > which makes me wonder whether we're calling DEFLATE the wrong way or whether > we hit a DEFLATE bug. I've looked at it so much that I may be missing the > most obvious stuff. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9500) Did we hit a DEFLATE bug?
[ https://issues.apache.org/jira/browse/LUCENE-9500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17189940#comment-17189940 ] Uwe Schindler commented on LUCENE-9500: --- I am wondering: Does this really happen with JDK 11? From my analysis, only later versions should be affected! Unless it was backported! > Did we hit a DEFLATE bug? > - > > Key: LUCENE-9500 > URL: https://issues.apache.org/jira/browse/LUCENE-9500 > Project: Lucene - Core > Issue Type: Bug >Reporter: Adrien Grand >Priority: Minor > Attachments: PresetDictTest.java, test_data.txt > > > I've been digging > [https://ci-builds.apache.org/job/Lucene/job/Lucene-Solr-NightlyTests-master/23/] > all day and managed to isolate a simple reproduction that shows the problem. > I've been starring at it all day and can't find what we are doing wrong, > which makes me wonder whether we're calling DEFLATE the wrong way or whether > we hit a DEFLATE bug. I've looked at it so much that I may be missing the > most obvious stuff. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9500) Did we hit a DEFLATE bug?
[ https://issues.apache.org/jira/browse/LUCENE-9500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17189937#comment-17189937 ] Uwe Schindler commented on LUCENE-9500: --- Status info: Java 8 is fine, the bug was introduced later: http://hg.openjdk.java.net/jdk8u/jdk8u/jdk/file/45506343cb65/src/share/native/java/util/zip/Deflater.c#l116 I am investigating when this happened: According to the changelog: The addition of ByteBuffer methods for setDictionary was not the bad guy, it was some refactoring done later to reduce number of JNI calls. > Did we hit a DEFLATE bug? > - > > Key: LUCENE-9500 > URL: https://issues.apache.org/jira/browse/LUCENE-9500 > Project: Lucene - Core > Issue Type: Bug >Reporter: Adrien Grand >Priority: Minor > Attachments: PresetDictTest.java, test_data.txt > > > I've been digging > [https://ci-builds.apache.org/job/Lucene/job/Lucene-Solr-NightlyTests-master/23/] > all day and managed to isolate a simple reproduction that shows the problem. > I've been starring at it all day and can't find what we are doing wrong, > which makes me wonder whether we're calling DEFLATE the wrong way or whether > we hit a DEFLATE bug. I've looked at it so much that I may be missing the > most obvious stuff. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9500) Did we hit a DEFLATE bug?
[ https://issues.apache.org/jira/browse/LUCENE-9500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17189920#comment-17189920 ] Uwe Schindler commented on LUCENE-9500: --- Thanks Adrien, I attach it to the issue I am creating at the moment! > Did we hit a DEFLATE bug? > - > > Key: LUCENE-9500 > URL: https://issues.apache.org/jira/browse/LUCENE-9500 > Project: Lucene - Core > Issue Type: Bug >Reporter: Adrien Grand >Priority: Minor > Attachments: PresetDictTest.java, test_data.txt > > > I've been digging > [https://ci-builds.apache.org/job/Lucene/job/Lucene-Solr-NightlyTests-master/23/] > all day and managed to isolate a simple reproduction that shows the problem. > I've been starring at it all day and can't find what we are doing wrong, > which makes me wonder whether we're calling DEFLATE the wrong way or whether > we hit a DEFLATE bug. I've looked at it so much that I may be missing the > most obvious stuff. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9500) Did we hit a DEFLATE bug?
[ https://issues.apache.org/jira/browse/LUCENE-9500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17189847#comment-17189847 ] Dawid Weiss commented on LUCENE-9500: - bq. parameter off seems unused Oh, great catch! bq. will report this one tomorrow morning after waking up. UweSays(tm): the only person who replies to e-mails in his/ her sleep. :) > Did we hit a DEFLATE bug? > - > > Key: LUCENE-9500 > URL: https://issues.apache.org/jira/browse/LUCENE-9500 > Project: Lucene - Core > Issue Type: Bug >Reporter: Adrien Grand >Priority: Minor > Attachments: PresetDictTest.java, test_data.txt > > > I've been digging > [https://ci-builds.apache.org/job/Lucene/job/Lucene-Solr-NightlyTests-master/23/] > all day and managed to isolate a simple reproduction that shows the problem. > I've been starring at it all day and can't find what we are doing wrong, > which makes me wonder whether we're calling DEFLATE the wrong way or whether > we hit a DEFLATE bug. I've looked at it so much that I may be missing the > most obvious stuff. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9500) Did we hit a DEFLATE bug?
[ https://issues.apache.org/jira/browse/LUCENE-9500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17189741#comment-17189741 ] Uwe Schindler commented on LUCENE-9500: --- Hi, will report this one tomorrow morning after waking up. I found another workaround, which - unfortunately - does not work. The alternative is to use a ByteBuffer, where the code in native c code looks fine. But unfortunately it won't work with heap-based ByteBuffer: If the buffer has an array / is not direct, it calls the same method that ignores heap. So I'd suggest to arraycopy the code to beginning of byte[] as workaround. It may take years until this is fixed, especially not in Java 8, not Java 11! [~jpountz]: The only workaround is: Copy the dictionary to a fresh byte array to offset 0 and use the methods without offset. > Did we hit a DEFLATE bug? > - > > Key: LUCENE-9500 > URL: https://issues.apache.org/jira/browse/LUCENE-9500 > Project: Lucene - Core > Issue Type: Bug >Reporter: Adrien Grand >Priority: Minor > Attachments: PresetDictTest.java, test_data.txt > > > I've been digging > [https://ci-builds.apache.org/job/Lucene/job/Lucene-Solr-NightlyTests-master/23/] > all day and managed to isolate a simple reproduction that shows the problem. > I've been starring at it all day and can't find what we are doing wrong, > which makes me wonder whether we're calling DEFLATE the wrong way or whether > we hit a DEFLATE bug. I've looked at it so much that I may be missing the > most obvious stuff. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9500) Did we hit a DEFLATE bug?
[ https://issues.apache.org/jira/browse/LUCENE-9500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17189738#comment-17189738 ] Robert Muir commented on LUCENE-9500: - I think as a workaround we should arraycopy preset dictionary to a byte[] so that we'll have a 0-based offset. We'll need to keep it around for quite some time to avoid corruption if someone has an impacted JDK. It seems to only impact compression too, not decompression. Inflater.setDictionary actually uses its "off" parameter and we already always use a 0-based offset for inflation, so we don't need any workaround at query time, just when indexing. > Did we hit a DEFLATE bug? > - > > Key: LUCENE-9500 > URL: https://issues.apache.org/jira/browse/LUCENE-9500 > Project: Lucene - Core > Issue Type: Bug >Reporter: Adrien Grand >Priority: Minor > Attachments: PresetDictTest.java, test_data.txt > > > I've been digging > [https://ci-builds.apache.org/job/Lucene/job/Lucene-Solr-NightlyTests-master/23/] > all day and managed to isolate a simple reproduction that shows the problem. > I've been starring at it all day and can't find what we are doing wrong, > which makes me wonder whether we're calling DEFLATE the wrong way or whether > we hit a DEFLATE bug. I've looked at it so much that I may be missing the > most obvious stuff. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9500) Did we hit a DEFLATE bug?
[ https://issues.apache.org/jira/browse/LUCENE-9500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17189726#comment-17189726 ] Robert Muir commented on LUCENE-9500: - [~uschindler] this is a good one to report to the openjdk, because you can see the bug just looking at the code. > Did we hit a DEFLATE bug? > - > > Key: LUCENE-9500 > URL: https://issues.apache.org/jira/browse/LUCENE-9500 > Project: Lucene - Core > Issue Type: Bug >Reporter: Adrien Grand >Priority: Minor > Attachments: PresetDictTest.java, test_data.txt > > > I've been digging > [https://ci-builds.apache.org/job/Lucene/job/Lucene-Solr-NightlyTests-master/23/] > all day and managed to isolate a simple reproduction that shows the problem. > I've been starring at it all day and can't find what we are doing wrong, > which makes me wonder whether we're calling DEFLATE the wrong way or whether > we hit a DEFLATE bug. I've looked at it so much that I may be missing the > most obvious stuff. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9500) Did we hit a DEFLATE bug?
[
https://issues.apache.org/jira/browse/LUCENE-9500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17189723#comment-17189723
]
Robert Muir commented on LUCENE-9500:
-
https://github.com/openjdk/jdk/blob/1643bc3defa241aef2cad53d0f11076366c3620d/src/java.base/share/native/libzip/Deflater.c#L100-L111
parameter {{off}} seems unused :)
> Did we hit a DEFLATE bug?
> -
>
> Key: LUCENE-9500
> URL: https://issues.apache.org/jira/browse/LUCENE-9500
> Project: Lucene - Core
> Issue Type: Bug
>Reporter: Adrien Grand
>Priority: Minor
> Attachments: PresetDictTest.java, test_data.txt
>
>
> I've been digging
> [https://ci-builds.apache.org/job/Lucene/job/Lucene-Solr-NightlyTests-master/23/]
> all day and managed to isolate a simple reproduction that shows the problem.
> I've been starring at it all day and can't find what we are doing wrong,
> which makes me wonder whether we're calling DEFLATE the wrong way or whether
> we hit a DEFLATE bug. I've looked at it so much that I may be missing the
> most obvious stuff.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9500) Did we hit a DEFLATE bug?
[ https://issues.apache.org/jira/browse/LUCENE-9500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17189356#comment-17189356 ] Dawid Weiss commented on LUCENE-9500: - No idea. Send a repro to core-libs-...@openjdk.java.net? > Did we hit a DEFLATE bug? > - > > Key: LUCENE-9500 > URL: https://issues.apache.org/jira/browse/LUCENE-9500 > Project: Lucene - Core > Issue Type: Bug >Reporter: Adrien Grand >Priority: Minor > Attachments: PresetDictTest.java, test_data.txt > > > I've been digging > [https://ci-builds.apache.org/job/Lucene/job/Lucene-Solr-NightlyTests-master/23/] > all day and managed to isolate a simple reproduction that shows the problem. > I've been starring at it all day and can't find what we are doing wrong, > which makes me wonder whether we're calling DEFLATE the wrong way or whether > we hit a DEFLATE bug. I've looked at it so much that I may be missing the > most obvious stuff. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9500) Did we hit a DEFLATE bug?
[
https://issues.apache.org/jira/browse/LUCENE-9500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17189332#comment-17189332
]
Adrien Grand commented on LUCENE-9500:
--
I just attached the test and test data I used that recreates the issue. I get
the following output:
{noformat}
Compressed to 115 bytes
First mismatch byte 0
First original bytes [c0 9a e2 81 24 d5 fc 9 9 49 f c0 9a e2 81 24 d5 fc 9 9 49
f c0 9a e2 81 24 d5 fc 9]
First restored bytes [9 9 49 f c0 9a e2 81 24 d5 fc 9 9 49 f c0 9a e2 81 24 d5
fc 9 9 49 f c0 9a e2 81]
Exception in thread "main" java.lang.Error
at PresetDictTest.main(PresetDictTest.java:87)
{noformat}
If I remove the call to setDictionary on the compression side, then no error.
> Did we hit a DEFLATE bug?
> -
>
> Key: LUCENE-9500
> URL: https://issues.apache.org/jira/browse/LUCENE-9500
> Project: Lucene - Core
> Issue Type: Bug
>Reporter: Adrien Grand
>Priority: Minor
> Attachments: PresetDictTest.java, test_data.txt
>
>
> I've been digging
> [https://ci-builds.apache.org/job/Lucene/job/Lucene-Solr-NightlyTests-master/23/]
> all day and managed to isolate a simple reproduction that shows the problem.
> I've been starring at it all day and can't find what we are doing wrong,
> which makes me wonder whether we're calling DEFLATE the wrong way or whether
> we hit a DEFLATE bug. I've looked at it so much that I may be missing the
> most obvious stuff.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
