[jira] [Commented] (LUCENE-7440) Document skipping on large indexes is broken
[
https://issues.apache.org/jira/browse/LUCENE-7440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15857817#comment-15857817
]
Adrien Grand commented on LUCENE-7440:
--
For the record, I verified that Test2BPostings passes on the 5.5 branch.
> Document skipping on large indexes is broken
>
>
> Key: LUCENE-7440
> URL: https://issues.apache.org/jira/browse/LUCENE-7440
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/search
>Affects Versions: 2.2
>Reporter: Yonik Seeley
>Assignee: Yonik Seeley
>Priority: Critical
> Fix For: master (7.0), 6.3, 5.5.4, 6.2.1
>
> Attachments: LUCENE-7440.patch, LUCENE-7440.patch
>
>
> Large skips on large indexes fail.
> Anything that uses skips (such as a boolean query, filtered queries, faceted
> queries, join queries, etc) can trigger this bug on a sufficiently large
> index.
> The bug is a numeric overflow in MultiLevelSkipList that has been present
> since inception (Lucene 2.2). It may not manifest until one has a single
> segment with more than ~1.8B documents, and a large skip is performed on that
> segment.
> Typical stack trace on Lucene7-dev:
> {code}
> java.lang.ArrayIndexOutOfBoundsException: 110
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader$SkipBuffer.readByte(MultiLevelSkipListReader.java:297)
> at org.apache.lucene.store.DataInput.readVInt(DataInput.java:125)
> at
> org.apache.lucene.codecs.lucene50.Lucene50SkipReader.readSkipData(Lucene50SkipReader.java:180)
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader.loadNextSkip(MultiLevelSkipListReader.java:163)
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader.skipTo(MultiLevelSkipListReader.java:133)
> at
> org.apache.lucene.codecs.lucene50.Lucene50PostingsReader$BlockDocsEnum.advance(Lucene50PostingsReader.java:421)
> at YCS_skip7$1.testSkip(YCS_skip7.java:307)
> {code}
> Typical stack trace on Lucene4.10.3:
> {code}
> 6-08-31 18:57:17,460 ERROR org.apache.solr.servlet.SolrDispatchFilter:
> null:java.lang.ArrayIndexOutOfBoundsException: 75
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader$SkipBuffer.readByte(MultiLevelSkipListReader.java:301)
> at org.apache.lucene.store.DataInput.readVInt(DataInput.java:122)
> at
> org.apache.lucene.codecs.lucene41.Lucene41SkipReader.readSkipData(Lucene41SkipReader.java:194)
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader.loadNextSkip(MultiLevelSkipListReader.java:168)
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader.skipTo(MultiLevelSkipListReader.java:138)
> at
> org.apache.lucene.codecs.lucene41.Lucene41PostingsReader$BlockDocsEnum.advance(Lucene41PostingsReader.java:506)
> at org.apache.lucene.search.TermScorer.advance(TermScorer.java:85)
> [...]
> at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621)
> [...]
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2004)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-7440) Document skipping on large indexes is broken
[
https://issues.apache.org/jira/browse/LUCENE-7440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15857788#comment-15857788
]
ASF subversion and git services commented on LUCENE-7440:
-
Commit 1c219d2da692d8b41c02d8a52ffacd813cd44a7b in lucene-solr's branch
refs/heads/branch_5_5 from [~yo...@apache.org]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=1c219d2 ]
LUCENE-7440: fix MultiLevelSkipListReader overflow
> Document skipping on large indexes is broken
>
>
> Key: LUCENE-7440
> URL: https://issues.apache.org/jira/browse/LUCENE-7440
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/search
>Affects Versions: 2.2
>Reporter: Yonik Seeley
>Assignee: Yonik Seeley
>Priority: Critical
> Fix For: master (7.0), 6.3, 6.2.1
>
> Attachments: LUCENE-7440.patch, LUCENE-7440.patch
>
>
> Large skips on large indexes fail.
> Anything that uses skips (such as a boolean query, filtered queries, faceted
> queries, join queries, etc) can trigger this bug on a sufficiently large
> index.
> The bug is a numeric overflow in MultiLevelSkipList that has been present
> since inception (Lucene 2.2). It may not manifest until one has a single
> segment with more than ~1.8B documents, and a large skip is performed on that
> segment.
> Typical stack trace on Lucene7-dev:
> {code}
> java.lang.ArrayIndexOutOfBoundsException: 110
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader$SkipBuffer.readByte(MultiLevelSkipListReader.java:297)
> at org.apache.lucene.store.DataInput.readVInt(DataInput.java:125)
> at
> org.apache.lucene.codecs.lucene50.Lucene50SkipReader.readSkipData(Lucene50SkipReader.java:180)
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader.loadNextSkip(MultiLevelSkipListReader.java:163)
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader.skipTo(MultiLevelSkipListReader.java:133)
> at
> org.apache.lucene.codecs.lucene50.Lucene50PostingsReader$BlockDocsEnum.advance(Lucene50PostingsReader.java:421)
> at YCS_skip7$1.testSkip(YCS_skip7.java:307)
> {code}
> Typical stack trace on Lucene4.10.3:
> {code}
> 6-08-31 18:57:17,460 ERROR org.apache.solr.servlet.SolrDispatchFilter:
> null:java.lang.ArrayIndexOutOfBoundsException: 75
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader$SkipBuffer.readByte(MultiLevelSkipListReader.java:301)
> at org.apache.lucene.store.DataInput.readVInt(DataInput.java:122)
> at
> org.apache.lucene.codecs.lucene41.Lucene41SkipReader.readSkipData(Lucene41SkipReader.java:194)
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader.loadNextSkip(MultiLevelSkipListReader.java:168)
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader.skipTo(MultiLevelSkipListReader.java:138)
> at
> org.apache.lucene.codecs.lucene41.Lucene41PostingsReader$BlockDocsEnum.advance(Lucene41PostingsReader.java:506)
> at org.apache.lucene.search.TermScorer.advance(TermScorer.java:85)
> [...]
> at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621)
> [...]
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2004)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-7440) Document skipping on large indexes is broken
[
https://issues.apache.org/jira/browse/LUCENE-7440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15484994#comment-15484994
]
Yonik Seeley commented on LUCENE-7440:
--
bq. Would this be faster to test if we configure a larger top-level skip
distance?
The top-level skip distance sort of falls out from other factors, rather than
being explicitly configured.
For a quicker, more thorough test, It would probably be good to somehow test
the skip list logic itself w/o having it backed by an actual index. Even with
that, I think it's a good idea to also test real indexes.
> Document skipping on large indexes is broken
>
>
> Key: LUCENE-7440
> URL: https://issues.apache.org/jira/browse/LUCENE-7440
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/search
>Affects Versions: 2.2
>Reporter: Yonik Seeley
>Assignee: Yonik Seeley
>Priority: Critical
> Fix For: master (7.0), 6.3, 6.2.1
>
> Attachments: LUCENE-7440.patch, LUCENE-7440.patch
>
>
> Large skips on large indexes fail.
> Anything that uses skips (such as a boolean query, filtered queries, faceted
> queries, join queries, etc) can trigger this bug on a sufficiently large
> index.
> The bug is a numeric overflow in MultiLevelSkipList that has been present
> since inception (Lucene 2.2). It may not manifest until one has a single
> segment with more than ~1.8B documents, and a large skip is performed on that
> segment.
> Typical stack trace on Lucene7-dev:
> {code}
> java.lang.ArrayIndexOutOfBoundsException: 110
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader$SkipBuffer.readByte(MultiLevelSkipListReader.java:297)
> at org.apache.lucene.store.DataInput.readVInt(DataInput.java:125)
> at
> org.apache.lucene.codecs.lucene50.Lucene50SkipReader.readSkipData(Lucene50SkipReader.java:180)
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader.loadNextSkip(MultiLevelSkipListReader.java:163)
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader.skipTo(MultiLevelSkipListReader.java:133)
> at
> org.apache.lucene.codecs.lucene50.Lucene50PostingsReader$BlockDocsEnum.advance(Lucene50PostingsReader.java:421)
> at YCS_skip7$1.testSkip(YCS_skip7.java:307)
> {code}
> Typical stack trace on Lucene4.10.3:
> {code}
> 6-08-31 18:57:17,460 ERROR org.apache.solr.servlet.SolrDispatchFilter:
> null:java.lang.ArrayIndexOutOfBoundsException: 75
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader$SkipBuffer.readByte(MultiLevelSkipListReader.java:301)
> at org.apache.lucene.store.DataInput.readVInt(DataInput.java:122)
> at
> org.apache.lucene.codecs.lucene41.Lucene41SkipReader.readSkipData(Lucene41SkipReader.java:194)
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader.loadNextSkip(MultiLevelSkipListReader.java:168)
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader.skipTo(MultiLevelSkipListReader.java:138)
> at
> org.apache.lucene.codecs.lucene41.Lucene41PostingsReader$BlockDocsEnum.advance(Lucene41PostingsReader.java:506)
> at org.apache.lucene.search.TermScorer.advance(TermScorer.java:85)
> [...]
> at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621)
> [...]
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2004)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-7440) Document skipping on large indexes is broken
[
https://issues.apache.org/jira/browse/LUCENE-7440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15484634#comment-15484634
]
Mike Drob commented on LUCENE-7440:
---
bq. Regarding the 1.8B docs number... at least in my tests I saw the top-level
skip distance of ~268M w/ the default codec. Subtracting this from MAX_INT
gives around 1.8B, which is around the number I saw prior to the overflow. To
hit the bug, one also needs to be doing large skips toward the end of the index
as well, in order to use the top level(s) of the multi-level skip list. Having
a conjunction query of a highly unique term (or clause) in conjunction with a
common term has a good chance of triggering (example: +timestamp:39520928456494
+doctype:common)
Would this be faster to test if we configure a larger top-level skip distance?
i.e. set up a skip distance of ~1B and then we'd only need to get to ~1.1B docs
indexed (40% fewer docs, theoretically 40% faster?) or even set up a skip
distance of ~2B to only need to index very few documents?
Maybe this idea should be split into a separate issue to focus on improving the
test?
> Document skipping on large indexes is broken
>
>
> Key: LUCENE-7440
> URL: https://issues.apache.org/jira/browse/LUCENE-7440
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/search
>Affects Versions: 2.2
>Reporter: Yonik Seeley
>Assignee: Yonik Seeley
>Priority: Critical
> Fix For: master (7.0), 6.3, 6.2.1
>
> Attachments: LUCENE-7440.patch, LUCENE-7440.patch
>
>
> Large skips on large indexes fail.
> Anything that uses skips (such as a boolean query, filtered queries, faceted
> queries, join queries, etc) can trigger this bug on a sufficiently large
> index.
> The bug is a numeric overflow in MultiLevelSkipList that has been present
> since inception (Lucene 2.2). It may not manifest until one has a single
> segment with more than ~1.8B documents, and a large skip is performed on that
> segment.
> Typical stack trace on Lucene7-dev:
> {code}
> java.lang.ArrayIndexOutOfBoundsException: 110
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader$SkipBuffer.readByte(MultiLevelSkipListReader.java:297)
> at org.apache.lucene.store.DataInput.readVInt(DataInput.java:125)
> at
> org.apache.lucene.codecs.lucene50.Lucene50SkipReader.readSkipData(Lucene50SkipReader.java:180)
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader.loadNextSkip(MultiLevelSkipListReader.java:163)
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader.skipTo(MultiLevelSkipListReader.java:133)
> at
> org.apache.lucene.codecs.lucene50.Lucene50PostingsReader$BlockDocsEnum.advance(Lucene50PostingsReader.java:421)
> at YCS_skip7$1.testSkip(YCS_skip7.java:307)
> {code}
> Typical stack trace on Lucene4.10.3:
> {code}
> 6-08-31 18:57:17,460 ERROR org.apache.solr.servlet.SolrDispatchFilter:
> null:java.lang.ArrayIndexOutOfBoundsException: 75
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader$SkipBuffer.readByte(MultiLevelSkipListReader.java:301)
> at org.apache.lucene.store.DataInput.readVInt(DataInput.java:122)
> at
> org.apache.lucene.codecs.lucene41.Lucene41SkipReader.readSkipData(Lucene41SkipReader.java:194)
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader.loadNextSkip(MultiLevelSkipListReader.java:168)
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader.skipTo(MultiLevelSkipListReader.java:138)
> at
> org.apache.lucene.codecs.lucene41.Lucene41PostingsReader$BlockDocsEnum.advance(Lucene41PostingsReader.java:506)
> at org.apache.lucene.search.TermScorer.advance(TermScorer.java:85)
> [...]
> at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621)
> [...]
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2004)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-7440) Document skipping on large indexes is broken
[
https://issues.apache.org/jira/browse/LUCENE-7440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15483925#comment-15483925
]
ASF subversion and git services commented on LUCENE-7440:
-
Commit c7b3e9ae3695a13dacb81312db0d470ada273808 in lucene-solr's branch
refs/heads/branch_6_2 from [~yo...@apache.org]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=c7b3e9a ]
LUCENE-7440: fix MultiLevelSkipListReader overflow
(cherry picked from commit cf72eeb)
> Document skipping on large indexes is broken
>
>
> Key: LUCENE-7440
> URL: https://issues.apache.org/jira/browse/LUCENE-7440
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/search
>Affects Versions: 2.2
>Reporter: Yonik Seeley
>Priority: Critical
> Fix For: master (7.0), 6.3, 6.2.1
>
> Attachments: LUCENE-7440.patch, LUCENE-7440.patch
>
>
> Large skips on large indexes fail.
> Anything that uses skips (such as a boolean query, filtered queries, faceted
> queries, join queries, etc) can trigger this bug on a sufficiently large
> index.
> The bug is a numeric overflow in MultiLevelSkipList that has been present
> since inception (Lucene 2.2). It may not manifest until one has a single
> segment with more than ~1.8B documents, and a large skip is performed on that
> segment.
> Typical stack trace on Lucene7-dev:
> {code}
> java.lang.ArrayIndexOutOfBoundsException: 110
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader$SkipBuffer.readByte(MultiLevelSkipListReader.java:297)
> at org.apache.lucene.store.DataInput.readVInt(DataInput.java:125)
> at
> org.apache.lucene.codecs.lucene50.Lucene50SkipReader.readSkipData(Lucene50SkipReader.java:180)
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader.loadNextSkip(MultiLevelSkipListReader.java:163)
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader.skipTo(MultiLevelSkipListReader.java:133)
> at
> org.apache.lucene.codecs.lucene50.Lucene50PostingsReader$BlockDocsEnum.advance(Lucene50PostingsReader.java:421)
> at YCS_skip7$1.testSkip(YCS_skip7.java:307)
> {code}
> Typical stack trace on Lucene4.10.3:
> {code}
> 6-08-31 18:57:17,460 ERROR org.apache.solr.servlet.SolrDispatchFilter:
> null:java.lang.ArrayIndexOutOfBoundsException: 75
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader$SkipBuffer.readByte(MultiLevelSkipListReader.java:301)
> at org.apache.lucene.store.DataInput.readVInt(DataInput.java:122)
> at
> org.apache.lucene.codecs.lucene41.Lucene41SkipReader.readSkipData(Lucene41SkipReader.java:194)
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader.loadNextSkip(MultiLevelSkipListReader.java:168)
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader.skipTo(MultiLevelSkipListReader.java:138)
> at
> org.apache.lucene.codecs.lucene41.Lucene41PostingsReader$BlockDocsEnum.advance(Lucene41PostingsReader.java:506)
> at org.apache.lucene.search.TermScorer.advance(TermScorer.java:85)
> [...]
> at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621)
> [...]
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2004)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-7440) Document skipping on large indexes is broken
[
https://issues.apache.org/jira/browse/LUCENE-7440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15480369#comment-15480369
]
ASF subversion and git services commented on LUCENE-7440:
-
Commit cf72eebf75bb3c5c3bdd8ef2ee288e67f89b5538 in lucene-solr's branch
refs/heads/branch_6x from [~yo...@apache.org]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=cf72eeb ]
LUCENE-7440: fix MultiLevelSkipListReader overflow
> Document skipping on large indexes is broken
>
>
> Key: LUCENE-7440
> URL: https://issues.apache.org/jira/browse/LUCENE-7440
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/search
>Affects Versions: 2.2
>Reporter: Yonik Seeley
>Priority: Critical
> Attachments: LUCENE-7440.patch, LUCENE-7440.patch
>
>
> Large skips on large indexes fail.
> Anything that uses skips (such as a boolean query, filtered queries, faceted
> queries, join queries, etc) can trigger this bug on a sufficiently large
> index.
> The bug is a numeric overflow in MultiLevelSkipList that has been present
> since inception (Lucene 2.2). It may not manifest until one has a single
> segment with more than ~1.8B documents, and a large skip is performed on that
> segment.
> Typical stack trace on Lucene7-dev:
> {code}
> java.lang.ArrayIndexOutOfBoundsException: 110
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader$SkipBuffer.readByte(MultiLevelSkipListReader.java:297)
> at org.apache.lucene.store.DataInput.readVInt(DataInput.java:125)
> at
> org.apache.lucene.codecs.lucene50.Lucene50SkipReader.readSkipData(Lucene50SkipReader.java:180)
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader.loadNextSkip(MultiLevelSkipListReader.java:163)
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader.skipTo(MultiLevelSkipListReader.java:133)
> at
> org.apache.lucene.codecs.lucene50.Lucene50PostingsReader$BlockDocsEnum.advance(Lucene50PostingsReader.java:421)
> at YCS_skip7$1.testSkip(YCS_skip7.java:307)
> {code}
> Typical stack trace on Lucene4.10.3:
> {code}
> 6-08-31 18:57:17,460 ERROR org.apache.solr.servlet.SolrDispatchFilter:
> null:java.lang.ArrayIndexOutOfBoundsException: 75
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader$SkipBuffer.readByte(MultiLevelSkipListReader.java:301)
> at org.apache.lucene.store.DataInput.readVInt(DataInput.java:122)
> at
> org.apache.lucene.codecs.lucene41.Lucene41SkipReader.readSkipData(Lucene41SkipReader.java:194)
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader.loadNextSkip(MultiLevelSkipListReader.java:168)
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader.skipTo(MultiLevelSkipListReader.java:138)
> at
> org.apache.lucene.codecs.lucene41.Lucene41PostingsReader$BlockDocsEnum.advance(Lucene41PostingsReader.java:506)
> at org.apache.lucene.search.TermScorer.advance(TermScorer.java:85)
> [...]
> at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621)
> [...]
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2004)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-7440) Document skipping on large indexes is broken
[
https://issues.apache.org/jira/browse/LUCENE-7440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15480355#comment-15480355
]
ASF subversion and git services commented on LUCENE-7440:
-
Commit c929d0595c0ad2ef311054746dc24aa8704f55e6 in lucene-solr's branch
refs/heads/master from [~yo...@apache.org]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=c929d05 ]
LUCENE-7440: fix MultiLevelSkipListReader overflow
> Document skipping on large indexes is broken
>
>
> Key: LUCENE-7440
> URL: https://issues.apache.org/jira/browse/LUCENE-7440
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/search
>Affects Versions: 2.2
>Reporter: Yonik Seeley
>Priority: Critical
> Attachments: LUCENE-7440.patch, LUCENE-7440.patch
>
>
> Large skips on large indexes fail.
> Anything that uses skips (such as a boolean query, filtered queries, faceted
> queries, join queries, etc) can trigger this bug on a sufficiently large
> index.
> The bug is a numeric overflow in MultiLevelSkipList that has been present
> since inception (Lucene 2.2). It may not manifest until one has a single
> segment with more than ~1.8B documents, and a large skip is performed on that
> segment.
> Typical stack trace on Lucene7-dev:
> {code}
> java.lang.ArrayIndexOutOfBoundsException: 110
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader$SkipBuffer.readByte(MultiLevelSkipListReader.java:297)
> at org.apache.lucene.store.DataInput.readVInt(DataInput.java:125)
> at
> org.apache.lucene.codecs.lucene50.Lucene50SkipReader.readSkipData(Lucene50SkipReader.java:180)
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader.loadNextSkip(MultiLevelSkipListReader.java:163)
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader.skipTo(MultiLevelSkipListReader.java:133)
> at
> org.apache.lucene.codecs.lucene50.Lucene50PostingsReader$BlockDocsEnum.advance(Lucene50PostingsReader.java:421)
> at YCS_skip7$1.testSkip(YCS_skip7.java:307)
> {code}
> Typical stack trace on Lucene4.10.3:
> {code}
> 6-08-31 18:57:17,460 ERROR org.apache.solr.servlet.SolrDispatchFilter:
> null:java.lang.ArrayIndexOutOfBoundsException: 75
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader$SkipBuffer.readByte(MultiLevelSkipListReader.java:301)
> at org.apache.lucene.store.DataInput.readVInt(DataInput.java:122)
> at
> org.apache.lucene.codecs.lucene41.Lucene41SkipReader.readSkipData(Lucene41SkipReader.java:194)
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader.loadNextSkip(MultiLevelSkipListReader.java:168)
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader.skipTo(MultiLevelSkipListReader.java:138)
> at
> org.apache.lucene.codecs.lucene41.Lucene41PostingsReader$BlockDocsEnum.advance(Lucene41PostingsReader.java:506)
> at org.apache.lucene.search.TermScorer.advance(TermScorer.java:85)
> [...]
> at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621)
> [...]
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2004)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-7440) Document skipping on large indexes is broken
[
https://issues.apache.org/jira/browse/LUCENE-7440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15480081#comment-15480081
]
Uwe Schindler commented on LUCENE-7440:
---
+1
> Document skipping on large indexes is broken
>
>
> Key: LUCENE-7440
> URL: https://issues.apache.org/jira/browse/LUCENE-7440
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/search
>Affects Versions: 2.2
>Reporter: Yonik Seeley
>Priority: Critical
> Attachments: LUCENE-7440.patch, LUCENE-7440.patch
>
>
> Large skips on large indexes fail.
> Anything that uses skips (such as a boolean query, filtered queries, faceted
> queries, join queries, etc) can trigger this bug on a sufficiently large
> index.
> The bug is a numeric overflow in MultiLevelSkipList that has been present
> since inception (Lucene 2.2). It may not manifest until one has a single
> segment with more than ~1.8B documents, and a large skip is performed on that
> segment.
> Typical stack trace on Lucene7-dev:
> {code}
> java.lang.ArrayIndexOutOfBoundsException: 110
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader$SkipBuffer.readByte(MultiLevelSkipListReader.java:297)
> at org.apache.lucene.store.DataInput.readVInt(DataInput.java:125)
> at
> org.apache.lucene.codecs.lucene50.Lucene50SkipReader.readSkipData(Lucene50SkipReader.java:180)
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader.loadNextSkip(MultiLevelSkipListReader.java:163)
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader.skipTo(MultiLevelSkipListReader.java:133)
> at
> org.apache.lucene.codecs.lucene50.Lucene50PostingsReader$BlockDocsEnum.advance(Lucene50PostingsReader.java:421)
> at YCS_skip7$1.testSkip(YCS_skip7.java:307)
> {code}
> Typical stack trace on Lucene4.10.3:
> {code}
> 6-08-31 18:57:17,460 ERROR org.apache.solr.servlet.SolrDispatchFilter:
> null:java.lang.ArrayIndexOutOfBoundsException: 75
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader$SkipBuffer.readByte(MultiLevelSkipListReader.java:301)
> at org.apache.lucene.store.DataInput.readVInt(DataInput.java:122)
> at
> org.apache.lucene.codecs.lucene41.Lucene41SkipReader.readSkipData(Lucene41SkipReader.java:194)
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader.loadNextSkip(MultiLevelSkipListReader.java:168)
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader.skipTo(MultiLevelSkipListReader.java:138)
> at
> org.apache.lucene.codecs.lucene41.Lucene41PostingsReader$BlockDocsEnum.advance(Lucene41PostingsReader.java:506)
> at org.apache.lucene.search.TermScorer.advance(TermScorer.java:85)
> [...]
> at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621)
> [...]
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2004)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-7440) Document skipping on large indexes is broken
[
https://issues.apache.org/jira/browse/LUCENE-7440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15478450#comment-15478450
]
Uwe Schindler commented on LUCENE-7440:
---
+1 compareUnsigned is the way to go! It's an intrinsic, too.
> Document skipping on large indexes is broken
>
>
> Key: LUCENE-7440
> URL: https://issues.apache.org/jira/browse/LUCENE-7440
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/search
>Affects Versions: 2.2
>Reporter: Yonik Seeley
>Priority: Critical
> Attachments: LUCENE-7440.patch
>
>
> Large skips on large indexes fail.
> Anything that uses skips (such as a boolean query, filtered queries, faceted
> queries, join queries, etc) can trigger this bug on a sufficiently large
> index.
> The bug is a numeric overflow in MultiLevelSkipList that has been present
> since inception (Lucene 2.2). It may not manifest until one has a single
> segment with more than ~1.8B documents, and a large skip is performed on that
> segment.
> Typical stack trace on Lucene7-dev:
> {code}
> java.lang.ArrayIndexOutOfBoundsException: 110
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader$SkipBuffer.readByte(MultiLevelSkipListReader.java:297)
> at org.apache.lucene.store.DataInput.readVInt(DataInput.java:125)
> at
> org.apache.lucene.codecs.lucene50.Lucene50SkipReader.readSkipData(Lucene50SkipReader.java:180)
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader.loadNextSkip(MultiLevelSkipListReader.java:163)
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader.skipTo(MultiLevelSkipListReader.java:133)
> at
> org.apache.lucene.codecs.lucene50.Lucene50PostingsReader$BlockDocsEnum.advance(Lucene50PostingsReader.java:421)
> at YCS_skip7$1.testSkip(YCS_skip7.java:307)
> {code}
> Typical stack trace on Lucene4.10.3:
> {code}
> 6-08-31 18:57:17,460 ERROR org.apache.solr.servlet.SolrDispatchFilter:
> null:java.lang.ArrayIndexOutOfBoundsException: 75
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader$SkipBuffer.readByte(MultiLevelSkipListReader.java:301)
> at org.apache.lucene.store.DataInput.readVInt(DataInput.java:122)
> at
> org.apache.lucene.codecs.lucene41.Lucene41SkipReader.readSkipData(Lucene41SkipReader.java:194)
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader.loadNextSkip(MultiLevelSkipListReader.java:168)
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader.skipTo(MultiLevelSkipListReader.java:138)
> at
> org.apache.lucene.codecs.lucene41.Lucene41PostingsReader$BlockDocsEnum.advance(Lucene41PostingsReader.java:506)
> at org.apache.lucene.search.TermScorer.advance(TermScorer.java:85)
> [...]
> at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621)
> [...]
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2004)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-7440) Document skipping on large indexes is broken
[
https://issues.apache.org/jira/browse/LUCENE-7440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15477697#comment-15477697
]
Michael McCandless commented on LUCENE-7440:
bq. Another alternative would be to use Integer.compareUnsigned()
+1
These values should never overflow 2^32
> Document skipping on large indexes is broken
>
>
> Key: LUCENE-7440
> URL: https://issues.apache.org/jira/browse/LUCENE-7440
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/search
>Affects Versions: 2.2
>Reporter: Yonik Seeley
>Priority: Critical
> Attachments: LUCENE-7440.patch
>
>
> Large skips on large indexes fail.
> Anything that uses skips (such as a boolean query, filtered queries, faceted
> queries, join queries, etc) can trigger this bug on a sufficiently large
> index.
> The bug is a numeric overflow in MultiLevelSkipList that has been present
> since inception (Lucene 2.2). It may not manifest until one has a single
> segment with more than ~1.8B documents, and a large skip is performed on that
> segment.
> Typical stack trace on Lucene7-dev:
> {code}
> java.lang.ArrayIndexOutOfBoundsException: 110
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader$SkipBuffer.readByte(MultiLevelSkipListReader.java:297)
> at org.apache.lucene.store.DataInput.readVInt(DataInput.java:125)
> at
> org.apache.lucene.codecs.lucene50.Lucene50SkipReader.readSkipData(Lucene50SkipReader.java:180)
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader.loadNextSkip(MultiLevelSkipListReader.java:163)
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader.skipTo(MultiLevelSkipListReader.java:133)
> at
> org.apache.lucene.codecs.lucene50.Lucene50PostingsReader$BlockDocsEnum.advance(Lucene50PostingsReader.java:421)
> at YCS_skip7$1.testSkip(YCS_skip7.java:307)
> {code}
> Typical stack trace on Lucene4.10.3:
> {code}
> 6-08-31 18:57:17,460 ERROR org.apache.solr.servlet.SolrDispatchFilter:
> null:java.lang.ArrayIndexOutOfBoundsException: 75
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader$SkipBuffer.readByte(MultiLevelSkipListReader.java:301)
> at org.apache.lucene.store.DataInput.readVInt(DataInput.java:122)
> at
> org.apache.lucene.codecs.lucene41.Lucene41SkipReader.readSkipData(Lucene41SkipReader.java:194)
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader.loadNextSkip(MultiLevelSkipListReader.java:168)
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader.skipTo(MultiLevelSkipListReader.java:138)
> at
> org.apache.lucene.codecs.lucene41.Lucene41PostingsReader$BlockDocsEnum.advance(Lucene41PostingsReader.java:506)
> at org.apache.lucene.search.TermScorer.advance(TermScorer.java:85)
> [...]
> at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621)
> [...]
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2004)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-7440) Document skipping on large indexes is broken
[
https://issues.apache.org/jira/browse/LUCENE-7440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15477628#comment-15477628
]
Robert Muir commented on LUCENE-7440:
-
Another alternative would be to use Integer.compareUnsigned()
> Document skipping on large indexes is broken
>
>
> Key: LUCENE-7440
> URL: https://issues.apache.org/jira/browse/LUCENE-7440
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/search
>Affects Versions: 2.2
>Reporter: Yonik Seeley
>Priority: Critical
> Attachments: LUCENE-7440.patch
>
>
> Large skips on large indexes fail.
> Anything that uses skips (such as a boolean query, filtered queries, faceted
> queries, join queries, etc) can trigger this bug on a sufficiently large
> index.
> The bug is a numeric overflow in MultiLevelSkipList that has been present
> since inception (Lucene 2.2). It may not manifest until one has a single
> segment with more than ~1.8B documents, and a large skip is performed on that
> segment.
> Typical stack trace on Lucene7-dev:
> {code}
> java.lang.ArrayIndexOutOfBoundsException: 110
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader$SkipBuffer.readByte(MultiLevelSkipListReader.java:297)
> at org.apache.lucene.store.DataInput.readVInt(DataInput.java:125)
> at
> org.apache.lucene.codecs.lucene50.Lucene50SkipReader.readSkipData(Lucene50SkipReader.java:180)
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader.loadNextSkip(MultiLevelSkipListReader.java:163)
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader.skipTo(MultiLevelSkipListReader.java:133)
> at
> org.apache.lucene.codecs.lucene50.Lucene50PostingsReader$BlockDocsEnum.advance(Lucene50PostingsReader.java:421)
> at YCS_skip7$1.testSkip(YCS_skip7.java:307)
> {code}
> Typical stack trace on Lucene4.10.3:
> {code}
> 6-08-31 18:57:17,460 ERROR org.apache.solr.servlet.SolrDispatchFilter:
> null:java.lang.ArrayIndexOutOfBoundsException: 75
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader$SkipBuffer.readByte(MultiLevelSkipListReader.java:301)
> at org.apache.lucene.store.DataInput.readVInt(DataInput.java:122)
> at
> org.apache.lucene.codecs.lucene41.Lucene41SkipReader.readSkipData(Lucene41SkipReader.java:194)
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader.loadNextSkip(MultiLevelSkipListReader.java:168)
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader.skipTo(MultiLevelSkipListReader.java:138)
> at
> org.apache.lucene.codecs.lucene41.Lucene41PostingsReader$BlockDocsEnum.advance(Lucene41PostingsReader.java:506)
> at org.apache.lucene.search.TermScorer.advance(TermScorer.java:85)
> [...]
> at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621)
> [...]
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2004)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-7440) Document skipping on large indexes is broken
[
https://issues.apache.org/jira/browse/LUCENE-7440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15477613#comment-15477613
]
Michael McCandless commented on LUCENE-7440:
What a nice find!
I would rather we just use {{long[]}} here? The added memory cost is minor,
and I think understandable code is more important.
> Document skipping on large indexes is broken
>
>
> Key: LUCENE-7440
> URL: https://issues.apache.org/jira/browse/LUCENE-7440
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/search
>Affects Versions: 2.2
>Reporter: Yonik Seeley
>Priority: Critical
> Attachments: LUCENE-7440.patch
>
>
> Large skips on large indexes fail.
> Anything that uses skips (such as a boolean query, filtered queries, faceted
> queries, join queries, etc) can trigger this bug on a sufficiently large
> index.
> The bug is a numeric overflow in MultiLevelSkipList that has been present
> since inception (Lucene 2.2). It may not manifest until one has a single
> segment with more than ~1.8B documents, and a large skip is performed on that
> segment.
> Typical stack trace on Lucene7-dev:
> {code}
> java.lang.ArrayIndexOutOfBoundsException: 110
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader$SkipBuffer.readByte(MultiLevelSkipListReader.java:297)
> at org.apache.lucene.store.DataInput.readVInt(DataInput.java:125)
> at
> org.apache.lucene.codecs.lucene50.Lucene50SkipReader.readSkipData(Lucene50SkipReader.java:180)
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader.loadNextSkip(MultiLevelSkipListReader.java:163)
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader.skipTo(MultiLevelSkipListReader.java:133)
> at
> org.apache.lucene.codecs.lucene50.Lucene50PostingsReader$BlockDocsEnum.advance(Lucene50PostingsReader.java:421)
> at YCS_skip7$1.testSkip(YCS_skip7.java:307)
> {code}
> Typical stack trace on Lucene4.10.3:
> {code}
> 6-08-31 18:57:17,460 ERROR org.apache.solr.servlet.SolrDispatchFilter:
> null:java.lang.ArrayIndexOutOfBoundsException: 75
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader$SkipBuffer.readByte(MultiLevelSkipListReader.java:301)
> at org.apache.lucene.store.DataInput.readVInt(DataInput.java:122)
> at
> org.apache.lucene.codecs.lucene41.Lucene41SkipReader.readSkipData(Lucene41SkipReader.java:194)
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader.loadNextSkip(MultiLevelSkipListReader.java:168)
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader.skipTo(MultiLevelSkipListReader.java:138)
> at
> org.apache.lucene.codecs.lucene41.Lucene41PostingsReader$BlockDocsEnum.advance(Lucene41PostingsReader.java:506)
> at org.apache.lucene.search.TermScorer.advance(TermScorer.java:85)
> [...]
> at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621)
> [...]
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2004)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-7440) Document skipping on large indexes is broken
[
https://issues.apache.org/jira/browse/LUCENE-7440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15477238#comment-15477238
]
Uwe Schindler commented on LUCENE-7440:
---
Fix with negative check looks fine to me, this should be cheap. I'd suggest
that [~mikemccand] also looks into it, he knows this code better.
> Document skipping on large indexes is broken
>
>
> Key: LUCENE-7440
> URL: https://issues.apache.org/jira/browse/LUCENE-7440
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/search
>Affects Versions: 2.2
>Reporter: Yonik Seeley
>Priority: Critical
> Attachments: LUCENE-7440.patch
>
>
> Large skips on large indexes fail.
> Anything that uses skips (such as a boolean query, filtered queries, faceted
> queries, join queries, etc) can trigger this bug on a sufficiently large
> index.
> The bug is a numeric overflow in MultiLevelSkipList that has been present
> since inception (Lucene 2.2). It may not manifest until one has a single
> segment with more than ~1.8B documents, and a large skip is performed on that
> segment.
> Typical stack trace on Lucene7-dev:
> {code}
> java.lang.ArrayIndexOutOfBoundsException: 110
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader$SkipBuffer.readByte(MultiLevelSkipListReader.java:297)
> at org.apache.lucene.store.DataInput.readVInt(DataInput.java:125)
> at
> org.apache.lucene.codecs.lucene50.Lucene50SkipReader.readSkipData(Lucene50SkipReader.java:180)
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader.loadNextSkip(MultiLevelSkipListReader.java:163)
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader.skipTo(MultiLevelSkipListReader.java:133)
> at
> org.apache.lucene.codecs.lucene50.Lucene50PostingsReader$BlockDocsEnum.advance(Lucene50PostingsReader.java:421)
> at YCS_skip7$1.testSkip(YCS_skip7.java:307)
> {code}
> Typical stack trace on Lucene4.10.3:
> {code}
> 6-08-31 18:57:17,460 ERROR org.apache.solr.servlet.SolrDispatchFilter:
> null:java.lang.ArrayIndexOutOfBoundsException: 75
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader$SkipBuffer.readByte(MultiLevelSkipListReader.java:301)
> at org.apache.lucene.store.DataInput.readVInt(DataInput.java:122)
> at
> org.apache.lucene.codecs.lucene41.Lucene41SkipReader.readSkipData(Lucene41SkipReader.java:194)
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader.loadNextSkip(MultiLevelSkipListReader.java:168)
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader.skipTo(MultiLevelSkipListReader.java:138)
> at
> org.apache.lucene.codecs.lucene41.Lucene41PostingsReader$BlockDocsEnum.advance(Lucene41PostingsReader.java:506)
> at org.apache.lucene.search.TermScorer.advance(TermScorer.java:85)
> [...]
> at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621)
> [...]
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2004)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-7440) Document skipping on large indexes is broken
[
https://issues.apache.org/jira/browse/LUCENE-7440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15477144#comment-15477144
]
Yonik Seeley commented on LUCENE-7440:
--
Regarding the 1.8B docs number... at least in my tests I saw the top-level skip
distance of ~268M w/ the default codec. Subtracting this from MAX_INT gives
around 1.8B, which is around the number I saw prior to the overflow. To hit
the bug, one also needs to be doing *large* skips toward the end of the index
as well, in order to use the top level(s) of the multi-level skip list. Having
a conjunction query of a highly unique term (or clause) in conjunction with a
common term has a good chance of triggering (example:
+timestamp:39520928456494 +doctype:common)
> Document skipping on large indexes is broken
>
>
> Key: LUCENE-7440
> URL: https://issues.apache.org/jira/browse/LUCENE-7440
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/search
>Affects Versions: 2.2
>Reporter: Yonik Seeley
>Priority: Critical
> Attachments: LUCENE-7440.patch
>
>
> Large skips on large indexes fail.
> Anything that uses skips (such as a boolean query, filtered queries, faceted
> queries, join queries, etc) can trigger this bug on a sufficiently large
> index.
> The bug is a numeric overflow in MultiLevelSkipList that has been present
> since inception (Lucene 2.2). It may not manifest until one has a single
> segment with more than ~1.8B documents, and a large skip is performed on that
> segment.
> Typical stack trace on Lucene7-dev:
> {code}
> java.lang.ArrayIndexOutOfBoundsException: 110
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader$SkipBuffer.readByte(MultiLevelSkipListReader.java:297)
> at org.apache.lucene.store.DataInput.readVInt(DataInput.java:125)
> at
> org.apache.lucene.codecs.lucene50.Lucene50SkipReader.readSkipData(Lucene50SkipReader.java:180)
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader.loadNextSkip(MultiLevelSkipListReader.java:163)
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader.skipTo(MultiLevelSkipListReader.java:133)
> at
> org.apache.lucene.codecs.lucene50.Lucene50PostingsReader$BlockDocsEnum.advance(Lucene50PostingsReader.java:421)
> at YCS_skip7$1.testSkip(YCS_skip7.java:307)
> {code}
> Typical stack trace on Lucene4.10.3:
> {code}
> 6-08-31 18:57:17,460 ERROR org.apache.solr.servlet.SolrDispatchFilter:
> null:java.lang.ArrayIndexOutOfBoundsException: 75
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader$SkipBuffer.readByte(MultiLevelSkipListReader.java:301)
> at org.apache.lucene.store.DataInput.readVInt(DataInput.java:122)
> at
> org.apache.lucene.codecs.lucene41.Lucene41SkipReader.readSkipData(Lucene41SkipReader.java:194)
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader.loadNextSkip(MultiLevelSkipListReader.java:168)
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader.skipTo(MultiLevelSkipListReader.java:138)
> at
> org.apache.lucene.codecs.lucene41.Lucene41PostingsReader$BlockDocsEnum.advance(Lucene41PostingsReader.java:506)
> at org.apache.lucene.search.TermScorer.advance(TermScorer.java:85)
> [...]
> at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621)
> [...]
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2004)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-7440) Document skipping on large indexes is broken
[
https://issues.apache.org/jira/browse/LUCENE-7440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15477131#comment-15477131
]
Uwe Schindler commented on LUCENE-7440:
---
Nice findings. A really old bug!
> Document skipping on large indexes is broken
>
>
> Key: LUCENE-7440
> URL: https://issues.apache.org/jira/browse/LUCENE-7440
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/search
>Affects Versions: 2.2
>Reporter: Yonik Seeley
>Priority: Critical
> Attachments: LUCENE-7440.patch
>
>
> Large skips on large indexes fail.
> Anything that uses skips (such as a boolean query, filtered queries, faceted
> queries, join queries, etc) can trigger this bug on a sufficiently large
> index.
> The bug is a numeric overflow in MultiLevelSkipList that has been present
> since inception (Lucene 2.2). It may not manifest until one has a single
> segment with more than ~1.8B documents, and a large skip is performed on that
> segment.
> Typical stack trace on Lucene7-dev:
> {code}
> java.lang.ArrayIndexOutOfBoundsException: 110
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader$SkipBuffer.readByte(MultiLevelSkipListReader.java:297)
> at org.apache.lucene.store.DataInput.readVInt(DataInput.java:125)
> at
> org.apache.lucene.codecs.lucene50.Lucene50SkipReader.readSkipData(Lucene50SkipReader.java:180)
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader.loadNextSkip(MultiLevelSkipListReader.java:163)
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader.skipTo(MultiLevelSkipListReader.java:133)
> at
> org.apache.lucene.codecs.lucene50.Lucene50PostingsReader$BlockDocsEnum.advance(Lucene50PostingsReader.java:421)
> at YCS_skip7$1.testSkip(YCS_skip7.java:307)
> {code}
> Typical stack trace on Lucene4.10.3:
> {code}
> 6-08-31 18:57:17,460 ERROR org.apache.solr.servlet.SolrDispatchFilter:
> null:java.lang.ArrayIndexOutOfBoundsException: 75
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader$SkipBuffer.readByte(MultiLevelSkipListReader.java:301)
> at org.apache.lucene.store.DataInput.readVInt(DataInput.java:122)
> at
> org.apache.lucene.codecs.lucene41.Lucene41SkipReader.readSkipData(Lucene41SkipReader.java:194)
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader.loadNextSkip(MultiLevelSkipListReader.java:168)
> at
> org.apache.lucene.codecs.MultiLevelSkipListReader.skipTo(MultiLevelSkipListReader.java:138)
> at
> org.apache.lucene.codecs.lucene41.Lucene41PostingsReader$BlockDocsEnum.advance(Lucene41PostingsReader.java:506)
> at org.apache.lucene.search.TermScorer.advance(TermScorer.java:85)
> [...]
> at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621)
> [...]
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2004)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-7440) Document skipping on large indexes is broken
[ https://issues.apache.org/jira/browse/LUCENE-7440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15477039#comment-15477039 ] Yonik Seeley commented on LUCENE-7440: -- Patch coming... was waiting for tests to pass. > Document skipping on large indexes is broken > > > Key: LUCENE-7440 > URL: https://issues.apache.org/jira/browse/LUCENE-7440 > Project: Lucene - Core > Issue Type: Bug > Components: core/search >Affects Versions: master (7.0) >Reporter: Yonik Seeley >Priority: Critical > > Large skips on large indexes fail. > Anything that uses skips (such as a boolean query, filtered queries, faceted > queries, join queries, etc) can trigger this bug on a sufficiently large > index. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-7440) Document skipping on large indexes is broken
[ https://issues.apache.org/jira/browse/LUCENE-7440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15476155#comment-15476155 ] Dawid Weiss commented on LUCENE-7440: - Yonik, what exactly is broken? Do you have a stack trace or repro code? > Document skipping on large indexes is broken > > > Key: LUCENE-7440 > URL: https://issues.apache.org/jira/browse/LUCENE-7440 > Project: Lucene - Core > Issue Type: Bug > Components: core/search >Affects Versions: master (7.0) >Reporter: Yonik Seeley >Priority: Critical > > Large skips on large indexes fail. > Anything that uses skips (such as a boolean query, filtered queries, faceted > queries, join queries, etc) can trigger this bug on a sufficiently large > index. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
