[jira] [Updated] (LUCENE-4635) ArrayIndexOutOfBoundsException when a segment has many, many terms

2012-12-21 Thread Steven Rowe (JIRA) 

 [ 
https://issues.apache.org/jira/browse/LUCENE-4635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steven Rowe updated LUCENE-4635:


Fix Version/s: (was: 3.6)
   3.6.2

 ArrayIndexOutOfBoundsException when a segment has many, many terms
 --

 Key: LUCENE-4635
 URL: https://issues.apache.org/jira/browse/LUCENE-4635
 Project: Lucene - Core
  Issue Type: Bug
Reporter: Michael McCandless
Assignee: Michael McCandless
 Fix For: 3.6.2

 Attachments: LUCENE-4635.patch, LUCENE-4635.patch


 Spinoff from Tom Burton-West's java-user thread CheckIndex 
 ArrayIndexOutOfBounds error for merged index ( 
 http://markmail.org/message/fatijkotwucn7hvu ).
 I modified Test2BTerms to instead generate a little over 10B terms, ran it 
 (took 17 hours and created a 162 GB index) and hit a similar exception:
 {noformat}
 Time: 62,164.058
 There was 1 failure:
 1) test2BTerms(org.apache.lucene.index.Test2BTerms)
 java.lang.ArrayIndexOutOfBoundsException: 1246
   at 
 org.apache.lucene.index.TermInfosReaderIndex.compareField(TermInfosReaderIndex.java:249)
   at 
 org.apache.lucene.index.TermInfosReaderIndex.compareTo(TermInfosReaderIndex.java:225)
   at 
 org.apache.lucene.index.TermInfosReaderIndex.getIndexOffset(TermInfosReaderIndex.java:156)
   at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:232)
   at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:172)
   at org.apache.lucene.index.SegmentReader.docFreq(SegmentReader.java:539)
   at 
 org.apache.lucene.search.TermQuery$TermWeight$1.add(TermQuery.java:56)
   at org.apache.lucene.util.ReaderUtil$Gather.run(ReaderUtil.java:81)
   at org.apache.lucene.util.ReaderUtil$Gather.run(ReaderUtil.java:87)
   at org.apache.lucene.util.ReaderUtil$Gather.run(ReaderUtil.java:70)
   at 
 org.apache.lucene.search.TermQuery$TermWeight.init(TermQuery.java:53)
   at org.apache.lucene.search.TermQuery.createWeight(TermQuery.java:199)
   at 
 org.apache.lucene.search.Searcher.createNormalizedWeight(Searcher.java:168)
   at 
 org.apache.lucene.search.IndexSearcher.createNormalizedWeight(IndexSearcher.java:664)
   at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:342)
   at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:330)
   at 
 org.apache.lucene.index.Test2BTerms.testSavedTerms(Test2BTerms.java:205)
   at org.apache.lucene.index.Test2BTerms.test2BTerms(Test2BTerms.java:154)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 {noformat}
 The index actually succeeded building and optimizing, but it was only when we 
 went to run searches of the random terms we collected along the way that the 
 AIOOBE was hit.
 I suspect this is a bug somewhere in the compact in-RAM terms index ... I'll 
 dig.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-4635) ArrayIndexOutOfBoundsException when a segment has many, many terms

2012-12-18 Thread Michael McCandless (JIRA) 

 [ 
https://issues.apache.org/jira/browse/LUCENE-4635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael McCandless updated LUCENE-4635:
---

Attachment: LUCENE-4635.patch

I suspect this fixes the issue ... at least CheckIndex on my 162 GB index is 
getting beyond where it failed previously.

I'll make a separate Test2BPagedBytes test!

 ArrayIndexOutOfBoundsException when a segment has many, many terms
 --

 Key: LUCENE-4635
 URL: https://issues.apache.org/jira/browse/LUCENE-4635
 Project: Lucene - Core
  Issue Type: Bug
Reporter: Michael McCandless
Assignee: Michael McCandless
 Attachments: LUCENE-4635.patch


 Spinoff from Tom Burton-West's java-user thread CheckIndex 
 ArrayIndexOutOfBounds error for merged index ( 
 http://markmail.org/message/fatijkotwucn7hvu ).
 I modified Test2BTerms to instead generate a little over 10B terms, ran it 
 (took 17 hours and created a 162 GB index) and hit a similar exception:
 {noformat}
 Time: 62,164.058
 There was 1 failure:
 1) test2BTerms(org.apache.lucene.index.Test2BTerms)
 java.lang.ArrayIndexOutOfBoundsException: 1246
   at 
 org.apache.lucene.index.TermInfosReaderIndex.compareField(TermInfosReaderIndex.java:249)
   at 
 org.apache.lucene.index.TermInfosReaderIndex.compareTo(TermInfosReaderIndex.java:225)
   at 
 org.apache.lucene.index.TermInfosReaderIndex.getIndexOffset(TermInfosReaderIndex.java:156)
   at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:232)
   at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:172)
   at org.apache.lucene.index.SegmentReader.docFreq(SegmentReader.java:539)
   at 
 org.apache.lucene.search.TermQuery$TermWeight$1.add(TermQuery.java:56)
   at org.apache.lucene.util.ReaderUtil$Gather.run(ReaderUtil.java:81)
   at org.apache.lucene.util.ReaderUtil$Gather.run(ReaderUtil.java:87)
   at org.apache.lucene.util.ReaderUtil$Gather.run(ReaderUtil.java:70)
   at 
 org.apache.lucene.search.TermQuery$TermWeight.init(TermQuery.java:53)
   at org.apache.lucene.search.TermQuery.createWeight(TermQuery.java:199)
   at 
 org.apache.lucene.search.Searcher.createNormalizedWeight(Searcher.java:168)
   at 
 org.apache.lucene.search.IndexSearcher.createNormalizedWeight(IndexSearcher.java:664)
   at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:342)
   at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:330)
   at 
 org.apache.lucene.index.Test2BTerms.testSavedTerms(Test2BTerms.java:205)
   at org.apache.lucene.index.Test2BTerms.test2BTerms(Test2BTerms.java:154)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 {noformat}
 The index actually succeeded building and optimizing, but it was only when we 
 went to run searches of the random terms we collected along the way that the 
 AIOOBE was hit.
 I suspect this is a bug somewhere in the compact in-RAM terms index ... I'll 
 dig.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-4635) ArrayIndexOutOfBoundsException when a segment has many, many terms

2012-12-18 Thread Michael McCandless (JIRA) 

 [ 
https://issues.apache.org/jira/browse/LUCENE-4635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael McCandless updated LUCENE-4635:
---

Attachment: LUCENE-4635.patch

New patch, with test, and fixing another place where we could overflow int.

I think it's ready.

 ArrayIndexOutOfBoundsException when a segment has many, many terms
 --

 Key: LUCENE-4635
 URL: https://issues.apache.org/jira/browse/LUCENE-4635
 Project: Lucene - Core
  Issue Type: Bug
Reporter: Michael McCandless
Assignee: Michael McCandless
 Attachments: LUCENE-4635.patch, LUCENE-4635.patch


 Spinoff from Tom Burton-West's java-user thread CheckIndex 
 ArrayIndexOutOfBounds error for merged index ( 
 http://markmail.org/message/fatijkotwucn7hvu ).
 I modified Test2BTerms to instead generate a little over 10B terms, ran it 
 (took 17 hours and created a 162 GB index) and hit a similar exception:
 {noformat}
 Time: 62,164.058
 There was 1 failure:
 1) test2BTerms(org.apache.lucene.index.Test2BTerms)
 java.lang.ArrayIndexOutOfBoundsException: 1246
   at 
 org.apache.lucene.index.TermInfosReaderIndex.compareField(TermInfosReaderIndex.java:249)
   at 
 org.apache.lucene.index.TermInfosReaderIndex.compareTo(TermInfosReaderIndex.java:225)
   at 
 org.apache.lucene.index.TermInfosReaderIndex.getIndexOffset(TermInfosReaderIndex.java:156)
   at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:232)
   at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:172)
   at org.apache.lucene.index.SegmentReader.docFreq(SegmentReader.java:539)
   at 
 org.apache.lucene.search.TermQuery$TermWeight$1.add(TermQuery.java:56)
   at org.apache.lucene.util.ReaderUtil$Gather.run(ReaderUtil.java:81)
   at org.apache.lucene.util.ReaderUtil$Gather.run(ReaderUtil.java:87)
   at org.apache.lucene.util.ReaderUtil$Gather.run(ReaderUtil.java:70)
   at 
 org.apache.lucene.search.TermQuery$TermWeight.init(TermQuery.java:53)
   at org.apache.lucene.search.TermQuery.createWeight(TermQuery.java:199)
   at 
 org.apache.lucene.search.Searcher.createNormalizedWeight(Searcher.java:168)
   at 
 org.apache.lucene.search.IndexSearcher.createNormalizedWeight(IndexSearcher.java:664)
   at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:342)
   at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:330)
   at 
 org.apache.lucene.index.Test2BTerms.testSavedTerms(Test2BTerms.java:205)
   at org.apache.lucene.index.Test2BTerms.test2BTerms(Test2BTerms.java:154)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 {noformat}
 The index actually succeeded building and optimizing, but it was only when we 
 went to run searches of the random terms we collected along the way that the 
 AIOOBE was hit.
 I suspect this is a bug somewhere in the compact in-RAM terms index ... I'll 
 dig.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org