[
https://issues.apache.org/jira/browse/LUCENE-6192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael McCandless updated LUCENE-6192:
---------------------------------------
Attachment: LUCENE-6192.patch
Here's a patch, only for 4.10.x for starters (Lucene41SkipWriter) so Tom can
test it and see if it fixes his exception. The good thing about skip data is
it's ignored during merging, so to test this you just need to apply the patch,
compile & deploy Lucene core JAR, then optimize so the skip data is
regenerated...
On 5.x/trunk we must also fix Lucene50SkipWriter...
> Long overflow in LuceneXXSkipWriter can corrupt skip data
> ---------------------------------------------------------
>
> Key: LUCENE-6192
> URL: https://issues.apache.org/jira/browse/LUCENE-6192
> Project: Lucene - Core
> Issue Type: Bug
> Reporter: Michael McCandless
> Fix For: 5.0, Trunk, 4.x
>
> Attachments: LUCENE-6192.patch
>
>
> I've been iterating with Tom on this corruption that CheckIndex detects in
> his rather large index (720 GB in a single segment):
> {noformat}
> java -Xmx16G -Xms16G -cp $JAR -ea:org.apache.lucene...
> org.apache.lucene.index.CheckIndex /XXXX/shards/4/core-1/data/test_index
> -verbose 2>&1 |tee -a shard4_reoptimizedNewJava
> Opening index @ /htsolr/lss-reindex/shards/4/core-1/data/test_index
> Segments file=segments_e numSegments=1 version=4.10.2 format=
> userData={commitTimeMSec=1421479358825}
> 1 of 1: name=_8m8 docCount=1130856
> version=4.10.2
> codec=Lucene410
> compound=false
> numFiles=10
> size (MB)=719,967.32
> diagnostics = {timestamp=1421437320935, os=Linux,
> os.version=2.6.18-400.1.1.el5, mergeFactor=2, source=merge,
> lucene.version=4.10.2, os.arch=amd64, mergeMaxNumSegments=1,
> java.version=1.7.0_71, java.vendor=Oracle Corporation}
> no deletions
> test: open reader.........OK
> test: check integrity.....OK
> test: check live docs.....OK
> test: fields..............OK [80 fields]
> test: field norms.........OK [23 fields]
> test: terms, freq, prox...ERROR: java.lang.AssertionError: -96
> java.lang.AssertionError: -96
> at
> org.apache.lucene.codecs.lucene41.ForUtil.skipBlock(ForUtil.java:228)
> at
> org.apache.lucene.codecs.lucene41.Lucene41PostingsReader$BlockDocsAndPositionsEnum.skipPositions(Lucene41PostingsReader.java:925)
> at
> org.apache.lucene.codecs.lucene41.Lucene41PostingsReader$BlockDocsAndPositionsEnum.nextPosition(Lucene41PostingsReader.java:955)
> at
> org.apache.lucene.index.CheckIndex.checkFields(CheckIndex.java:1100)
> at
> org.apache.lucene.index.CheckIndex.testPostings(CheckIndex.java:1357)
> at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:655)
> at org.apache.lucene.index.CheckIndex.main(CheckIndex.java:2096)
> test: stored fields.......OK [67472796 total field count; avg 59.665
> fields per doc]
> test: term vectors........OK [0 total vector count; avg 0 term/freq
> vector fields per doc]
> test: docvalues...........OK [0 docvalues fields; 0 BINARY; 0 NUMERIC; 0
> SORTED; 0 SORTED_NUMERIC; 0 SORTED_SET]
> FAILED
> WARNING: fixIndex() would remove reference to this segment; full
> exception:
> java.lang.RuntimeException: Term Index test failed
> at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:670)
> at org.apache.lucene.index.CheckIndex.main(CheckIndex.java:2096)
> WARNING: 1 broken segments (containing 1130856 documents) detected
> WARNING: would write new segments file, and 1130856 documents would be lost,
> if -fix were specified
> {noformat}
> And Rob spotted long -> int casts in our skip list writers that look like
> they could cause such corruption if a single high-freq term with many
> positions required > 2.1 GB to write its positions into .pos.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
