Hi experts,
Recently one of our customer continuously seeing ArrayIndexOutOfBoundsException
which is thrown from Lucene.
Our production is full-text search engine built on top of Lucene, following is
the stack traces. The customer saying that they can reproduce the issue even
after re-index everything from scratch.
Caused by: java.lang.ArrayIndexOutOfBoundsException: 110
at
org.apache.lucene.codecs.lucene41.ForUtil.readBlock(ForUtil.java:196)
at
org.apache.lucene.codecs.lucene41.Lucene41PostingsReader$EverythingEnum.refillPositions(Lucene41PostingsReader.java:1284)
at
org.apache.lucene.codecs.lucene41.Lucene41PostingsReader$EverythingEnum.skipPositions(Lucene41PostingsReader.java:1505)
at
org.apache.lucene.codecs.lucene41.Lucene41PostingsReader$EverythingEnum.nextPosition(Lucene41PostingsReader.java:1548)
at
org.apache.lucene.search.spans.TermSpans.skipTo(TermSpans.java:82)
at
org.apache.lucene.search.spans.SpanScorer.advance(SpanScorer.java:63)
at
org.apache.lucene.search.ConjunctionScorer.doNext(ConjunctionScorer.java:69)
at
org.apache.lucene.search.ConjunctionScorer.nextDoc(ConjunctionScorer.java:100)
at org.apache.lucene.search.Scorer.score(Scorer.java:64)
at
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:627)
at com.xhive.lucene.executor.f.a(xdb:158)
at com.xhive.lucene.executor.f.search(xdb:145)
at com.xhive.lucene.subpath.e.a(xdb:313)
at com.xhive.lucene.subpath.e.a(xdb:264)
at com.xhive.lucene.subpath.e.a(xdb:183)
at com.xhive.lucene.executor.v.executeExternally(xdb:253)
at com.xhive.kernel.ay.externalIndexExecute(xdb:2791)
at com.xhive.core.index.ExternalIndex.executeExternally(xdb:485)
at com.xhive.core.index.XhiveMultiPathIndex.a(xdb:306)
at com.xhive.xquery.pathexpr.v$a.ci(xdb:124)
at com.xhive.xquery.pathexpr.ad$a.cp(xdb:104)
at com.xhive.xquery.pathexpr.ax.awP(xdb:39)
at com.xhive.xquery.pathexpr.ax.<init>(xdb:32)
at com.xhive.xquery.pathexpr.av.a(xdb:424)
at com.xhive.xquery.pathexpr.al$a.awk(xdb:61)
at com.xhive.xquery.pathexpr.ag.awj(xdb:28)
at com.xhive.xquery.pathexpr.al.Xo(xdb:26)
at com.xhive.xquery.pathexpr.aj.<init>(xdb:33)
at com.xhive.xquery.pathexpr.al.<init>(xdb:20)
at com.xhive.xquery.pathexpr.av.a(xdb:462)
at com.xhive.xquery.pathexpr.av.a(xdb:413)
at com.xhive.xquery.pathexpr.av.a(xdb:276)
at com.xhive.xquery.pathexpr.av.a(xdb:220)
==============================================================
following is CheckIndex output of corrupted segment. The full output is
attached.
Checking consistency of: [CHECK_INDEXES_CONSISTENCY]
Library child /dpwprd/dsearch/Data/Collection2 is not in consistent state,
errors report:
============================================================
Library child name=/dpwprd/dsearch/Data/Collection2 indexes
consistency report.
============================================================
check external index consistency [database name: xhivedb;
index name: dmftdoc; segment id:
EI-0ab89c0c-2a9d-4fe2-97b9-5f0c96678f13-510173395289107-master;
xhive index id id: 510173395289107]
check lucene indices
fail: lucene index LI-0001cd61-342c-4cfe-9898-c293eb1c8c09
is not consistent; Segments file=segments_2 numSegments=5
version=4.5.1 format=
1 of 5: name=_0 docCount=8341939
codec=Lucene45
compound=false
numFiles=26
size (MB)=16,446.152
diagnostics =
{timestamp=1514627603337, mergeFactor=6, os.version=6.1,
os=Windows Server 2008 R2, lucene.version=4.5.1 1533280 -
mark - 2013-10-17 21:37:01, source=merge, os.arch=amd64,
mergeMaxNumSegments=5, java.version=1.7.0_80,
java.vendor=Oracle Corporation}
has deletions
[delGen=70]
test: open reader.........OK [2295 deleted
docs]
test: fields..............OK [268 fields]
test: field norms.........OK [3 fields]
test: terms,
freq, prox...ERROR:
java.lang.ArrayIndexOutOfBoundsException
java.lang.ArrayIndexOutOfBoundsException
test: stored
fields.......OK [16679288 total field count; avg 2 fields
per doc]
test: term vectors........OK [0 total vector
count; avg 0 term/freq vector fields per doc]
test:
docvalues...........OK [0 docvalues fields; 0 BINARY; 0
NUMERIC; 0 SORTED; 0 SORTED_SET]
FAILED
WARNING:
fixIndex() would remove reference to this segment; full
exception:
java.lang.RuntimeException: Term Index test
failed
at
org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:638)
at
org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:372)
at com.xhive.lucene.executor.j.a(xdb:1190)
at
com.xhive.lucene.executor.j.aY(xdb:1166)
at
com.xhive.lucene.executor.v.checkIndexConsistency(xdb:370)
at
com.xhive.kernel.ay.externalIndexCheckConsistency(xdb:2523)
at com.xhive.kernel.bn.handleRequest(xdb:2544)
at
com.xhive.kernel.bn.run(xdb:222)
at
java.lang.Thread.run(Thread.java:745)
==============================================================
The corrupted payload stores a serialized hashmap which contains several
configurable metadata which is used to sort by condition.
The field of the corrupted payload is single term field, so the structure of
posting looks like a sequence of payload.
We also put freshness boost value into payload in another field, which have no
issues.
It is the first customer report the corruption after we used Lucene 4.5.1 and
released our product for many years.
Please let me know if you have any idea to this issue.
Thanks,
Tony Ma(马江)
Checking consistency of: [CHECK_INDEXES_CONSISTENCY]
Library child /dpwprd/dsearch/Data/Collection2 is not in consistent state,
errors report:
============================================================
Library child name=/dpwprd/dsearch/Data/Collection2 indexes
consistency report.
============================================================
check external index consistency [database name: xhivedb;
index name: dmftdoc; segment id:
EI-0ab89c0c-2a9d-4fe2-97b9-5f0c96678f13-510173395289107-master;
xhive index id id: 510173395289107]
check lucene indices
fail: lucene index LI-0001cd61-342c-4cfe-9898-c293eb1c8c09
is not consistent; Segments file=segments_2 numSegments=5
version=4.5.1 format=
1 of 5: name=_0 docCount=8341939
codec=Lucene45
compound=false
numFiles=26
size (MB)=16,446.152
diagnostics =
{timestamp=1514627603337, mergeFactor=6, os.version=6.1,
os=Windows Server 2008 R2, lucene.version=4.5.1 1533280 -
mark - 2013-10-17 21:37:01, source=merge, os.arch=amd64,
mergeMaxNumSegments=5, java.version=1.7.0_80,
java.vendor=Oracle Corporation}
has deletions
[delGen=70]
test: open reader.........OK [2295 deleted
docs]
test: fields..............OK [268 fields]
test: field norms.........OK [3 fields]
test: terms,
freq, prox...ERROR:
java.lang.ArrayIndexOutOfBoundsException
java.lang.ArrayIndexOutOfBoundsException
test: stored
fields.......OK [16679288 total field count; avg 2 fields
per doc]
test: term vectors........OK [0 total vector
count; avg 0 term/freq vector fields per doc]
test:
docvalues...........OK [0 docvalues fields; 0 BINARY; 0
NUMERIC; 0 SORTED; 0 SORTED_SET]
FAILED
WARNING:
fixIndex() would remove reference to this segment; full
exception:
java.lang.RuntimeException: Term Index test
failed
at
org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:638)
at
org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:372)
at com.xhive.lucene.executor.j.a(xdb:1190)
at
com.xhive.lucene.executor.j.aY(xdb:1166)
at
com.xhive.lucene.executor.v.checkIndexConsistency(xdb:370)
at
com.xhive.kernel.ay.externalIndexCheckConsistency(xdb:2523)
at com.xhive.kernel.bn.handleRequest(xdb:2544)
at
com.xhive.kernel.bn.run(xdb:222)
at
java.lang.Thread.run(Thread.java:745)
2 of 5: name=_1
docCount=2190595
codec=Lucene45
compound=false
numFiles=25
size (MB)=4,417.281
diagnostics =
{timestamp=1514771642294, mergeFactor=6, os.version=6.1,
os=Windows Server 2008 R2, lucene.version=4.5.1 1533280 -
mark - 2013-10-17 21:37:01, source=merge, os.arch=amd64,
mergeMaxNumSegments=5, java.version=1.7.0_80,
java.vendor=Oracle Corporation}
has deletions
[delGen=53]
test: open reader.........OK [1312 deleted
docs]
test: fields..............OK [262 fields]
test: field norms.........OK [3 fields]
test: terms,
freq, prox...OK [54401729 terms; 787177718 terms/docs
pairs; 624929292 tokens]
test (ignoring deletes):
terms, freq, prox...OK [54429967 terms; 787657042
terms/docs pairs; 625313964 tokens]
test: stored
fields.......OK [4378566 total field count; avg 2 fields
per doc]
test: term vectors........OK [0 total vector
count; avg 0 term/freq vector fields per doc]
test:
docvalues...........OK [0 docvalues fields; 0 BINARY; 0
NUMERIC; 0 SORTED; 0 SORTED_SET]
3 of 5: name=_2
docCount=851764
codec=Lucene45
compound=false
numFiles=25
size (MB)=1,711.874
diagnostics =
{timestamp=1514814824171, mergeFactor=9, os.version=6.1,
os=Windows Server 2008 R2, lucene.version=4.5.1 1533280 -
mark - 2013-10-17 21:37:01, source=merge, os.arch=amd64,
mergeMaxNumSegments=5, java.version=1.7.0_80,
java.vendor=Oracle Corporation}
has deletions
[delGen=63]
test: open reader.........OK [764 deleted
docs]
test: fields..............OK [254 fields]
test: field norms.........OK [3 fields]
test: terms,
freq, prox...OK [21881507 terms; 307102495 terms/docs
pairs; 243816854 tokens]
test (ignoring deletes):
terms, freq, prox...OK [21899561 terms; 307376274
terms/docs pairs; 244037071 tokens]
test: stored
fields.......OK [1702000 total field count; avg 2 fields
per doc]
test: term vectors........OK [0 total vector
count; avg 0 term/freq vector fields per doc]
test:
docvalues...........OK [0 docvalues fields; 0 BINARY; 0
NUMERIC; 0 SORTED; 0 SORTED_SET]
4 of 5: name=_3
docCount=553102
codec=Lucene45
compound=false
numFiles=25
size (MB)=1,122.343
diagnostics =
{timestamp=1514858042321, mergeFactor=3, os.version=6.1,
os=Windows Server 2008 R2, lucene.version=4.5.1 1533280 -
mark - 2013-10-17 21:37:01, source=merge, os.arch=amd64,
mergeMaxNumSegments=5, java.version=1.7.0_80,
java.vendor=Oracle Corporation}
has deletions
[delGen=70]
test: open reader.........OK [4955 deleted
docs]
test: fields..............OK [254 fields]
test: field norms.........OK [3 fields]
test: terms,
freq, prox...OK [14773729 terms; 197409595 terms/docs
pairs; 156425802 tokens]
test (ignoring deletes):
terms, freq, prox...OK [14890759 terms; 199279426
terms/docs pairs; 157925779 tokens]
test: stored
fields.......OK [1096294 total field count; avg 2 fields
per doc]
test: term vectors........OK [0 total vector
count; avg 0 term/freq vector fields per doc]
test:
docvalues...........OK [0 docvalues fields; 0 BINARY; 0
NUMERIC; 0 SORTED; 0 SORTED_SET]
5 of 5: name=_4
docCount=805370
codec=Lucene45
compound=false
numFiles=11
size (MB)=1,651.581
diagnostics =
{timestamp=1515667638318, mergeFactor=7, os.version=6.1,
os=Windows Server 2008 R2, lucene.version=4.5.1 1533280 -
mark - 2013-10-17 21:37:01, source=merge, os.arch=amd64,
mergeMaxNumSegments=5, java.version=1.7.0_80,
java.vendor=Oracle Corporation}
no deletions
test: open reader.........OK
test:
fields..............OK [254 fields]
test: field
norms.........OK [3 fields]
test: terms, freq,
prox...OK [21490057 terms; 290045363 terms/docs pairs;
230248681 tokens]
test: stored fields.......OK
[1610740 total field count; avg 2 fields per doc]
test: term vectors........OK [0 total vector count; avg 0
term/freq vector fields per doc]
test:
docvalues...........OK [0 docvalues fields; 0 BINARY; 0
NUMERIC; 0 SORTED; 0 SORTED_SET]
WARNING: 1 broken
segments (containing 8339644 documents) detected
FAILED
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
