Hi experts,
Still about this issue, is there any known bug which will cause payload file
corruption? The stack trace indicates that the fisrt byte of input should be an
Integer <= 32, but actually it is 110.
Our customers seeing this kind of corruption several times, and all of the
corruption is from payload. Is there any possibility that the bytes put into
payload being incompatible with payload codec?
void readBlock(IndexInput in, byte[] encoded, int[] decoded) throws
IOException {
final int numBits = in.readByte();
assert numBits <= 32 : numBits;
if (numBits == ALL_VALUES_EQUAL) {
final int value = in.readVInt();
Arrays.fill(decoded, 0, BLOCK_SIZE, value);
return;
}
final int encodedSize = encodedSizes[numBits];
in.readBytes(encoded, 0, encodedSize);
From: 马江 <[email protected]>
Reply-To: "[email protected]" <[email protected]>
Date: Tuesday, January 16, 2018 at 11:16 AM
To: "[email protected]" <[email protected]>
Subject: [EXTERNAL] - Lucene 4.5.1 payload corruption -
ArrayIndexOutOfBoundsException
Hi experts,
Recently one of our customer continuously seeing ArrayIndexOutOfBoundsException
which is thrown from Lucene.
Our production is full-text search engine built on top of Lucene, following is
the stack traces. The customer saying that they can reproduce the issue even
after re-index everything from scratch.
Caused by: java.lang.ArrayIndexOutOfBoundsException: 110
at
org.apache.lucene.codecs.lucene41.ForUtil.readBlock(ForUtil.java:196)
at
org.apache.lucene.codecs.lucene41.Lucene41PostingsReader$EverythingEnum.refillPositions(Lucene41PostingsReader.java:1284)
at
org.apache.lucene.codecs.lucene41.Lucene41PostingsReader$EverythingEnum.skipPositions(Lucene41PostingsReader.java:1505)
at
org.apache.lucene.codecs.lucene41.Lucene41PostingsReader$EverythingEnum.nextPosition(Lucene41PostingsReader.java:1548)
at
org.apache.lucene.search.spans.TermSpans.skipTo(TermSpans.java:82)
at
org.apache.lucene.search.spans.SpanScorer.advance(SpanScorer.java:63)
at
org.apache.lucene.search.ConjunctionScorer.doNext(ConjunctionScorer.java:69)
at
org.apache.lucene.search.ConjunctionScorer.nextDoc(ConjunctionScorer.java:100)
at org.apache.lucene.search.Scorer.score(Scorer.java:64)
at
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:627)
at com.xhive.lucene.executor.f.a(xdb:158)
at com.xhive.lucene.executor.f.search(xdb:145)
at com.xhive.lucene.subpath.e.a(xdb:313)
at com.xhive.lucene.subpath.e.a(xdb:264)
at com.xhive.lucene.subpath.e.a(xdb:183)
at com.xhive.lucene.executor.v.executeExternally(xdb:253)
at com.xhive.kernel.ay.externalIndexExecute(xdb:2791)
at com.xhive.core.index.ExternalIndex.executeExternally(xdb:485)
at com.xhive.core.index.XhiveMultiPathIndex.a(xdb:306)
at com.xhive.xquery.pathexpr.v$a.ci(xdb:124)
at com.xhive.xquery.pathexpr.ad$a.cp(xdb:104)
at com.xhive.xquery.pathexpr.ax.awP(xdb:39)
at com.xhive.xquery.pathexpr.ax.<init>(xdb:32)
at com.xhive.xquery.pathexpr.av.a(xdb:424)
at com.xhive.xquery.pathexpr.al$a.awk(xdb:61)
at com.xhive.xquery.pathexpr.ag.awj(xdb:28)
at com.xhive.xquery.pathexpr.al.Xo(xdb:26)
at com.xhive.xquery.pathexpr.aj.<init>(xdb:33)
at com.xhive.xquery.pathexpr.al.<init>(xdb:20)
at com.xhive.xquery.pathexpr.av.a(xdb:462)
at com.xhive.xquery.pathexpr.av.a(xdb:413)
at com.xhive.xquery.pathexpr.av.a(xdb:276)
at com.xhive.xquery.pathexpr.av.a(xdb:220)
==============================================================
following is CheckIndex output of corrupted segment. The full output is
attached.
Checking consistency of: [CHECK_INDEXES_CONSISTENCY]
Library child /dpwprd/dsearch/Data/Collection2 is not in consistent state,
errors report:
============================================================
Library child name=/dpwprd/dsearch/Data/Collection2 indexes
consistency report.
============================================================
check external index consistency [database name: xhivedb;
index name: dmftdoc; segment id:
EI-0ab89c0c-2a9d-4fe2-97b9-5f0c96678f13-510173395289107-master;
xhive index id id: 510173395289107]
check lucene indices
fail: lucene index LI-0001cd61-342c-4cfe-9898-c293eb1c8c09
is not consistent; Segments file=segments_2 numSegments=5
version=4.5.1 format=
1 of 5: name=_0 docCount=8341939
codec=Lucene45
compound=false
numFiles=26
size (MB)=16,446.152
diagnostics =
{timestamp=1514627603337, mergeFactor=6, os.version=6.1,
os=Windows Server 2008 R2, lucene.version=4.5.1 1533280 -
mark - 2013-10-17 21:37:01, source=merge, os.arch=amd64,
mergeMaxNumSegments=5, java.version=1.7.0_80,
java.vendor=Oracle Corporation}
has deletions
[delGen=70]
test: open reader.........OK [2295 deleted
docs]
test: fields..............OK [268 fields]
test: field norms.........OK [3 fields]
test: terms,
freq, prox...ERROR:
java.lang.ArrayIndexOutOfBoundsException
java.lang.ArrayIndexOutOfBoundsException
test: stored
fields.......OK [16679288 total field count; avg 2 fields
per doc]
test: term vectors........OK [0 total vector
count; avg 0 term/freq vector fields per doc]
test:
docvalues...........OK [0 docvalues fields; 0 BINARY; 0
NUMERIC; 0 SORTED; 0 SORTED_SET]
FAILED
WARNING:
fixIndex() would remove reference to this segment; full
exception:
java.lang.RuntimeException: Term Index test
failed
at
org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:638)
at
org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:372)
at com.xhive.lucene.executor.j.a(xdb:1190)
at
com.xhive.lucene.executor.j.aY(xdb:1166)
at
com.xhive.lucene.executor.v.checkIndexConsistency(xdb:370)
at
com.xhive.kernel.ay.externalIndexCheckConsistency(xdb:2523)
at com.xhive.kernel.bn.handleRequest(xdb:2544)
at
com.xhive.kernel.bn.run(xdb:222)
at
java.lang.Thread.run(Thread.java:745)
==============================================================
The corrupted payload stores a serialized hashmap which contains several
configurable metadata which is used to sort by condition.
The field of the corrupted payload is single term field, so the structure of
posting looks like a sequence of payload.
We also put freshness boost value into payload in another field, which have no
issues.
It is the first customer report the corruption after we used Lucene 4.5.1 and
released our product for many years.
Please let me know if you have any idea to this issue.
Thanks,
Tony Ma(马江)
