Ansgar Kapels created SOLR-8331:
-----------------------------------
Summary: CorruptIndexException after auto commit
Key: SOLR-8331
URL: https://issues.apache.org/jira/browse/SOLR-8331
Project: Solr
Issue Type: Bug
Components: update
Affects Versions: 4.10.4
Environment: OS: SUSE Linux Enterprise Server 11 SP3
File system: ext3
Application server: Tomcat 7
Reporter: Ansgar Kapels
Priority: Critical
While adding many new documents to Solr (via solrJ) it happens that the index
files get corrupted.
This problem is occurring on different virtual servers (all same OS and
configuration). Especially when adding many new (or updated) documents to Solr
in a short time.
Here's the exception:
{code}
org.apache.solr.common.SolrException; auto commit
error...:org.apache.lucene.index.CorruptIndexException: codec header mismatch:
actual header=1970145651 vs expected header=1071082519 (resource:
BufferedChecksumIndexInput(MMapIndexInput(path="/data/solr/data1/index/_gru.fnm")))
at org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:136)
at
org.apache.lucene.codecs.lucene46.Lucene46FieldInfosReader.read(Lucene46FieldInfosReader.java:57)
at
org.apache.lucene.index.SegmentReader.readFieldInfos(SegmentReader.java:289)
at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:107)
at
org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:145)
at
org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:282)
at
org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3312)
at
org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3303)
at
org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2989)
at
org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3134)
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3101)
at
org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:582)
at org.apache.solr.update.CommitTracker.run(CommitTracker.java:216)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{code}
I noticed that Solr runs stable much longer when I modify merge settings:
Reducing
<maxMergeDocs>100000</maxMergeDocs>
to
<maxMergeDocs>10000</maxMergeDocs>
has a positive effect but at some point the index still gets corrupted.
Same when setting a higher mergeFactor. So it seems I can delay the issue for a
while but certainly it will reach a critical point after a few days or weeks it
can't handle anymore. Maybe it is related to a certain file size or something?
The index's total size (data directory) is about 23 GB with 1,600,000 documents.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
