Ansgar Kapels created SOLR-8331: ----------------------------------- Summary: CorruptIndexException after auto commit Key: SOLR-8331 URL: https://issues.apache.org/jira/browse/SOLR-8331 Project: Solr Issue Type: Bug Components: update Affects Versions: 4.10.4 Environment: OS: SUSE Linux Enterprise Server 11 SP3 File system: ext3 Application server: Tomcat 7 Reporter: Ansgar Kapels Priority: Critical
While adding many new documents to Solr (via solrJ) it happens that the index files get corrupted. This problem is occurring on different virtual servers (all same OS and configuration). Especially when adding many new (or updated) documents to Solr in a short time. Here's the exception: {code} org.apache.solr.common.SolrException; auto commit error...:org.apache.lucene.index.CorruptIndexException: codec header mismatch: actual header=1970145651 vs expected header=1071082519 (resource: BufferedChecksumIndexInput(MMapIndexInput(path="/data/solr/data1/index/_gru.fnm"))) at org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:136) at org.apache.lucene.codecs.lucene46.Lucene46FieldInfosReader.read(Lucene46FieldInfosReader.java:57) at org.apache.lucene.index.SegmentReader.readFieldInfos(SegmentReader.java:289) at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:107) at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:145) at org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:282) at org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3312) at org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3303) at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2989) at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3134) at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3101) at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:582) at org.apache.solr.update.CommitTracker.run(CommitTracker.java:216) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) {code} I noticed that Solr runs stable much longer when I modify merge settings: Reducing <maxMergeDocs>100000</maxMergeDocs> to <maxMergeDocs>10000</maxMergeDocs> has a positive effect but at some point the index still gets corrupted. Same when setting a higher mergeFactor. So it seems I can delay the issue for a while but certainly it will reach a critical point after a few days or weeks it can't handle anymore. Maybe it is related to a certain file size or something? The index's total size (data directory) is about 23 GB with 1,600,000 documents. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org