Han-Wen NIenhuys created LUCENE-7499: ----------------------------------------
Summary: IndexWriter race condition Key: LUCENE-7499 URL: https://issues.apache.org/jira/browse/LUCENE-7499 Project: Lucene - Core Issue Type: Bug Components: core/index Affects Versions: 6.2, 5.5.2 Reporter: Han-Wen NIenhuys Priority: Minor separated out from https://issues.apache.org/jira/browse/LUCENE-2585 I've seen race condition diagnostics for Lucene code that could be related to this. Line numbers are for Lucene 5.5.2, but the problem seems present in master too. AFAICT: StandardDirectoryReader has an IndexWriter, and it checks things are up-to-date by checking if there are pending in-memory updates by looking at IndexWriter.SegmentInfos.version. That read is not synchronized relative to its write from IndexWriter.updateDocument I'm not enough of a lucene expert to suggest how to fix this in detail. If I would guess at how Lucene roughly works, I would suggest the following * when applying pending in-memory changes, this takes a lock, and under lock there is a "changed = true" assignment. * when checking for changes, take lock, return changed. * when writing pending in-memory changes, take lock, write out file, set "changed=false", release lock. The problem can be reproduced by running some of the Gerrit acceptance tests under Threadsanitizer. I don't know of open-source tooling to set that up, unfortunately. WARNING: ThreadSanitizer: data race (pid=21636) Write of size 8 at 0x7f24ad9d3210 by thread T36 (mutexes: write M431922409782456832): #0 org.apache.lucene.index.SegmentInfos.changed()V (SegmentInfos.java:944) #1 org.apache.lucene.index.IndexWriter.newSegmentName()Ljava/lang/String; (IndexWriter.java:1652) #2 org.apache.lucene.index.DocumentsWriter.ensureInitialized(Lorg/apache/lucene/index/DocumentsWriterPerThreadPool$ThreadState;)V (DocumentsWriter.java:391) #3 org.apache.lucene.index.DocumentsWriter.updateDocument(Ljava/lang/Iterable;Lorg/apache/lucene/analysis/Analyzer;Lorg/apache/lucene/index/Term;)Z (DocumentsWriter.java:445) #4 org.apache.lucene.index.IndexWriter.updateDocument(Lorg/apache/lucene/index/Term;Ljava/lang/Iterable;)V (IndexWriter.java:1477) #5 com.google.gerrit.lucene.AutoCommitWriter.updateDocument(Lorg/apache/lucene/index/Term;Ljava/lang/Iterable;)V (AutoCommitWriter.java:100) #6 org.apache.lucene.index.TrackingIndexWriter.updateDocument(Lorg/apache/lucene/index/Term;Ljava/lang/Iterable;)J (TrackingIndexWriter.java:55) #7 com.google.gerrit.lucene.AbstractLuceneIndex$4.call()Ljava/lang/Long; (AbstractLuceneIndex.java:250) #8 com.google.gerrit.lucene.AbstractLuceneIndex$4.call()Ljava/lang/Object; (AbstractLuceneIndex.java:247) #9 com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly()V (TrustedListenableFutureTask.java:108) #10 com.google.common.util.concurrent.InterruptibleTask.run()V (InterruptibleTask.java:41) #11 com.google.common.util.concurrent.TrustedListenableFutureTask.run()V (TrustedListenableFutureTask.java:77) #12 java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V (ThreadPoolExecutor.java:1142) #13 java.util.concurrent.ThreadPoolExecutor$Worker.run()V (ThreadPoolExecutor.java:617) #14 java.lang.Thread.run()V (Thread.java:745) #15 (Generated Stub) Previous read of size 8 at 0x7f24ad9d3210 by thread T29 (mutexes: write M1060737507754061632): #0 org.apache.lucene.index.IndexWriter.nrtIsCurrent(Lorg/apache/lucene/index/SegmentInfos;)Z (IndexWriter.java:4592) #1 org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(Lorg/apache/lucene/index/IndexCommit;)Lorg/apache/lucene/index/DirectoryReader; (StandardDirectoryReader.java:282) #2 org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(Lorg/apache/lucene/index/IndexCommit;)Lorg/apache/lucene/index/DirectoryReader; (StandardDirectoryReader.java:261) #3 org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged()Lorg/apache/lucene/index/DirectoryReader; (StandardDirectoryReader.java:251) #4 org.apache.lucene.index.DirectoryReader.openIfChanged(Lorg/apache/lucene/index/DirectoryReader;)Lorg/apache/lucene/index/DirectoryReader; (DirectoryReader.java:137) #5 com.google.gerrit.lucene.WrappableSearcherManager.refreshIfNeeded(Lorg/apache/lucene/search/IndexSearcher;)Lorg/apache/lucene/search/IndexSearcher; (WrappableSearcherManager.java:148) #6 com.google.gerrit.lucene.WrappableSearcherManager.refreshIfNeeded(Ljava/lang/Object;)Ljava/lang/Object; (WrappableSearcherManager.java:68) #7 org.apache.lucene.search.ReferenceManager.doMaybeRefresh()V (ReferenceManager.java:176) #8 org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking()V (ReferenceManager.java:253) #9 org.apache.lucene.search.ControlledRealTimeReopenThread.run()V (ControlledRealTimeReopenThread.java:245) #10 (Generated Stub) -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org