mike can you try to reproduce it - i can't reproduce it with this patch:
Index: lucene/core/src/java/org/apache/lucene/index/IndexWriter.java
===================================================================
--- lucene/core/src/java/org/apache/lucene/index/IndexWriter.java
(revision 1551902)
+++ lucene/core/src/java/org/apache/lucene/index/IndexWriter.java (working copy)
@@ -1029,7 +1029,7 @@
if (doFlush) {
commitInternal();
}
-
+ processEvents(false, true);
synchronized(this) {
// commitInternal calls ReaderPool.commit, which
// writes any pending liveDocs from ReaderPool, so
@@ -1064,7 +1064,6 @@
}
// finally, restore interrupt status:
if (interrupted) Thread.currentThread().interrupt();
- processEvents(false, true);
}
}
I think we call it too late though it needs to be called fare more above
I will open an issue if your beast is happy too?
simon
On Wed, Dec 18, 2013 at 2:33 PM, Simon Willnauer <[email protected]> wrote:
> I have a fix for it - will open an issue later
>
> On Wed, Dec 18, 2013 at 11:44 AM, Simon Willnauer <[email protected]> wrote:
>> it's not trying to flush a segment it tries to publish it. It already
>> is flushed but still scary. I will update the job description @Robert
>>
>> On Tue, Dec 17, 2013 at 9:27 PM, Michael McCandless
>> <[email protected]> wrote:
>>> It actually repros with java7 too, if you beast it.
>>>
>>> It's a scary failure: it seems like, after closing, the very last
>>> thing IW does in closeInternal's finally clause is to process events,
>>> and one of those events is trying to flush a segment ... weird.
>>>
>>> Mike McCandless
>>>
>>> http://blog.mikemccandless.com
>>>
>>>
>>> On Tue, Dec 17, 2013 at 3:21 PM, Robert Muir <[email protected]> wrote:
>>>> I just want to point out this confused me a bit. the jenkins job says
>>>> java7 but it really runs java6.
>>>>
>>>> On Tue, Dec 17, 2013 at 1:43 PM, <[email protected]> wrote:
>>>>> Build: builds.flonkings.com/job/Lucene-4x-Linux-Java7-64-test-only/6907/
>>>>>
>>>>> 1 tests failed.
>>>>> REGRESSION:
>>>>> org.apache.lucene.index.TestIndexWriterWithThreads.testRollbackAndCommitWithThreads
>>>>>
>>>>> Error Message:
>>>>> Captured an uncaught exception in thread: Thread[id=193, name=Thread-133,
>>>>> state=RUNNABLE, group=TGRP-TestIndexWriterWithThreads]
>>>>>
>>>>> Stack Trace:
>>>>> com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an
>>>>> uncaught exception in thread: Thread[id=193, name=Thread-133,
>>>>> state=RUNNABLE, group=TGRP-TestIndexWriterWithThreads]
>>>>> Caused by: java.lang.RuntimeException:
>>>>> org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
>>>>> at __randomizedtesting.SeedInfo.seed([3FAF37E1AFFB2502]:0)
>>>>> at
>>>>> org.apache.lucene.index.TestIndexWriterWithThreads$1.run(TestIndexWriterWithThreads.java:619)
>>>>> Caused by: org.apache.lucene.store.AlreadyClosedException: this
>>>>> IndexWriter is closed
>>>>> at
>>>>> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:645)
>>>>> at
>>>>> org.apache.lucene.index.IndexWriter.numDeletedDocs(IndexWriter.java:622)
>>>>> at
>>>>> org.apache.lucene.index.IndexWriter.segString(IndexWriter.java:4265)
>>>>> at
>>>>> org.apache.lucene.index.IndexWriter.publishFlushedSegment(IndexWriter.java:2324)
>>>>> at
>>>>> org.apache.lucene.index.DocumentsWriterFlushQueue$FlushTicket.publishFlushedSegment(DocumentsWriterFlushQueue.java:198)
>>>>> at
>>>>> org.apache.lucene.index.DocumentsWriterFlushQueue$FlushTicket.finishFlush(DocumentsWriterFlushQueue.java:213)
>>>>> at
>>>>> org.apache.lucene.index.DocumentsWriterFlushQueue$SegmentFlushTicket.publish(DocumentsWriterFlushQueue.java:249)
>>>>> at
>>>>> org.apache.lucene.index.DocumentsWriterFlushQueue.innerPurge(DocumentsWriterFlushQueue.java:116)
>>>>> at
>>>>> org.apache.lucene.index.DocumentsWriterFlushQueue.forcePurge(DocumentsWriterFlushQueue.java:138)
>>>>> at
>>>>> org.apache.lucene.index.DocumentsWriter.purgeBuffer(DocumentsWriter.java:185)
>>>>> at
>>>>> org.apache.lucene.index.IndexWriter.purge(IndexWriter.java:4634)
>>>>> at
>>>>> org.apache.lucene.index.DocumentsWriter$ForcedPurgeEvent.process(DocumentsWriter.java:701)
>>>>> at
>>>>> org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4665)
>>>>> at
>>>>> org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4657)
>>>>> at
>>>>> org.apache.lucene.index.IndexWriter.closeInternal(IndexWriter.java:1067)
>>>>> at
>>>>> org.apache.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:2106)
>>>>> at
>>>>> org.apache.lucene.index.IndexWriter.rollback(IndexWriter.java:2024)
>>>>> at
>>>>> org.apache.lucene.index.TestIndexWriterWithThreads$1.run(TestIndexWriterWithThreads.java:575)
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Build Log:
>>>>> [...truncated 782 lines...]
>>>>> [junit4] Suite: org.apache.lucene.index.TestIndexWriterWithThreads
>>>>> [junit4] 2> 17 déc. 2013 19:41:23
>>>>> com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler
>>>>> uncaughtException
>>>>> [junit4] 2> ATTENTION: Uncaught exception in thread:
>>>>> Thread[Thread-133,5,TGRP-TestIndexWriterWithThreads]
>>>>> [junit4] 2> java.lang.RuntimeException:
>>>>> org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
>>>>> [junit4] 2> at
>>>>> __randomizedtesting.SeedInfo.seed([3FAF37E1AFFB2502]:0)
>>>>> [junit4] 2> at
>>>>> org.apache.lucene.index.TestIndexWriterWithThreads$1.run(TestIndexWriterWithThreads.java:619)
>>>>> [junit4] 2> Caused by:
>>>>> org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
>>>>> [junit4] 2> at
>>>>> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:645)
>>>>> [junit4] 2> at
>>>>> org.apache.lucene.index.IndexWriter.numDeletedDocs(IndexWriter.java:622)
>>>>> [junit4] 2> at
>>>>> org.apache.lucene.index.IndexWriter.segString(IndexWriter.java:4265)
>>>>> [junit4] 2> at
>>>>> org.apache.lucene.index.IndexWriter.publishFlushedSegment(IndexWriter.java:2324)
>>>>> [junit4] 2> at
>>>>> org.apache.lucene.index.DocumentsWriterFlushQueue$FlushTicket.publishFlushedSegment(DocumentsWriterFlushQueue.java:198)
>>>>> [junit4] 2> at
>>>>> org.apache.lucene.index.DocumentsWriterFlushQueue$FlushTicket.finishFlush(DocumentsWriterFlushQueue.java:213)
>>>>> [junit4] 2> at
>>>>> org.apache.lucene.index.DocumentsWriterFlushQueue$SegmentFlushTicket.publish(DocumentsWriterFlushQueue.java:249)
>>>>> [junit4] 2> at
>>>>> org.apache.lucene.index.DocumentsWriterFlushQueue.innerPurge(DocumentsWriterFlushQueue.java:116)
>>>>> [junit4] 2> at
>>>>> org.apache.lucene.index.DocumentsWriterFlushQueue.forcePurge(DocumentsWriterFlushQueue.java:138)
>>>>> [junit4] 2> at
>>>>> org.apache.lucene.index.DocumentsWriter.purgeBuffer(DocumentsWriter.java:185)
>>>>> [junit4] 2> at
>>>>> org.apache.lucene.index.IndexWriter.purge(IndexWriter.java:4634)
>>>>> [junit4] 2> at
>>>>> org.apache.lucene.index.DocumentsWriter$ForcedPurgeEvent.process(DocumentsWriter.java:701)
>>>>> [junit4] 2> at
>>>>> org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4665)
>>>>> [junit4] 2> at
>>>>> org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4657)
>>>>> [junit4] 2> at
>>>>> org.apache.lucene.index.IndexWriter.closeInternal(IndexWriter.java:1067)
>>>>> [junit4] 2> at
>>>>> org.apache.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:2106)
>>>>> [junit4] 2> at
>>>>> org.apache.lucene.index.IndexWriter.rollback(IndexWriter.java:2024)
>>>>> [junit4] 2> at
>>>>> org.apache.lucene.index.TestIndexWriterWithThreads$1.run(TestIndexWriterWithThreads.java:575)
>>>>> [junit4] 2> 17 déc. 2013 19:41:23
>>>>> com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler
>>>>> uncaughtException
>>>>> [junit4] 2> ATTENTION: Uncaught exception in thread:
>>>>> Thread[Thread-135,5,TGRP-TestIndexWriterWithThreads]
>>>>> [junit4] 2> java.lang.RuntimeException:
>>>>> org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
>>>>> [junit4] 2> at
>>>>> __randomizedtesting.SeedInfo.seed([3FAF37E1AFFB2502]:0)
>>>>> [junit4] 2> at
>>>>> org.apache.lucene.index.TestIndexWriterWithThreads$1.run(TestIndexWriterWithThreads.java:619)
>>>>> [junit4] 2> Caused by:
>>>>> org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
>>>>> [junit4] 2> at
>>>>> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:645)
>>>>> [junit4] 2> at
>>>>> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:659)
>>>>> [junit4] 2> at
>>>>> org.apache.lucene.index.IndexWriter.rollback(IndexWriter.java:2018)
>>>>> [junit4] 2> at
>>>>> org.apache.lucene.index.TestIndexWriterWithThreads$1.run(TestIndexWriterWithThreads.java:575)
>>>>> [junit4] 2> NOTE: reproduce with: ant test
>>>>> -Dtestcase=TestIndexWriterWithThreads
>>>>> -Dtests.method=testRollbackAndCommitWithThreads
>>>>> -Dtests.seed=3FAF37E1AFFB2502 -Dtests.slow=true -Dtests.locale=fr_FR
>>>>> -Dtests.timezone=Africa/Malabo -Dtests.file.encoding=UTF-8
>>>>> [junit4] ERROR 0.27s J5 |
>>>>> TestIndexWriterWithThreads.testRollbackAndCommitWithThreads <<<
>>>>> [junit4] > Throwable #1: java.lang.AssertionError
>>>>> [junit4] > at
>>>>> org.apache.lucene.index.TestIndexWriterWithThreads.testRollbackAndCommitWithThreads(TestIndexWriterWithThreads.java:631)
>>>>> [junit4] > at java.lang.Thread.run(Thread.java:662)Throwable
>>>>> #2: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured
>>>>> an uncaught exception in thread: Thread[id=193, name=Thread-133,
>>>>> state=RUNNABLE, group=TGRP-TestIndexWriterWithThreads]
>>>>> [junit4] > Caused by: java.lang.RuntimeException:
>>>>> org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
>>>>> [junit4] > at
>>>>> __randomizedtesting.SeedInfo.seed([3FAF37E1AFFB2502]:0)
>>>>> [junit4] > at
>>>>> org.apache.lucene.index.TestIndexWriterWithThreads$1.run(TestIndexWriterWithThreads.java:619)
>>>>> [junit4] > Caused by:
>>>>> org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
>>>>> [junit4] > at
>>>>> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:645)
>>>>> [junit4] > at
>>>>> org.apache.lucene.index.IndexWriter.numDeletedDocs(IndexWriter.java:622)
>>>>> [junit4] > at
>>>>> org.apache.lucene.index.IndexWriter.segString(IndexWriter.java:4265)
>>>>> [junit4] > at
>>>>> org.apache.lucene.index.IndexWriter.publishFlushedSegment(IndexWriter.java:2324)
>>>>> [junit4] > at
>>>>> org.apache.lucene.index.DocumentsWriterFlushQueue$FlushTicket.publishFlushedSegment(DocumentsWriterFlushQueue.java:198)
>>>>> [junit4] > at
>>>>> org.apache.lucene.index.DocumentsWriterFlushQueue$FlushTicket.finishFlush(DocumentsWriterFlushQueue.java:213)
>>>>> [junit4] > at
>>>>> org.apache.lucene.index.DocumentsWriterFlushQueue$SegmentFlushTicket.publish(DocumentsWriterFlushQueue.java:249)
>>>>> [junit4] > at
>>>>> org.apache.lucene.index.DocumentsWriterFlushQueue.innerPurge(DocumentsWriterFlushQueue.java:116)
>>>>> [junit4] > at
>>>>> org.apache.lucene.index.DocumentsWriterFlushQueue.forcePurge(DocumentsWriterFlushQueue.java:138)
>>>>> [junit4] > at
>>>>> org.apache.lucene.index.DocumentsWriter.purgeBuffer(DocumentsWriter.java:185)
>>>>> [junit4] > at
>>>>> org.apache.lucene.index.IndexWriter.purge(IndexWriter.java:4634)
>>>>> [junit4] > at
>>>>> org.apache.lucene.index.DocumentsWriter$ForcedPurgeEvent.process(DocumentsWriter.java:701)
>>>>> [junit4] > at
>>>>> org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4665)
>>>>> [junit4] > at
>>>>> org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4657)
>>>>> [junit4] > at
>>>>> org.apache.lucene.index.IndexWriter.closeInternal(IndexWriter.java:1067)
>>>>> [junit4] > at
>>>>> org.apache.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:2106)
>>>>> [junit4] > at
>>>>> org.apache.lucene.index.IndexWriter.rollback(IndexWriter.java:2024)
>>>>> [junit4] > at
>>>>> org.apache.lucene.index.TestIndexWriterWithThreads$1.run(TestIndexWriterWithThreads.java:575)Throwable
>>>>> #3: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured
>>>>> an uncaught exception in thread: Thread[id=195, name=Thread-135,
>>>>> state=RUNNABLE, group=TGRP-TestIndexWriterWithThreads]
>>>>> [junit4] > Caused by: java.lang.RuntimeException:
>>>>> org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
>>>>> [junit4] > at
>>>>> __randomizedtesting.SeedInfo.seed([3FAF37E1AFFB2502]:0)
>>>>> [junit4] > at
>>>>> org.apache.lucene.index.TestIndexWriterWithThreads$1.run(TestIndexWriterWithThreads.java:619)
>>>>> [junit4] > Caused by:
>>>>> org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
>>>>> [junit4] > at
>>>>> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:645)
>>>>> [junit4] > at
>>>>> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:659)
>>>>> [junit4] > at
>>>>> org.apache.lucene.index.IndexWriter.rollback(IndexWriter.java:2018)
>>>>> [junit4] > at
>>>>> org.apache.lucene.index.TestIndexWriterWithThreads$1.run(TestIndexWriterWithThreads.java:575)
>>>>> [junit4] 2> NOTE: test params are: codec=Lucene45,
>>>>> sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {field=DFR
>>>>> I(F)L3(800.0), body=DFR I(n)B1, titleTokenized=DFR I(F)B1}, locale=fr_FR,
>>>>> timezone=Africa/Malabo
>>>>> [junit4] 2> NOTE: Linux 3.2.0-26-generic amd64/Sun Microsystems Inc.
>>>>> 1.6.0_45 (64-bit)/cpus=8,threads=1,free=78129256,total=170721280
>>>>> [junit4] 2> NOTE: All tests run in this JVM: [Test2BTerms,
>>>>> TestCompoundFile, TestAtomicUpdate, TestByteSlices, TestWindowsMMap,
>>>>> TestEliasFanoDocIdSet, TestSnapshotDeletionPolicy,
>>>>> TestCrashCausesCorruptIndex, TestBooleanOr, TestCachingTokenFilter,
>>>>> TestDocIdSet, TestFieldInfos, TestAssertions, TestBytesStore, TestSearch,
>>>>> TestPrefixInBooleanQuery, TestDocument, TestRateLimiter,
>>>>> TestEarlyTermination, TestPositionIncrement, TestCachingCollector,
>>>>> TestSpecialOperations, TestReaderClosed, TestPerFieldDocValuesFormat,
>>>>> TestFSTs, TestIndexWriterWithThreads]
>>>>> [junit4] Completed on J5 in 6.95s, 12 tests, 1 error <<< FAILURES!
>>>>>
>>>>> [...truncated 636 lines...]
>>>>> BUILD FAILED
>>>>> /var/lib/jenkins/workspace/Lucene-4x-Linux-Java7-64-test-only/checkout/lucene/build.xml:49:
>>>>> The following error occurred while executing this line:
>>>>> /var/lib/jenkins/workspace/Lucene-4x-Linux-Java7-64-test-only/checkout/lucene/common-build.xml:1307:
>>>>> The following error occurred while executing this line:
>>>>> /var/lib/jenkins/workspace/Lucene-4x-Linux-Java7-64-test-only/checkout/lucene/common-build.xml:940:
>>>>> There were test failures: 379 suites, 2758 tests, 1 error, 81 ignored
>>>>> (68 assumptions)
>>>>>
>>>>> Total time: 5 minutes 33 seconds
>>>>> Build step 'Invoke Ant' marked build as failure
>>>>> Recording test results
>>>>> Email was triggered for: Failure
>>>>> Sending email for trigger: Failure
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: [email protected]
>>>>> For additional commands, e-mail: [email protected]
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: [email protected]
>>>> For additional commands, e-mail: [email protected]
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: [email protected]
>>> For additional commands, e-mail: [email protected]
>>>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
