I opened https://issues.apache.org/jira/browse/LUCENE-5374
On Wed, Dec 18, 2013 at 7:26 PM, Michael McCandless <[email protected]> wrote: > Patch seems to work! I'm up to 827 iters and no fail ... previously > it would fail much more quickly. > > +1 > > Mike McCandless > > http://blog.mikemccandless.com > > > On Wed, Dec 18, 2013 at 12:19 PM, Simon Willnauer <[email protected]> wrote: >> mike can you try to reproduce it - i can't reproduce it with this patch: >> >> Index: lucene/core/src/java/org/apache/lucene/index/IndexWriter.java >> =================================================================== >> --- lucene/core/src/java/org/apache/lucene/index/IndexWriter.java >> (revision 1551902) >> +++ lucene/core/src/java/org/apache/lucene/index/IndexWriter.java (working >> copy) >> @@ -1029,7 +1029,7 @@ >> if (doFlush) { >> commitInternal(); >> } >> - >> + processEvents(false, true); >> synchronized(this) { >> // commitInternal calls ReaderPool.commit, which >> // writes any pending liveDocs from ReaderPool, so >> @@ -1064,7 +1064,6 @@ >> } >> // finally, restore interrupt status: >> if (interrupted) Thread.currentThread().interrupt(); >> - processEvents(false, true); >> } >> } >> >> >> >> I think we call it too late though it needs to be called fare more above >> I will open an issue if your beast is happy too? >> >> simon >> >> On Wed, Dec 18, 2013 at 2:33 PM, Simon Willnauer <[email protected]> wrote: >>> I have a fix for it - will open an issue later >>> >>> On Wed, Dec 18, 2013 at 11:44 AM, Simon Willnauer <[email protected]> wrote: >>>> it's not trying to flush a segment it tries to publish it. It already >>>> is flushed but still scary. I will update the job description @Robert >>>> >>>> On Tue, Dec 17, 2013 at 9:27 PM, Michael McCandless >>>> <[email protected]> wrote: >>>>> It actually repros with java7 too, if you beast it. >>>>> >>>>> It's a scary failure: it seems like, after closing, the very last >>>>> thing IW does in closeInternal's finally clause is to process events, >>>>> and one of those events is trying to flush a segment ... weird. >>>>> >>>>> Mike McCandless >>>>> >>>>> http://blog.mikemccandless.com >>>>> >>>>> >>>>> On Tue, Dec 17, 2013 at 3:21 PM, Robert Muir <[email protected]> wrote: >>>>>> I just want to point out this confused me a bit. the jenkins job says >>>>>> java7 but it really runs java6. >>>>>> >>>>>> On Tue, Dec 17, 2013 at 1:43 PM, <[email protected]> wrote: >>>>>>> Build: builds.flonkings.com/job/Lucene-4x-Linux-Java7-64-test-only/6907/ >>>>>>> >>>>>>> 1 tests failed. >>>>>>> REGRESSION: >>>>>>> org.apache.lucene.index.TestIndexWriterWithThreads.testRollbackAndCommitWithThreads >>>>>>> >>>>>>> Error Message: >>>>>>> Captured an uncaught exception in thread: Thread[id=193, >>>>>>> name=Thread-133, state=RUNNABLE, group=TGRP-TestIndexWriterWithThreads] >>>>>>> >>>>>>> Stack Trace: >>>>>>> com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an >>>>>>> uncaught exception in thread: Thread[id=193, name=Thread-133, >>>>>>> state=RUNNABLE, group=TGRP-TestIndexWriterWithThreads] >>>>>>> Caused by: java.lang.RuntimeException: >>>>>>> org.apache.lucene.store.AlreadyClosedException: this IndexWriter is >>>>>>> closed >>>>>>> at __randomizedtesting.SeedInfo.seed([3FAF37E1AFFB2502]:0) >>>>>>> at >>>>>>> org.apache.lucene.index.TestIndexWriterWithThreads$1.run(TestIndexWriterWithThreads.java:619) >>>>>>> Caused by: org.apache.lucene.store.AlreadyClosedException: this >>>>>>> IndexWriter is closed >>>>>>> at >>>>>>> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:645) >>>>>>> at >>>>>>> org.apache.lucene.index.IndexWriter.numDeletedDocs(IndexWriter.java:622) >>>>>>> at >>>>>>> org.apache.lucene.index.IndexWriter.segString(IndexWriter.java:4265) >>>>>>> at >>>>>>> org.apache.lucene.index.IndexWriter.publishFlushedSegment(IndexWriter.java:2324) >>>>>>> at >>>>>>> org.apache.lucene.index.DocumentsWriterFlushQueue$FlushTicket.publishFlushedSegment(DocumentsWriterFlushQueue.java:198) >>>>>>> at >>>>>>> org.apache.lucene.index.DocumentsWriterFlushQueue$FlushTicket.finishFlush(DocumentsWriterFlushQueue.java:213) >>>>>>> at >>>>>>> org.apache.lucene.index.DocumentsWriterFlushQueue$SegmentFlushTicket.publish(DocumentsWriterFlushQueue.java:249) >>>>>>> at >>>>>>> org.apache.lucene.index.DocumentsWriterFlushQueue.innerPurge(DocumentsWriterFlushQueue.java:116) >>>>>>> at >>>>>>> org.apache.lucene.index.DocumentsWriterFlushQueue.forcePurge(DocumentsWriterFlushQueue.java:138) >>>>>>> at >>>>>>> org.apache.lucene.index.DocumentsWriter.purgeBuffer(DocumentsWriter.java:185) >>>>>>> at >>>>>>> org.apache.lucene.index.IndexWriter.purge(IndexWriter.java:4634) >>>>>>> at >>>>>>> org.apache.lucene.index.DocumentsWriter$ForcedPurgeEvent.process(DocumentsWriter.java:701) >>>>>>> at >>>>>>> org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4665) >>>>>>> at >>>>>>> org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4657) >>>>>>> at >>>>>>> org.apache.lucene.index.IndexWriter.closeInternal(IndexWriter.java:1067) >>>>>>> at >>>>>>> org.apache.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:2106) >>>>>>> at >>>>>>> org.apache.lucene.index.IndexWriter.rollback(IndexWriter.java:2024) >>>>>>> at >>>>>>> org.apache.lucene.index.TestIndexWriterWithThreads$1.run(TestIndexWriterWithThreads.java:575) >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> Build Log: >>>>>>> [...truncated 782 lines...] >>>>>>> [junit4] Suite: org.apache.lucene.index.TestIndexWriterWithThreads >>>>>>> [junit4] 2> 17 déc. 2013 19:41:23 >>>>>>> com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler >>>>>>> uncaughtException >>>>>>> [junit4] 2> ATTENTION: Uncaught exception in thread: >>>>>>> Thread[Thread-133,5,TGRP-TestIndexWriterWithThreads] >>>>>>> [junit4] 2> java.lang.RuntimeException: >>>>>>> org.apache.lucene.store.AlreadyClosedException: this IndexWriter is >>>>>>> closed >>>>>>> [junit4] 2> at >>>>>>> __randomizedtesting.SeedInfo.seed([3FAF37E1AFFB2502]:0) >>>>>>> [junit4] 2> at >>>>>>> org.apache.lucene.index.TestIndexWriterWithThreads$1.run(TestIndexWriterWithThreads.java:619) >>>>>>> [junit4] 2> Caused by: >>>>>>> org.apache.lucene.store.AlreadyClosedException: this IndexWriter is >>>>>>> closed >>>>>>> [junit4] 2> at >>>>>>> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:645) >>>>>>> [junit4] 2> at >>>>>>> org.apache.lucene.index.IndexWriter.numDeletedDocs(IndexWriter.java:622) >>>>>>> [junit4] 2> at >>>>>>> org.apache.lucene.index.IndexWriter.segString(IndexWriter.java:4265) >>>>>>> [junit4] 2> at >>>>>>> org.apache.lucene.index.IndexWriter.publishFlushedSegment(IndexWriter.java:2324) >>>>>>> [junit4] 2> at >>>>>>> org.apache.lucene.index.DocumentsWriterFlushQueue$FlushTicket.publishFlushedSegment(DocumentsWriterFlushQueue.java:198) >>>>>>> [junit4] 2> at >>>>>>> org.apache.lucene.index.DocumentsWriterFlushQueue$FlushTicket.finishFlush(DocumentsWriterFlushQueue.java:213) >>>>>>> [junit4] 2> at >>>>>>> org.apache.lucene.index.DocumentsWriterFlushQueue$SegmentFlushTicket.publish(DocumentsWriterFlushQueue.java:249) >>>>>>> [junit4] 2> at >>>>>>> org.apache.lucene.index.DocumentsWriterFlushQueue.innerPurge(DocumentsWriterFlushQueue.java:116) >>>>>>> [junit4] 2> at >>>>>>> org.apache.lucene.index.DocumentsWriterFlushQueue.forcePurge(DocumentsWriterFlushQueue.java:138) >>>>>>> [junit4] 2> at >>>>>>> org.apache.lucene.index.DocumentsWriter.purgeBuffer(DocumentsWriter.java:185) >>>>>>> [junit4] 2> at >>>>>>> org.apache.lucene.index.IndexWriter.purge(IndexWriter.java:4634) >>>>>>> [junit4] 2> at >>>>>>> org.apache.lucene.index.DocumentsWriter$ForcedPurgeEvent.process(DocumentsWriter.java:701) >>>>>>> [junit4] 2> at >>>>>>> org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4665) >>>>>>> [junit4] 2> at >>>>>>> org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4657) >>>>>>> [junit4] 2> at >>>>>>> org.apache.lucene.index.IndexWriter.closeInternal(IndexWriter.java:1067) >>>>>>> [junit4] 2> at >>>>>>> org.apache.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:2106) >>>>>>> [junit4] 2> at >>>>>>> org.apache.lucene.index.IndexWriter.rollback(IndexWriter.java:2024) >>>>>>> [junit4] 2> at >>>>>>> org.apache.lucene.index.TestIndexWriterWithThreads$1.run(TestIndexWriterWithThreads.java:575) >>>>>>> [junit4] 2> 17 déc. 2013 19:41:23 >>>>>>> com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler >>>>>>> uncaughtException >>>>>>> [junit4] 2> ATTENTION: Uncaught exception in thread: >>>>>>> Thread[Thread-135,5,TGRP-TestIndexWriterWithThreads] >>>>>>> [junit4] 2> java.lang.RuntimeException: >>>>>>> org.apache.lucene.store.AlreadyClosedException: this IndexWriter is >>>>>>> closed >>>>>>> [junit4] 2> at >>>>>>> __randomizedtesting.SeedInfo.seed([3FAF37E1AFFB2502]:0) >>>>>>> [junit4] 2> at >>>>>>> org.apache.lucene.index.TestIndexWriterWithThreads$1.run(TestIndexWriterWithThreads.java:619) >>>>>>> [junit4] 2> Caused by: >>>>>>> org.apache.lucene.store.AlreadyClosedException: this IndexWriter is >>>>>>> closed >>>>>>> [junit4] 2> at >>>>>>> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:645) >>>>>>> [junit4] 2> at >>>>>>> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:659) >>>>>>> [junit4] 2> at >>>>>>> org.apache.lucene.index.IndexWriter.rollback(IndexWriter.java:2018) >>>>>>> [junit4] 2> at >>>>>>> org.apache.lucene.index.TestIndexWriterWithThreads$1.run(TestIndexWriterWithThreads.java:575) >>>>>>> [junit4] 2> NOTE: reproduce with: ant test >>>>>>> -Dtestcase=TestIndexWriterWithThreads >>>>>>> -Dtests.method=testRollbackAndCommitWithThreads >>>>>>> -Dtests.seed=3FAF37E1AFFB2502 -Dtests.slow=true -Dtests.locale=fr_FR >>>>>>> -Dtests.timezone=Africa/Malabo -Dtests.file.encoding=UTF-8 >>>>>>> [junit4] ERROR 0.27s J5 | >>>>>>> TestIndexWriterWithThreads.testRollbackAndCommitWithThreads <<< >>>>>>> [junit4] > Throwable #1: java.lang.AssertionError >>>>>>> [junit4] > at >>>>>>> org.apache.lucene.index.TestIndexWriterWithThreads.testRollbackAndCommitWithThreads(TestIndexWriterWithThreads.java:631) >>>>>>> [junit4] > at >>>>>>> java.lang.Thread.run(Thread.java:662)Throwable #2: >>>>>>> com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an >>>>>>> uncaught exception in thread: Thread[id=193, name=Thread-133, >>>>>>> state=RUNNABLE, group=TGRP-TestIndexWriterWithThreads] >>>>>>> [junit4] > Caused by: java.lang.RuntimeException: >>>>>>> org.apache.lucene.store.AlreadyClosedException: this IndexWriter is >>>>>>> closed >>>>>>> [junit4] > at >>>>>>> __randomizedtesting.SeedInfo.seed([3FAF37E1AFFB2502]:0) >>>>>>> [junit4] > at >>>>>>> org.apache.lucene.index.TestIndexWriterWithThreads$1.run(TestIndexWriterWithThreads.java:619) >>>>>>> [junit4] > Caused by: >>>>>>> org.apache.lucene.store.AlreadyClosedException: this IndexWriter is >>>>>>> closed >>>>>>> [junit4] > at >>>>>>> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:645) >>>>>>> [junit4] > at >>>>>>> org.apache.lucene.index.IndexWriter.numDeletedDocs(IndexWriter.java:622) >>>>>>> [junit4] > at >>>>>>> org.apache.lucene.index.IndexWriter.segString(IndexWriter.java:4265) >>>>>>> [junit4] > at >>>>>>> org.apache.lucene.index.IndexWriter.publishFlushedSegment(IndexWriter.java:2324) >>>>>>> [junit4] > at >>>>>>> org.apache.lucene.index.DocumentsWriterFlushQueue$FlushTicket.publishFlushedSegment(DocumentsWriterFlushQueue.java:198) >>>>>>> [junit4] > at >>>>>>> org.apache.lucene.index.DocumentsWriterFlushQueue$FlushTicket.finishFlush(DocumentsWriterFlushQueue.java:213) >>>>>>> [junit4] > at >>>>>>> org.apache.lucene.index.DocumentsWriterFlushQueue$SegmentFlushTicket.publish(DocumentsWriterFlushQueue.java:249) >>>>>>> [junit4] > at >>>>>>> org.apache.lucene.index.DocumentsWriterFlushQueue.innerPurge(DocumentsWriterFlushQueue.java:116) >>>>>>> [junit4] > at >>>>>>> org.apache.lucene.index.DocumentsWriterFlushQueue.forcePurge(DocumentsWriterFlushQueue.java:138) >>>>>>> [junit4] > at >>>>>>> org.apache.lucene.index.DocumentsWriter.purgeBuffer(DocumentsWriter.java:185) >>>>>>> [junit4] > at >>>>>>> org.apache.lucene.index.IndexWriter.purge(IndexWriter.java:4634) >>>>>>> [junit4] > at >>>>>>> org.apache.lucene.index.DocumentsWriter$ForcedPurgeEvent.process(DocumentsWriter.java:701) >>>>>>> [junit4] > at >>>>>>> org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4665) >>>>>>> [junit4] > at >>>>>>> org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4657) >>>>>>> [junit4] > at >>>>>>> org.apache.lucene.index.IndexWriter.closeInternal(IndexWriter.java:1067) >>>>>>> [junit4] > at >>>>>>> org.apache.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:2106) >>>>>>> [junit4] > at >>>>>>> org.apache.lucene.index.IndexWriter.rollback(IndexWriter.java:2024) >>>>>>> [junit4] > at >>>>>>> org.apache.lucene.index.TestIndexWriterWithThreads$1.run(TestIndexWriterWithThreads.java:575)Throwable >>>>>>> #3: com.carrotsearch.randomizedtesting.UncaughtExceptionError: >>>>>>> Captured an uncaught exception in thread: Thread[id=195, >>>>>>> name=Thread-135, state=RUNNABLE, group=TGRP-TestIndexWriterWithThreads] >>>>>>> [junit4] > Caused by: java.lang.RuntimeException: >>>>>>> org.apache.lucene.store.AlreadyClosedException: this IndexWriter is >>>>>>> closed >>>>>>> [junit4] > at >>>>>>> __randomizedtesting.SeedInfo.seed([3FAF37E1AFFB2502]:0) >>>>>>> [junit4] > at >>>>>>> org.apache.lucene.index.TestIndexWriterWithThreads$1.run(TestIndexWriterWithThreads.java:619) >>>>>>> [junit4] > Caused by: >>>>>>> org.apache.lucene.store.AlreadyClosedException: this IndexWriter is >>>>>>> closed >>>>>>> [junit4] > at >>>>>>> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:645) >>>>>>> [junit4] > at >>>>>>> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:659) >>>>>>> [junit4] > at >>>>>>> org.apache.lucene.index.IndexWriter.rollback(IndexWriter.java:2018) >>>>>>> [junit4] > at >>>>>>> org.apache.lucene.index.TestIndexWriterWithThreads$1.run(TestIndexWriterWithThreads.java:575) >>>>>>> [junit4] 2> NOTE: test params are: codec=Lucene45, >>>>>>> sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {field=DFR >>>>>>> I(F)L3(800.0), body=DFR I(n)B1, titleTokenized=DFR I(F)B1}, >>>>>>> locale=fr_FR, timezone=Africa/Malabo >>>>>>> [junit4] 2> NOTE: Linux 3.2.0-26-generic amd64/Sun Microsystems >>>>>>> Inc. 1.6.0_45 (64-bit)/cpus=8,threads=1,free=78129256,total=170721280 >>>>>>> [junit4] 2> NOTE: All tests run in this JVM: [Test2BTerms, >>>>>>> TestCompoundFile, TestAtomicUpdate, TestByteSlices, TestWindowsMMap, >>>>>>> TestEliasFanoDocIdSet, TestSnapshotDeletionPolicy, >>>>>>> TestCrashCausesCorruptIndex, TestBooleanOr, TestCachingTokenFilter, >>>>>>> TestDocIdSet, TestFieldInfos, TestAssertions, TestBytesStore, >>>>>>> TestSearch, TestPrefixInBooleanQuery, TestDocument, TestRateLimiter, >>>>>>> TestEarlyTermination, TestPositionIncrement, TestCachingCollector, >>>>>>> TestSpecialOperations, TestReaderClosed, TestPerFieldDocValuesFormat, >>>>>>> TestFSTs, TestIndexWriterWithThreads] >>>>>>> [junit4] Completed on J5 in 6.95s, 12 tests, 1 error <<< FAILURES! >>>>>>> >>>>>>> [...truncated 636 lines...] >>>>>>> BUILD FAILED >>>>>>> /var/lib/jenkins/workspace/Lucene-4x-Linux-Java7-64-test-only/checkout/lucene/build.xml:49: >>>>>>> The following error occurred while executing this line: >>>>>>> /var/lib/jenkins/workspace/Lucene-4x-Linux-Java7-64-test-only/checkout/lucene/common-build.xml:1307: >>>>>>> The following error occurred while executing this line: >>>>>>> /var/lib/jenkins/workspace/Lucene-4x-Linux-Java7-64-test-only/checkout/lucene/common-build.xml:940: >>>>>>> There were test failures: 379 suites, 2758 tests, 1 error, 81 ignored >>>>>>> (68 assumptions) >>>>>>> >>>>>>> Total time: 5 minutes 33 seconds >>>>>>> Build step 'Invoke Ant' marked build as failure >>>>>>> Recording test results >>>>>>> Email was triggered for: Failure >>>>>>> Sending email for trigger: Failure >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> --------------------------------------------------------------------- >>>>>>> To unsubscribe, e-mail: [email protected] >>>>>>> For additional commands, e-mail: [email protected] >>>>>> >>>>>> --------------------------------------------------------------------- >>>>>> To unsubscribe, e-mail: [email protected] >>>>>> For additional commands, e-mail: [email protected] >>>>>> >>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe, e-mail: [email protected] >>>>> For additional commands, e-mail: [email protected] >>>>> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
