[ https://issues.apache.org/jira/browse/LUCENE-8639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16744168#comment-16744168 ]
ASF subversion and git services commented on LUCENE-8639: --------------------------------------------------------- Commit 0d306b40b77ba85d1602c60621e93c5b190b6ef9 in lucene-solr's branch refs/heads/branch_7x from Simon Willnauer [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=0d306b4 ] LUCENE-8639: Prevent new threadstates from being created while we cut over to a new delete queue (#535) This prevents an edge case where suddenly a lot of threads start indexing while we carry over sequence ids from the previous to the new delete queue. We now lock creation of new thread states for a very short time until we created and assigned a new delete queue. > SeqNo accounting in IW is broken if many threads start indexing while we > flush. > ------------------------------------------------------------------------------- > > Key: LUCENE-8639 > URL: https://issues.apache.org/jira/browse/LUCENE-8639 > Project: Lucene - Core > Issue Type: Improvement > Reporter: Simon Willnauer > Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > While this is rare in the wild we have a test failure that shows that our > seqNo accounting is broken when we carry over seqNo to a new delete queue. > We had this test-failure: > {noformat} > 6:06:08 [junit4] Suite: org.apache.lucene.index.TestIndexTooManyDocs > 16:06:08 [junit4] 2> ??? 14, 2019 9:05:46 ? > com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler > uncaughtException > 16:06:08 [junit4] 2> WARNING: Uncaught exception in thread: > Thread[Thread-8,5,TGRP-TestIndexTooManyDocs] > 16:06:08 [junit4] 2> java.lang.AssertionError: seqNo=7 vs maxSeqNo=6 > 16:06:08 [junit4] 2> at > __randomizedtesting.SeedInfo.seed([43B7C75B765AFEBD]:0) > 16:06:08 [junit4] 2> at > org.apache.lucene.index.DocumentsWriterDeleteQueue.getNextSequenceNumber(DocumentsWriterDeleteQueue.java:482) > 16:06:08 [junit4] 2> at > org.apache.lucene.index.DocumentsWriterDeleteQueue.add(DocumentsWriterDeleteQueue.java:168) > 16:06:08 [junit4] 2> at > org.apache.lucene.index.DocumentsWriterDeleteQueue.add(DocumentsWriterDeleteQueue.java:146) > 16:06:08 [junit4] 2> at > org.apache.lucene.index.DocumentsWriterPerThread.finishDocument(DocumentsWriterPerThread.java:362) > 16:06:08 [junit4] 2> at > org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:264) > 16:06:08 [junit4] 2> at > org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:494) > 16:06:08 [junit4] 2> at > org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1594) > 16:06:08 [junit4] 2> at > org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1586) > 16:06:08 [junit4] 2> at > org.apache.lucene.index.TestIndexTooManyDocs.lambda$testIndexTooManyDocs$0(TestIndexTooManyDocs.java:70) > 16:06:08 [junit4] 2> at java.lang.Thread.run(Thread.java:748) > 16:06:08 [junit4] 2> > 16:06:08 [junit4] 2> ??? 14, 2019 9:05:46 ? > com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler > uncaughtException > 16:06:08 [junit4] 2> WARNING: Uncaught exception in thread: > Thread[Thread-9,5,TGRP-TestIndexTooManyDocs] > 16:06:08 [junit4] 2> java.lang.AssertionError: seqNo=6 vs maxSeqNo=6 > 16:06:08 [junit4] 2> at > __randomizedtesting.SeedInfo.seed([43B7C75B765AFEBD]:0) > 16:06:08 [junit4] 2> at > org.apache.lucene.index.DocumentsWriterDeleteQueue.getNextSequenceNumber(DocumentsWriterDeleteQueue.java:482) > 16:06:08 [junit4] 2> at > org.apache.lucene.index.DocumentsWriterDeleteQueue.add(DocumentsWriterDeleteQueue.java:168) > 16:06:08 [junit4] 2> at > org.apache.lucene.index.DocumentsWriterDeleteQueue.add(DocumentsWriterDeleteQueue.java:146) > 16:06:08 [junit4] 2> at > org.apache.lucene.index.DocumentsWriterPerThread.finishDocument(DocumentsWriterPerThread.java:362) > 16:06:08 [junit4] 2> at > org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:264) > 16:06:08 [junit4] 2> at > org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:494) > 16:06:08 [junit4] 2> at > org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1594) > 16:06:08 [junit4] 2> at > org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1586) > 16:06:08 [junit4] 2> at > org.apache.lucene.index.TestIndexTooManyDocs.lambda$testIndexTooManyDocs$0(TestIndexTooManyDocs.java:70) > 16:06:08 [junit4] 2> at java.lang.Thread.run(Thread.java:748) > 16:06:08 [junit4] 2> > 16:06:08 [junit4] 2> ??? 14, 2019 11:05:45 ? > com.carrotsearch.randomizedtesting.ThreadLeakControl$2 evaluate > 16:06:08 [junit4] 2> WARNING: Suite execution timed out: > org.apache.lucene.index.TestIndexTooManyDocs > 16:06:08 [junit4] 2> 1) Thread[id=20, > name=SUITE-TestIndexTooManyDocs-seed#[43B7C75B765AFEBD], state=RUNNABLE, > group=TGRP-TestIndexTooManyDocs] > 16:06:08 [junit4] 2> at > java.lang.Thread.getStackTrace(Thread.java:1559) > 16:06:08 [junit4] 2> at > com.carrotsearch.randomizedtesting.ThreadLeakControl$4.run(ThreadLeakControl.java:696) > 16:06:08 [junit4] 2> at > com.carrotsearch.randomizedtesting.ThreadLeakControl$4.run(ThreadLeakControl.java:693) > 16:06:08 [junit4] 2> at > java.security.AccessController.doPrivileged(Native Method) > 16:06:08 [junit4] 2> at > com.carrotsearch.randomizedtesting.ThreadLeakControl.getStackTrace(ThreadLeakControl.java:693) > 16:06:08 [junit4] 2> at > com.carrotsearch.randomizedtesting.ThreadLeakControl.getThreadsWithTraces(ThreadLeakControl.java:709) > 16:06:08 [junit4] 2> at > com.carrotsearch.randomizedtesting.ThreadLeakControl.formatThreadStacksFull(ThreadLeakControl.java:689) > 16:06:08 [junit4] 2> at > com.carrotsearch.randomizedtesting.ThreadLeakControl.access$1000(ThreadLeakControl.java:65) > 16:06:08 [junit4] 2> at > com.carrotsearch.randomizedtesting.ThreadLeakControl$2.evaluate(ThreadLeakControl.java:415) > 16:06:08 [junit4] 2> at > com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:708) > 16:06:08 [junit4] 2> at > com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:138) > 16:06:08 [junit4] 2> at > com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:629) > 16:06:08 [junit4] 2> 2) Thread[id=23, name=Thread-2, state=BLOCKED, > group=TGRP-TestIndexTooManyDocs] > 16:06:08 [junit4] 2> at > org.apache.lucene.index.IndexWriter.nrtIsCurrent(IndexWriter.java:4942) > 16:06:08 [junit4] 2> at > org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:290) > 16:06:08 [junit4] 2> at > org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:269) > 16:06:08 [junit4] 2> at > org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:259) > 16:06:08 [junit4] 2> at > org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:140) > 16:06:08 [junit4] 2> at > org.apache.lucene.index.TestIndexTooManyDocs.lambda$testIndexTooManyDocs$1(TestIndexTooManyDocs.java:86) > 16:06:08 [junit4] 2> at > org.apache.lucene.index.TestIndexTooManyDocs$$Lambda$8/1612361439.run(Unknown > Source) > 16:06:08 [junit4] 2> at java.lang.Thread.run(Thread.java:748) > 16:06:08 [junit4] 2> 3) Thread[id=18, name=JUnit4-serializer-daemon, > state=TIMED_WAITING, group=main] > 16:06:08 [junit4] 2> at java.lang.Thread.sleep(Native Method) > 16:06:08 [junit4] 2> at > com.carrotsearch.ant.tasks.junit4.events.Serializer$1.run(Serializer.java:50) > 16:06:08 [junit4] 2> 4) Thread[id=22, name=Thread-1, state=RUNNABLE, > group=TGRP-TestIndexTooManyDocs] > 16:06:08 [junit4] 2> at > org.apache.lucene.index.ReaderPool.anyDocValuesChanges(ReaderPool.java:354) > 16:06:08 [junit4] 2> at > org.apache.lucene.index.IndexWriter.nrtIsCurrent(IndexWriter.java:4946) > 16:06:08 [junit4] 2> at > org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:290) > 16:06:08 [junit4] 2> at > org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:269) > 16:06:08 [junit4] 2> at > org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:259) > 16:06:08 [junit4] 2> at > org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:140) > 16:06:08 [junit4] 2> at > org.apache.lucene.index.TestIndexTooManyDocs.lambda$testIndexTooManyDocs$1(TestIndexTooManyDocs.java:86) > 16:06:08 [junit4] 2> at > org.apache.lucene.index.TestIndexTooManyDocs$$Lambda$8/1612361439.run(Unknown > Source) > 16:06:08 [junit4] 2> at java.lang.Thread.run(Thread.java:748) > 16:06:08 [junit4] 2> 5) Thread[id=1, name=main, state=WAITING, > group=main] > 16:06:08 [junit4] 2> at java.lang.Object.wait(Native Method) > 16:06:08 [junit4] 2> at java.lang.Thread.join(Thread.java:1252) > 16:06:08 [junit4] 2> at java.lang.Thread.join(Thread.java:1326) > 16:06:08 [junit4] 2> at > com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:639) > 16:06:08 [junit4] 2> at > com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunner.java:496) > 16:06:08 [junit4] 2> at > com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:269) > 16:06:08 [junit4] 2> at > com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:394) > 16:06:08 [junit4] 2> at > com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:13) > 16:06:08 [junit4] 2> 6) Thread[id=21, > name=TEST-TestIndexTooManyDocs.testIndexTooManyDocs-seed#[43B7C75B765AFEBD], > state=WAITING, group=TGRP-TestIndexTooManyDocs] > 16:06:08 [junit4] 2> at sun.misc.Unsafe.park(Native Method) > 16:06:08 [junit4] 2> at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > 16:06:08 [junit4] 2> at > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) > 16:06:08 [junit4] 2> at > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) > 16:06:08 [junit4] 2> at > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) > 16:06:08 [junit4] 2> at > java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231) > 16:06:08 [junit4] 2> at > org.apache.lucene.index.TestIndexTooManyDocs.testIndexTooManyDocs(TestIndexTooManyDocs.java:101) > 16:06:08 [junit4] 2> at > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > 16:06:08 [junit4] 2> at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > 16:06:08 [junit4] 2> at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > 16:06:08 [junit4] 2> at > java.lang.reflect.Method.invoke(Method.java:498) > 16:06:08 [junit4] 2> at > com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750) > 16:06:08 [junit4] 2> at > com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938) > 16:06:08 [junit4] 2> at > com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974) > 16:06:08 [junit4] 2> at > com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988) > 16:06:08 [junit4] 2> at > org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) > 16:06:08 [junit4] 2> at > org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) > 16:06:08 [junit4] 2> at > org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) > 16:06:08 [junit4] 2> at > org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) > 16:06:08 [junit4] 2> at > org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) > 16:06:08 [junit4] 2> at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > 16:06:08 [junit4] 2> at > com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) > 16:06:08 [junit4] 2> at > com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) > 16:06:08 [junit4] 2> at > com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) > 16:06:08 [junit4] 2> at > com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947) > 16:06:08 [junit4] 2> at > com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832) > 16:06:08 [junit4] 2> at > com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883) > 16:06:08 [junit4] 2> at > com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894) > 16:06:08 [junit4] 2> at > org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) > 16:06:08 [junit4] 2> at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > 16:06:08 [junit4] 2> at > org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) > 16:06:08 [junit4] 2> at > com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) > 16:06:08 [junit4] 2> at > com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) > 16:06:08 [junit4] 2> at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > 16:06:08 [junit4] 2> at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > 16:06:08 [junit4] 2> at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > 16:06:08 [junit4] 2> at > org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) > 16:06:08 [junit4] 2> at > org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) > 16:06:08 [junit4] 2> at > org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) > 16:06:08 [junit4] 2> at > org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) > 16:06:08 [junit4] 2> at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > 16:06:08 [junit4] 2> at > com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) > 16:06:08 [junit4] 2> at java.lang.Thread.run(Thread.java:748) > 16:06:08 [junit4] 2> NOTE: reproduce with: ant test > -Dtestcase=TestIndexTooManyDocs -Dtests.method=testIndexTooManyDocs > -Dtests.seed=43B7C75B765AFEBD -Dtests.slow=true -Dtests.badapples=true > -Dtests.locale=ar-MA -Dtests.timezone=America/Jamaica -Dtests.asserts=true > -Dtests.file.encoding=US-ASCII > 16:06:08 [junit4] ERROR 7200s J3 | > TestIndexTooManyDocs.testIndexTooManyDocs <<< > 16:06:08 [junit4] > Throwable #1: java.lang.Exception: Test abandoned > because suite timeout was reached. > 16:06:08 [junit4] > at > __randomizedtesting.SeedInfo.seed([43B7C75B765AFEBD]:0) > 16:06:08 [junit4] 2> ??? 14, 2019 11:05:45 ? > com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks > 16:06:08 [junit4] 2> WARNING: Will linger awaiting termination of 3 > leaked thread(s). > 16:06:08 [junit4] 2> ??? 14, 2019 11:06:05 ? > com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks > 16:06:08 [junit4] 2> SEVERE: 3 threads leaked from SUITE scope at > org.apache.lucene.index.TestIndexTooManyDocs: > 16:06:08 [junit4] 2> 1) Thread[id=23, name=Thread-2, state=BLOCKED, > group=TGRP-TestIndexTooManyDocs] > 16:06:08 [junit4] 2> at > org.apache.lucene.index.IndexWriter.nrtIsCurrent(IndexWriter.java:4942) > 16:06:08 [junit4] 2> at > org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:290) > 16:06:08 [junit4] 2> at > org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:269) > 16:06:08 [junit4] 2> at > org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:259) > 16:06:08 [junit4] 2> at > org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:140) > 16:06:08 [junit4] 2> at > org.apache.lucene.index.TestIndexTooManyDocs.lambda$testIndexTooManyDocs$1(TestIndexTooManyDocs.java:86) > 16:06:08 [junit4] 2> at > org.apache.lucene.index.TestIndexTooManyDocs$$Lambda$8/1612361439.run(Unknown > Source) > 16:06:08 [junit4] 2> at java.lang.Thread.run(Thread.java:748) > 16:06:08 [junit4] 2> 2) Thread[id=22, name=Thread-1, state=BLOCKED, > group=TGRP-TestIndexTooManyDocs] > 16:06:08 [junit4] 2> at > org.apache.lucene.index.IndexWriter.nrtIsCurrent(IndexWriter.java:4942) > 16:06:08 [junit4] 2> at > org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:290) > 16:06:08 [junit4] 2> at > org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:269) > 16:06:08 [junit4] 2> at > org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:259) > 16:06:08 [junit4] 2> at > org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:140) > 16:06:08 [junit4] 2> at > org.apache.lucene.index.TestIndexTooManyDocs.lambda$testIndexTooManyDocs$1(TestIndexTooManyDocs.java:86) > 16:06:08 [junit4] 2> at > org.apache.lucene.index.TestIndexTooManyDocs$$Lambda$8/1612361439.run(Unknown > Source) > 16:06:08 [junit4] 2> at java.lang.Thread.run(Thread.java:748) > 16:06:08 [junit4] 2> 3) Thread[id=21, > name=TEST-TestIndexTooManyDocs.testIndexTooManyDocs-seed#[43B7C75B765AFEBD], > state=WAITING, group=TGRP-TestIndexTooManyDocs] > 16:06:08 [junit4] 2> at sun.misc.Unsafe.park(Native Method) > 16:06:08 [junit4] 2> at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > 16:06:08 [junit4] 2> at > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) > 16:06:08 [junit4] 2> at > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) > 16:06:08 [junit4] 2> at > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) > 16:06:08 [junit4] 2> at > java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231) > 16:06:08 [junit4] 2> at > org.apache.lucene.index.TestIndexTooManyDocs.testIndexTooManyDocs(TestIndexTooManyDocs.java:101) > 16:06:08 [junit4] 2> at > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > 16:06:08 [junit4] 2> at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > 16:06:08 [junit4] 2> at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > 16:06:08 [junit4] 2> at > java.lang.reflect.Method.invoke(Method.java:498) > 16:06:08 [junit4] 2> at > com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750) > 16:06:08 [junit4] 2> at > com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938) > 16:06:08 [junit4] 2> at > com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974) > 16:06:08 [junit4] 2> at > com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988) > 16:06:08 [junit4] 2> at > org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) > 16:06:08 [junit4] 2> at > org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) > 16:06:08 [junit4] 2> at > org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) > 16:06:08 [junit4] 2> at > org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) > 16:06:08 [junit4] 2> at > org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) > 16:06:08 [junit4] 2> at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > 16:06:08 [junit4] 2> at > com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) > 16:06:08 [junit4] 2> at > com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) > 16:06:08 [junit4] 2> at > com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) > 16:06:08 [junit4] 2> at > com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947) > 16:06:08 [junit4] 2> at > com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832) > 16:06:08 [junit4] 2> at > com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883) > 16:06:08 [junit4] 2> at > com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894) > 16:06:08 [junit4] 2> at > org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) > 16:06:08 [junit4] 2> at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > 16:06:08 [junit4] 2> at > org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) > 16:06:08 [junit4] 2> at > com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) > 16:06:08 [junit4] 2> at > com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) > 16:06:08 [junit4] 2> at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > 16:06:08 [junit4] 2> at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > 16:06:08 [junit4] 2> at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > 16:06:08 [junit4] 2> at > org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) > 16:06:08 [junit4] 2> at > org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) > 16:06:08 [junit4] 2> at > org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) > 16:06:08 [junit4] 2> at > org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) > 16:06:08 [junit4] 2> at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > 16:06:08 [junit4] 2> at > com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) > 16:06:08 [junit4] 2> at java.lang.Thread.run(Thread.java:748) > 16:06:08 [junit4] 2> ??? 14, 2019 11:06:05 ? > com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll > 16:06:08 [junit4] 2> INFO: Starting to interrupt leaked threads: > 16:06:08 [junit4] 2> 1) Thread[id=23, name=Thread-2, state=BLOCKED, > group=TGRP-TestIndexTooManyDocs] > 16:06:08 [junit4] 2> 2) Thread[id=22, name=Thread-1, state=RUNNABLE, > group=TGRP-TestIndexTooManyDocs] > 16:06:08 [junit4] 2> 3) Thread[id=21, > name=TEST-TestIndexTooManyDocs.testIndexTooManyDocs-seed#[43B7C75B765AFEBD], > state=WAITING, group=TGRP-TestIndexTooManyDocs] > 16:06:08 [junit4] 2> Jan 14, 2019 4:06:08 PM > com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll > 16:06:08 [junit4] 2> SEVERE: There are still zombie threads that > couldn't be terminated: > 16:06:08 [junit4] 2> 1) Thread[id=23, name=Thread-2, state=RUNNABLE, > group=TGRP-TestIndexTooManyDocs] > 16:06:08 [junit4] 2> at > org.apache.lucene.index.IndexWriter.nrtIsCurrent(IndexWriter.java:4942) > 16:06:08 [junit4] 2> at > org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:290) > 16:06:08 [junit4] 2> at > org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:269) > 16:06:08 [junit4] 2> at > org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:259) > 16:06:08 [junit4] 2> at > org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:140) > 16:06:08 [junit4] 2> at > org.apache.lucene.index.TestIndexTooManyDocs.lambda$testIndexTooManyDocs$1(TestIndexTooManyDocs.java:86) > 16:06:08 [junit4] 2> at > org.apache.lucene.index.TestIndexTooManyDocs$$Lambda$8/1612361439.run(Unknown > Source) > 16:06:08 [junit4] 2> at java.lang.Thread.run(Thread.java:748) > 16:06:08 [junit4] 2> 2) Thread[id=22, name=Thread-1, state=BLOCKED, > group=TGRP-TestIndexTooManyDocs] > 16:06:08 [junit4] 2> at > org.apache.lucene.index.IndexWriter.nrtIsCurrent(IndexWriter.java:4942) > 16:06:08 [junit4] 2> at > org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:290) > 16:06:08 [junit4] 2> at > org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:269) > 16:06:08 [junit4] 2> at > org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:259) > 16:06:08 [junit4] 2> at > org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:140) > 16:06:08 [junit4] 2> at > org.apache.lucene.index.TestIndexTooManyDocs.lambda$testIndexTooManyDocs$1(TestIndexTooManyDocs.java:86) > 16:06:08 [junit4] 2> at > org.apache.lucene.index.TestIndexTooManyDocs$$Lambda$8/1612361439.run(Unknown > Source) > 16:06:08 [junit4] 2> at java.lang.Thread.run(Thread.java:748) > 16:06:08 [junit4] 2> NOTE: test params are: codec=Asserting(Lucene80): > {id=PostingsFormat(name=Direct)}, docValues:{}, maxPointsInLeafNode=848, > maxMBSortInHeap=7.086705758151636, > sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@41eeb709), > locale=ar-MA, timezone=America/Jamaica > 16:06:08 [junit4] 2> NOTE: Linux 4.15.0-1026-gcp amd64/Oracle > Corporation 1.8.0_191 > (64-bit)/cpus=16,threads=3,free=456394640,total=506462208 > 16:06:08 [junit4] 2> NOTE: All tests run in this JVM: > [TestIndexTooManyDocs] > 16:06:08 [junit4] 2> NOTE: reproduce with: ant test > -Dtestcase=TestIndexTooManyDocs -Dtests.seed=43B7C75B765AFEBD > -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=ar-MA > -Dtests.timezone=America/Jamaica -Dtests.asserts=true > -Dtests.file.encoding=US-ASCII > 16:06:08 [junit4] ERROR 0.00s J3 | TestIndexTooManyDocs (suite) <<< > 16:06:08 [junit4] > Throwable #1: java.lang.Exception: Suite timeout > exceeded (>= 7200000 msec). > 16:06:08 [junit4] > at > __randomizedtesting.SeedInfo.seed([43B7C75B765AFEBD]:0)Throwable #2: > com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an > uncaught exception in thread: Thread[id=30, name=Thread-9, state=RUNNABLE, > group=TGRP-TestIndexTooManyDocs] > 16:06:08 [junit4] > Caused by: java.lang.AssertionError: seqNo=6 vs > maxSeqNo=6 > 16:06:08 [junit4] > at > __randomizedtesting.SeedInfo.seed([43B7C75B765AFEBD]:0) > 16:06:08 [junit4] > at > org.apache.lucene.index.DocumentsWriterDeleteQueue.getNextSequenceNumber(DocumentsWriterDeleteQueue.java:482) > 16:06:08 [junit4] > at > org.apache.lucene.index.DocumentsWriterDeleteQueue.add(DocumentsWriterDeleteQueue.java:168) > 16:06:08 [junit4] > at > org.apache.lucene.index.DocumentsWriterDeleteQueue.add(DocumentsWriterDeleteQueue.java:146) > 16:06:08 [junit4] > at > org.apache.lucene.index.DocumentsWriterPerThread.finishDocument(DocumentsWriterPerThread.java:362) > 16:06:08 [junit4] > at > org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:264) > 16:06:08 [junit4] > at > org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:494) > 16:06:08 [junit4] > at > org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1594) > 16:06:08 [junit4] > at > org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1586) > 16:06:08 [junit4] > at > org.apache.lucene.index.TestIndexTooManyDocs.lambda$testIndexTooManyDocs$0(TestIndexTooManyDocs.java:70) > 16:06:08 [junit4] > at > java.lang.Thread.run(Thread.java:748)Throwable #3: > com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an > uncaught exception in thread: Thread[id=29, name=Thread-8, state=RUNNABLE, > group=TGRP-TestIndexTooManyDocs] > 16:06:08 [junit4] > Caused by: java.lang.AssertionError: seqNo=7 vs > maxSeqNo=6 > 16:06:08 [junit4] > at > __randomizedtesting.SeedInfo.seed([43B7C75B765AFEBD]:0) > 16:06:08 [junit4] > at > org.apache.lucene.index.DocumentsWriterDeleteQueue.getNextSequenceNumber(DocumentsWriterDeleteQueue.java:482) > 16:06:08 [junit4] > at > org.apache.lucene.index.DocumentsWriterDeleteQueue.add(DocumentsWriterDeleteQueue.java:168) > 16:06:08 [junit4] > at > org.apache.lucene.index.DocumentsWriterDeleteQueue.add(DocumentsWriterDeleteQueue.java:146) > 16:06:08 [junit4] > at > org.apache.lucene.index.DocumentsWriterPerThread.finishDocument(DocumentsWriterPerThread.java:362) > 16:06:08 [junit4] > at > org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:264) > 16:06:08 [junit4] > at > org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:494) > 16:06:08 [junit4] > at > org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1594) > 16:06:08 [junit4] > at > org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1586) > 16:06:08 [junit4] > at > org.apache.lucene.index.TestIndexTooManyDocs.lambda$testIndexTooManyDocs$0(TestIndexTooManyDocs.java:70) > {noformat} > The reason seems to be that we don't prevent new threadstates from being > created while we move over to a new delete queue. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org