Oh! Thank you; after reading it I see folks have progressed to a root cause and a proposed solution.
On Sat, Nov 27, 2021 at 4:24 PM Dawid Weiss <[email protected]> wrote: > > > It's this issue, Mike: > https://issues.apache.org/jira/browse/LUCENE-10088 > > On Sat, Nov 27, 2021 at 8:11 PM Michael Sokolov <[email protected]> wrote: >> >> I tried reducing 500 full flushes to 250 and 5000 iters to 2500, and >> then the test would fairly reliably finish in 1 minute or so, although >> I did once see >> >> * Exception: java.lang.IllegalArgumentException thrown >> from the UncaughtExceptionHandler in thread "Lucene Merge Thread #102" >> >> I added an UncaughtExceptionHandler with printStackTrace to threads >> created by ConcurrentMergeScheduler but was never able to reproduce >> that one >> >> with the larger limits, this test generally seems to fail with OOM, or >> else takes so long that I kill it, so I think we should reduce the >> limits; then perhaps we might see some more interesting failure modes >> >> On Sat, Nov 27, 2021 at 1:30 PM Michael Sokolov <[email protected]> wrote: >> > >> > Hmm, that test is taking a very long time on my laptop (running >> > Ubuntu) with JDK11, so doesn't seem to be a Windows or JDK17 issue >> > >> > ./gradlew :lucene:core:test --tests >> > "org.apache.lucene.index.TestIndexWriterMergePolicy.testStressUpdateSameDocumentWithMergeOnGetReader" >> > -Dtests.nightly=true -Dtests.seed=B67ECC7381FE35B >> > >> > after 12m44s I killed it >> > >> > >> > On Sat, Nov 27, 2021 at 11:59 AM Michael Sokolov <[email protected]> >> > wrote: >> > > >> > > I see that we periodically get various test failures from this test. >> > > Has anybody been tracking this more carefully than me, and if so, do >> > > you remember if it's always on Windows where these too-many-open-files >> > > (and sometimes out of disk space) errors show up? I wonder if we >> > > should reduce the cardinality of these test settings: >> > > >> > > AtomicInteger iters = new AtomicInteger(100 + >> > > random().nextInt(TEST_NIGHTLY ? 5000 : 1000)); >> > > AtomicInteger numFullFlushes = >> > > new AtomicInteger(10 + random().nextInt(TEST_NIGHTLY ? 500 : >> > > 100)); >> > > >> > > On Sat, Nov 27, 2021 at 5:54 AM Policeman Jenkins Server >> > > <[email protected]> wrote: >> > > > >> > > > Build: https://jenkins.thetaphi.de/job/Lucene-jdk17panama-Windows/701/ >> > > > Java: 64bit/jdk-17 -XX:+UseCompressedOops -XX:+UseG1GC >> > > > >> > > > 2 tests failed. >> > > > FAILED: org.apache.lucene.index.TestIndexWriterMergePolicy.classMethod >> > > > >> > > > Error Message: >> > > > java.lang.AssertionError: The test or suite printed 10871 bytes to >> > > > stdout and stderr, even though the limit was set to 8192 bytes. >> > > > Increase the limit with @Limit, ignore it completely with >> > > > @SuppressSysoutChecks or run with -Dtests.verbose=true >> > > > >> > > > Stack Trace: >> > > > java.lang.AssertionError: The test or suite printed 10871 bytes to >> > > > stdout and stderr, even though the limit was set to 8192 bytes. >> > > > Increase the limit with @Limit, ignore it completely with >> > > > @SuppressSysoutChecks or run with -Dtests.verbose=true >> > > > at __randomizedtesting.SeedInfo.seed([B67ECC7381FE35B]:0) >> > > > at >> > > > org.apache.lucene.util.TestRuleLimitSysouts.afterIfSuccessful(TestRuleLimitSysouts.java:283) >> > > > at >> > > > com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.afterIfSuccessful(TestRuleAdapter.java:36) >> > > > at >> > > > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:37) >> > > > at >> > > > org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) >> > > > at >> > > > org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) >> > > > at >> > > > org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44) >> > > > at >> > > > org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60) >> > > > at >> > > > org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47) >> > > > at org.junit.rules.RunRules.evaluate(RunRules.java:20) >> > > > at >> > > > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) >> > > > at >> > > > com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370) >> > > > at >> > > > com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:826) >> > > > at java.base/java.lang.Thread.run(Thread.java:833) >> > > > >> > > > >> > > > FAILED: >> > > > org.apache.lucene.index.TestIndexWriterMergePolicy.testStressUpdateSameDocumentWithMergeOnGetReader >> > > > >> > > > Error Message: >> > > > com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an >> > > > uncaught exception in thread: Thread[id=869, name=Thread-739, >> > > > state=RUNNABLE, group=TGRP-TestIndexWriterMergePolicy] >> > > > >> > > > Stack Trace: >> > > > com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an >> > > > uncaught exception in thread: Thread[id=869, name=Thread-739, >> > > > state=RUNNABLE, group=TGRP-TestIndexWriterMergePolicy] >> > > > Caused by: java.lang.AssertionError: >> > > > java.nio.file.FileSystemException: >> > > > C:\Users\jenkins\workspace\Lucene-jdk17panama-Windows\lucene\core\build\tmp\tests-tmp\lucene.index.TestIndexWriterMergePolicy_B67ECC7381FE35B-001\index-MMapDirectory-015\_na_Lucene90FieldsIndex-doc_ids_1ak.tmp: >> > > > Too many open files >> > > > at __randomizedtesting.SeedInfo.seed([B67ECC7381FE35B]:0) >> > > > at >> > > > org.apache.lucene.index.TestIndexWriterMergePolicy.lambda$stressUpdateSameDocumentWithMergeOnX$4(TestIndexWriterMergePolicy.java:752) >> > > > at java.base/java.lang.Thread.run(Thread.java:833) >> > > > Caused by: java.nio.file.FileSystemException: >> > > > C:\Users\jenkins\workspace\Lucene-jdk17panama-Windows\lucene\core\build\tmp\tests-tmp\lucene.index.TestIndexWriterMergePolicy_B67ECC7381FE35B-001\index-MMapDirectory-015\_na_Lucene90FieldsIndex-doc_ids_1ak.tmp: >> > > > Too many open files >> > > > at >> > > > org.apache.lucene.mockfile.HandleLimitFS.onOpen(HandleLimitFS.java:45) >> > > > at >> > > > org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:82) >> > > > at >> > > > org.apache.lucene.mockfile.HandleTrackingFS.newOutputStream(HandleTrackingFS.java:163) >> > > > at >> > > > org.apache.lucene.mockfile.FilterFileSystemProvider.newOutputStream(FilterFileSystemProvider.java:194) >> > > > at >> > > > java.base/java.nio.file.Files.newOutputStream(Files.java:228) >> > > > at >> > > > org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:392) >> > > > at >> > > > org.apache.lucene.store.FSDirectory.createTempOutput(FSDirectory.java:234) >> > > > at >> > > > org.apache.lucene.store.MockDirectoryWrapper.createTempOutput(MockDirectoryWrapper.java:745) >> > > > at >> > > > org.apache.lucene.store.FilterDirectory.createTempOutput(FilterDirectory.java:81) >> > > > at >> > > > org.apache.lucene.store.TrackingDirectoryWrapper.createTempOutput(TrackingDirectoryWrapper.java:49) >> > > > at >> > > > org.apache.lucene.codecs.lucene90.compressing.FieldsIndexWriter.<init>(FieldsIndexWriter.java:83) >> > > > at >> > > > org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsWriter.<init>(Lucene90CompressingStoredFieldsWriter.java:148) >> > > > at >> > > > org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsFormat.fieldsWriter(Lucene90CompressingStoredFieldsFormat.java:140) >> > > > at >> > > > org.apache.lucene.codecs.lucene90.Lucene90StoredFieldsFormat.fieldsWriter(Lucene90StoredFieldsFormat.java:154) >> > > > at >> > > > org.apache.lucene.codecs.asserting.AssertingStoredFieldsFormat.fieldsWriter(AssertingStoredFieldsFormat.java:48) >> > > > at >> > > > org.apache.lucene.index.StoredFieldsConsumer.initStoredFieldsWriter(StoredFieldsConsumer.java:49) >> > > > at >> > > > org.apache.lucene.index.StoredFieldsConsumer.startDocument(StoredFieldsConsumer.java:56) >> > > > at >> > > > org.apache.lucene.index.IndexingChain.startStoredFields(IndexingChain.java:556) >> > > > at >> > > > org.apache.lucene.index.IndexingChain.processDocument(IndexingChain.java:587) >> > > > at >> > > > org.apache.lucene.index.DocumentsWriterPerThread.updateDocuments(DocumentsWriterPerThread.java:239) >> > > > at >> > > > org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:432) >> > > > at >> > > > org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1530) >> > > > at >> > > > org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1815) >> > > > at >> > > > org.apache.lucene.index.RandomIndexWriter.updateDocument(RandomIndexWriter.java:305) >> > > > at >> > > > org.apache.lucene.index.TestIndexWriterMergePolicy.lambda$stressUpdateSameDocumentWithMergeOnX$4(TestIndexWriterMergePolicy.java:746) >> > > > ... 1 more >> > > > >> > > > --------------------------------------------------------------------- >> > > > To unsubscribe, e-mail: [email protected] >> > > > For additional commands, e-mail: [email protected] >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] >> --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
