Sneaky! Maybe this test (and any other "across IW sessions" tests could call an MDW method each time a new IW is about to open, to clear any successfully deleted files from the "already written to" map?
Mike McCandless http://blog.mikemccandless.com On Sun, Sep 7, 2014 at 12:09 PM, Robert Muir <[email protected]> wrote: > This is possible when e.g. you close but don't wait for merges, and > its actually able to delete unreferenced files. > then the next IW session will use the same file names, which MDW catches. > > we already fixed the possible bug case where, if the files cant be > deleted for some reason, the next IW will "skip past those > generations" so it wont cause issues. > > But I don't know how to fix the test.. We could allow this case when > we successfully deleted a file (only successful, to ensure above bug > case "works"). > But really this should only be allowed across IW sessions, and most > tests don't do that, so we lose a lot of safety checks. > > On Sun, Sep 7, 2014 at 3:52 AM, Apache Jenkins Server > <[email protected]> wrote: >> Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-4.x/613/ >> >> 1 tests failed. >> REGRESSION: org.apache.lucene.index.TestIndexWriterMerging.testNoWaitClose >> >> Error Message: >> file "_au.fdx" was already written to >> >> Stack Trace: >> java.io.IOException: file "_au.fdx" was already written to >> at >> __randomizedtesting.SeedInfo.seed([D9426C8ADCEA197A:FEB3CD5B0FFCC98E]:0) >> at >> org.apache.lucene.store.MockDirectoryWrapper.createOutput(MockDirectoryWrapper.java:528) >> at >> org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:44) >> at >> org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.<init>(CompressingStoredFieldsWriter.java:113) >> at >> org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsWriter(CompressingStoredFieldsFormat.java:120) >> at >> org.apache.lucene.codecs.asserting.AssertingStoredFieldsFormat.fieldsWriter(AssertingStoredFieldsFormat.java:47) >> at >> org.apache.lucene.index.DefaultIndexingChain.initStoredFieldsWriter(DefaultIndexingChain.java:84) >> at >> org.apache.lucene.index.DefaultIndexingChain.startStoredFields(DefaultIndexingChain.java:254) >> at >> org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:298) >> at >> org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:242) >> at >> org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:454) >> at >> org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1542) >> at >> org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1257) >> at >> org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1238) >> at >> org.apache.lucene.index.TestIndexWriterMerging.testNoWaitClose(TestIndexWriterMerging.java:388) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:606) >> at >> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618) >> at >> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827) >> at >> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) >> at >> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877) >> at >> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50) >> at >> org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51) >> at >> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) >> at >> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) >> at >> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49) >> at >> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) >> at >> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) >> at >> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) >> at >> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) >> at >> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798) >> at >> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458) >> at >> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836) >> at >> com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738) >> at >> com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772) >> at >> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783) >> at >> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) >> at >> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42) >> at >> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) >> at >> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) >> at >> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) >> at >> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) >> at >> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) >> at >> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) >> at >> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43) >> at >> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) >> at >> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) >> at >> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55) >> at >> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) >> at >> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) >> at java.lang.Thread.run(Thread.java:745) >> >> >> >> >> Build Log: >> [...truncated 991 lines...] >> [junit4] Suite: org.apache.lucene.index.TestIndexWriterMerging >> [junit4] 2> NOTE: download the large Jenkins line-docs file by running >> 'ant get-jenkins-line-docs' in the lucene directory. >> [junit4] 2> NOTE: reproduce with: ant test >> -Dtestcase=TestIndexWriterMerging -Dtests.method=testNoWaitClose >> -Dtests.seed=D9426C8ADCEA197A -Dtests.multiplier=2 -Dtests.nightly=true >> -Dtests.slow=true >> -Dtests.linedocsfile=/home/jenkins/lucene-data/enwiki.random.lines.txt >> -Dtests.locale=ar_IQ -Dtests.timezone=Europe/Dublin >> -Dtests.file.encoding=ISO-8859-1 >> [junit4] ERROR 2.54s J3 | TestIndexWriterMerging.testNoWaitClose <<< >> [junit4] > Throwable #1: java.io.IOException: file "_au.fdx" was >> already written to >> [junit4] > at >> __randomizedtesting.SeedInfo.seed([D9426C8ADCEA197A:FEB3CD5B0FFCC98E]:0) >> [junit4] > at >> org.apache.lucene.store.MockDirectoryWrapper.createOutput(MockDirectoryWrapper.java:528) >> [junit4] > at >> org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:44) >> [junit4] > at >> org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.<init>(CompressingStoredFieldsWriter.java:113) >> [junit4] > at >> org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsWriter(CompressingStoredFieldsFormat.java:120) >> [junit4] > at >> org.apache.lucene.codecs.asserting.AssertingStoredFieldsFormat.fieldsWriter(AssertingStoredFieldsFormat.java:47) >> [junit4] > at >> org.apache.lucene.index.DefaultIndexingChain.initStoredFieldsWriter(DefaultIndexingChain.java:84) >> [junit4] > at >> org.apache.lucene.index.DefaultIndexingChain.startStoredFields(DefaultIndexingChain.java:254) >> [junit4] > at >> org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:298) >> [junit4] > at >> org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:242) >> [junit4] > at >> org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:454) >> [junit4] > at >> org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1542) >> [junit4] > at >> org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1257) >> [junit4] > at >> org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1238) >> [junit4] > at >> org.apache.lucene.index.TestIndexWriterMerging.testNoWaitClose(TestIndexWriterMerging.java:388) >> [junit4] > at java.lang.Thread.run(Thread.java:745) >> [junit4] 2> NOTE: leaving temporary files on disk at: >> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/build/core/test/J3/./temp/lucene.index.TestIndexWriterMerging-D9426C8ADCEA197A-001 >> [junit4] 2> NOTE: test params are: codec=Asserting(Lucene410), >> sim=DefaultSimilarity, locale=ar_IQ, timezone=Europe/Dublin >> [junit4] 2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation >> 1.7.0_65 (64-bit)/cpus=16,threads=1,free=287705952,total=468713472 >> [junit4] 2> NOTE: All tests run in this JVM: [TestCharsRefBuilder, >> TestLockFactory, TestExceedMaxTermLength, TestDoubleBarrelLRUCache, >> TestMergeSchedulerExternal, TestOpenBitSet, TestTermdocPerf, >> TestPrefixFilter, TestTermVectorsFormat, NestedSetupChain, >> NestedTeardownChain, TestTimeLimitingCollector, >> TestBackwardsCompatibility3x, TestTopFieldCollector, TestFieldValueFilter, >> TestForceMergeForever, TestReaderClosed, TestSimpleExplanations, >> TestIndexWriterNRTIsCurrent, TestQueryBuilder, TestMultiMMap, TestLock, >> TestSortDocValues, TestMultiLevelSkipList, TestSpanExplanationsOfNonMatches, >> TestFieldType, TestDisjunctionMaxQuery, TestAtomicUpdate, TestVirtualMethod, >> TestBooleanMinShouldMatch, TestLucene41StoredFieldsFormat, TestFSTs, >> TestUnicodeUtil, TestStoredFieldsFormat, TestRegexpQuery, >> TestPayloadNearQuery, TestSearch, TestFilterIterator, TestFieldReuse, >> TestDeterminizeLexicon, TestLogMergePolicy, TestSortRescorer, >> TestMutableValues, TestRamUsageEstimatorOnWildAnimals, TestStressNRT, >> TestPayloadSpans, TestWAH8DocIdSet, TestPositionIncrement, >> TestCachingCollector, TestLucene410DocValuesFormat, TestNot, TestAddIndexes, >> TestPrefixQuery, TestIndexWriterReader, TestIndexWriterMerging] >> [junit4] Completed on J3 in 8.98s, 6 tests, 1 error <<< FAILURES! >> >> [...truncated 985 lines...] >> BUILD FAILED >> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-4.x/build.xml:498: >> The following error occurred while executing this line: >> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-4.x/build.xml:471: >> The following error occurred while executing this line: >> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-4.x/build.xml:60: >> The following error occurred while executing this line: >> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-4.x/extra-targets.xml:39: >> The following error occurred while executing this line: >> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/build.xml:49: >> The following error occurred while executing this line: >> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/common-build.xml:1362: >> The following error occurred while executing this line: >> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/common-build.xml:965: >> There were test failures: 418 suites, 3731 tests, 1 error, 120 ignored (109 >> assumptions) >> >> Total time: 66 minutes 5 seconds >> Build step 'Invoke Ant' marked build as failure >> Archiving artifacts >> Sending artifact delta relative to Lucene-Solr-NightlyTests-4.x #610 >> Archived 5 artifacts >> Archive block size is 32768 >> Received 0 blocks and 4845127 bytes >> Compression is 0.0% >> Took 2.9 sec >> Recording test results >> Email was triggered for: Failure >> Sending email for trigger: Failure >> >> >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
