this is an ugly test bug: i opened an issue for it ( https://issues.apache.org/jira/browse/LUCENE-5066)
I think the patch is ok to fix this test fail, but as noted the issue, we should probably add a simple low-level test for this to all real directories. also, i think it would be good to move DocumentsWriterPerThread.MAX_TERM_LENGTH_UTF8 somewhere else (e.g. IndexWriter), so we can add a similar assert to all the codecs: This way instead of OOM we know stuff is really jacked up. On Tue, Jun 18, 2013 at 5:13 PM, Robert Muir <[email protected]> wrote: > somehow this relates to the delegation in faultyindexinput (NOT its > exception-generation, which isnt yet happening....) > > and the surrogates dance is somehow tickling the delegator-bug in such a > way that it thinks a single term is 262MB. thats why it only fails on 3.x > > > On Tue, Jun 18, 2013 at 4:38 PM, Robert Muir <[email protected]> wrote: > >> This has failed several times, since LUCENE-5038 was committed. But that >> commit didn't really change the test, except to setCFSRatio(0.0) >> >> and this test indexes only one document!!!!!!!!!! >> >> I'll look at this later tonight if nobody beats me to it, seems like >> something is really wrong. >> >> On Tue, Jun 18, 2013 at 4:21 PM, Policeman Jenkins Server < >> [email protected]> wrote: >> >>> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/6128/ >>> Java: 32bit/jdk1.8.0-ea-b93 -server -XX:+UseG1GC >>> >>> 2 tests failed. >>> FAILED: >>> junit.framework.TestSuite.org.apache.lucene.index.TestFieldsReader >>> >>> Error Message: >>> Captured an uncaught exception in thread: Thread[id=298, name=Lucene >>> Merge Thread #0, state=RUNNABLE, group=TGRP-TestFieldsReader] >>> >>> Stack Trace: >>> com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an >>> uncaught exception in thread: Thread[id=298, name=Lucene Merge Thread #0, >>> state=RUNNABLE, group=TGRP-TestFieldsReader] >>> Caused by: org.apache.lucene.index.MergePolicy$MergeException: >>> java.lang.OutOfMemoryError: Java heap space >>> at __randomizedtesting.SeedInfo.seed([2EE02ABE17F63E4B]:0) >>> at >>> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:541) >>> at >>> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:514) >>> Caused by: java.lang.OutOfMemoryError: Java heap space >>> at org.apache.lucene.util.BytesRef.copyBytes(BytesRef.java:196) >>> at org.apache.lucene.util.BytesRef.deepCopyOf(BytesRef.java:343) >>> at >>> org.apache.lucene.codecs.lucene3x.TermBuffer.toTerm(TermBuffer.java:113) >>> at >>> org.apache.lucene.codecs.lucene3x.SegmentTermEnum.term(SegmentTermEnum.java:184) >>> at >>> org.apache.lucene.codecs.lucene3x.Lucene3xFields$PreTermsEnum.next(Lucene3xFields.java:863) >>> at >>> org.apache.lucene.index.MultiTermsEnum.pushTop(MultiTermsEnum.java:292) >>> at >>> org.apache.lucene.index.MultiTermsEnum.next(MultiTermsEnum.java:318) >>> at >>> org.apache.lucene.codecs.TermsConsumer.merge(TermsConsumer.java:103) >>> at >>> org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:72) >>> at >>> org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:365) >>> at >>> org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:98) >>> at >>> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3767) >>> at >>> org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3371) >>> at >>> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:401) >>> at >>> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:478) >>> >>> >>> REGRESSION: org.apache.lucene.index.TestFieldsReader.testExceptions >>> >>> Error Message: >>> this writer hit an OutOfMemoryError; cannot complete forceMerge >>> >>> Stack Trace: >>> java.lang.IllegalStateException: this writer hit an OutOfMemoryError; >>> cannot complete forceMerge >>> at >>> __randomizedtesting.SeedInfo.seed([2EE02ABE17F63E4B:58E17816BF8B40FD]:0) >>> at >>> org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1704) >>> at >>> org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1650) >>> at >>> org.apache.lucene.index.TestFieldsReader.testExceptions(TestFieldsReader.java:204) >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>> at >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >>> at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> at java.lang.reflect.Method.invoke(Method.java:491) >>> at >>> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559) >>> at >>> com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79) >>> at >>> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737) >>> at >>> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773) >>> at >>> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787) >>> at >>> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50) >>> at >>> org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51) >>> at >>> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) >>> at >>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) >>> at >>> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49) >>> at >>> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70) >>> at >>> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) >>> at >>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) >>> at >>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358) >>> at >>> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782) >>> at >>> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442) >>> at >>> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746) >>> at >>> com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648) >>> at >>> com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682) >>> at >>> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693) >>> at >>> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) >>> at >>> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42) >>> at >>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) >>> at >>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) >>> at >>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) >>> at >>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) >>> at >>> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43) >>> at >>> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) >>> at >>> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70) >>> at >>> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55) >>> at >>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) >>> at >>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358) >>> at java.lang.Thread.run(Thread.java:724) >>> >>> >>> >>> >>> Build Log: >>> [...truncated 395 lines...] >>> [junit4:junit4] Suite: org.apache.lucene.index.TestFieldsReader >>> [junit4:junit4] 2> NOTE: reproduce with: ant test >>> -Dtestcase=TestFieldsReader -Dtests.method=testExceptions >>> -Dtests.seed=2EE02ABE17F63E4B -Dtests.multiplier=3 -Dtests.slow=true >>> -Dtests.locale=ar_LB -Dtests.timezone=America/Boa_Vista >>> -Dtests.file.encoding=UTF-8 >>> [junit4:junit4] ERROR 1.28s J0 | TestFieldsReader.testExceptions <<< >>> [junit4:junit4] > Throwable #1: java.lang.IllegalStateException: this >>> writer hit an OutOfMemoryError; cannot complete forceMerge >>> [junit4:junit4] > at >>> __randomizedtesting.SeedInfo.seed([2EE02ABE17F63E4B:58E17816BF8B40FD]:0) >>> [junit4:junit4] > at >>> org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1704) >>> [junit4:junit4] > at >>> org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1650) >>> [junit4:junit4] > at >>> org.apache.lucene.index.TestFieldsReader.testExceptions(TestFieldsReader.java:204) >>> [junit4:junit4] > at java.lang.Thread.run(Thread.java:724) >>> [junit4:junit4] 2> Jun 18, 2013 11:14:11 PM >>> com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks >>> [junit4:junit4] 2> WARNING: Will linger awaiting termination of 1 >>> leaked thread(s). >>> [junit4:junit4] 2> Jun 18, 2013 11:14:11 PM >>> com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler >>> uncaughtException >>> [junit4:junit4] 2> WARNING: Uncaught exception in thread: >>> Thread[Lucene Merge Thread #0,6,TGRP-TestFieldsReader] >>> [junit4:junit4] 2> org.apache.lucene.index.MergePolicy$MergeException: >>> java.lang.OutOfMemoryError: Java heap space >>> [junit4:junit4] 2> at >>> __randomizedtesting.SeedInfo.seed([2EE02ABE17F63E4B]:0) >>> [junit4:junit4] 2> at >>> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:541) >>> [junit4:junit4] 2> at >>> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:514) >>> [junit4:junit4] 2> Caused by: java.lang.OutOfMemoryError: Java heap >>> space >>> [junit4:junit4] 2> at >>> org.apache.lucene.util.BytesRef.copyBytes(BytesRef.java:196) >>> [junit4:junit4] 2> at >>> org.apache.lucene.util.BytesRef.deepCopyOf(BytesRef.java:343) >>> [junit4:junit4] 2> at >>> org.apache.lucene.codecs.lucene3x.TermBuffer.toTerm(TermBuffer.java:113) >>> [junit4:junit4] 2> at >>> org.apache.lucene.codecs.lucene3x.SegmentTermEnum.term(SegmentTermEnum.java:184) >>> [junit4:junit4] 2> at >>> org.apache.lucene.codecs.lucene3x.Lucene3xFields$PreTermsEnum.next(Lucene3xFields.java:863) >>> [junit4:junit4] 2> at >>> org.apache.lucene.index.MultiTermsEnum.pushTop(MultiTermsEnum.java:292) >>> [junit4:junit4] 2> at >>> org.apache.lucene.index.MultiTermsEnum.next(MultiTermsEnum.java:318) >>> [junit4:junit4] 2> at >>> org.apache.lucene.codecs.TermsConsumer.merge(TermsConsumer.java:103) >>> [junit4:junit4] 2> at >>> org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:72) >>> [junit4:junit4] 2> at >>> org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:365) >>> [junit4:junit4] 2> at >>> org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:98) >>> [junit4:junit4] 2> at >>> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3767) >>> [junit4:junit4] 2> at >>> org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3371) >>> [junit4:junit4] 2> at >>> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:401) >>> [junit4:junit4] 2> at >>> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:478) >>> [junit4:junit4] 2> >>> [junit4:junit4] 2> NOTE: test params are: codec=Lucene3x, >>> sim=DefaultSimilarity, locale=ar_LB, timezone=America/Boa_Vista >>> [junit4:junit4] 2> NOTE: Linux 3.2.0-48-generic i386/Oracle >>> Corporation 1.8.0-ea >>> (32-bit)/cpus=8,threads=1,free=505999600,total=520093696 >>> [junit4:junit4] 2> NOTE: All tests run in this JVM: >>> [TestLucene40PostingsFormat, TestSimilarity2, Test2BPostingsBytes, >>> TestDemo, TestStressAdvance, TestIndexWriterForceMerge, TestArrayUtil, >>> TestDocTermOrds, TestSentinelIntSet, TestFieldCacheSanityChecker, >>> TestDateTools, TestIndexCommit, TestLucene40PostingsReader, >>> TestCachingCollector, TestShardSearching, TestStressNRT, >>> TestNoMergeScheduler, TestTopFieldCollector, TestIntroSorter, >>> TestNRTThreads, TestCompressingTermVectorsFormat, TestTermInfosReaderIndex, >>> TestFastDecompressionMode, TestPayloadTermQuery, TestFieldsReader] >>> [junit4:junit4] 2> NOTE: reproduce with: ant test >>> -Dtestcase=TestFieldsReader -Dtests.seed=2EE02ABE17F63E4B >>> -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ar_LB >>> -Dtests.timezone=America/Boa_Vista -Dtests.file.encoding=UTF-8 >>> [junit4:junit4] ERROR 0.00s J0 | TestFieldsReader (suite) <<< >>> [junit4:junit4] > Throwable #1: >>> com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an >>> uncaught exception in thread: Thread[id=298, name=Lucene Merge Thread #0, >>> state=RUNNABLE, group=TGRP-TestFieldsReader] >>> [junit4:junit4] > Caused by: >>> org.apache.lucene.index.MergePolicy$MergeException: >>> java.lang.OutOfMemoryError: Java heap space >>> [junit4:junit4] > at >>> __randomizedtesting.SeedInfo.seed([2EE02ABE17F63E4B]:0) >>> [junit4:junit4] > at >>> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:541) >>> [junit4:junit4] > at >>> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:514) >>> [junit4:junit4] > Caused by: java.lang.OutOfMemoryError: Java heap >>> space >>> [junit4:junit4] > at >>> org.apache.lucene.util.BytesRef.copyBytes(BytesRef.java:196) >>> [junit4:junit4] > at >>> org.apache.lucene.util.BytesRef.deepCopyOf(BytesRef.java:343) >>> [junit4:junit4] > at >>> org.apache.lucene.codecs.lucene3x.TermBuffer.toTerm(TermBuffer.java:113) >>> [junit4:junit4] > at >>> org.apache.lucene.codecs.lucene3x.SegmentTermEnum.term(SegmentTermEnum.java:184) >>> [junit4:junit4] > at >>> org.apache.lucene.codecs.lucene3x.Lucene3xFields$PreTermsEnum.next(Lucene3xFields.java:863) >>> [junit4:junit4] > at >>> org.apache.lucene.index.MultiTermsEnum.pushTop(MultiTermsEnum.java:292) >>> [junit4:junit4] > at >>> org.apache.lucene.index.MultiTermsEnum.next(MultiTermsEnum.java:318) >>> [junit4:junit4] > at >>> org.apache.lucene.codecs.TermsConsumer.merge(TermsConsumer.java:103) >>> [junit4:junit4] > at >>> org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:72) >>> [junit4:junit4] > at >>> org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:365) >>> [junit4:junit4] > at >>> org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:98) >>> [junit4:junit4] > at >>> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3767) >>> [junit4:junit4] > at >>> org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3371) >>> [junit4:junit4] > at >>> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:401) >>> [junit4:junit4] > at >>> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:478) >>> [junit4:junit4] Completed on J0 in 1.73s, 2 tests, 2 errors <<< FAILURES! >>> >>> [...truncated 984 lines...] >>> BUILD FAILED >>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:392: The >>> following error occurred while executing this line: >>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:372: The >>> following error occurred while executing this line: >>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The >>> following error occurred while executing this line: >>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/build.xml:49: >>> The following error occurred while executing this line: >>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1246: >>> The following error occurred while executing this line: >>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:890: >>> There were test failures: 364 suites, 2312 tests, 1 suite-level error, 1 >>> error, 47 ignored (34 assumptions) >>> >>> Total time: 9 minutes 17 seconds >>> Build step 'Invoke Ant' marked build as failure >>> Description set: Java: 32bit/jdk1.8.0-ea-b93 -server -XX:+UseG1GC >>> Archiving artifacts >>> Recording test results >>> Email was triggered for: Failure >>> Sending email for trigger: Failure >>> >>> >>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: [email protected] >>> For additional commands, e-mail: [email protected] >>> >> >> >
