I have seen some issues like that with a 3rd party collection library lately going away as users move away from G1GC. So I think if that is the case it might be G1GC that is the problem here! thanks for tracking it down to a reproducible state!
simon On Mon, Aug 12, 2013 at 10:04 PM, Robert Muir <[email protected]> wrote: > I can now reproduce this, but in a crazy way. (I reproduced it twice, > and first time it failed in FastVectorHighlighter, second time in > PostingsHighlighter!!!!!) > > So i think this really looks like a JVM bug (i have not tried other > possibilties or combinations, i will open an issue). > > REPRO #1: > rmuir@beast:~/workspace/branch_4x$ svn up -r 1512807 > rmuir@beast:~/workspace/branch_4x$ ant clean > rmuir@beast:~/workspace/branch_4x$ rm -rf .caches #this is important, > otherwise master seed does not work! > rmuir@beast:~/workspace/branch_4x/lucene/highlighter$ ant test > -Dtests.jvms=2 -Dtests.seed=EBBFA6F4E80A7365 -Dargs="-server > -XX:+UseG1GC" > > [junit4] Suite: > org.apache.lucene.search.vectorhighlight.FastVectorHighlighterTest > [junit4] 2> NOTE: reproduce with: ant test > -Dtestcase=FastVectorHighlighterTest > -Dtests.method=testCommonTermsQueryHighlightTest > -Dtests.seed=EBBFA6F4E80A7365 -Dtests.slow=true -Dtests.locale=es_PA > -Dtests.timezone=America/Indiana/Vincennes -Dtests.file.encoding=UTF-8 > [junit4] FAILURE 0.02s J1 | > FastVectorHighlighterTest.testCommonTermsQueryHighlightTest <<< > [junit4] > Throwable #1: java.lang.AssertionError > [junit4] > at > __randomizedtesting.SeedInfo.seed([EBBFA6F4E80A7365:D307BBE9A713DA33]:0) > [junit4] > at > org.apache.lucene.index.ByteSliceReader.readByte(ByteSliceReader.java:73) > [junit4] > at > org.apache.lucene.store.DataInput.readVInt(DataInput.java:108) > [junit4] > at > org.apache.lucene.index.FreqProxTermsWriterPerField.flush(FreqProxTermsWriterPerField.java:453) > [junit4] > at > org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:85) > [junit4] > at > org.apache.lucene.index.TermsHash.flush(TermsHash.java:116) > [junit4] > at > org.apache.lucene.index.DocInverter.flush(DocInverter.java:53) > [junit4] > at > org.apache.lucene.index.DocFieldProcessor.flush(DocFieldProcessor.java:81) > [junit4] > at > org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:501) > [junit4] > at > org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:478) > [junit4] > at > org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:615) > [junit4] > at > org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:365) > [junit4] > at > org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:111) > [junit4] > at > org.apache.lucene.search.vectorhighlight.FastVectorHighlighterTest.testCommonTermsQueryHighlightTest(FastVectorHighlighterTest.java:285) > [junit4] > at java.lang.Thread.run(Thread.java:724) > [junit4] 2> NOTE: test params are: codec=Lucene3x, > sim=DefaultSimilarity, locale=es_PA, > timezone=America/Indiana/Vincennes > [junit4] 2> NOTE: Linux 3.5.0-27-generic i386/Oracle Corporation > 1.7.0_25 (32-bit)/cpus=8,threads=1,free=48330984,total=67108864 > [junit4] 2> NOTE: All tests run in this JVM: > [WeightedFragListBuilderTest, IndexTimeSynonymTest, > ScoreOrderFragmentsBuilderTest, HighlighterTest, FieldPhraseListTest, > HighlighterPhraseTest, SimpleFragListBuilderTest, FieldTermStackTest, > SimpleFragmentsBuilderTest, FieldQueryTest, TestPostingsHighlighter, > FastVectorHighlighterTest] > [junit4] Completed on J1 in 0.16s, 6 tests, 1 failure <<< FAILURES! > > REPRO #2: > rmuir@beast:~/workspace/branch_4x$ rm -rf .caches #this is important, > otherwise master seed does not work! > rmuir@beast:~/workspace/branch_4x/lucene/highlighter$ ant test > -Dtests.jvms=2 -Dtests.seed=EBBFA6F4E80A7365 -Dargs="-server > -XX:+UseG1GC" > > [junit4] Suite: > org.apache.lucene.search.postingshighlight.TestPostingsHighlighter > [junit4] 2> NOTE: reproduce with: ant test > -Dtestcase=TestPostingsHighlighter > -Dtests.method=testUserFailedToIndexOffsets > -Dtests.seed=EBBFA6F4E80A7365 -Dtests.slow=true -Dtests.locale=lt_LT > -Dtests.timezone=Europe/Isle_of_Man -Dtests.file.encoding=UTF-8 > [junit4] FAILURE 0.02s J1 | > TestPostingsHighlighter.testUserFailedToIndexOffsets <<< > [junit4] > Throwable #1: java.lang.AssertionError > [junit4] > at > __randomizedtesting.SeedInfo.seed([EBBFA6F4E80A7365:1FBF811885F2D611]:0) > [junit4] > at > org.apache.lucene.index.ByteSliceReader.readByte(ByteSliceReader.java:73) > [junit4] > at > org.apache.lucene.store.DataInput.readVInt(DataInput.java:108) > [junit4] > at > org.apache.lucene.index.FreqProxTermsWriterPerField.flush(FreqProxTermsWriterPerField.java:453) > [junit4] > at > org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:85) > [junit4] > at > org.apache.lucene.index.TermsHash.flush(TermsHash.java:116) > [junit4] > at > org.apache.lucene.index.DocInverter.flush(DocInverter.java:53) > [junit4] > at > org.apache.lucene.index.DocFieldProcessor.flush(DocFieldProcessor.java:81) > [junit4] > at > org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:501) > [junit4] > at > org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:478) > [junit4] > at > org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:615) > [junit4] > at > org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2760) > [junit4] > at > org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2909) > [junit4] > at > org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2884) > [junit4] > at > org.apache.lucene.index.RandomIndexWriter.getReader(RandomIndexWriter.java:312) > [junit4] > at > org.apache.lucene.index.RandomIndexWriter.getReader(RandomIndexWriter.java:249) > [junit4] > at > org.apache.lucene.search.postingshighlight.TestPostingsHighlighter.testUserFailedToIndexOffsets(TestPostingsHighlighter.java:295) > [junit4] > at java.lang.Thread.run(Thread.java:724) > [junit4] 2> NOTE: test params are: codec=Lucene42: > {id=Pulsing41(freqCutoff=3 minBlockSize=2 maxBlockSize=9), > body=Lucene41(blocksize=128), > title=PostingsFormat(name=Lucene41WithOrds)}, docValues:{}, > sim=DefaultSimilarity, locale=lt_LT, timezone=Europe/Isle_of_Man > [junit4] 2> NOTE: Linux 3.5.0-27-generic i386/Oracle Corporation > 1.7.0_25 (32-bit)/cpus=8,threads=1,free=22664728,total=67108864 > [junit4] 2> NOTE: All tests run in this JVM: > [WeightedFragListBuilderTest, IndexTimeSynonymTest, > ScoreOrderFragmentsBuilderTest, HighlighterTest, FieldPhraseListTest, > HighlighterPhraseTest, SimpleFragListBuilderTest, FieldTermStackTest, > SimpleFragmentsBuilderTest, FieldQueryTest, TestPostingsHighlighter] > > > On Sun, Aug 11, 2013 at 5:20 PM, Robert Muir <[email protected]> wrote: >> I tried even harder (with exact JVM, tests.jvms, master seed, svn rev, >> and same JVM args): >> >> At revision 1512807 >> rmuir@beast:~/workspace/branch_4x/lucene/highlighter$ export >> JAVA_HOME=/home/rmuir/Downloads/32bit7/jdk1.7.0_25 >> rmuir@beast:~/workspace/branch_4x/lucene/highlighter$ ant test >> -Dtests.jvms=2 -Dtests.seed=EBBFA6F4E80A7365 -Dargs="-server >> -XX:+UseG1GC" >> >> BUILD SUCCESSFUL >> Total time: 18 seconds >> >> On Sun, Aug 11, 2013 at 2:17 PM, Robert Muir <[email protected]> wrote: >>> I cannot reproduce this. >>> >>> On Sat, Aug 10, 2013 at 1:44 PM, Policeman Jenkins Server >>> <[email protected]> wrote: >>>> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/6874/ >>>> Java: 32bit/jdk1.7.0_25 -server -XX:+UseG1GC >>>> >>>> 1 tests failed. >>>> REGRESSION: >>>> org.apache.lucene.search.postingshighlight.TestPostingsHighlighter.testUserFailedToIndexOffsets >>>> >>>> Error Message: >>>> >>>> >>>> Stack Trace: >>>> java.lang.AssertionError >>>> at >>>> __randomizedtesting.SeedInfo.seed([EBBFA6F4E80A7365:1FBF811885F2D611]:0) >>>> at >>>> org.apache.lucene.index.ByteSliceReader.readByte(ByteSliceReader.java:73) >>>> at org.apache.lucene.store.DataInput.readVInt(DataInput.java:108) >>>> at >>>> org.apache.lucene.index.FreqProxTermsWriterPerField.flush(FreqProxTermsWriterPerField.java:453) >>>> at >>>> org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:85) >>>> at org.apache.lucene.index.TermsHash.flush(TermsHash.java:116) >>>> at org.apache.lucene.index.DocInverter.flush(DocInverter.java:53) >>>> at >>>> org.apache.lucene.index.DocFieldProcessor.flush(DocFieldProcessor.java:81) >>>> at >>>> org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:501) >>>> at >>>> org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:478) >>>> at >>>> org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:615) >>>> at >>>> org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2760) >>>> at >>>> org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2909) >>>> at >>>> org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2884) >>>> at >>>> org.apache.lucene.index.RandomIndexWriter.getReader(RandomIndexWriter.java:312) >>>> at >>>> org.apache.lucene.index.RandomIndexWriter.getReader(RandomIndexWriter.java:249) >>>> at >>>> org.apache.lucene.search.postingshighlight.TestPostingsHighlighter.testUserFailedToIndexOffsets(TestPostingsHighlighter.java:295) >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>> at >>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >>>> at >>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>> at java.lang.reflect.Method.invoke(Method.java:606) >>>> at >>>> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559) >>>> at >>>> com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79) >>>> at >>>> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737) >>>> at >>>> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773) >>>> at >>>> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787) >>>> at >>>> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50) >>>> at >>>> org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51) >>>> at >>>> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) >>>> at >>>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) >>>> at >>>> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49) >>>> at >>>> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70) >>>> at >>>> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) >>>> at >>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) >>>> at >>>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358) >>>> at >>>> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782) >>>> at >>>> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442) >>>> at >>>> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746) >>>> at >>>> com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648) >>>> at >>>> com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682) >>>> at >>>> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693) >>>> at >>>> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) >>>> at >>>> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42) >>>> at >>>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) >>>> at >>>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) >>>> at >>>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) >>>> at >>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) >>>> at >>>> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43) >>>> at >>>> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) >>>> at >>>> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70) >>>> at >>>> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55) >>>> at >>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) >>>> at >>>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358) >>>> at java.lang.Thread.run(Thread.java:724) >>>> >>>> >>>> >>>> >>>> Build Log: >>>> [...truncated 7569 lines...] >>>> [junit4] Suite: >>>> org.apache.lucene.search.postingshighlight.TestPostingsHighlighter >>>> [junit4] 2> NOTE: reproduce with: ant test >>>> -Dtestcase=TestPostingsHighlighter >>>> -Dtests.method=testUserFailedToIndexOffsets -Dtests.seed=EBBFA6F4E80A7365 >>>> -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=lt_LT >>>> -Dtests.timezone=Europe/Isle_of_Man -Dtests.file.encoding=UTF-8 >>>> [junit4] FAILURE 0.04s J1 | >>>> TestPostingsHighlighter.testUserFailedToIndexOffsets <<< >>>> [junit4] > Throwable #1: java.lang.AssertionError >>>> [junit4] > at >>>> __randomizedtesting.SeedInfo.seed([EBBFA6F4E80A7365:1FBF811885F2D611]:0) >>>> [junit4] > at >>>> org.apache.lucene.index.ByteSliceReader.readByte(ByteSliceReader.java:73) >>>> [junit4] > at >>>> org.apache.lucene.store.DataInput.readVInt(DataInput.java:108) >>>> [junit4] > at >>>> org.apache.lucene.index.FreqProxTermsWriterPerField.flush(FreqProxTermsWriterPerField.java:453) >>>> [junit4] > at >>>> org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:85) >>>> [junit4] > at >>>> org.apache.lucene.index.TermsHash.flush(TermsHash.java:116) >>>> [junit4] > at >>>> org.apache.lucene.index.DocInverter.flush(DocInverter.java:53) >>>> [junit4] > at >>>> org.apache.lucene.index.DocFieldProcessor.flush(DocFieldProcessor.java:81) >>>> [junit4] > at >>>> org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:501) >>>> [junit4] > at >>>> org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:478) >>>> [junit4] > at >>>> org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:615) >>>> [junit4] > at >>>> org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2760) >>>> [junit4] > at >>>> org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2909) >>>> [junit4] > at >>>> org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2884) >>>> [junit4] > at >>>> org.apache.lucene.index.RandomIndexWriter.getReader(RandomIndexWriter.java:312) >>>> [junit4] > at >>>> org.apache.lucene.index.RandomIndexWriter.getReader(RandomIndexWriter.java:249) >>>> [junit4] > at >>>> org.apache.lucene.search.postingshighlight.TestPostingsHighlighter.testUserFailedToIndexOffsets(TestPostingsHighlighter.java:295) >>>> [junit4] > at java.lang.Thread.run(Thread.java:724) >>>> [junit4] 2> NOTE: test params are: codec=Lucene42: >>>> {id=Pulsing41(freqCutoff=3 minBlockSize=2 maxBlockSize=9), >>>> body=Lucene41(blocksize=128), >>>> title=PostingsFormat(name=Lucene41WithOrds)}, docValues:{}, >>>> sim=DefaultSimilarity, locale=lt_LT, timezone=Europe/Isle_of_Man >>>> [junit4] 2> NOTE: Linux 3.2.0-51-generic i386/Oracle Corporation >>>> 1.7.0_25 (32-bit)/cpus=8,threads=1,free=37286400,total=67108864 >>>> [junit4] 2> NOTE: All tests run in this JVM: >>>> [WeightedFragListBuilderTest, IndexTimeSynonymTest, >>>> ScoreOrderFragmentsBuilderTest, HighlighterTest, FieldPhraseListTest, >>>> HighlighterPhraseTest, SimpleFragListBuilderTest, FieldTermStackTest, >>>> SimpleFragmentsBuilderTest, FieldQueryTest, TestPostingsHighlighter] >>>> [junit4] Completed on J1 in 2.21s, 25 tests, 1 failure <<< FAILURES! >>>> >>>> [...truncated 25 lines...] >>>> BUILD FAILED >>>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:395: The >>>> following error occurred while executing this line: >>>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:375: The >>>> following error occurred while executing this line: >>>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The >>>> following error occurred while executing this line: >>>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/build.xml:556: The >>>> following error occurred while executing this line: >>>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1887: >>>> The following error occurred while executing this line: >>>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/module-build.xml:61: >>>> The following error occurred while executing this line: >>>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1233: >>>> The following error occurred while executing this line: >>>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:876: >>>> There were test failures: 20 suites, 208 tests, 1 failure >>>> >>>> Total time: 16 minutes 46 seconds >>>> Build step 'Invoke Ant' marked build as failure >>>> Description set: Java: 32bit/jdk1.7.0_25 -server -XX:+UseG1GC >>>> Archiving artifacts >>>> Recording test results >>>> Email was triggered for: Failure >>>> Sending email for trigger: Failure >>>> >>>> >>>> >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: [email protected] >>>> For additional commands, e-mail: [email protected] > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
