Thanks Adrien. It's particularly curious that it fails even with -verbose.
It's actually a curious failure: the mergeFactor is 50, and CMS has maxThreadCount 1, which means after flushing 50 segs, a merge (bg thread) kicks off, but then if we manage to flush another 50 segments before that first thread finishes, another merge is requested but then when we ask the merge scheduler to run that merge, it should block the main thread and not return until one of the merges has finished. So now I don't get why it fails under jrockit :) I'll go download it and try to figure it out. Mike McCandless http://blog.mikemccandless.com On Thu, Nov 21, 2013 at 6:36 AM, Adrien Grand <[email protected]> wrote: > Thanks Mike! I was looking at the info stream to see what happens, and > merging seems to work fine, it just seems to be unable to keep up with > segment creations. If someone wants to have a look at the info stream, > I uploaded it at > http://people.apache.org/~jpountz/org.apache.lucene.index.TestIndexWriterReader.testTooManySegments.log > > On Thu, Nov 21, 2013 at 12:17 PM, Michael McCandless > <[email protected]> wrote: >> I think the problem is that it got a LogByteSizeMergePolicy w/ >> mergeFactor=50; not sure why it would fail under JRockit but not other >> JVMs; maybe a thread scheduling difference? >> >> I committed a fix to not use newIndexWriterConfig, so we always get a >> sane (the default) merge policy. >> >> Mike McCandless >> >> http://blog.mikemccandless.com >> >> >> On Thu, Nov 21, 2013 at 5:49 AM, Adrien Grand <[email protected]> wrote: >>> This one is easily reproducible with a JRockit JDK but unreproducible >>> with an Oracle JDK or OpenJDK. >>> >>> On Thu, Nov 21, 2013 at 7:03 AM, Policeman Jenkins Server >>> <[email protected]> wrote: >>>> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/8305/ >>>> Java: 64bit/jrockit-jdk1.6.0_45-R28.2.7-4.1.0 -XnoOpt >>>> >>>> 1 tests failed. >>>> REGRESSION: >>>> org.apache.lucene.index.TestIndexWriterReader.testTooManySegments >>>> >>>> Error Message: >>>> >>>> >>>> Stack Trace: >>>> java.lang.AssertionError >>>> at >>>> __randomizedtesting.SeedInfo.seed([2694FC274A8D4839:6007370D936AF0EA]:0) >>>> at org.junit.Assert.fail(Assert.java:92) >>>> at org.junit.Assert.assertTrue(Assert.java:43) >>>> at org.junit.Assert.assertTrue(Assert.java:54) >>>> at >>>> org.apache.lucene.index.TestIndexWriterReader.testTooManySegments(TestIndexWriterReader.java:1160) >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>> at >>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >>>> at >>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>>> at java.lang.reflect.Method.invoke(Method.java:597) >>>> at >>>> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559) >>>> at >>>> com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79) >>>> at >>>> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:738) >>>> at >>>> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:774) >>>> at >>>> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787) >>>> at >>>> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50) >>>> at >>>> org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51) >>>> at >>>> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) >>>> at >>>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) >>>> at >>>> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49) >>>> at >>>> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70) >>>> at >>>> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) >>>> at >>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) >>>> at >>>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358) >>>> at >>>> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782) >>>> at >>>> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442) >>>> at >>>> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746) >>>> at >>>> com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648) >>>> at >>>> com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:683) >>>> at >>>> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693) >>>> at >>>> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) >>>> at >>>> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42) >>>> at >>>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) >>>> at >>>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) >>>> at >>>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) >>>> at >>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) >>>> at >>>> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:44) >>>> at >>>> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) >>>> at >>>> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70) >>>> at >>>> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:56) >>>> at >>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) >>>> at >>>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358) >>>> at java.lang.Thread.run(Thread.java:662) >>>> >>>> >>>> >>>> >>>> Build Log: >>>> [...truncated 995 lines...] >>>> [junit4] Suite: org.apache.lucene.index.TestIndexWriterReader >>>> [junit4] 2> NOTE: reproduce with: ant test >>>> -Dtestcase=TestIndexWriterReader -Dtests.method=testTooManySegments >>>> -Dtests.seed=2694FC274A8D4839 -Dtests.multiplier=3 -Dtests.slow=true >>>> -Dtests.locale=pl -Dtests.timezone=Asia/Gaza -Dtests.file.encoding=UTF-8 >>>> [junit4] FAILURE 16.1s J1 | TestIndexWriterReader.testTooManySegments >>>> <<< >>>> [junit4] > Throwable #1: java.lang.AssertionError >>>> [junit4] > at >>>> __randomizedtesting.SeedInfo.seed([2694FC274A8D4839:6007370D936AF0EA]:0) >>>> [junit4] > at >>>> org.apache.lucene.index.TestIndexWriterReader.testTooManySegments(TestIndexWriterReader.java:1160) >>>> [junit4] > at java.lang.Thread.run(Thread.java:662) >>>> [junit4] 2> NOTE: test params are: codec=Lucene46: >>>> {f=Pulsing41(freqCutoff=18 minBlockSize=88 maxBlockSize=246), >>>> field5=Lucene41(blocksize=128), id=Lucene41(blocksize=128), >>>> field=Pulsing41(freqCutoff=18 minBlockSize=88 maxBlockSize=246), >>>> field4=Pulsing41(freqCutoff=18 minBlockSize=88 maxBlockSize=246), >>>> field3=PostingsFormat(name=Asserting), field2=Pulsing41(freqCutoff=18 >>>> minBlockSize=88 maxBlockSize=246), foo=Pulsing41(freqCutoff=18 >>>> minBlockSize=88 maxBlockSize=246), field1=Lucene41(blocksize=128), >>>> indexname=PostingsFormat(name=Asserting)}, docValues:{}, >>>> sim=DefaultSimilarity, locale=pl, timezone=Asia/Gaza >>>> [junit4] 2> NOTE: Linux 3.8.0-32-generic amd64/Oracle Corporation >>>> 1.6.0_45 (64-bit)/cpus=8,threads=1,free=183934704,total=536870912 >>>> [junit4] 2> NOTE: All tests run in this JVM: [TestCopyBytes, >>>> TestBasics, TestBinaryTerms, TestMockDirectoryWrapper, >>>> TestCachingTokenFilter, TestIndexWriterUnicode, TestCodecs, >>>> TestAssertions, TestToken, Nested, TestNot, TestMultiDocValues, >>>> TestSegmentMerger, TestBufferedIndexInput, Before3, Before3, >>>> TestSloppyPhraseQuery2, FuzzyTermOnShortTermsTest, TestDuelingCodecs, >>>> TestPayloadNearQuery, TestIndexSearcher, TestAttributeSource, >>>> TestPrefixQuery, TestIntsRef, TestQueryWrapperFilter, TestRegexpRandom2, >>>> TestOpenBitSet, TestMinimize, TestDateFilter, Test2BDocs, >>>> TestLongPostings, MultiCollectorTest, TestPerFieldDocValuesFormat, >>>> TestPerFieldPostingsFormat2, TestDocValuesIndexing, >>>> TestIndexWriterNRTIsCurrent, TestCodecHoldsOpenFiles, >>>> TestIndexWriterWithThreads, TestBackwardsCompatibility3x, >>>> TestIdentityHashSet, TestSimpleSearchEquivalence, TestElevationComparator, >>>> TestHugeRamFile, TestIndexWriterDelete, TestDocValuesFormat, >>>> TestBooleanQuery, TestSnapshotDeletionPolicy, TestIndexWriterOnJRECrash, >>>> TestBackwardsCompatibility, TestDoc, TestIndexInput, TestWAH8DocIdSet, >>>> TestBooleanQueryVisitSubscorers, TestGraphTokenizers, NestedSetupChain, >>>> NestedTeardownChain, TestIntBlockPool, TestDocCount, >>>> TestControlledRealTimeReopenThread, TestCachingWrapperFilter, >>>> TestTermRangeFilter, TestNamedSPILoader, TestConjunctions, >>>> Test4GBStoredFields, TestDocument, TestThreadedForceMerge, TestFieldCache, >>>> TestRateLimiter, TestLucene3xTermVectorsFormat, TestSizeBoundedForceMerge, >>>> TestPositionIncrement, TestRecyclingIntBlockAllocator, >>>> TestFieldMaskingSpanQuery, TestCharsRef, TestSloppyMath, >>>> TestTermVectorsReader, TestSimilarityBase, TestRAMDirectory, Nested1, >>>> TestSortDocValues, TestTransactions, TestTotalHitCountCollector, >>>> TestSimpleExplanationsOfNonMatches, TestFileSwitchDirectory, TestFSTs, >>>> TestTerm, TestNumericTokenStream, TestMultiMMap, Nested, >>>> TestSpanSearchEquivalence, TestDocsAndPositions, TestIntroSorter, >>>> TestSpanExplanations, TestOmitNorms, TestPayloadExplanations, >>>> TestEarlyTermination, TestIndexWriterReader] >>>> [junit4] Completed on J1 in 38.24s, 23 tests, 1 failure <<< FAILURES! >>>> >>>> [...truncated 676 lines...] >>>> BUILD FAILED >>>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:426: The >>>> following error occurred while executing this line: >>>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:406: The >>>> following error occurred while executing this line: >>>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The >>>> following error occurred while executing this line: >>>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/extra-targets.xml:37: The >>>> following error occurred while executing this line: >>>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/build.xml:49: The >>>> following error occurred while executing this line: >>>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1276: >>>> The following error occurred while executing this line: >>>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:912: >>>> There were test failures: 379 suites, 2758 tests, 1 failure, 105 ignored >>>> (92 assumptions) >>>> >>>> Total time: 21 minutes 14 seconds >>>> Build step 'Invoke Ant' marked build as failure >>>> Description set: Java: 64bit/jrockit-jdk1.6.0_45-R28.2.7-4.1.0 -XnoOpt >>>> Archiving artifacts >>>> Recording test results >>>> Email was triggered for: Failure >>>> Sending email for trigger: Failure >>>> >>>> >>>> >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: [email protected] >>>> For additional commands, e-mail: [email protected] >>> >>> >>> >>> -- >>> Adrien >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: [email protected] >>> For additional commands, e-mail: [email protected] >>> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] >> > > > > -- > Adrien > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
