Also this test finishes in 15 seconds if i turn off doRandomForceMerge in RandomIndexWriter. but im not sure if i mess up the seed, i need to do more tests.
I think the current situation is something like this: 1. test gets AssertingCodec, which means no bulk merge for stored fields 2. thousands of documents with just "a" as the value, compressing VERY well into one chunk. 3. merging (since it cannot use bulk merge), turns into a O(n^2) since its scanning thru too many docs. On Thu, Apr 11, 2013 at 2:49 PM, Robert Muir <[email protected]> wrote: > this seems to be very very slow indexing/merging its index of ~40k docs. i > ran it and waited for its heart to beat and then dumped stack: > > [junit4:junit4] > "TEST-TestBooleanOr.testBooleanScorerMax-seed#[FB4CB2F81A74301E]" prio=5 > tid=0x00007faaa40f4800 nid=0x5003 runnable [0x000000011161a000] > [junit4:junit4] java.lang.Thread.State: RUNNABLE > [junit4:junit4] at > org.apache.lucene.store.MockIndexInputWrapper.readBytes(MockIndexInputWrapper.java:118) > [junit4:junit4] at > org.apache.lucene.util.packed.PackedReaderIterator.next(PackedReaderIterator.java:76) > [junit4:junit4] at > org.apache.lucene.util.packed.PackedInts$ReaderIteratorImpl.next(PackedInts.java:531) > [junit4:junit4] at > org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.visitDocument(CompressingStoredFieldsReader.java:238) > [junit4:junit4] at > org.apache.lucene.codecs.asserting.AssertingStoredFieldsFormat$AssertingStoredFieldsReader.visitDocument(AssertingStoredFieldsFormat.java:67) > [junit4:junit4] at > org.apache.lucene.index.SegmentReader.document(SegmentReader.java:133) > [junit4:junit4] at > org.apache.lucene.index.IndexReader.document(IndexReader.java:346) > [junit4:junit4] at > org.apache.lucene.codecs.StoredFieldsWriter.merge(StoredFieldsWriter.java:100) > [junit4:junit4] at > org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:298) > [junit4:junit4] at > org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:86) > [junit4:junit4] at > org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3664) > [junit4:junit4] at > org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3268) > [junit4:junit4] at > org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40) > [junit4:junit4] - locked <0x00000007e07019d0> (a > org.apache.lucene.index.SerialMergeScheduler) > > > On Thu, Apr 11, 2013 at 11:37 AM, Apache Jenkins Server < > [email protected]> wrote: > >> Build: https://builds.apache.org/job/Lucene-Solr-Tests-trunk-java7/3877/ >> >> 2 tests failed. >> REGRESSION: org.apache.lucene.search.TestBooleanOr.testBooleanScorerMax >> >> Error Message: >> Test abandoned because suite timeout was reached. >> >> Stack Trace: >> java.lang.Exception: Test abandoned because suite timeout was reached. >> at __randomizedtesting.SeedInfo.seed([FB4CB2F81A74301E]:0) >> >> >> FAILED: junit.framework.TestSuite.org.apache.lucene.search.TestBooleanOr >> >> Error Message: >> Suite timeout exceeded (>= 7200000 msec). >> >> Stack Trace: >> java.lang.Exception: Suite timeout exceeded (>= 7200000 msec). >> at __randomizedtesting.SeedInfo.seed([FB4CB2F81A74301E]:0) >> >> >> >> >> Build Log: >> [...truncated 1472 lines...] >> [junit4:junit4] Suite: org.apache.lucene.search.TestBooleanOr >> [junit4:junit4] 2> NOTE: reproduce with: ant test >> -Dtestcase=TestBooleanOr -Dtests.method=testBooleanScorerMax >> -Dtests.seed=FB4CB2F81A74301E -Dtests.multiplier=3 -Dtests.slow=true >> -Dtests.locale=sr_RS_#Latn -Dtests.timezone=Europe/Amsterdam >> -Dtests.file.encoding=US-ASCII >> [junit4:junit4] ERROR 7200s J0 | TestBooleanOr.testBooleanScorerMax <<< >> [junit4:junit4] > Throwable #1: java.lang.Exception: Test abandoned >> because suite timeout was reached. >> [junit4:junit4] > at >> __randomizedtesting.SeedInfo.seed([FB4CB2F81A74301E]:0) >> [junit4:junit4] 2> NOTE: test params are: codec=Asserting, >> sim=RandomSimilarityProvider(queryNorm=false,coord=crazy): {field=DFR >> I(F)B1, T=DFR I(ne)B2, C=DFR GB2}, locale=sr_RS_#Latn, >> timezone=Europe/Amsterdam >> [junit4:junit4] 2> NOTE: FreeBSD 9.0-RELEASE amd64/Oracle Corporation >> 1.7.0_17 (64-bit)/cpus=16,threads=2,free=322256320,total=505413632 >> [junit4:junit4] 2> NOTE: All tests run in this JVM: [TestSearchAfter, >> TestBlockPostingsFormat3, TestAutomatonQuery, TestMockAnalyzer, >> MultiCollectorTest, Nested1, ThrowInUncaught, TestIndexWriterUnicode, >> TestCompressingStoredFieldsFormat, TestSloppyPhraseQuery2, >> TestIdentityHashSet, TestStressNRT, TestSumDocFreq, TestVersionComparator, >> TestTermVectorsReader, TestTermsEnum, TestDocCount, TestExplanations, >> TestIndexWriter, TestBooleanQuery, TestPersistentSnapshotDeletionPolicy, >> TestRollingUpdates, TestPerFieldPostingsFormat2, TestNumericRangeQuery32, >> TestFieldMaskingSpanQuery, TestNearSpansOrdered, TestDateTools, >> TestDocsAndPositions, TestSegmentReader, TestNoDeletionPolicy, >> TestNGramPhraseQuery, TestNRTManager, TestSmallFloat, >> TestWorstCaseTestBehavior, TestIndexWriterCommit, TestNewestSegment, >> TestSpanExplanationsOfNonMatches, TestDocument, TestToken, >> TestTopDocsCollector, TestForUtil, TestWeakIdentityMap, TestIntsRef, >> TestCollectionUtil, TestSimilarityBase, TestDocumentsWriterDeleteQueue, >> TestPayloadTermQuery, TestDoc, TestSpanMultiTermQueryWrapper, >> TestConstantScoreQuery, TestFlushByRamOrCountsPolicy, >> TestSpanSearchEquivalence, TestSimilarity2, TestIndexWriterConfig, >> TestPayloadsOnVectors, TestSimilarityProvider, TestLazyProxSkipping, >> Test2BDocs, TestLucene42DocValuesFormat, TestDocumentsWriterStallControl, >> Before3, Before3, TestSearch, TestFieldInfos, TestCustomNorms, >> TestLiveFieldValues, Test2BTerms, TestIndexFileDeleter, TestPriorityQueue, >> TestBytesStore, TestTerm, TestSorterTemplate, TestDocIdSet, >> TestIndexWriterMerging, TestUnicodeUtil, TestLucene40DocValuesFormat, >> TestPositionIncrement, TestFilteredSearch, TestTwoPhaseCommitTool, >> TestOmitPositions, TestSubScorerFreqs, TestCompressingTermVectorsFormat, >> TestFieldCacheRewriteMethod, TestMergeSchedulerExternal, >> TestCharTermAttributeImpl, TestSloppyPhraseQuery, Test2BPositions, >> TestIndexWriterReader, TestLucene40TermVectorsFormat, TestFSTs, >> TestNumericRangeQuery64, TestLookaheadTokenFilter, TestShardSearching, >> TestDeletionPolicy, TestBooleanOr] >> [junit4:junit4] 2> NOTE: reproduce with: ant test >> -Dtestcase=TestBooleanOr -Dtests.seed=FB4CB2F81A74301E >> -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=sr_RS_#Latn >> -Dtests.timezone=Europe/Amsterdam -Dtests.file.encoding=US-ASCII >> [junit4:junit4] ERROR 0.00s J0 | TestBooleanOr (suite) <<< >> [junit4:junit4] > Throwable #1: java.lang.Exception: Suite timeout >> exceeded (>= 7200000 msec). >> [junit4:junit4] > at >> __randomizedtesting.SeedInfo.seed([FB4CB2F81A74301E]:0) >> [junit4:junit4] Completed on J0 in 7223.29s, 5 tests, 2 errors <<< >> FAILURES! >> >> [...truncated 11 lines...] >> BUILD FAILED >> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/build.xml:375: >> The following error occurred while executing this line: >> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/build.xml:355: >> The following error occurred while executing this line: >> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/build.xml:39: >> The following error occurred while executing this line: >> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/lucene/build.xml:50: >> The following error occurred while executing this line: >> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/lucene/common-build.xml:1231: >> The following error occurred while executing this line: >> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/lucene/common-build.xml:875: >> There were test failures: 351 suites, 2210 tests, 1 suite-level error, 1 >> error, 42 ignored (30 assumptions) >> >> Total time: 128 minutes 16 seconds >> Build step 'Invoke Ant' marked build as failure >> Archiving artifacts >> Recording test results >> Email was triggered for: Failure >> Sending email for trigger: Failure >> >> >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] >> > >
