limiting the compressor to only put 128 docs in a chunk works for this: I think Adrien's issue is the best solution https://issues.apache.org/jira/browse/LUCENE-4928
On Thu, Apr 11, 2013 at 6:51 PM, Robert Muir <[email protected]> wrote: > Also this test finishes in 15 seconds if i turn off doRandomForceMerge in > RandomIndexWriter. but im not sure if i mess up the seed, i need to do more > tests. > > I think the current situation is something like this: > 1. test gets AssertingCodec, which means no bulk merge for stored fields > 2. thousands of documents with just "a" as the value, compressing VERY > well into one chunk. > 3. merging (since it cannot use bulk merge), turns into a O(n^2) since its > scanning thru too many docs. > > > On Thu, Apr 11, 2013 at 2:49 PM, Robert Muir <[email protected]> wrote: > >> this seems to be very very slow indexing/merging its index of ~40k docs. >> i ran it and waited for its heart to beat and then dumped stack: >> >> [junit4:junit4] >> "TEST-TestBooleanOr.testBooleanScorerMax-seed#[FB4CB2F81A74301E]" prio=5 >> tid=0x00007faaa40f4800 nid=0x5003 runnable [0x000000011161a000] >> [junit4:junit4] java.lang.Thread.State: RUNNABLE >> [junit4:junit4] at >> org.apache.lucene.store.MockIndexInputWrapper.readBytes(MockIndexInputWrapper.java:118) >> [junit4:junit4] at >> org.apache.lucene.util.packed.PackedReaderIterator.next(PackedReaderIterator.java:76) >> [junit4:junit4] at >> org.apache.lucene.util.packed.PackedInts$ReaderIteratorImpl.next(PackedInts.java:531) >> [junit4:junit4] at >> org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.visitDocument(CompressingStoredFieldsReader.java:238) >> [junit4:junit4] at >> org.apache.lucene.codecs.asserting.AssertingStoredFieldsFormat$AssertingStoredFieldsReader.visitDocument(AssertingStoredFieldsFormat.java:67) >> [junit4:junit4] at >> org.apache.lucene.index.SegmentReader.document(SegmentReader.java:133) >> [junit4:junit4] at >> org.apache.lucene.index.IndexReader.document(IndexReader.java:346) >> [junit4:junit4] at >> org.apache.lucene.codecs.StoredFieldsWriter.merge(StoredFieldsWriter.java:100) >> [junit4:junit4] at >> org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:298) >> [junit4:junit4] at >> org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:86) >> [junit4:junit4] at >> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3664) >> [junit4:junit4] at >> org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3268) >> [junit4:junit4] at >> org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40) >> [junit4:junit4] - locked <0x00000007e07019d0> (a >> org.apache.lucene.index.SerialMergeScheduler) >> >> >> On Thu, Apr 11, 2013 at 11:37 AM, Apache Jenkins Server < >> [email protected]> wrote: >> >>> Build: https://builds.apache.org/job/Lucene-Solr-Tests-trunk-java7/3877/ >>> >>> 2 tests failed. >>> REGRESSION: org.apache.lucene.search.TestBooleanOr.testBooleanScorerMax >>> >>> Error Message: >>> Test abandoned because suite timeout was reached. >>> >>> Stack Trace: >>> java.lang.Exception: Test abandoned because suite timeout was reached. >>> at __randomizedtesting.SeedInfo.seed([FB4CB2F81A74301E]:0) >>> >>> >>> FAILED: junit.framework.TestSuite.org.apache.lucene.search.TestBooleanOr >>> >>> Error Message: >>> Suite timeout exceeded (>= 7200000 msec). >>> >>> Stack Trace: >>> java.lang.Exception: Suite timeout exceeded (>= 7200000 msec). >>> at __randomizedtesting.SeedInfo.seed([FB4CB2F81A74301E]:0) >>> >>> >>> >>> >>> Build Log: >>> [...truncated 1472 lines...] >>> [junit4:junit4] Suite: org.apache.lucene.search.TestBooleanOr >>> [junit4:junit4] 2> NOTE: reproduce with: ant test >>> -Dtestcase=TestBooleanOr -Dtests.method=testBooleanScorerMax >>> -Dtests.seed=FB4CB2F81A74301E -Dtests.multiplier=3 -Dtests.slow=true >>> -Dtests.locale=sr_RS_#Latn -Dtests.timezone=Europe/Amsterdam >>> -Dtests.file.encoding=US-ASCII >>> [junit4:junit4] ERROR 7200s J0 | TestBooleanOr.testBooleanScorerMax <<< >>> [junit4:junit4] > Throwable #1: java.lang.Exception: Test abandoned >>> because suite timeout was reached. >>> [junit4:junit4] > at >>> __randomizedtesting.SeedInfo.seed([FB4CB2F81A74301E]:0) >>> [junit4:junit4] 2> NOTE: test params are: codec=Asserting, >>> sim=RandomSimilarityProvider(queryNorm=false,coord=crazy): {field=DFR >>> I(F)B1, T=DFR I(ne)B2, C=DFR GB2}, locale=sr_RS_#Latn, >>> timezone=Europe/Amsterdam >>> [junit4:junit4] 2> NOTE: FreeBSD 9.0-RELEASE amd64/Oracle Corporation >>> 1.7.0_17 (64-bit)/cpus=16,threads=2,free=322256320,total=505413632 >>> [junit4:junit4] 2> NOTE: All tests run in this JVM: [TestSearchAfter, >>> TestBlockPostingsFormat3, TestAutomatonQuery, TestMockAnalyzer, >>> MultiCollectorTest, Nested1, ThrowInUncaught, TestIndexWriterUnicode, >>> TestCompressingStoredFieldsFormat, TestSloppyPhraseQuery2, >>> TestIdentityHashSet, TestStressNRT, TestSumDocFreq, TestVersionComparator, >>> TestTermVectorsReader, TestTermsEnum, TestDocCount, TestExplanations, >>> TestIndexWriter, TestBooleanQuery, TestPersistentSnapshotDeletionPolicy, >>> TestRollingUpdates, TestPerFieldPostingsFormat2, TestNumericRangeQuery32, >>> TestFieldMaskingSpanQuery, TestNearSpansOrdered, TestDateTools, >>> TestDocsAndPositions, TestSegmentReader, TestNoDeletionPolicy, >>> TestNGramPhraseQuery, TestNRTManager, TestSmallFloat, >>> TestWorstCaseTestBehavior, TestIndexWriterCommit, TestNewestSegment, >>> TestSpanExplanationsOfNonMatches, TestDocument, TestToken, >>> TestTopDocsCollector, TestForUtil, TestWeakIdentityMap, TestIntsRef, >>> TestCollectionUtil, TestSimilarityBase, TestDocumentsWriterDeleteQueue, >>> TestPayloadTermQuery, TestDoc, TestSpanMultiTermQueryWrapper, >>> TestConstantScoreQuery, TestFlushByRamOrCountsPolicy, >>> TestSpanSearchEquivalence, TestSimilarity2, TestIndexWriterConfig, >>> TestPayloadsOnVectors, TestSimilarityProvider, TestLazyProxSkipping, >>> Test2BDocs, TestLucene42DocValuesFormat, TestDocumentsWriterStallControl, >>> Before3, Before3, TestSearch, TestFieldInfos, TestCustomNorms, >>> TestLiveFieldValues, Test2BTerms, TestIndexFileDeleter, TestPriorityQueue, >>> TestBytesStore, TestTerm, TestSorterTemplate, TestDocIdSet, >>> TestIndexWriterMerging, TestUnicodeUtil, TestLucene40DocValuesFormat, >>> TestPositionIncrement, TestFilteredSearch, TestTwoPhaseCommitTool, >>> TestOmitPositions, TestSubScorerFreqs, TestCompressingTermVectorsFormat, >>> TestFieldCacheRewriteMethod, TestMergeSchedulerExternal, >>> TestCharTermAttributeImpl, TestSloppyPhraseQuery, Test2BPositions, >>> TestIndexWriterReader, TestLucene40TermVectorsFormat, TestFSTs, >>> TestNumericRangeQuery64, TestLookaheadTokenFilter, TestShardSearching, >>> TestDeletionPolicy, TestBooleanOr] >>> [junit4:junit4] 2> NOTE: reproduce with: ant test >>> -Dtestcase=TestBooleanOr -Dtests.seed=FB4CB2F81A74301E >>> -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=sr_RS_#Latn >>> -Dtests.timezone=Europe/Amsterdam -Dtests.file.encoding=US-ASCII >>> [junit4:junit4] ERROR 0.00s J0 | TestBooleanOr (suite) <<< >>> [junit4:junit4] > Throwable #1: java.lang.Exception: Suite timeout >>> exceeded (>= 7200000 msec). >>> [junit4:junit4] > at >>> __randomizedtesting.SeedInfo.seed([FB4CB2F81A74301E]:0) >>> [junit4:junit4] Completed on J0 in 7223.29s, 5 tests, 2 errors <<< >>> FAILURES! >>> >>> [...truncated 11 lines...] >>> BUILD FAILED >>> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/build.xml:375: >>> The following error occurred while executing this line: >>> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/build.xml:355: >>> The following error occurred while executing this line: >>> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/build.xml:39: >>> The following error occurred while executing this line: >>> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/lucene/build.xml:50: >>> The following error occurred while executing this line: >>> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/lucene/common-build.xml:1231: >>> The following error occurred while executing this line: >>> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/lucene/common-build.xml:875: >>> There were test failures: 351 suites, 2210 tests, 1 suite-level error, 1 >>> error, 42 ignored (30 assumptions) >>> >>> Total time: 128 minutes 16 seconds >>> Build step 'Invoke Ant' marked build as failure >>> Archiving artifacts >>> Recording test results >>> Email was triggered for: Failure >>> Sending email for trigger: Failure >>> >>> >>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: [email protected] >>> For additional commands, e-mail: [email protected] >>> >> >> >
