[
https://issues.apache.org/jira/browse/LUCENE-4561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Simon Willnauer updated LUCENE-4561:
------------------------------------
Attachment: LUCENE-4561.patch
Phew... glad it reproduced....
So the problem here is the same as it was the last time that the assert is
tripped when multiple threads already crossed the stall check before we
actually stalled. I fixed this last time with checking the number of active
thread states in the pool to get an upper bound but that is actually error
prone. In this test here the thread pool only had one thread state but some of
the indexing threads already went past the stall check and waited on the
DWPTPool to lock that single state. At the same time flushing was so slow
(simple text + throttle) that memory piled up and we stalled. But the
calculation was simply wrong since there were 3 threads waiting for the DWPT
lock but we counted only one in the assert. I added a counter that checks how
many updates we see during a stall period. This give a tighter bound but its
independent of the # of thread states in the pool
> DWPT assert tripped again
> -------------------------
>
> Key: LUCENE-4561
> URL: https://issues.apache.org/jira/browse/LUCENE-4561
> Project: Lucene - Core
> Issue Type: Bug
> Reporter: Michael McCandless
> Assignee: Simon Willnauer
> Attachments: LUCENE-4561.patch
>
>
> TestBagOfPositions tripped the spooky DWPT ram used on flush assert in
> http://jenkins.sd-datasolutions.de/job/Lucene-Solr-4.x-Linux/2472/
> It reproduces for me:
> {noformat}
> ant test -Dtestcase=TestBagOfPositions -Dtests.method=test
> -Dtests.seed=730E05D38A0E4AFA -Dtests.multiplier=3 -Dtests.slow=true
> -Dtests.locale=de_DE_PREEURO -Dtests.timezone=America/Metlakatla
> -Dtests.file.encoding=UTF-8
> {noformat}
> Full failure:
> {noformat}
> [junit4:junit4] 2> NOTE: reproduce with: ant test
> -Dtestcase=TestBagOfPositions -Dtests.method=test
> -Dtests.seed=730E05D38A0E4AFA -Dtests.multiplier=3 -Dtests.slow=true
> -Dtests.locale=de_DE_PREEURO -Dtests.timezone=America/Metlakatla
> -Dtests.file.encoding=UTF-8
> [junit4:junit4] ERROR 63.2s J0 | TestBagOfPositions.test <<<
> [junit4:junit4] > Throwable #1:
> com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an
> uncaught exception in thread: Thread[id=2163, name=Thread-1670,
> state=RUNNABLE, group=TGRP-TestBagOfPositions]
> [junit4:junit4] > Caused by: java.lang.AssertionError: actual mem:
> 33763152 byte, expected mem: 33755888 byte, flush mem: 33610208, active mem:
> 152944, pending DWPT: 0, flushing DWPT: 2, blocked DWPT: 0, peakDelta mem:
> 67152 byte
> [junit4:junit4] > at
> __randomizedtesting.SeedInfo.seed([730E05D38A0E4AFA]:0)
> [junit4:junit4] > at
> org.apache.lucene.index.DocumentsWriterFlushControl.assertMemory(DocumentsWriterFlushControl.java:120)
> [junit4:junit4] > at
> org.apache.lucene.index.DocumentsWriterFlushControl.doAfterDocument(DocumentsWriterFlushControl.java:187)
> [junit4:junit4] > at
> org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:384)
> [junit4:junit4] > at
> org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1451)
> [junit4:junit4] > at
> org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1126)
> [junit4:junit4] > at
> org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:201)
> [junit4:junit4] > at
> org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:160)
> [junit4:junit4] > at
> org.apache.lucene.index.TestBagOfPositions$1.run(TestBagOfPositions.java:111)
> [junit4:junit4] > Throwable #2:
> com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an
> uncaught exception in thread: Thread[id=2164, name=Thread-1671,
> state=RUNNABLE, group=TGRP-TestBagOfPositions]
> [junit4:junit4] > Caused by: java.lang.AssertionError: actual mem:
> 33763152 byte, expected mem: 33755888 byte, flush mem: 33610208, active mem:
> 152944, pending DWPT: 0, flushing DWPT: 2, blocked DWPT: 0, peakDelta mem:
> 67152 byte
> [junit4:junit4] > at
> __randomizedtesting.SeedInfo.seed([730E05D38A0E4AFA]:0)
> [junit4:junit4] > at
> org.apache.lucene.index.DocumentsWriterFlushControl.assertMemory(DocumentsWriterFlushControl.java:120)
> [junit4:junit4] > at
> org.apache.lucene.index.DocumentsWriterFlushControl.doAfterDocument(DocumentsWriterFlushControl.java:187)
> [junit4:junit4] > at
> org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:384)
> [junit4:junit4] > at
> org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1451)
> [junit4:junit4] > at
> org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1126)
> [junit4:junit4] > at
> org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:201)
> [junit4:junit4] > at
> org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:160)
> [junit4:junit4] > at
> org.apache.lucene.index.TestBagOfPositions$1.run(TestBagOfPositions.java:111)
> [junit4:junit4] > Throwable #3:
> com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an
> uncaught exception in thread: Thread[id=2166, name=Thread-1673,
> state=RUNNABLE, group=TGRP-TestBagOfPositions]
> [junit4:junit4] > Caused by: java.lang.AssertionError: actual mem:
> 33763152 byte, expected mem: 33755888 byte, flush mem: 33610208, active mem:
> 152944, pending DWPT: 0, flushing DWPT: 2, blocked DWPT: 0, peakDelta mem:
> 67152 byte
> [junit4:junit4] > at
> __randomizedtesting.SeedInfo.seed([730E05D38A0E4AFA]:0)
> [junit4:junit4] > at
> org.apache.lucene.index.DocumentsWriterFlushControl.assertMemory(DocumentsWriterFlushControl.java:120)
> [junit4:junit4] > at
> org.apache.lucene.index.DocumentsWriterFlushControl.doAfterDocument(DocumentsWriterFlushControl.java:187)
> [junit4:junit4] > at
> org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:384)
> [junit4:junit4] > at
> org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1451)
> [junit4:junit4] > at
> org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1126)
> [junit4:junit4] > at
> org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:201)
> [junit4:junit4] > at
> org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:160)
> [junit4:junit4] > at
> org.apache.lucene.index.TestBagOfPositions$1.run(TestBagOfPositions.java:111)
> [junit4:junit4] 2> NOTE: test params are: codec=SimpleText,
> sim=RandomSimilarityProvider(queryNorm=false,coord=no): {field=DFR I(ne)1},
> locale=de_DE_PREEURO, timezone=America/Metlakatla
> [junit4:junit4] 2> NOTE: Linux 3.2.0-32-generic amd64/IBM Corporation 1.6.0
> (64-bit)/cpus=8,threads=1,free=55064736,total=63793152
> [junit4:junit4] 2> NOTE: All tests run in this JVM: [Nested1,
> TestIndexWriterWithThreads, TestLevenshteinAutomata, TestBasicOperations,
> TestStressNRT, TestIndexWriterExceptions, TestConjunctions,
> TestMultiLevelSkipList, TestConcurrentMergeScheduler,
> TestIndexWriterOnDiskFull, TestBytesRef, TestThreadedForceMerge,
> TestDocument, TestCopyBytes, TestIndexWriterNRTIsCurrent,
> TestScoreCachingWrappingScorer, TestScorerPerf,
> TestDocValuesTypeCompatibility, TestPrefixCodedTerms, TestRollingBuffer,
> TestPhrasePrefixQuery, Test4GBStoredFields, TestCustomSearcherSort,
> TestExplanations, TestIndexInput, TestMultiThreadTermVectors,
> TestTermInfosReaderIndex, TestSearchAfter, Test2BPositions, TestField,
> TestSimpleAttributeImpl, TestFlushByRamOrCountsPolicy, TestSimilarityBase,
> TestByteSlices, TestFlex, TestRecyclingByteBlockAllocator, TestCrash,
> Test2BTerms, TestSearcherManager, TestDeterminism, TestDemo,
> TestSpanExplanationsOfNonMatches, TestSubScorerFreqs, TestDocIdSet,
> TestFieldValueFilter, TestSpanMultiTermQueryWrapper, TestTermVectors,
> TestSurrogates, TestSimilarity2, Nested1, TestParallelReaderEmptyIndex,
> TestShardSearching, TestBlockPostingsFormat3, TestPagedBytes,
> TestCustomNorms, Before3, Before3, TestSegmentReader, TestMatchAllDocsQuery,
> TestTopDocsCollector, TestNoMergeScheduler, TestDeletionPolicy, Nested1,
> ThrowInUncaught, TestMultiTermConstantScore, TestPhraseQuery,
> TestGraphTokenizers, TestBitVector, TestPerFieldPostingsFormat2,
> TestSegmentTermEnum, TestVersion, TestNGramPhraseQuery,
> TestBackwardsCompatibility3x, TestRegexpRandom2, TestDirectoryReaderReopen,
> TestCompoundFile, TestBlockPostingsFormat, TestNRTThreads, TestPayloads,
> TestTransactions, TestTermRangeQuery, TestSmallFloat, TestFSTs, TestNorms,
> TestLookaheadTokenFilter, TestDuelingCodecs, TestAtomicUpdate, TestTermsEnum,
> TestMultiMMap, TestTimeLimitingCollector, TestTopDocsMerge, TestNRTManager,
> TestArrayUtil, TestBufferedIndexInput, TestIndexWriterForceMerge,
> TestIndexWriterCommit, TestWeakIdentityMap, TestTypePromotion,
> TestSimpleExplanations, TestStressIndexing, TestSnapshotDeletionPolicy,
> TestNRTReaderWithThreads, TestTieredMergePolicy, TestConsistentFieldNumbers,
> TestCrashCausesCorruptIndex, TestNumericUtils,
> TestMultiValuedNumericRangeQuery, TestCharTermAttributeImpl,
> TestRollingUpdates, TestPrefixInBooleanQuery, TestBytesRefHash,
> TestRamUsageEstimatorOnWildAnimals, TestFieldCacheRangeFilter,
> TestPayloadSpans, TestMixedCodecs, TestSegmentTermDocs,
> TestFieldCacheSanityChecker, TestDoc, TestMergeSchedulerExternal, TestOmitTf,
> TestDisjunctionMaxQuery, TestSimpleSearchEquivalence, Nested, Nested, Nested,
> Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested,
> Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested,
> TestPayloadNearQuery, TestFilteredQuery, TestPayloadExplanations,
> TestDocsAndPositions, TestTransactionRollback, TestCompiledAutomaton,
> TestSentinelIntSet, TestIndexableField, TestBooleanQuery, TestAutomatonQuery,
> TestComplexExplanationsOfNonMatches, TestRegexpQuery, TestDocCount,
> TestSearchForDuplicates, TestFilteredSearch, NestedSetupChain,
> NestedTeardownChain, Nested, Nested, TestDateFilter, TestSpansAdvanced,
> TestConstantScoreQuery, TestDateTools, TestSearch, TestReaderClosed,
> TestElevationComparator, TestAutomatonQueryUnicode, TestDateSort,
> TestBinaryDocument, TestSpanFirstQuery, TestPriorityQueue, TestDocBoost,
> TestMockCharFilter, TestIsCurrent, TestPrefixFilter, TestBitUtil,
> TestVersionComparator, TestCachingTokenFilter, TestTermdocPerf, TestTerm,
> TestLucene40PostingsFormat, TestAllFilesHaveCodecHeader, TestBagOfPositions]
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
