[
https://issues.apache.org/jira/browse/LUCENE-8358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Simon Willnauer updated LUCENE-8358:
------------------------------------
Attachment: LUCENE-8358.patch
> Asserting trips when IW tries to free ram by writing DV updates
> ---------------------------------------------------------------
>
> Key: LUCENE-8358
> URL: https://issues.apache.org/jira/browse/LUCENE-8358
> Project: Lucene - Core
> Issue Type: Bug
> Affects Versions: 7.4, master (8.0), 7.5
> Reporter: Simon Willnauer
> Priority: Blocker
> Fix For: 7.4, master (8.0), 7.5
>
> Attachments: LUCENE-8358.patch, LUCENE-8358.patch
>
>
> This assertion is pretty new I think we need to relax is since there are
> chances that this situation is valid ie. if a ReadersAndUpdates instance gets
> concurrently written and readerpooling is off. That is just fine since this
> is best effort anyway. I will attach a patch.
> {noformat}
> 07:35:14 [junit4] Suite: org.apache.lucene.index.TestBinaryDocValuesUpdates
> 07:35:14 [junit4] IGNOR/A 0.01s J0 |
> TestBinaryDocValuesUpdates.testTonsOfUpdates
> 07:35:14 [junit4] > Assumption #1: 'nightly' test group is disabled
> (@Nightly())
> 07:35:14 [junit4] 1> TEST: isNRT=false
> reader1=StandardDirectoryReader(segments_1:4 _0(7.5.0):c2)
> 07:35:14 [junit4] 1> TEST: now reopen
> 07:35:14 [junit4] 2> NOTE: reproduce with: ant test
> -Dtestcase=TestBinaryDocValuesUpdates -Dtests.method=testUpdatesAreFlushed
> -Dtests.seed=B8D5250C8CAA9010 -Dtests.slow=true -Dtests.badapples=true
> -Dtests.locale=en-IE -Dtests.timezone=SystemV/CST6 -Dtests.asserts=true
> -Dtests.file.encoding=UTF8
> 07:35:14 [junit4] FAILURE 0.04s J0 |
> TestBinaryDocValuesUpdates.testUpdatesAreFlushed <<<
> 07:35:14 [junit4] > Throwable #1: java.lang.AssertionError: Segment
> [_2(7.5.0):c1] is not dropped yet
> 07:35:14 [junit4] > at
> __randomizedtesting.SeedInfo.seed([B8D5250C8CAA9010:8228ECC925943F29]:0)
> 07:35:14 [junit4] > at
> org.apache.lucene.index.IndexWriter.writeSomeDocValuesUpdates(IndexWriter.java:613)
> 07:35:14 [junit4] > at
> org.apache.lucene.index.FrozenBufferedUpdates.apply(FrozenBufferedUpdates.java:298)
> 07:35:14 [junit4] > at
> org.apache.lucene.index.IndexWriter.lambda$publishFrozenUpdates$3(IndexWriter.java:2594)
> 07:35:14 [junit4] > at
> org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5064)
> 07:35:14 [junit4] > at
> org.apache.lucene.index.IndexWriter.updateBinaryDocValue(IndexWriter.java:1742)
> 07:35:14 [junit4] > at
> org.apache.lucene.index.TestBinaryDocValuesUpdates.testUpdatesAreFlushed(TestBinaryDocValuesUpdates.java:100)
> 07:35:14 [junit4] > at java.lang.Thread.run(Thread.java:748)
> 07:35:14 [junit4] 2> Jun 15, 2018 1:35:14 AM
> com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler
> uncaughtException
> 07:35:14 [junit4] 2> WARNING: Uncaught exception in thread:
> Thread[Lucene Merge Thread #1,5,TGRP-TestBinaryDocValuesUpdates]
> 07:35:14 [junit4] 2> org.apache.lucene.index.MergePolicy$MergeException:
> java.lang.IllegalStateException: this writer hit an unrecoverable error;
> cannot merge
> 07:35:14 [junit4] 2> at
> __randomizedtesting.SeedInfo.seed([B8D5250C8CAA9010]:0)
> 07:35:14 [junit4] 2> at
> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:704)
> 07:35:14 [junit4] 2> at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:684)
> 07:35:14 [junit4] 2> Caused by: java.lang.IllegalStateException: this
> writer hit an unrecoverable error; cannot merge
> 07:35:14 [junit4] 2> at
> org.apache.lucene.index.IndexWriter._mergeInit(IndexWriter.java:4222)
> 07:35:14 [junit4] 2> at
> org.apache.lucene.index.IndexWriter.mergeInit(IndexWriter.java:4202)
> 07:35:14 [junit4] 2> at
> org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4054)
> 07:35:14 [junit4] 2> at
> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:625)
> 07:35:14 [junit4] 2> at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:662)
> 07:35:14 [junit4] 2> Caused by: java.lang.AssertionError: Segment
> [_2(7.5.0):c1] is not dropped yet
> 07:35:14 [junit4] 2> at
> __randomizedtesting.SeedInfo.seed([B8D5250C8CAA9010:8228ECC925943F29]:0)
> 07:35:14 [junit4] 2> at
> org.apache.lucene.index.IndexWriter.writeSomeDocValuesUpdates(IndexWriter.java:613)
> 07:35:14 [junit4] 2> at
> org.apache.lucene.index.FrozenBufferedUpdates.apply(FrozenBufferedUpdates.java:298)
> 07:35:14 [junit4] 2> at
> org.apache.lucene.index.IndexWriter.lambda$publishFrozenUpdates$3(IndexWriter.java:2594)
> 07:35:14 [junit4] 2> at
> org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5064)
> 07:35:14 [junit4] 2> at
> org.apache.lucene.index.IndexWriter.updateBinaryDocValue(IndexWriter.java:1742)
> 07:35:14 [junit4] 2> at
> org.apache.lucene.index.TestBinaryDocValuesUpdates.testUpdatesAreFlushed(TestBinaryDocValuesUpdates.java:100)
> 07:35:14 [junit4] 2> at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 07:35:14 [junit4] 2> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 07:35:14 [junit4] 2> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 07:35:14 [junit4] 2> at
> java.lang.reflect.Method.invoke(Method.java:498)
> 07:35:14 [junit4] 2> at
> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
> 07:35:14 [junit4] 2> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
> 07:35:14 [junit4] 2> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
> 07:35:14 [junit4] 2> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
> 07:35:14 [junit4] 2> at
> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
> 07:35:14 [junit4] 2> at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
> 07:35:14 [junit4] 2> at
> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
> 07:35:14 [junit4] 2> at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
> 07:35:14 [junit4] 2> at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
> 07:35:14 [junit4] 2> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> 07:35:14 [junit4] 2> at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
> 07:35:14 [junit4] 2> at
> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
> 07:35:14 [junit4] 2> at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
> 07:35:14 [junit4] 2> at
> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
> 07:35:14 [junit4] 2> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
> 07:35:14 [junit4] 2> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
> 07:35:14 [junit4] 2> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
> 07:35:14 [junit4] 2> at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
> 07:35:14 [junit4] 2> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> 07:35:14 [junit4] 2> at
> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
> 07:35:14 [junit4] 2> at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> 07:35:14 [junit4] 2> at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> 07:35:14 [junit4] 2> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> 07:35:14 [junit4] 2> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> 07:35:14 [junit4] 2> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> 07:35:14 [junit4] 2> at
> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
> 07:35:14 [junit4] 2> at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
> 07:35:14 [junit4] 2> at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
> 07:35:14 [junit4] 2> at
> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
> 07:35:14 [junit4] 2> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> 07:35:14 [junit4] 2> at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
> 07:35:14 [junit4] 2> at java.lang.Thread.run(Thread.java:748)
> 07:35:14 [junit4] 2>
> 07:35:14 [junit4] 2> NOTE: reproduce with: ant test
> -Dtestcase=TestBinaryDocValuesUpdates
> -Dtests.method=testUpdateTwoNonexistingTerms -Dtests.seed=B8D5250C8CAA9010
> -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=en-IE
> -Dtests.timezone=SystemV/CST6 -Dtests.asserts=true -Dtests.file.encoding=UTF8
> 07:35:14 [junit4] ERROR 0.01s J0 |
> TestBinaryDocValuesUpdates.testUpdateTwoNonexistingTerms <<<
> 07:35:14 [junit4] > Throwable #1:
> com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an
> uncaught exception in thread: Thread[id=806, name=Lucene Merge Thread #1,
> state=RUNNABLE, group=TGRP-TestBinaryDocValuesUpdates]
> 07:35:14 [junit4] > at
> __randomizedtesting.SeedInfo.seed([B8D5250C8CAA9010:C6CD8D9072E639EE]:0)
> 07:35:14 [junit4] > Caused by:
> org.apache.lucene.index.MergePolicy$MergeException:
> java.lang.IllegalStateException: this writer hit an unrecoverable error;
> cannot merge
> 07:35:14 [junit4] > at
> __randomizedtesting.SeedInfo.seed([B8D5250C8CAA9010]:0)
> 07:35:14 [junit4] > at
> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:704)
> 07:35:14 [junit4] > at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:684)
> 07:35:14 [junit4] > Caused by: java.lang.IllegalStateException: this
> writer hit an unrecoverable error; cannot merge
> 07:35:14 [junit4] > at
> org.apache.lucene.index.IndexWriter._mergeInit(IndexWriter.java:4222)
> 07:35:14 [junit4] > at
> org.apache.lucene.index.IndexWriter.mergeInit(IndexWriter.java:4202)
> 07:35:14 [junit4] > at
> org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4054)
> 07:35:14 [junit4] > at
> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:625)
> 07:35:14 [junit4] > at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:662)
> 07:35:14 [junit4] > Caused by: java.lang.AssertionError: Segment
> [_2(7.5.0):c1] is not dropped yet
> 07:35:14 [junit4] > at
> __randomizedtesting.SeedInfo.seed([B8D5250C8CAA9010:8228ECC925943F29]:0)
> 07:35:14 [junit4] > at
> org.apache.lucene.index.IndexWriter.writeSomeDocValuesUpdates(IndexWriter.java:613)
> 07:35:14 [junit4] > at
> org.apache.lucene.index.FrozenBufferedUpdates.apply(FrozenBufferedUpdates.java:298)
> 07:35:14 [junit4] > at
> org.apache.lucene.index.IndexWriter.lambda$publishFrozenUpdates$3(IndexWriter.java:2594)
> 07:35:14 [junit4] > at
> org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5064)
> 07:35:14 [junit4] > at
> org.apache.lucene.index.IndexWriter.updateBinaryDocValue(IndexWriter.java:1742)
> 07:35:14 [junit4] > at
> org.apache.lucene.index.TestBinaryDocValuesUpdates.testUpdatesAreFlushed(TestBinaryDocValuesUpdates.java:100)
> 07:35:14 [junit4] > at java.lang.Thread.run(Thread.java:748)
> 07:35:14 [junit4] 2> NOTE: test params are: codec=Asserting(Lucene70):
> {bdv=BlockTreeOrds(blocksize=128), f=PostingsFormat(name=Asserting),
> k1=PostingsFormat(name=Asserting), foo=PostingsFormat(name=Asserting),
> dvUpdateKey=PostingsFormat(name=Asserting), k2=Lucene50(blocksize=128),
> upd=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))),
> updKey=BlockTreeOrds(blocksize=128), id=Lucene50(blocksize=128),
> key=Lucene50(blocksize=128)}, docValues:{val=DocValuesFormat(name=Direct),
> ndv=DocValuesFormat(name=Lucene70), cf=DocValuesFormat(name=Asserting),
> ssdv=DocValuesFormat(name=Lucene70), sdv=DocValuesFormat(name=Direct),
> f=DocValuesFormat(name=Lucene70), control=DocValuesFormat(name=Direct),
> f0=DocValuesFormat(name=Lucene70), sort=DocValuesFormat(name=Lucene70),
> f1=DocValuesFormat(name=Asserting), f2=DocValuesFormat(name=Lucene70),
> cf0=DocValuesFormat(name=Direct), f3=DocValuesFormat(name=Direct),
> f4=DocValuesFormat(name=Lucene70), bdv2=DocValuesFormat(name=Lucene70),
> number=DocValuesFormat(name=Direct), bdv1=DocValuesFormat(name=Direct),
> bdv=DocValuesFormat(name=Lucene70), id=DocValuesFormat(name=Asserting),
> key=DocValuesFormat(name=Asserting)}, maxPointsInLeafNode=747,
> maxMBSortInHeap=5.830077125785203, sim=RandomSimilarity(queryNorm=true): {},
> locale=en-IE, timezone=SystemV/CST6
> 07:35:14 [junit4] 2> NOTE: Linux 4.4.0-1057-aws amd64/Oracle Corporation
> 1.8.0_171 (64-bit)/cpus=16,threads=1,free=140024800,total=343932928
> 07:35:14 [junit4] 2> NOTE: All tests run in this JVM:
> [TestSimilarityBase, TestMatchAllDocsQuery, TestSortRescorer,
> TestBagOfPostings, TestNumericUtils, TestThreadedForceMerge,
> TestLucene50LiveDocsFormat, TestIndexableField,
> TestIndexWriterThreadsToSegments, TestSnapshotDeletionPolicy,
> TestFilterDirectoryReader, TestWildcardRandom, TestLongPostings,
> TestTermScorer, TestFutureObjects, TestLucene70NormsFormat,
> TestFixedBitDocIdSet, TestParallelLeafReader, TestFieldInfos,
> TestLazyProxSkipping, TestBoolean2, TermInSetQueryTest,
> TestExitableDirectoryReader, TestPositiveScoresOnlyCollector,
> TestMixedCodecs, TestDisjunctionMaxQuery, TestSimpleSearchEquivalence,
> TestDocValuesQueries, TestSpanMultiTermQueryWrapper, TestMultiset,
> TestManyFields, TestDocValuesFieldUpdates, TestCustomNorms,
> TestFloatRangeFieldQueries, TestMultiThreadTermVectors,
> TestSimpleFSDirectory, TestDateSort, TestRAMDirectory, TestOrdinalMap,
> TestAxiomaticSimilarity, TestFilterSpans, TestNoMergeScheduler,
> TestGeoEncodingUtils, TestDateTools, TestFieldType,
> TestParallelCompositeReader, TestLucene60FieldInfoFormat,
> TestBufferedIndexInput, TestPrefixQuery, TestIndexWriterWithThreads,
> TestSpansEnum, TestSynonymQuery, TestIndexWriterExceptions, TestIntroSorter,
> TestBooleanOr, TestTrackingDirectoryWrapper, Test2BNumericDocValues,
> TestBinaryDocument, TestStressAdvance, TestReusableStringReader,
> TestIndexWriter, TestGraphTokenizers, TestShardSearching, TestNRTThreads,
> TestStandardAnalyzer, TestNeverDelete, TestTieredMergePolicy, TestByteSlices,
> TestLockFactory, TestPhraseQuery, TestFixedBitSet, TestComplexExplanations,
> TestRegexpRandom2, TestSegmentReader, TestIndexWriterNRTIsCurrent,
> TestPagedBytes, TestCrash, TestFieldsReader, TestTermsEnum2, Test2BDocs,
> TestIndexWriterOnDiskFull, TestFlex, TestFastCompressionMode,
> TestDeterminism, TestCharTermAttributeImpl, TestOmitTf, TestNearSpansOrdered,
> TestTermRangeQuery, TestDocumentWriter, TestLiveFieldValues,
> TestForTooMuchCloning, TestFuzzyQuery, TestTopDocsCollector,
> TestComplexExplanationsOfNonMatches, TestParallelReaderEmptyIndex,
> TestTopFieldCollector, TestDirectory, TestCheckIndex, TestPositionIncrement,
> TestVirtualMethod, TestSetOnce, TestCachingTokenFilter, TestDocCount,
> TestTotalHitCountCollector, TestSpanFirstQuery, TestNoMergePolicy,
> TestByteBlockPool, TestRollback, Test4GBStoredFields,
> TestScoreCachingWrappingScorer, Test2BPositions, TestByteArrayDataInput,
> TestDelegatingAnalyzerWrapper, TestPackedTokenAttributeImpl,
> TestLucene50CompoundFormat, TestIndexedDISI, TestBinaryDocValuesUpdates]
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
