[ 
https://issues.apache.org/jira/browse/LUCENE-8243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael McCandless updated LUCENE-8243:
---------------------------------------
    Attachment: LUCENE-8243.patch

> IndexWriter might delete DV update files if addIndices are invovled
> -------------------------------------------------------------------
>
>                 Key: LUCENE-8243
>                 URL: https://issues.apache.org/jira/browse/LUCENE-8243
>             Project: Lucene - Core
>          Issue Type: Bug
>    Affects Versions: 7.4, master (8.0)
>            Reporter: Simon Willnauer
>            Assignee: Michael McCandless
>            Priority: Major
>             Fix For: 7.4, master (8.0)
>
>         Attachments: LUCENE-8243.patch, broken_dv_update.patch
>
>
> the attached test fails with this output:
> {noformat}
> /Library/Java/JavaVirtualMachines/jdk-9.0.1.jdk/Contents/Home/bin/java -ea 
> -Djava.security.egd=file:/dev/./urandom 
> -Didea.test.cyclic.buffer.size=1048576 -Dfile.encoding=UTF-8 -classpath 
> "/Applications/IntelliJ 
> IDEA.app/Contents/lib/idea_rt.jar:/Applications/IntelliJ 
> IDEA.app/Contents/plugins/junit/lib/junit-rt.jar:/Applications/IntelliJ 
> IDEA.app/Contents/plugins/junit/lib/junit5-rt.jar:/Users/simonw/projects/lucene-solr/idea-build/lucene/test-framework/classes/test:/Users/simonw/projects/lucene-solr/idea-build/lucene/test-framework/classes/java:/Users/simonw/projects/lucene-solr/lucene/test-framework/lib/junit-4.10.jar:/Users/simonw/projects/lucene-solr/lucene/test-framework/lib/randomizedtesting-runner-2.5.3.jar:/Users/simonw/projects/lucene-solr/idea-build/lucene/codecs/classes/java:/Users/simonw/projects/lucene-solr/idea-build/lucene/core/classes/java:/Users/simonw/projects/lucene-solr/idea-build/lucene/core/classes/test"
>  com.intellij.rt.execution.junit.JUnitStarter -ideVersion5 -junit4 
> org.apache.lucene.index.TestAddIndexes,testAddIndexesDVUpdate
> IFD 0 [2018-04-06T19:27:27.176036Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> current segments file is "segments_1"; 
> deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@27cf18f0
> IFD 0 [2018-04-06T19:27:27.188066Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> load commit "segments_1"
> IFD 0 [2018-04-06T19:27:27.189800Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> seg=_0 set nextWriteDelGen=2 vs current=1
> IFD 0 [2018-04-06T19:27:27.190053Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> removing unreferenced file "_0_1_Lucene70_0.dvd"
> IFD 0 [2018-04-06T19:27:27.190224Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> removing unreferenced file "_0_1.fnm"
> IFD 0 [2018-04-06T19:27:27.190371Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> removing unreferenced file "_0_1_Lucene70_0.dvm"
> IFD 0 [2018-04-06T19:27:27.190528Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: delete 
> [_0_1_Lucene70_0.dvd, _0_1.fnm, _0_1_Lucene70_0.dvm]
> IFD 0 [2018-04-06T19:27:27.192558Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: now 
> checkpoint "_0(8.0.0):C1:fieldInfosGen=1:dvGen=1" [1 segments ; isCommit = 
> false]
> IFD 0 [2018-04-06T19:27:27.192806Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: 0 msec 
> to checkpoint
> IW 0 [2018-04-06T19:27:27.193012Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> create=false
> IW 0 [2018-04-06T19:27:27.193428Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: 
> dir=MockDirectoryWrapper(RAMDirectory@79d3c690 
> lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@795a0c8b)
> index=_0(8.0.0):C1:fieldInfosGen=1:dvGen=1
> version=8.0.0
> analyzer=org.apache.lucene.analysis.MockAnalyzer
> ramBufferSizeMB=16.0
> maxBufferedDocs=503
> mergedSegmentWarmer=null
> delPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy
> commit=null
> openMode=CREATE_OR_APPEND
> similarity=org.apache.lucene.search.similarities.AssertingSimilarity
> mergeScheduler=org.apache.lucene.index.SerialMergeScheduler@2f3feff6
> codec=FastDecompressionCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST_DECOMPRESSION,
>  chunkSize=8, maxDocsPerChunk=6, blockSize=201), 
> termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST_DECOMPRESSION,
>  chunkSize=8, blockSize=201))
> infoStream=org.apache.lucene.util.PrintStreamInfoStream
> mergePolicy=[TieredMergePolicy: maxMergeAtOnce=41, maxMergeAtOnceExplicit=44, 
> maxMergedSegmentMB=6.255859375, floorSegmentMB=0.38671875, 
> forceMergeDeletesPctAllowed=4.456652110760543, segmentsPerTier=31.0, 
> maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.877330376985384
> indexerThreadPool=org.apache.lucene.index.DocumentsWriterPerThreadPool@257ebcdb
> readerPooling=true
> perThreadHardLimitMB=1945
> useCompoundFile=true
> commitOnClose=true
> indexSort=null
> checkPendingFlushOnUpdate=true
> softDeletesField=null
> writer=org.apache.lucene.index.IndexWriter@17b77ed5
> IW 0 [2018-04-06T19:27:27.194085Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: 
> MMapDirectory.UNMAP_SUPPORTED=true
> IW 0 [2018-04-06T19:27:27.194347Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: now 
> flush at close
> IW 0 [2018-04-06T19:27:27.194630Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]:   start 
> flush: applyAllDeletes=true
> IW 0 [2018-04-06T19:27:27.194864Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]:   index 
> before flush _0(8.0.0):C1:fieldInfosGen=1:dvGen=1
> DW 0 [2018-04-06T19:27:27.194964Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: 
> startFullFlush
> DW 0 [2018-04-06T19:27:27.195096Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD] 
> finishFullFlush success=true
> IW 0 [2018-04-06T19:27:27.195181Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: now 
> apply all deletes for all segments buffered updates bytesUsed=0 reader pool 
> bytesUsed=0
> BD 0 [2018-04-06T19:27:27.195260Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: 
> waitApply: no deletes to apply
> IW 0 [2018-04-06T19:27:27.195330Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: 
> waitForMerges
> IW 0 [2018-04-06T19:27:27.195385Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: 
> waitForMerges done
> IW 0 [2018-04-06T19:27:27.195448Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: commit: 
> start
> IW 0 [2018-04-06T19:27:27.195504Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: commit: 
> enter lock
> IW 0 [2018-04-06T19:27:27.195560Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: commit: 
> now prepare
> IW 0 [2018-04-06T19:27:27.195779Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: 
> prepareCommit: flush
> IW 0 [2018-04-06T19:27:27.195873Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]:   index 
> before flush _0(8.0.0):C1:fieldInfosGen=1:dvGen=1
> DW 0 [2018-04-06T19:27:27.195954Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: 
> startFullFlush
> IW 0 [2018-04-06T19:27:27.196076Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: now 
> apply all deletes for all segments buffered updates bytesUsed=0 reader pool 
> bytesUsed=0
> BD 0 [2018-04-06T19:27:27.196170Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: 
> waitApply: no deletes to apply
> DW 0 [2018-04-06T19:27:27.196299Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD] 
> finishFullFlush success=true
> IW 0 [2018-04-06T19:27:27.196382Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: 
> startCommit(): start
> IW 0 [2018-04-06T19:27:27.196455Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]:   skip 
> startCommit(): no changes pending
> IW 0 [2018-04-06T19:27:27.196542Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: commit: 
> pendingCommit == null; skip
> IW 0 [2018-04-06T19:27:27.196717Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: commit: 
> took 0.8 msec
> IW 0 [2018-04-06T19:27:27.196791Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: commit: 
> done
> IW 0 [2018-04-06T19:27:27.196950Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: rollback
> IW 0 [2018-04-06T19:27:27.197032Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: all 
> running merges have aborted
> IW 0 [2018-04-06T19:27:27.197326Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: 
> rollback: done finish merges
> DW 0 [2018-04-06T19:27:27.197411Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: abort
> DW 0 [2018-04-06T19:27:27.197486Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: done 
> abort success=true
> IW 0 [2018-04-06T19:27:27.197605Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: 
> rollback: infos=_0(8.0.0):C1:fieldInfosGen=1:dvGen=1
> IFD 0 [2018-04-06T19:27:27.197706Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: now 
> checkpoint "_0(8.0.0):C1:fieldInfosGen=1:dvGen=1" [1 segments ; isCommit = 
> false]
> IFD 0 [2018-04-06T19:27:27.197837Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: 0 msec 
> to checkpoint
> java.nio.file.NoSuchFileException: _0_1.fnm in dir=RAMDirectory@79d3c690 
> lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@795a0c8b
>       at 
> __randomizedtesting.SeedInfo.seed([9F04EE6B720B6BFD:E6EC23304E67E247]:0)
>       at 
> org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:750)
>       at 
> org.apache.lucene.store.Directory.openChecksumInput(Directory.java:119)
>       at 
> org.apache.lucene.store.MockDirectoryWrapper.openChecksumInput(MockDirectoryWrapper.java:1072)
>       at 
> org.apache.lucene.codecs.lucene60.Lucene60FieldInfosFormat.read(Lucene60FieldInfosFormat.java:113)
>       at 
> org.apache.lucene.index.IndexWriter.readFieldInfos(IndexWriter.java:1207)
>       at 
> org.apache.lucene.index.IndexWriter.getFieldNumberMap(IndexWriter.java:1227)
>       at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1114)
>       at 
> org.apache.lucene.index.TestAddIndexes.testAddIndexesDVUpdate(TestAddIndexes.java:1364)
>       at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>       at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>       at 
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>       at java.base/java.lang.reflect.Method.invoke(Method.java:564)
>       at 
> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
>       at 
> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
>       at 
> com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
>       at 
> com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
>       at 
> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
>       at 
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>       at 
> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
>       at 
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>       at 
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>       at 
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>       at 
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
>       at 
> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
>       at 
> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
>       at 
> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
>       at 
> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
>       at 
> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
>       at 
> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
>       at 
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>       at 
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>       at 
> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
>       at 
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>       at 
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>       at 
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>       at 
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>       at 
> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
>       at 
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>       at 
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>       at 
> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
>       at 
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>       at 
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
>       at java.base/java.lang.Thread.run(Thread.java:844)
> NOTE: reproduce with: ant test  -Dtestcase=TestAddIndexes 
> -Dtests.method=testAddIndexesDVUpdate -Dtests.seed=9F04EE6B720B6BFD 
> -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=es-HN 
> -Dtests.timezone=Asia/Singapore -Dtests.asserts=true 
> -Dtests.file.encoding=UTF-8
> NOTE: test params are: 
> codec=FastDecompressionCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST_DECOMPRESSION,
>  chunkSize=8, maxDocsPerChunk=6, blockSize=201), 
> termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST_DECOMPRESSION,
>  chunkSize=8, blockSize=201)), 
> sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@55f3c81b),
>  locale=es-HN, timezone=Asia/Singapore
> NOTE: Mac OS X 10.13.3 x86_64/Oracle Corporation 9.0.1 
> (64-bit)/cpus=4,threads=1,free=231833088,total=268435456
> NOTE: All tests run in this JVM: [TestAddIndexes]
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Reply via email to