[jira] [Commented] (LUCENE-8243) IndexWriter might delete DV update files if addIndices are invovled

2018-04-09 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/LUCENE-8243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16430173#comment-16430173
 ] 

ASF subversion and git services commented on LUCENE-8243:
-

Commit 7140e432594b67a8d6c93d6bf7ae2103717263ed in lucene-solr's branch 
refs/heads/branch_7x from [~simonw]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=7140e43 ]

LUCENE-8243: Add original finder to the attribution list in the CHANGES.TXT


> IndexWriter might delete DV update files if addIndices are invovled
> ---
>
> Key: LUCENE-8243
> URL: https://issues.apache.org/jira/browse/LUCENE-8243
> Project: Lucene - Core
>  Issue Type: Bug
>Affects Versions: 7.4, master (8.0)
>Reporter: Simon Willnauer
>Assignee: Michael McCandless
>Priority: Major
> Fix For: 7.4, master (8.0)
>
> Attachments: LUCENE-8243.patch, LUCENE-8243.patch, 
> broken_dv_update.patch
>
>
> the attached test fails with this output:
> {noformat}
> /Library/Java/JavaVirtualMachines/jdk-9.0.1.jdk/Contents/Home/bin/java -ea 
> -Djava.security.egd=file:/dev/./urandom 
> -Didea.test.cyclic.buffer.size=1048576 -Dfile.encoding=UTF-8 -classpath 
> "/Applications/IntelliJ 
> IDEA.app/Contents/lib/idea_rt.jar:/Applications/IntelliJ 
> IDEA.app/Contents/plugins/junit/lib/junit-rt.jar:/Applications/IntelliJ 
> IDEA.app/Contents/plugins/junit/lib/junit5-rt.jar:/Users/simonw/projects/lucene-solr/idea-build/lucene/test-framework/classes/test:/Users/simonw/projects/lucene-solr/idea-build/lucene/test-framework/classes/java:/Users/simonw/projects/lucene-solr/lucene/test-framework/lib/junit-4.10.jar:/Users/simonw/projects/lucene-solr/lucene/test-framework/lib/randomizedtesting-runner-2.5.3.jar:/Users/simonw/projects/lucene-solr/idea-build/lucene/codecs/classes/java:/Users/simonw/projects/lucene-solr/idea-build/lucene/core/classes/java:/Users/simonw/projects/lucene-solr/idea-build/lucene/core/classes/test"
>  com.intellij.rt.execution.junit.JUnitStarter -ideVersion5 -junit4 
> org.apache.lucene.index.TestAddIndexes,testAddIndexesDVUpdate
> IFD 0 [2018-04-06T19:27:27.176036Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> current segments file is "segments_1"; 
> deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@27cf18f0
> IFD 0 [2018-04-06T19:27:27.188066Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> load commit "segments_1"
> IFD 0 [2018-04-06T19:27:27.189800Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> seg=_0 set nextWriteDelGen=2 vs current=1
> IFD 0 [2018-04-06T19:27:27.190053Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> removing unreferenced file "_0_1_Lucene70_0.dvd"
> IFD 0 [2018-04-06T19:27:27.190224Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> removing unreferenced file "_0_1.fnm"
> IFD 0 [2018-04-06T19:27:27.190371Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> removing unreferenced file "_0_1_Lucene70_0.dvm"
> IFD 0 [2018-04-06T19:27:27.190528Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: delete 
> [_0_1_Lucene70_0.dvd, _0_1.fnm, _0_1_Lucene70_0.dvm]
> IFD 0 [2018-04-06T19:27:27.192558Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: now 
> checkpoint "_0(8.0.0):C1:fieldInfosGen=1:dvGen=1" [1 segments ; isCommit = 
> false]
> IFD 0 [2018-04-06T19:27:27.192806Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: 0 msec 
> to checkpoint
> IW 0 [2018-04-06T19:27:27.193012Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> create=false
> IW 0 [2018-04-06T19:27:27.193428Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: 
> dir=MockDirectoryWrapper(RAMDirectory@79d3c690 
> lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@795a0c8b)
> index=_0(8.0.0):C1:fieldInfosGen=1:dvGen=1
> version=8.0.0
> analyzer=org.apache.lucene.analysis.MockAnalyzer
> ramBufferSizeMB=16.0
> maxBufferedDocs=503
> mergedSegmentWarmer=null
> delPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy
> commit=null
> openMode=CREATE_OR_APPEND
> similarity=org.apache.lucene.search.similarities.AssertingSimilarity
> mergeScheduler=org.apache.lucene.index.SerialMergeScheduler@2f3feff6
> codec=FastDecompressionCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST_DECOMPRESSION,
>  chunkSize=8, maxDocsPerChunk=6, blockSize=201), 
> termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST_DECOMPRESSION,
>  chunkSize=8, blockSize=201))
> infoStream=org.apache.lucene.util.PrintS

[jira] [Commented] (LUCENE-8243) IndexWriter might delete DV update files if addIndices are invovled

2018-04-09 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/LUCENE-8243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16430174#comment-16430174
 ] 

ASF subversion and git services commented on LUCENE-8243:
-

Commit a49543b48aaec44c0735f6f41e6e455354720b9c in lucene-solr's branch 
refs/heads/master from [~simonw]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=a49543b ]

LUCENE-8243: Add original finder to the attribution list in the CHANGES.TXT


> IndexWriter might delete DV update files if addIndices are invovled
> ---
>
> Key: LUCENE-8243
> URL: https://issues.apache.org/jira/browse/LUCENE-8243
> Project: Lucene - Core
>  Issue Type: Bug
>Affects Versions: 7.4, master (8.0)
>Reporter: Simon Willnauer
>Assignee: Michael McCandless
>Priority: Major
> Fix For: 7.4, master (8.0)
>
> Attachments: LUCENE-8243.patch, LUCENE-8243.patch, 
> broken_dv_update.patch
>
>
> the attached test fails with this output:
> {noformat}
> /Library/Java/JavaVirtualMachines/jdk-9.0.1.jdk/Contents/Home/bin/java -ea 
> -Djava.security.egd=file:/dev/./urandom 
> -Didea.test.cyclic.buffer.size=1048576 -Dfile.encoding=UTF-8 -classpath 
> "/Applications/IntelliJ 
> IDEA.app/Contents/lib/idea_rt.jar:/Applications/IntelliJ 
> IDEA.app/Contents/plugins/junit/lib/junit-rt.jar:/Applications/IntelliJ 
> IDEA.app/Contents/plugins/junit/lib/junit5-rt.jar:/Users/simonw/projects/lucene-solr/idea-build/lucene/test-framework/classes/test:/Users/simonw/projects/lucene-solr/idea-build/lucene/test-framework/classes/java:/Users/simonw/projects/lucene-solr/lucene/test-framework/lib/junit-4.10.jar:/Users/simonw/projects/lucene-solr/lucene/test-framework/lib/randomizedtesting-runner-2.5.3.jar:/Users/simonw/projects/lucene-solr/idea-build/lucene/codecs/classes/java:/Users/simonw/projects/lucene-solr/idea-build/lucene/core/classes/java:/Users/simonw/projects/lucene-solr/idea-build/lucene/core/classes/test"
>  com.intellij.rt.execution.junit.JUnitStarter -ideVersion5 -junit4 
> org.apache.lucene.index.TestAddIndexes,testAddIndexesDVUpdate
> IFD 0 [2018-04-06T19:27:27.176036Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> current segments file is "segments_1"; 
> deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@27cf18f0
> IFD 0 [2018-04-06T19:27:27.188066Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> load commit "segments_1"
> IFD 0 [2018-04-06T19:27:27.189800Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> seg=_0 set nextWriteDelGen=2 vs current=1
> IFD 0 [2018-04-06T19:27:27.190053Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> removing unreferenced file "_0_1_Lucene70_0.dvd"
> IFD 0 [2018-04-06T19:27:27.190224Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> removing unreferenced file "_0_1.fnm"
> IFD 0 [2018-04-06T19:27:27.190371Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> removing unreferenced file "_0_1_Lucene70_0.dvm"
> IFD 0 [2018-04-06T19:27:27.190528Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: delete 
> [_0_1_Lucene70_0.dvd, _0_1.fnm, _0_1_Lucene70_0.dvm]
> IFD 0 [2018-04-06T19:27:27.192558Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: now 
> checkpoint "_0(8.0.0):C1:fieldInfosGen=1:dvGen=1" [1 segments ; isCommit = 
> false]
> IFD 0 [2018-04-06T19:27:27.192806Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: 0 msec 
> to checkpoint
> IW 0 [2018-04-06T19:27:27.193012Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> create=false
> IW 0 [2018-04-06T19:27:27.193428Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: 
> dir=MockDirectoryWrapper(RAMDirectory@79d3c690 
> lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@795a0c8b)
> index=_0(8.0.0):C1:fieldInfosGen=1:dvGen=1
> version=8.0.0
> analyzer=org.apache.lucene.analysis.MockAnalyzer
> ramBufferSizeMB=16.0
> maxBufferedDocs=503
> mergedSegmentWarmer=null
> delPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy
> commit=null
> openMode=CREATE_OR_APPEND
> similarity=org.apache.lucene.search.similarities.AssertingSimilarity
> mergeScheduler=org.apache.lucene.index.SerialMergeScheduler@2f3feff6
> codec=FastDecompressionCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST_DECOMPRESSION,
>  chunkSize=8, maxDocsPerChunk=6, blockSize=201), 
> termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST_DECOMPRESSION,
>  chunkSize=8, blockSize=201))
> infoStream=org.apache.lucene.util.PrintStre

[jira] [Commented] (LUCENE-8243) IndexWriter might delete DV update files if addIndices are invovled

2018-04-08 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/LUCENE-8243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16429795#comment-16429795
 ] 

ASF subversion and git services commented on LUCENE-8243:
-

Commit 99364584fffcb8352ca9cd566c3044ed04732582 in lucene-solr's branch 
refs/heads/master from Mike McCandless
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=9936458 ]

LUCENE-8243: fix IndexWriter.addIndexes(Directory[]) to properly preserve index 
file names for updated doc values fields


> IndexWriter might delete DV update files if addIndices are invovled
> ---
>
> Key: LUCENE-8243
> URL: https://issues.apache.org/jira/browse/LUCENE-8243
> Project: Lucene - Core
>  Issue Type: Bug
>Affects Versions: 7.4, master (8.0)
>Reporter: Simon Willnauer
>Assignee: Michael McCandless
>Priority: Major
> Fix For: 7.4, master (8.0)
>
> Attachments: LUCENE-8243.patch, LUCENE-8243.patch, 
> broken_dv_update.patch
>
>
> the attached test fails with this output:
> {noformat}
> /Library/Java/JavaVirtualMachines/jdk-9.0.1.jdk/Contents/Home/bin/java -ea 
> -Djava.security.egd=file:/dev/./urandom 
> -Didea.test.cyclic.buffer.size=1048576 -Dfile.encoding=UTF-8 -classpath 
> "/Applications/IntelliJ 
> IDEA.app/Contents/lib/idea_rt.jar:/Applications/IntelliJ 
> IDEA.app/Contents/plugins/junit/lib/junit-rt.jar:/Applications/IntelliJ 
> IDEA.app/Contents/plugins/junit/lib/junit5-rt.jar:/Users/simonw/projects/lucene-solr/idea-build/lucene/test-framework/classes/test:/Users/simonw/projects/lucene-solr/idea-build/lucene/test-framework/classes/java:/Users/simonw/projects/lucene-solr/lucene/test-framework/lib/junit-4.10.jar:/Users/simonw/projects/lucene-solr/lucene/test-framework/lib/randomizedtesting-runner-2.5.3.jar:/Users/simonw/projects/lucene-solr/idea-build/lucene/codecs/classes/java:/Users/simonw/projects/lucene-solr/idea-build/lucene/core/classes/java:/Users/simonw/projects/lucene-solr/idea-build/lucene/core/classes/test"
>  com.intellij.rt.execution.junit.JUnitStarter -ideVersion5 -junit4 
> org.apache.lucene.index.TestAddIndexes,testAddIndexesDVUpdate
> IFD 0 [2018-04-06T19:27:27.176036Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> current segments file is "segments_1"; 
> deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@27cf18f0
> IFD 0 [2018-04-06T19:27:27.188066Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> load commit "segments_1"
> IFD 0 [2018-04-06T19:27:27.189800Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> seg=_0 set nextWriteDelGen=2 vs current=1
> IFD 0 [2018-04-06T19:27:27.190053Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> removing unreferenced file "_0_1_Lucene70_0.dvd"
> IFD 0 [2018-04-06T19:27:27.190224Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> removing unreferenced file "_0_1.fnm"
> IFD 0 [2018-04-06T19:27:27.190371Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> removing unreferenced file "_0_1_Lucene70_0.dvm"
> IFD 0 [2018-04-06T19:27:27.190528Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: delete 
> [_0_1_Lucene70_0.dvd, _0_1.fnm, _0_1_Lucene70_0.dvm]
> IFD 0 [2018-04-06T19:27:27.192558Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: now 
> checkpoint "_0(8.0.0):C1:fieldInfosGen=1:dvGen=1" [1 segments ; isCommit = 
> false]
> IFD 0 [2018-04-06T19:27:27.192806Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: 0 msec 
> to checkpoint
> IW 0 [2018-04-06T19:27:27.193012Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> create=false
> IW 0 [2018-04-06T19:27:27.193428Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: 
> dir=MockDirectoryWrapper(RAMDirectory@79d3c690 
> lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@795a0c8b)
> index=_0(8.0.0):C1:fieldInfosGen=1:dvGen=1
> version=8.0.0
> analyzer=org.apache.lucene.analysis.MockAnalyzer
> ramBufferSizeMB=16.0
> maxBufferedDocs=503
> mergedSegmentWarmer=null
> delPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy
> commit=null
> openMode=CREATE_OR_APPEND
> similarity=org.apache.lucene.search.similarities.AssertingSimilarity
> mergeScheduler=org.apache.lucene.index.SerialMergeScheduler@2f3feff6
> codec=FastDecompressionCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST_DECOMPRESSION,
>  chunkSize=8, maxDocsPerChunk=6, blockSize=201), 
> termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST_DECOMPRESSION,
>  chunkSize=8, blockSize

[jira] [Commented] (LUCENE-8243) IndexWriter might delete DV update files if addIndices are invovled

2018-04-08 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/LUCENE-8243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16429785#comment-16429785
 ] 

ASF subversion and git services commented on LUCENE-8243:
-

Commit 27ad64d672670d1ece372fbbb20847d6bd9f2a38 in lucene-solr's branch 
refs/heads/branch_7x from Mike McCandless
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=27ad64d ]

LUCENE-8243: fix IndexWriter.addIndexes(Directory[]) to properly preserve index 
file names for updated doc values fields


> IndexWriter might delete DV update files if addIndices are invovled
> ---
>
> Key: LUCENE-8243
> URL: https://issues.apache.org/jira/browse/LUCENE-8243
> Project: Lucene - Core
>  Issue Type: Bug
>Affects Versions: 7.4, master (8.0)
>Reporter: Simon Willnauer
>Assignee: Michael McCandless
>Priority: Major
> Fix For: 7.4, master (8.0)
>
> Attachments: LUCENE-8243.patch, LUCENE-8243.patch, 
> broken_dv_update.patch
>
>
> the attached test fails with this output:
> {noformat}
> /Library/Java/JavaVirtualMachines/jdk-9.0.1.jdk/Contents/Home/bin/java -ea 
> -Djava.security.egd=file:/dev/./urandom 
> -Didea.test.cyclic.buffer.size=1048576 -Dfile.encoding=UTF-8 -classpath 
> "/Applications/IntelliJ 
> IDEA.app/Contents/lib/idea_rt.jar:/Applications/IntelliJ 
> IDEA.app/Contents/plugins/junit/lib/junit-rt.jar:/Applications/IntelliJ 
> IDEA.app/Contents/plugins/junit/lib/junit5-rt.jar:/Users/simonw/projects/lucene-solr/idea-build/lucene/test-framework/classes/test:/Users/simonw/projects/lucene-solr/idea-build/lucene/test-framework/classes/java:/Users/simonw/projects/lucene-solr/lucene/test-framework/lib/junit-4.10.jar:/Users/simonw/projects/lucene-solr/lucene/test-framework/lib/randomizedtesting-runner-2.5.3.jar:/Users/simonw/projects/lucene-solr/idea-build/lucene/codecs/classes/java:/Users/simonw/projects/lucene-solr/idea-build/lucene/core/classes/java:/Users/simonw/projects/lucene-solr/idea-build/lucene/core/classes/test"
>  com.intellij.rt.execution.junit.JUnitStarter -ideVersion5 -junit4 
> org.apache.lucene.index.TestAddIndexes,testAddIndexesDVUpdate
> IFD 0 [2018-04-06T19:27:27.176036Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> current segments file is "segments_1"; 
> deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@27cf18f0
> IFD 0 [2018-04-06T19:27:27.188066Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> load commit "segments_1"
> IFD 0 [2018-04-06T19:27:27.189800Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> seg=_0 set nextWriteDelGen=2 vs current=1
> IFD 0 [2018-04-06T19:27:27.190053Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> removing unreferenced file "_0_1_Lucene70_0.dvd"
> IFD 0 [2018-04-06T19:27:27.190224Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> removing unreferenced file "_0_1.fnm"
> IFD 0 [2018-04-06T19:27:27.190371Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> removing unreferenced file "_0_1_Lucene70_0.dvm"
> IFD 0 [2018-04-06T19:27:27.190528Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: delete 
> [_0_1_Lucene70_0.dvd, _0_1.fnm, _0_1_Lucene70_0.dvm]
> IFD 0 [2018-04-06T19:27:27.192558Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: now 
> checkpoint "_0(8.0.0):C1:fieldInfosGen=1:dvGen=1" [1 segments ; isCommit = 
> false]
> IFD 0 [2018-04-06T19:27:27.192806Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: 0 msec 
> to checkpoint
> IW 0 [2018-04-06T19:27:27.193012Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> create=false
> IW 0 [2018-04-06T19:27:27.193428Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: 
> dir=MockDirectoryWrapper(RAMDirectory@79d3c690 
> lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@795a0c8b)
> index=_0(8.0.0):C1:fieldInfosGen=1:dvGen=1
> version=8.0.0
> analyzer=org.apache.lucene.analysis.MockAnalyzer
> ramBufferSizeMB=16.0
> maxBufferedDocs=503
> mergedSegmentWarmer=null
> delPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy
> commit=null
> openMode=CREATE_OR_APPEND
> similarity=org.apache.lucene.search.similarities.AssertingSimilarity
> mergeScheduler=org.apache.lucene.index.SerialMergeScheduler@2f3feff6
> codec=FastDecompressionCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST_DECOMPRESSION,
>  chunkSize=8, maxDocsPerChunk=6, blockSize=201), 
> termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST_DECOMPRESSION,
>  chunkSize=8, blockS

[jira] [Commented] (LUCENE-8243) IndexWriter might delete DV update files if addIndices are invovled

2018-04-08 Thread Michael McCandless (JIRA) 

[ 
https://issues.apache.org/jira/browse/LUCENE-8243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16429782#comment-16429782
 ] 

Michael McCandless commented on LUCENE-8243:


OK I attached new patch w/ a new test that does write a new segment name and it 
passes!

It turns out the {{SegmentCommitInfo}} APIs for setting the file names rewrite 
to their segment name, so it's a non issue.

I'll push shortly.

> IndexWriter might delete DV update files if addIndices are invovled
> ---
>
> Key: LUCENE-8243
> URL: https://issues.apache.org/jira/browse/LUCENE-8243
> Project: Lucene - Core
>  Issue Type: Bug
>Affects Versions: 7.4, master (8.0)
>Reporter: Simon Willnauer
>Assignee: Michael McCandless
>Priority: Major
> Fix For: 7.4, master (8.0)
>
> Attachments: LUCENE-8243.patch, LUCENE-8243.patch, 
> broken_dv_update.patch
>
>
> the attached test fails with this output:
> {noformat}
> /Library/Java/JavaVirtualMachines/jdk-9.0.1.jdk/Contents/Home/bin/java -ea 
> -Djava.security.egd=file:/dev/./urandom 
> -Didea.test.cyclic.buffer.size=1048576 -Dfile.encoding=UTF-8 -classpath 
> "/Applications/IntelliJ 
> IDEA.app/Contents/lib/idea_rt.jar:/Applications/IntelliJ 
> IDEA.app/Contents/plugins/junit/lib/junit-rt.jar:/Applications/IntelliJ 
> IDEA.app/Contents/plugins/junit/lib/junit5-rt.jar:/Users/simonw/projects/lucene-solr/idea-build/lucene/test-framework/classes/test:/Users/simonw/projects/lucene-solr/idea-build/lucene/test-framework/classes/java:/Users/simonw/projects/lucene-solr/lucene/test-framework/lib/junit-4.10.jar:/Users/simonw/projects/lucene-solr/lucene/test-framework/lib/randomizedtesting-runner-2.5.3.jar:/Users/simonw/projects/lucene-solr/idea-build/lucene/codecs/classes/java:/Users/simonw/projects/lucene-solr/idea-build/lucene/core/classes/java:/Users/simonw/projects/lucene-solr/idea-build/lucene/core/classes/test"
>  com.intellij.rt.execution.junit.JUnitStarter -ideVersion5 -junit4 
> org.apache.lucene.index.TestAddIndexes,testAddIndexesDVUpdate
> IFD 0 [2018-04-06T19:27:27.176036Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> current segments file is "segments_1"; 
> deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@27cf18f0
> IFD 0 [2018-04-06T19:27:27.188066Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> load commit "segments_1"
> IFD 0 [2018-04-06T19:27:27.189800Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> seg=_0 set nextWriteDelGen=2 vs current=1
> IFD 0 [2018-04-06T19:27:27.190053Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> removing unreferenced file "_0_1_Lucene70_0.dvd"
> IFD 0 [2018-04-06T19:27:27.190224Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> removing unreferenced file "_0_1.fnm"
> IFD 0 [2018-04-06T19:27:27.190371Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> removing unreferenced file "_0_1_Lucene70_0.dvm"
> IFD 0 [2018-04-06T19:27:27.190528Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: delete 
> [_0_1_Lucene70_0.dvd, _0_1.fnm, _0_1_Lucene70_0.dvm]
> IFD 0 [2018-04-06T19:27:27.192558Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: now 
> checkpoint "_0(8.0.0):C1:fieldInfosGen=1:dvGen=1" [1 segments ; isCommit = 
> false]
> IFD 0 [2018-04-06T19:27:27.192806Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: 0 msec 
> to checkpoint
> IW 0 [2018-04-06T19:27:27.193012Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> create=false
> IW 0 [2018-04-06T19:27:27.193428Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: 
> dir=MockDirectoryWrapper(RAMDirectory@79d3c690 
> lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@795a0c8b)
> index=_0(8.0.0):C1:fieldInfosGen=1:dvGen=1
> version=8.0.0
> analyzer=org.apache.lucene.analysis.MockAnalyzer
> ramBufferSizeMB=16.0
> maxBufferedDocs=503
> mergedSegmentWarmer=null
> delPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy
> commit=null
> openMode=CREATE_OR_APPEND
> similarity=org.apache.lucene.search.similarities.AssertingSimilarity
> mergeScheduler=org.apache.lucene.index.SerialMergeScheduler@2f3feff6
> codec=FastDecompressionCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST_DECOMPRESSION,
>  chunkSize=8, maxDocsPerChunk=6, blockSize=201), 
> termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST_DECOMPRESSION,
>  chunkSize=8, blockSize=201))
> infoStream=org.apache.lucene.util.PrintStreamInfoStream
> mergePolicy=[TieredMergePolicy: m

[jira] [Commented] (LUCENE-8243) IndexWriter might delete DV update files if addIndices are invovled

2018-04-08 Thread Michael McCandless (JIRA) 

[ 
https://issues.apache.org/jira/browse/LUCENE-8243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16429778#comment-16429778
 ] 

Michael McCandless commented on LUCENE-8243:


Hmm actually I'm worried about my fix: this method changes the name of segment, 
but my change just blindly copies the file names over (from old to new segment) 
w/o also updating the segment name?  Maybe we need a better test that actually 
changes the segment name, to see this fix is not sufficient?  I'll try.

> IndexWriter might delete DV update files if addIndices are invovled
> ---
>
> Key: LUCENE-8243
> URL: https://issues.apache.org/jira/browse/LUCENE-8243
> Project: Lucene - Core
>  Issue Type: Bug
>Affects Versions: 7.4, master (8.0)
>Reporter: Simon Willnauer
>Assignee: Michael McCandless
>Priority: Major
> Fix For: 7.4, master (8.0)
>
> Attachments: LUCENE-8243.patch, broken_dv_update.patch
>
>
> the attached test fails with this output:
> {noformat}
> /Library/Java/JavaVirtualMachines/jdk-9.0.1.jdk/Contents/Home/bin/java -ea 
> -Djava.security.egd=file:/dev/./urandom 
> -Didea.test.cyclic.buffer.size=1048576 -Dfile.encoding=UTF-8 -classpath 
> "/Applications/IntelliJ 
> IDEA.app/Contents/lib/idea_rt.jar:/Applications/IntelliJ 
> IDEA.app/Contents/plugins/junit/lib/junit-rt.jar:/Applications/IntelliJ 
> IDEA.app/Contents/plugins/junit/lib/junit5-rt.jar:/Users/simonw/projects/lucene-solr/idea-build/lucene/test-framework/classes/test:/Users/simonw/projects/lucene-solr/idea-build/lucene/test-framework/classes/java:/Users/simonw/projects/lucene-solr/lucene/test-framework/lib/junit-4.10.jar:/Users/simonw/projects/lucene-solr/lucene/test-framework/lib/randomizedtesting-runner-2.5.3.jar:/Users/simonw/projects/lucene-solr/idea-build/lucene/codecs/classes/java:/Users/simonw/projects/lucene-solr/idea-build/lucene/core/classes/java:/Users/simonw/projects/lucene-solr/idea-build/lucene/core/classes/test"
>  com.intellij.rt.execution.junit.JUnitStarter -ideVersion5 -junit4 
> org.apache.lucene.index.TestAddIndexes,testAddIndexesDVUpdate
> IFD 0 [2018-04-06T19:27:27.176036Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> current segments file is "segments_1"; 
> deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@27cf18f0
> IFD 0 [2018-04-06T19:27:27.188066Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> load commit "segments_1"
> IFD 0 [2018-04-06T19:27:27.189800Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> seg=_0 set nextWriteDelGen=2 vs current=1
> IFD 0 [2018-04-06T19:27:27.190053Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> removing unreferenced file "_0_1_Lucene70_0.dvd"
> IFD 0 [2018-04-06T19:27:27.190224Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> removing unreferenced file "_0_1.fnm"
> IFD 0 [2018-04-06T19:27:27.190371Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> removing unreferenced file "_0_1_Lucene70_0.dvm"
> IFD 0 [2018-04-06T19:27:27.190528Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: delete 
> [_0_1_Lucene70_0.dvd, _0_1.fnm, _0_1_Lucene70_0.dvm]
> IFD 0 [2018-04-06T19:27:27.192558Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: now 
> checkpoint "_0(8.0.0):C1:fieldInfosGen=1:dvGen=1" [1 segments ; isCommit = 
> false]
> IFD 0 [2018-04-06T19:27:27.192806Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: 0 msec 
> to checkpoint
> IW 0 [2018-04-06T19:27:27.193012Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> create=false
> IW 0 [2018-04-06T19:27:27.193428Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: 
> dir=MockDirectoryWrapper(RAMDirectory@79d3c690 
> lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@795a0c8b)
> index=_0(8.0.0):C1:fieldInfosGen=1:dvGen=1
> version=8.0.0
> analyzer=org.apache.lucene.analysis.MockAnalyzer
> ramBufferSizeMB=16.0
> maxBufferedDocs=503
> mergedSegmentWarmer=null
> delPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy
> commit=null
> openMode=CREATE_OR_APPEND
> similarity=org.apache.lucene.search.similarities.AssertingSimilarity
> mergeScheduler=org.apache.lucene.index.SerialMergeScheduler@2f3feff6
> codec=FastDecompressionCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST_DECOMPRESSION,
>  chunkSize=8, maxDocsPerChunk=6, blockSize=201), 
> termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST_DECOMPRESSION,
>  chunkSize=8, blockSize=201))
> infoStream=org.apache.lucene.util

[jira] [Commented] (LUCENE-8243) IndexWriter might delete DV update files if addIndices are invovled

2018-04-06 Thread Simon Willnauer (JIRA) 

[ 
https://issues.apache.org/jira/browse/LUCENE-8243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16428896#comment-16428896
 ] 

Simon Willnauer commented on LUCENE-8243:
-

LGTM thanks mike!

> IndexWriter might delete DV update files if addIndices are invovled
> ---
>
> Key: LUCENE-8243
> URL: https://issues.apache.org/jira/browse/LUCENE-8243
> Project: Lucene - Core
>  Issue Type: Bug
>Affects Versions: 7.4, master (8.0)
>Reporter: Simon Willnauer
>Assignee: Michael McCandless
>Priority: Major
> Fix For: 7.4, master (8.0)
>
> Attachments: LUCENE-8243.patch, broken_dv_update.patch
>
>
> the attached test fails with this output:
> {noformat}
> /Library/Java/JavaVirtualMachines/jdk-9.0.1.jdk/Contents/Home/bin/java -ea 
> -Djava.security.egd=file:/dev/./urandom 
> -Didea.test.cyclic.buffer.size=1048576 -Dfile.encoding=UTF-8 -classpath 
> "/Applications/IntelliJ 
> IDEA.app/Contents/lib/idea_rt.jar:/Applications/IntelliJ 
> IDEA.app/Contents/plugins/junit/lib/junit-rt.jar:/Applications/IntelliJ 
> IDEA.app/Contents/plugins/junit/lib/junit5-rt.jar:/Users/simonw/projects/lucene-solr/idea-build/lucene/test-framework/classes/test:/Users/simonw/projects/lucene-solr/idea-build/lucene/test-framework/classes/java:/Users/simonw/projects/lucene-solr/lucene/test-framework/lib/junit-4.10.jar:/Users/simonw/projects/lucene-solr/lucene/test-framework/lib/randomizedtesting-runner-2.5.3.jar:/Users/simonw/projects/lucene-solr/idea-build/lucene/codecs/classes/java:/Users/simonw/projects/lucene-solr/idea-build/lucene/core/classes/java:/Users/simonw/projects/lucene-solr/idea-build/lucene/core/classes/test"
>  com.intellij.rt.execution.junit.JUnitStarter -ideVersion5 -junit4 
> org.apache.lucene.index.TestAddIndexes,testAddIndexesDVUpdate
> IFD 0 [2018-04-06T19:27:27.176036Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> current segments file is "segments_1"; 
> deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@27cf18f0
> IFD 0 [2018-04-06T19:27:27.188066Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> load commit "segments_1"
> IFD 0 [2018-04-06T19:27:27.189800Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> seg=_0 set nextWriteDelGen=2 vs current=1
> IFD 0 [2018-04-06T19:27:27.190053Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> removing unreferenced file "_0_1_Lucene70_0.dvd"
> IFD 0 [2018-04-06T19:27:27.190224Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> removing unreferenced file "_0_1.fnm"
> IFD 0 [2018-04-06T19:27:27.190371Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> removing unreferenced file "_0_1_Lucene70_0.dvm"
> IFD 0 [2018-04-06T19:27:27.190528Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: delete 
> [_0_1_Lucene70_0.dvd, _0_1.fnm, _0_1_Lucene70_0.dvm]
> IFD 0 [2018-04-06T19:27:27.192558Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: now 
> checkpoint "_0(8.0.0):C1:fieldInfosGen=1:dvGen=1" [1 segments ; isCommit = 
> false]
> IFD 0 [2018-04-06T19:27:27.192806Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: 0 msec 
> to checkpoint
> IW 0 [2018-04-06T19:27:27.193012Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> create=false
> IW 0 [2018-04-06T19:27:27.193428Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: 
> dir=MockDirectoryWrapper(RAMDirectory@79d3c690 
> lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@795a0c8b)
> index=_0(8.0.0):C1:fieldInfosGen=1:dvGen=1
> version=8.0.0
> analyzer=org.apache.lucene.analysis.MockAnalyzer
> ramBufferSizeMB=16.0
> maxBufferedDocs=503
> mergedSegmentWarmer=null
> delPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy
> commit=null
> openMode=CREATE_OR_APPEND
> similarity=org.apache.lucene.search.similarities.AssertingSimilarity
> mergeScheduler=org.apache.lucene.index.SerialMergeScheduler@2f3feff6
> codec=FastDecompressionCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST_DECOMPRESSION,
>  chunkSize=8, maxDocsPerChunk=6, blockSize=201), 
> termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST_DECOMPRESSION,
>  chunkSize=8, blockSize=201))
> infoStream=org.apache.lucene.util.PrintStreamInfoStream
> mergePolicy=[TieredMergePolicy: maxMergeAtOnce=41, maxMergeAtOnceExplicit=44, 
> maxMergedSegmentMB=6.255859375, floorSegmentMB=0.38671875, 
> forceMergeDeletesPctAllowed=4.456652110760543, segmentsPerTier=31.0, 
> maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8773303

[jira] [Commented] (LUCENE-8243) IndexWriter might delete DV update files if addIndices are invovled

2018-04-06 Thread Michael McCandless (JIRA) 

[ 
https://issues.apache.org/jira/browse/LUCENE-8243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16428866#comment-16428866
 ] 

Michael McCandless commented on LUCENE-8243:


I just attached a patch that I think fixes the issue – the problem was 
{{IW.addIndexes(Directory[])}} was creating a new {{SegmentCommitInfo}} for the 
copied segment, but was failing to preserve the files associated with doc 
values updates.  All Lucene tests, and {{ant precommit}} pass.

> IndexWriter might delete DV update files if addIndices are invovled
> ---
>
> Key: LUCENE-8243
> URL: https://issues.apache.org/jira/browse/LUCENE-8243
> Project: Lucene - Core
>  Issue Type: Bug
>Affects Versions: 7.4, master (8.0)
>Reporter: Simon Willnauer
>Assignee: Michael McCandless
>Priority: Major
> Fix For: 7.4, master (8.0)
>
> Attachments: LUCENE-8243.patch, broken_dv_update.patch
>
>
> the attached test fails with this output:
> {noformat}
> /Library/Java/JavaVirtualMachines/jdk-9.0.1.jdk/Contents/Home/bin/java -ea 
> -Djava.security.egd=file:/dev/./urandom 
> -Didea.test.cyclic.buffer.size=1048576 -Dfile.encoding=UTF-8 -classpath 
> "/Applications/IntelliJ 
> IDEA.app/Contents/lib/idea_rt.jar:/Applications/IntelliJ 
> IDEA.app/Contents/plugins/junit/lib/junit-rt.jar:/Applications/IntelliJ 
> IDEA.app/Contents/plugins/junit/lib/junit5-rt.jar:/Users/simonw/projects/lucene-solr/idea-build/lucene/test-framework/classes/test:/Users/simonw/projects/lucene-solr/idea-build/lucene/test-framework/classes/java:/Users/simonw/projects/lucene-solr/lucene/test-framework/lib/junit-4.10.jar:/Users/simonw/projects/lucene-solr/lucene/test-framework/lib/randomizedtesting-runner-2.5.3.jar:/Users/simonw/projects/lucene-solr/idea-build/lucene/codecs/classes/java:/Users/simonw/projects/lucene-solr/idea-build/lucene/core/classes/java:/Users/simonw/projects/lucene-solr/idea-build/lucene/core/classes/test"
>  com.intellij.rt.execution.junit.JUnitStarter -ideVersion5 -junit4 
> org.apache.lucene.index.TestAddIndexes,testAddIndexesDVUpdate
> IFD 0 [2018-04-06T19:27:27.176036Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> current segments file is "segments_1"; 
> deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@27cf18f0
> IFD 0 [2018-04-06T19:27:27.188066Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> load commit "segments_1"
> IFD 0 [2018-04-06T19:27:27.189800Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> seg=_0 set nextWriteDelGen=2 vs current=1
> IFD 0 [2018-04-06T19:27:27.190053Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> removing unreferenced file "_0_1_Lucene70_0.dvd"
> IFD 0 [2018-04-06T19:27:27.190224Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> removing unreferenced file "_0_1.fnm"
> IFD 0 [2018-04-06T19:27:27.190371Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> removing unreferenced file "_0_1_Lucene70_0.dvm"
> IFD 0 [2018-04-06T19:27:27.190528Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: delete 
> [_0_1_Lucene70_0.dvd, _0_1.fnm, _0_1_Lucene70_0.dvm]
> IFD 0 [2018-04-06T19:27:27.192558Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: now 
> checkpoint "_0(8.0.0):C1:fieldInfosGen=1:dvGen=1" [1 segments ; isCommit = 
> false]
> IFD 0 [2018-04-06T19:27:27.192806Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: 0 msec 
> to checkpoint
> IW 0 [2018-04-06T19:27:27.193012Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> create=false
> IW 0 [2018-04-06T19:27:27.193428Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: 
> dir=MockDirectoryWrapper(RAMDirectory@79d3c690 
> lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@795a0c8b)
> index=_0(8.0.0):C1:fieldInfosGen=1:dvGen=1
> version=8.0.0
> analyzer=org.apache.lucene.analysis.MockAnalyzer
> ramBufferSizeMB=16.0
> maxBufferedDocs=503
> mergedSegmentWarmer=null
> delPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy
> commit=null
> openMode=CREATE_OR_APPEND
> similarity=org.apache.lucene.search.similarities.AssertingSimilarity
> mergeScheduler=org.apache.lucene.index.SerialMergeScheduler@2f3feff6
> codec=FastDecompressionCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST_DECOMPRESSION,
>  chunkSize=8, maxDocsPerChunk=6, blockSize=201), 
> termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST_DECOMPRESSION,
>  chunkSize=8, blockSize=201))
> infoStream=org.apache.lucene.util.PrintStreamInfoStream

[jira] [Commented] (LUCENE-8243) IndexWriter might delete DV update files if addIndices are invovled

2018-04-06 Thread Michael McCandless (JIRA) 

[ 
https://issues.apache.org/jira/browse/LUCENE-8243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16428842#comment-16428842
 ] 

Michael McCandless commented on LUCENE-8243:


I'll dig ...

> IndexWriter might delete DV update files if addIndices are invovled
> ---
>
> Key: LUCENE-8243
> URL: https://issues.apache.org/jira/browse/LUCENE-8243
> Project: Lucene - Core
>  Issue Type: Bug
>Affects Versions: 7.4, master (8.0)
>Reporter: Simon Willnauer
>Priority: Major
> Fix For: 7.4, master (8.0)
>
> Attachments: broken_dv_update.patch
>
>
> the attached test fails with this output:
> {noformat}
> /Library/Java/JavaVirtualMachines/jdk-9.0.1.jdk/Contents/Home/bin/java -ea 
> -Djava.security.egd=file:/dev/./urandom 
> -Didea.test.cyclic.buffer.size=1048576 -Dfile.encoding=UTF-8 -classpath 
> "/Applications/IntelliJ 
> IDEA.app/Contents/lib/idea_rt.jar:/Applications/IntelliJ 
> IDEA.app/Contents/plugins/junit/lib/junit-rt.jar:/Applications/IntelliJ 
> IDEA.app/Contents/plugins/junit/lib/junit5-rt.jar:/Users/simonw/projects/lucene-solr/idea-build/lucene/test-framework/classes/test:/Users/simonw/projects/lucene-solr/idea-build/lucene/test-framework/classes/java:/Users/simonw/projects/lucene-solr/lucene/test-framework/lib/junit-4.10.jar:/Users/simonw/projects/lucene-solr/lucene/test-framework/lib/randomizedtesting-runner-2.5.3.jar:/Users/simonw/projects/lucene-solr/idea-build/lucene/codecs/classes/java:/Users/simonw/projects/lucene-solr/idea-build/lucene/core/classes/java:/Users/simonw/projects/lucene-solr/idea-build/lucene/core/classes/test"
>  com.intellij.rt.execution.junit.JUnitStarter -ideVersion5 -junit4 
> org.apache.lucene.index.TestAddIndexes,testAddIndexesDVUpdate
> IFD 0 [2018-04-06T19:27:27.176036Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> current segments file is "segments_1"; 
> deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@27cf18f0
> IFD 0 [2018-04-06T19:27:27.188066Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> load commit "segments_1"
> IFD 0 [2018-04-06T19:27:27.189800Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> seg=_0 set nextWriteDelGen=2 vs current=1
> IFD 0 [2018-04-06T19:27:27.190053Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> removing unreferenced file "_0_1_Lucene70_0.dvd"
> IFD 0 [2018-04-06T19:27:27.190224Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> removing unreferenced file "_0_1.fnm"
> IFD 0 [2018-04-06T19:27:27.190371Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> removing unreferenced file "_0_1_Lucene70_0.dvm"
> IFD 0 [2018-04-06T19:27:27.190528Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: delete 
> [_0_1_Lucene70_0.dvd, _0_1.fnm, _0_1_Lucene70_0.dvm]
> IFD 0 [2018-04-06T19:27:27.192558Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: now 
> checkpoint "_0(8.0.0):C1:fieldInfosGen=1:dvGen=1" [1 segments ; isCommit = 
> false]
> IFD 0 [2018-04-06T19:27:27.192806Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: 0 msec 
> to checkpoint
> IW 0 [2018-04-06T19:27:27.193012Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> create=false
> IW 0 [2018-04-06T19:27:27.193428Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: 
> dir=MockDirectoryWrapper(RAMDirectory@79d3c690 
> lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@795a0c8b)
> index=_0(8.0.0):C1:fieldInfosGen=1:dvGen=1
> version=8.0.0
> analyzer=org.apache.lucene.analysis.MockAnalyzer
> ramBufferSizeMB=16.0
> maxBufferedDocs=503
> mergedSegmentWarmer=null
> delPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy
> commit=null
> openMode=CREATE_OR_APPEND
> similarity=org.apache.lucene.search.similarities.AssertingSimilarity
> mergeScheduler=org.apache.lucene.index.SerialMergeScheduler@2f3feff6
> codec=FastDecompressionCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST_DECOMPRESSION,
>  chunkSize=8, maxDocsPerChunk=6, blockSize=201), 
> termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST_DECOMPRESSION,
>  chunkSize=8, blockSize=201))
> infoStream=org.apache.lucene.util.PrintStreamInfoStream
> mergePolicy=[TieredMergePolicy: maxMergeAtOnce=41, maxMergeAtOnceExplicit=44, 
> maxMergedSegmentMB=6.255859375, floorSegmentMB=0.38671875, 
> forceMergeDeletesPctAllowed=4.456652110760543, segmentsPerTier=31.0, 
> maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.877330376985384
> indexerThreadPool=org.apache.lucene.index.Documen

[jira] [Commented] (LUCENE-8243) IndexWriter might delete DV update files if addIndices are invovled

2018-04-06 Thread Simon Willnauer (JIRA) 

[ 
https://issues.apache.org/jira/browse/LUCENE-8243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16428833#comment-16428833
 ] 

Simon Willnauer commented on LUCENE-8243:
-

this was found by Nhat Nguyen I just open the issue (https://github.com/dnhatn/)

> IndexWriter might delete DV update files if addIndices are invovled
> ---
>
> Key: LUCENE-8243
> URL: https://issues.apache.org/jira/browse/LUCENE-8243
> Project: Lucene - Core
>  Issue Type: Bug
>Affects Versions: 7.4, master (8.0)
>Reporter: Simon Willnauer
>Priority: Major
> Fix For: 7.4, master (8.0)
>
> Attachments: broken_dv_update.patch
>
>
> the attached test fails with this output:
> {noformat}
> /Library/Java/JavaVirtualMachines/jdk-9.0.1.jdk/Contents/Home/bin/java -ea 
> -Djava.security.egd=file:/dev/./urandom 
> -Didea.test.cyclic.buffer.size=1048576 -Dfile.encoding=UTF-8 -classpath 
> "/Applications/IntelliJ 
> IDEA.app/Contents/lib/idea_rt.jar:/Applications/IntelliJ 
> IDEA.app/Contents/plugins/junit/lib/junit-rt.jar:/Applications/IntelliJ 
> IDEA.app/Contents/plugins/junit/lib/junit5-rt.jar:/Users/simonw/projects/lucene-solr/idea-build/lucene/test-framework/classes/test:/Users/simonw/projects/lucene-solr/idea-build/lucene/test-framework/classes/java:/Users/simonw/projects/lucene-solr/lucene/test-framework/lib/junit-4.10.jar:/Users/simonw/projects/lucene-solr/lucene/test-framework/lib/randomizedtesting-runner-2.5.3.jar:/Users/simonw/projects/lucene-solr/idea-build/lucene/codecs/classes/java:/Users/simonw/projects/lucene-solr/idea-build/lucene/core/classes/java:/Users/simonw/projects/lucene-solr/idea-build/lucene/core/classes/test"
>  com.intellij.rt.execution.junit.JUnitStarter -ideVersion5 -junit4 
> org.apache.lucene.index.TestAddIndexes,testAddIndexesDVUpdate
> IFD 0 [2018-04-06T19:27:27.176036Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> current segments file is "segments_1"; 
> deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@27cf18f0
> IFD 0 [2018-04-06T19:27:27.188066Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> load commit "segments_1"
> IFD 0 [2018-04-06T19:27:27.189800Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> seg=_0 set nextWriteDelGen=2 vs current=1
> IFD 0 [2018-04-06T19:27:27.190053Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> removing unreferenced file "_0_1_Lucene70_0.dvd"
> IFD 0 [2018-04-06T19:27:27.190224Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> removing unreferenced file "_0_1.fnm"
> IFD 0 [2018-04-06T19:27:27.190371Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> removing unreferenced file "_0_1_Lucene70_0.dvm"
> IFD 0 [2018-04-06T19:27:27.190528Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: delete 
> [_0_1_Lucene70_0.dvd, _0_1.fnm, _0_1_Lucene70_0.dvm]
> IFD 0 [2018-04-06T19:27:27.192558Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: now 
> checkpoint "_0(8.0.0):C1:fieldInfosGen=1:dvGen=1" [1 segments ; isCommit = 
> false]
> IFD 0 [2018-04-06T19:27:27.192806Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: 0 msec 
> to checkpoint
> IW 0 [2018-04-06T19:27:27.193012Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: 
> create=false
> IW 0 [2018-04-06T19:27:27.193428Z; 
> TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: 
> dir=MockDirectoryWrapper(RAMDirectory@79d3c690 
> lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@795a0c8b)
> index=_0(8.0.0):C1:fieldInfosGen=1:dvGen=1
> version=8.0.0
> analyzer=org.apache.lucene.analysis.MockAnalyzer
> ramBufferSizeMB=16.0
> maxBufferedDocs=503
> mergedSegmentWarmer=null
> delPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy
> commit=null
> openMode=CREATE_OR_APPEND
> similarity=org.apache.lucene.search.similarities.AssertingSimilarity
> mergeScheduler=org.apache.lucene.index.SerialMergeScheduler@2f3feff6
> codec=FastDecompressionCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST_DECOMPRESSION,
>  chunkSize=8, maxDocsPerChunk=6, blockSize=201), 
> termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST_DECOMPRESSION,
>  chunkSize=8, blockSize=201))
> infoStream=org.apache.lucene.util.PrintStreamInfoStream
> mergePolicy=[TieredMergePolicy: maxMergeAtOnce=41, maxMergeAtOnceExplicit=44, 
> maxMergedSegmentMB=6.255859375, floorSegmentMB=0.38671875, 
> forceMergeDeletesPctAllowed=4.456652110760543, segmentsPerTier=31.0, 
> maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.87733