Simon Willnauer created LUCENE-9309:
---------------------------------------
Summary: IW#addIndices(CodecReader) might delete files
concurrently to IW#rollback
Key: LUCENE-9309
URL: https://issues.apache.org/jira/browse/LUCENE-9309
Project: Lucene - Core
Issue Type: Bug
Reporter: Simon Willnauer
During work on LUCENE-9304 [~mikemccand] ran into a failure:
{noformat}
org.apache.lucene.index.TestAddIndexes > test suite's output saved to
/home/mike/src/simon/lucene/core/build/test-results/test/outputs/OUTPUT-org.apache.lucene.index.TestAddIndexes.txt,
copied below:
> java.nio.file.NoSuchFileException:
_gt_Lucene85FieldsIndex-doc_ids_6u.tmp
> at
__randomizedtesting.SeedInfo.seed([4760FA81FBD4B2CE:A147156E5F7BD9B0]:0)
> at
org.apache.lucene.store.ByteBuffersDirectory.deleteFile(ByteBuffersDirectory.java:148)
> at
org.apache.lucene.store.MockDirectoryWrapper.deleteFile(MockDirectoryWrapper.java:607)
> at
org.apache.lucene.store.LockValidatingDirectoryWrapper.deleteFile(LockValidatingDirectoryWrapper.java:38)
> at
org.apache.lucene.index.IndexFileDeleter.deleteFile(IndexFileDeleter.java:696)
> at
org.apache.lucene.index.IndexFileDeleter.deleteFiles(IndexFileDeleter.java:690)
> at
org.apache.lucene.index.IndexFileDeleter.refresh(IndexFileDeleter.java:449)
> at
org.apache.lucene.index.IndexWriter.rollbackInternalNoCommit(IndexWriter.java:2334)
> at
org.apache.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:2275)
> at
org.apache.lucene.index.IndexWriter.rollback(IndexWriter.java:2268)
> at
org.apache.lucene.index.TestAddIndexes.testAddIndexesWithRollback(TestAddIndexes.java:974)
2> NOTE: reproduce with: ant test -Dtestcase=TestAddIndexes
-Dtests.method=testAddIndexesWithRollback -Dtests.seed=4760FA81FBD4B2CE
-Dtests.slow=true -Dtests.badapples=true -Dtests.locale=fr-GP -Dtests.t\
imezone=Asia/Tbilisi -Dtests.asserts=true -Dtests.file.encoding=UTF-8
2> NOTE: test params are: codec=Asserting(Lucene84):
{c=PostingsFormat(name=LuceneFixedGap), id=PostingsFormat(name=LuceneFixedGap),
f1=PostingsFormat(name=LuceneFixedGap), f2=BlockTreeOrds(blocksize=128)\
, version=BlockTreeOrds(blocksize=128), content=FST50},
docValues:{dv=DocValuesFormat(name=Lucene80),
soft_delete=DocValuesFormat(name=Lucene80), doc=DocValuesFormat(name=Lucene80),
id=DocValuesFormat(name=\
Asserting), content=DocValuesFormat(name=Asserting),
doc2d=DocValuesFormat(name=Lucene80)}, maxPointsInLeafNode=982,
maxMBSortInHeap=5.837219998050092,
sim=Asserting(org.apache.lucene.search.similarities.As\
sertingSimilarity@6ce38471), locale=fr-GP, timezone=Asia/Tbilisi
{noformat}
While this unfortunately doesn't reproduce it's likely a bug that exists for
quite some time but never showed up until LUCENE-9147 which uses a temporary
output. That's fine but with IW#addIndices(CodecReader...) not registering the
merge it does in the IW we never wait for the merge to finish while rollback
and if that merge finishes concurrently it will also remove these .tmp files.
There are many ways to fix this and I can work on a patch, but hey do we really
need to be able to add indices while we index and do that on an open and live
IW or can it be a tool on top of it?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
