Simon Willnauer created LUCENE-8310:
---------------------------------------
Summary: Relax IWs check on pending deletes
Key: LUCENE-8310
URL: https://issues.apache.org/jira/browse/LUCENE-8310
Project: Lucene - Core
Issue Type: Improvement
Affects Versions: 7.4, master (8.0)
Reporter: Simon Willnauer
Fix For: 7.4, master (8.0)
I recently fixed the check in IW to fail if there are pending deletes. After
upgrading to a snapshot I realized the consequences of this check. It made most
of our usage of IW to for instance prepare commit metadata, rollback to safe
commit-points etc. impossible since we have to now busy wait on top of
directory util all deletes are actually gone even though that we can guarantee
that our history always goes forward. ie we are truly append-only in the sense
of never reusing segment generations. The fix that I made was basically return
false from a _checkPendingDeletions_ in a directory wrapper to work around it.
I do expect this to happen to a lot of lucene users even if they use IW
correctly. My proposal is to make the check in IW a bit more sophisticated and
only fail if there are pending deletes that are in the future from a generation
perspective. We really don't care about files from the past. My patch checks
the segment generation of each pending file which is safe since that is the
same procedure we apply in IndexFileDeleter to inc reference etc. and only fail
if the pending delete is in the future.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
