[jira] [Commented] (LUCENE-8310) Relax IWs check on pending deletes
[ https://issues.apache.org/jira/browse/LUCENE-8310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16477222#comment-16477222 ] ASF subversion and git services commented on LUCENE-8310: - Commit 4c45d11c9091f595d7fbb8292ae40103c8775dea in lucene-solr's branch refs/heads/branch_7x from [~simonw] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=4c45d11 ] LUCENE-8310: Remove suppress WindowsFS IW can now handle pending deletes. > Relax IWs check on pending deletes > -- > > Key: LUCENE-8310 > URL: https://issues.apache.org/jira/browse/LUCENE-8310 > Project: Lucene - Core > Issue Type: Improvement >Affects Versions: 7.4, master (8.0) >Reporter: Simon Willnauer >Priority: Major > Fix For: 7.4, master (8.0) > > Attachments: LUCENE-8310.patch, LUCENE-8310.patch, LUCENE-8310.patch, > LUCENE-8310.patch > > > I recently fixed the check in IW to fail if there are pending deletes. After > upgrading to a snapshot I realized the consequences of this check. It made > most of our usage of IW to for instance prepare commit metadata, rollback to > safe commit-points etc. impossible since we have to now busy wait on top of > directory util all deletes are actually gone even though that we can > guarantee that our history always goes forward. ie we are truly append-only > in the sense of never reusing segment generations. The fix that I made was > basically return false from a _checkPendingDeletions_ in a directory wrapper > to work around it. > I do expect this to happen to a lot of lucene users even if they use IW > correctly. My proposal is to make the check in IW a bit more sophisticated > and only fail if there are pending deletes that are in the future from a > generation perspective. We really don't care about files from the past. My > patch checks the segment generation of each pending file which is safe since > that is the same procedure we apply in IndexFileDeleter to inc reference etc. > and only fail if the pending delete is in the future. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8310) Relax IWs check on pending deletes
[ https://issues.apache.org/jira/browse/LUCENE-8310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16477220#comment-16477220 ] ASF subversion and git services commented on LUCENE-8310: - Commit b3894d7f11ac49e6cc0d3f9d888145659b804544 in lucene-solr's branch refs/heads/master from [~simonw] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=b3894d7 ] LUCENE-8310: Remove supporess WindowsFS IW can now handle pending deletes. > Relax IWs check on pending deletes > -- > > Key: LUCENE-8310 > URL: https://issues.apache.org/jira/browse/LUCENE-8310 > Project: Lucene - Core > Issue Type: Improvement >Affects Versions: 7.4, master (8.0) >Reporter: Simon Willnauer >Priority: Major > Fix For: 7.4, master (8.0) > > Attachments: LUCENE-8310.patch, LUCENE-8310.patch, LUCENE-8310.patch, > LUCENE-8310.patch > > > I recently fixed the check in IW to fail if there are pending deletes. After > upgrading to a snapshot I realized the consequences of this check. It made > most of our usage of IW to for instance prepare commit metadata, rollback to > safe commit-points etc. impossible since we have to now busy wait on top of > directory util all deletes are actually gone even though that we can > guarantee that our history always goes forward. ie we are truly append-only > in the sense of never reusing segment generations. The fix that I made was > basically return false from a _checkPendingDeletions_ in a directory wrapper > to work around it. > I do expect this to happen to a lot of lucene users even if they use IW > correctly. My proposal is to make the check in IW a bit more sophisticated > and only fail if there are pending deletes that are in the future from a > generation perspective. We really don't care about files from the past. My > patch checks the segment generation of each pending file which is safe since > that is the same procedure we apply in IndexFileDeleter to inc reference etc. > and only fail if the pending delete is in the future. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8310) Relax IWs check on pending deletes
[ https://issues.apache.org/jira/browse/LUCENE-8310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16477164#comment-16477164 ] ASF subversion and git services commented on LUCENE-8310: - Commit b4239ed51f3552b74b30c9b2cb70cf6a09161a69 in lucene-solr's branch refs/heads/branch_7x from [~simonw] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=b4239ed ] LUCENE-8310: Ensure IndexFileDeleter accounts for pending deletes Today we fail creating the IndexWriter when the directory has a pending delete. Yet, this is mainly done to prevent writing still existing files more than once. IndexFileDeleter already accounts for that for existing files which we can now use to also take pending deletes into account which ensures that all file generations per segment always go forward. > Relax IWs check on pending deletes > -- > > Key: LUCENE-8310 > URL: https://issues.apache.org/jira/browse/LUCENE-8310 > Project: Lucene - Core > Issue Type: Improvement >Affects Versions: 7.4, master (8.0) >Reporter: Simon Willnauer >Priority: Major > Fix For: 7.4, master (8.0) > > Attachments: LUCENE-8310.patch, LUCENE-8310.patch, LUCENE-8310.patch, > LUCENE-8310.patch > > > I recently fixed the check in IW to fail if there are pending deletes. After > upgrading to a snapshot I realized the consequences of this check. It made > most of our usage of IW to for instance prepare commit metadata, rollback to > safe commit-points etc. impossible since we have to now busy wait on top of > directory util all deletes are actually gone even though that we can > guarantee that our history always goes forward. ie we are truly append-only > in the sense of never reusing segment generations. The fix that I made was > basically return false from a _checkPendingDeletions_ in a directory wrapper > to work around it. > I do expect this to happen to a lot of lucene users even if they use IW > correctly. My proposal is to make the check in IW a bit more sophisticated > and only fail if there are pending deletes that are in the future from a > generation perspective. We really don't care about files from the past. My > patch checks the segment generation of each pending file which is safe since > that is the same procedure we apply in IndexFileDeleter to inc reference etc. > and only fail if the pending delete is in the future. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8310) Relax IWs check on pending deletes
[ https://issues.apache.org/jira/browse/LUCENE-8310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16477142#comment-16477142 ] ASF subversion and git services commented on LUCENE-8310: - Commit 585952797cfe715aefa29380f08eb93a25acc55e in lucene-solr's branch refs/heads/master from [~simonw] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=5859527 ] LUCENE-8310: Ensure IndexFileDeleter accounts for pending deletes Today we fail creating the IndexWriter when the directory has a pending delete. Yet, this is mainly done to prevent writing still existing files more than once. IndexFileDeleter already accounts for that for existing files which we can now use to also take pending deletes into account which ensures that all file generations per segment always go forward. > Relax IWs check on pending deletes > -- > > Key: LUCENE-8310 > URL: https://issues.apache.org/jira/browse/LUCENE-8310 > Project: Lucene - Core > Issue Type: Improvement >Affects Versions: 7.4, master (8.0) >Reporter: Simon Willnauer >Priority: Major > Fix For: 7.4, master (8.0) > > Attachments: LUCENE-8310.patch, LUCENE-8310.patch, LUCENE-8310.patch, > LUCENE-8310.patch > > > I recently fixed the check in IW to fail if there are pending deletes. After > upgrading to a snapshot I realized the consequences of this check. It made > most of our usage of IW to for instance prepare commit metadata, rollback to > safe commit-points etc. impossible since we have to now busy wait on top of > directory util all deletes are actually gone even though that we can > guarantee that our history always goes forward. ie we are truly append-only > in the sense of never reusing segment generations. The fix that I made was > basically return false from a _checkPendingDeletions_ in a directory wrapper > to work around it. > I do expect this to happen to a lot of lucene users even if they use IW > correctly. My proposal is to make the check in IW a bit more sophisticated > and only fail if there are pending deletes that are in the future from a > generation perspective. We really don't care about files from the past. My > patch checks the segment generation of each pending file which is safe since > that is the same procedure we apply in IndexFileDeleter to inc reference etc. > and only fail if the pending delete is in the future. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8310) Relax IWs check on pending deletes
[ https://issues.apache.org/jira/browse/LUCENE-8310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16476524#comment-16476524 ] Michael McCandless commented on LUCENE-8310: +1 > Relax IWs check on pending deletes > -- > > Key: LUCENE-8310 > URL: https://issues.apache.org/jira/browse/LUCENE-8310 > Project: Lucene - Core > Issue Type: Improvement >Affects Versions: 7.4, master (8.0) >Reporter: Simon Willnauer >Priority: Major > Fix For: 7.4, master (8.0) > > Attachments: LUCENE-8310.patch, LUCENE-8310.patch, LUCENE-8310.patch, > LUCENE-8310.patch > > > I recently fixed the check in IW to fail if there are pending deletes. After > upgrading to a snapshot I realized the consequences of this check. It made > most of our usage of IW to for instance prepare commit metadata, rollback to > safe commit-points etc. impossible since we have to now busy wait on top of > directory util all deletes are actually gone even though that we can > guarantee that our history always goes forward. ie we are truly append-only > in the sense of never reusing segment generations. The fix that I made was > basically return false from a _checkPendingDeletions_ in a directory wrapper > to work around it. > I do expect this to happen to a lot of lucene users even if they use IW > correctly. My proposal is to make the check in IW a bit more sophisticated > and only fail if there are pending deletes that are in the future from a > generation perspective. We really don't care about files from the past. My > patch checks the segment generation of each pending file which is safe since > that is the same procedure we apply in IndexFileDeleter to inc reference etc. > and only fail if the pending delete is in the future. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8310) Relax IWs check on pending deletes
[ https://issues.apache.org/jira/browse/LUCENE-8310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16476015#comment-16476015 ] Simon Willnauer commented on LUCENE-8310: - done! > Relax IWs check on pending deletes > -- > > Key: LUCENE-8310 > URL: https://issues.apache.org/jira/browse/LUCENE-8310 > Project: Lucene - Core > Issue Type: Improvement >Affects Versions: 7.4, master (8.0) >Reporter: Simon Willnauer >Priority: Major > Fix For: 7.4, master (8.0) > > Attachments: LUCENE-8310.patch, LUCENE-8310.patch, LUCENE-8310.patch, > LUCENE-8310.patch > > > I recently fixed the check in IW to fail if there are pending deletes. After > upgrading to a snapshot I realized the consequences of this check. It made > most of our usage of IW to for instance prepare commit metadata, rollback to > safe commit-points etc. impossible since we have to now busy wait on top of > directory util all deletes are actually gone even though that we can > guarantee that our history always goes forward. ie we are truly append-only > in the sense of never reusing segment generations. The fix that I made was > basically return false from a _checkPendingDeletions_ in a directory wrapper > to work around it. > I do expect this to happen to a lot of lucene users even if they use IW > correctly. My proposal is to make the check in IW a bit more sophisticated > and only fail if there are pending deletes that are in the future from a > generation perspective. We really don't care about files from the past. My > patch checks the segment generation of each pending file which is safe since > that is the same procedure we apply in IndexFileDeleter to inc reference etc. > and only fail if the pending delete is in the future. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8310) Relax IWs check on pending deletes
[
https://issues.apache.org/jira/browse/LUCENE-8310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16475993#comment-16475993
]
Michael McCandless commented on LUCENE-8310:
+1, much cleaner!
You can make {{SegmentInfos.getNextPendingGeneration}} private again?
> Relax IWs check on pending deletes
> --
>
> Key: LUCENE-8310
> URL: https://issues.apache.org/jira/browse/LUCENE-8310
> Project: Lucene - Core
> Issue Type: Improvement
>Affects Versions: 7.4, master (8.0)
>Reporter: Simon Willnauer
>Priority: Major
> Fix For: 7.4, master (8.0)
>
> Attachments: LUCENE-8310.patch, LUCENE-8310.patch, LUCENE-8310.patch
>
>
> I recently fixed the check in IW to fail if there are pending deletes. After
> upgrading to a snapshot I realized the consequences of this check. It made
> most of our usage of IW to for instance prepare commit metadata, rollback to
> safe commit-points etc. impossible since we have to now busy wait on top of
> directory util all deletes are actually gone even though that we can
> guarantee that our history always goes forward. ie we are truly append-only
> in the sense of never reusing segment generations. The fix that I made was
> basically return false from a _checkPendingDeletions_ in a directory wrapper
> to work around it.
> I do expect this to happen to a lot of lucene users even if they use IW
> correctly. My proposal is to make the check in IW a bit more sophisticated
> and only fail if there are pending deletes that are in the future from a
> generation perspective. We really don't care about files from the past. My
> patch checks the segment generation of each pending file which is safe since
> that is the same procedure we apply in IndexFileDeleter to inc reference etc.
> and only fail if the pending delete is in the future.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8310) Relax IWs check on pending deletes
[ https://issues.apache.org/jira/browse/LUCENE-8310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16475921#comment-16475921 ] Simon Willnauer commented on LUCENE-8310: - new patch, I think it's ready! I also removed checkPendingDeletions entirely > Relax IWs check on pending deletes > -- > > Key: LUCENE-8310 > URL: https://issues.apache.org/jira/browse/LUCENE-8310 > Project: Lucene - Core > Issue Type: Improvement >Affects Versions: 7.4, master (8.0) >Reporter: Simon Willnauer >Priority: Major > Fix For: 7.4, master (8.0) > > Attachments: LUCENE-8310.patch, LUCENE-8310.patch, LUCENE-8310.patch > > > I recently fixed the check in IW to fail if there are pending deletes. After > upgrading to a snapshot I realized the consequences of this check. It made > most of our usage of IW to for instance prepare commit metadata, rollback to > safe commit-points etc. impossible since we have to now busy wait on top of > directory util all deletes are actually gone even though that we can > guarantee that our history always goes forward. ie we are truly append-only > in the sense of never reusing segment generations. The fix that I made was > basically return false from a _checkPendingDeletions_ in a directory wrapper > to work around it. > I do expect this to happen to a lot of lucene users even if they use IW > correctly. My proposal is to make the check in IW a bit more sophisticated > and only fail if there are pending deletes that are in the future from a > generation perspective. We really don't care about files from the past. My > patch checks the segment generation of each pending file which is safe since > that is the same procedure we apply in IndexFileDeleter to inc reference etc. > and only fail if the pending delete is in the future. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8310) Relax IWs check on pending deletes
[ https://issues.apache.org/jira/browse/LUCENE-8310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16475907#comment-16475907 ] Adrien Grand commented on LUCENE-8310: -- Much nicer. +1 > Relax IWs check on pending deletes > -- > > Key: LUCENE-8310 > URL: https://issues.apache.org/jira/browse/LUCENE-8310 > Project: Lucene - Core > Issue Type: Improvement >Affects Versions: 7.4, master (8.0) >Reporter: Simon Willnauer >Priority: Major > Fix For: 7.4, master (8.0) > > Attachments: LUCENE-8310.patch, LUCENE-8310.patch > > > I recently fixed the check in IW to fail if there are pending deletes. After > upgrading to a snapshot I realized the consequences of this check. It made > most of our usage of IW to for instance prepare commit metadata, rollback to > safe commit-points etc. impossible since we have to now busy wait on top of > directory util all deletes are actually gone even though that we can > guarantee that our history always goes forward. ie we are truly append-only > in the sense of never reusing segment generations. The fix that I made was > basically return false from a _checkPendingDeletions_ in a directory wrapper > to work around it. > I do expect this to happen to a lot of lucene users even if they use IW > correctly. My proposal is to make the check in IW a bit more sophisticated > and only fail if there are pending deletes that are in the future from a > generation perspective. We really don't care about files from the past. My > patch checks the segment generation of each pending file which is safe since > that is the same procedure we apply in IndexFileDeleter to inc reference etc. > and only fail if the pending delete is in the future. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8310) Relax IWs check on pending deletes
[ https://issues.apache.org/jira/browse/LUCENE-8310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16475887#comment-16475887 ] Adrien Grand commented on LUCENE-8310: -- Maybe we can get rid of checkPendingDeletions now that we have getPendingDeletions? > Relax IWs check on pending deletes > -- > > Key: LUCENE-8310 > URL: https://issues.apache.org/jira/browse/LUCENE-8310 > Project: Lucene - Core > Issue Type: Improvement >Affects Versions: 7.4, master (8.0) >Reporter: Simon Willnauer >Priority: Major > Fix For: 7.4, master (8.0) > > Attachments: LUCENE-8310.patch, LUCENE-8310.patch > > > I recently fixed the check in IW to fail if there are pending deletes. After > upgrading to a snapshot I realized the consequences of this check. It made > most of our usage of IW to for instance prepare commit metadata, rollback to > safe commit-points etc. impossible since we have to now busy wait on top of > directory util all deletes are actually gone even though that we can > guarantee that our history always goes forward. ie we are truly append-only > in the sense of never reusing segment generations. The fix that I made was > basically return false from a _checkPendingDeletions_ in a directory wrapper > to work around it. > I do expect this to happen to a lot of lucene users even if they use IW > correctly. My proposal is to make the check in IW a bit more sophisticated > and only fail if there are pending deletes that are in the future from a > generation perspective. We really don't care about files from the past. My > patch checks the segment generation of each pending file which is safe since > that is the same procedure we apply in IndexFileDeleter to inc reference etc. > and only fail if the pending delete is in the future. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8310) Relax IWs check on pending deletes
[ https://issues.apache.org/jira/browse/LUCENE-8310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16475885#comment-16475885 ] Simon Willnauer commented on LUCENE-8310: - [~mikemccand] I had a better Idea and I managed to remove the exception altogether and configure IW to adopt it's gens to pending deletes. > Relax IWs check on pending deletes > -- > > Key: LUCENE-8310 > URL: https://issues.apache.org/jira/browse/LUCENE-8310 > Project: Lucene - Core > Issue Type: Improvement >Affects Versions: 7.4, master (8.0) >Reporter: Simon Willnauer >Priority: Major > Fix For: 7.4, master (8.0) > > Attachments: LUCENE-8310.patch, LUCENE-8310.patch > > > I recently fixed the check in IW to fail if there are pending deletes. After > upgrading to a snapshot I realized the consequences of this check. It made > most of our usage of IW to for instance prepare commit metadata, rollback to > safe commit-points etc. impossible since we have to now busy wait on top of > directory util all deletes are actually gone even though that we can > guarantee that our history always goes forward. ie we are truly append-only > in the sense of never reusing segment generations. The fix that I made was > basically return false from a _checkPendingDeletions_ in a directory wrapper > to work around it. > I do expect this to happen to a lot of lucene users even if they use IW > correctly. My proposal is to make the check in IW a bit more sophisticated > and only fail if there are pending deletes that are in the future from a > generation perspective. We really don't care about files from the past. My > patch checks the segment generation of each pending file which is safe since > that is the same procedure we apply in IndexFileDeleter to inc reference etc. > and only fail if the pending delete is in the future. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8310) Relax IWs check on pending deletes
[
https://issues.apache.org/jira/browse/LUCENE-8310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16475667#comment-16475667
]
Michael McCandless commented on LUCENE-8310:
{{+1}}
{{futur}} -> {{future}}
{{segment_s2}} --> {{segments_2}}
Can we mark the new {{Directory.getPendingDeletions}} API as
{{@lucene.internal}}?
Maybe factor out the new code in {{IndexWriter}}'s ctor to a package private
static method in {{IndexFileDeleter}} like {{verifyPendingDeletes}} since it's
that class's logic that really cares about from-the-future pending deletes?
Do we need to handle the case of a from-the-future new live docs generation?
I.e. every time we write a new live docs file for a given segment we write it
to a new generation for that one segment (e.g. {{_x_N.liv}}). Doc values
updates also write to generation files for that one segment.
> Relax IWs check on pending deletes
> --
>
> Key: LUCENE-8310
> URL: https://issues.apache.org/jira/browse/LUCENE-8310
> Project: Lucene - Core
> Issue Type: Improvement
>Affects Versions: 7.4, master (8.0)
>Reporter: Simon Willnauer
>Priority: Major
> Fix For: 7.4, master (8.0)
>
> Attachments: LUCENE-8310.patch
>
>
> I recently fixed the check in IW to fail if there are pending deletes. After
> upgrading to a snapshot I realized the consequences of this check. It made
> most of our usage of IW to for instance prepare commit metadata, rollback to
> safe commit-points etc. impossible since we have to now busy wait on top of
> directory util all deletes are actually gone even though that we can
> guarantee that our history always goes forward. ie we are truly append-only
> in the sense of never reusing segment generations. The fix that I made was
> basically return false from a _checkPendingDeletions_ in a directory wrapper
> to work around it.
> I do expect this to happen to a lot of lucene users even if they use IW
> correctly. My proposal is to make the check in IW a bit more sophisticated
> and only fail if there are pending deletes that are in the future from a
> generation perspective. We really don't care about files from the past. My
> patch checks the segment generation of each pending file which is safe since
> that is the same procedure we apply in IndexFileDeleter to inc reference etc.
> and only fail if the pending delete is in the future.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8310) Relax IWs check on pending deletes
[ https://issues.apache.org/jira/browse/LUCENE-8310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16475640#comment-16475640 ] Simon Willnauer commented on LUCENE-8310: - here is a patch > Relax IWs check on pending deletes > -- > > Key: LUCENE-8310 > URL: https://issues.apache.org/jira/browse/LUCENE-8310 > Project: Lucene - Core > Issue Type: Improvement >Affects Versions: 7.4, master (8.0) >Reporter: Simon Willnauer >Priority: Major > Fix For: 7.4, master (8.0) > > Attachments: LUCENE-8310.patch > > > I recently fixed the check in IW to fail if there are pending deletes. After > upgrading to a snapshot I realized the consequences of this check. It made > most of our usage of IW to for instance prepare commit metadata, rollback to > safe commit-points etc. impossible since we have to now busy wait on top of > directory util all deletes are actually gone even though that we can > guarantee that our history always goes forward. ie we are truly append-only > in the sense of never reusing segment generations. The fix that I made was > basically return false from a _checkPendingDeletions_ in a directory wrapper > to work around it. > I do expect this to happen to a lot of lucene users even if they use IW > correctly. My proposal is to make the check in IW a bit more sophisticated > and only fail if there are pending deletes that are in the future from a > generation perspective. We really don't care about files from the past. My > patch checks the segment generation of each pending file which is safe since > that is the same procedure we apply in IndexFileDeleter to inc reference etc. > and only fail if the pending delete is in the future. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
