richiebachala commented on issue #2195:
URL: https://github.com/apache/iceberg-go/issues/2195#issuecomment-6727928156
@badalprasadsingh thanks, I'll take it. Sorry for the slow reply.
Agreed on `ReplaceDataFiles`. Since every operation shares the guard, I'll
make the rule depend on the operation, not just on whether the commit removes a
DV:
1. Keep `noReplay` as the default for any commit that removes a DV.
2. Lift it only when every removed DV belongs to a data file that the same
commit removes and that a conflict validator covers:
- copy-on-write delete and overwrite (the `rewriteValidator(removed)` in
`transaction_delete.go`);
- `ReplaceFiles` (`rewriteValidatorWithReferencedDataFiles` in
`replaceFiles`);
- merge-on-read whole-file drops, once they register a check that rejects
a concurrent DV on a dropped file.
3. `ReplaceDataFiles` registers no validator, so it stays non-replayable. So
does the merge-on-read partial-delete merge, where the data file stays live.
Tests in `no_replay_removals_test.go`:
- A CoW Delete/Overwrite of a file with a DV now lands after a concurrent
unrelated append.
- `ReplaceDataFiles` that removes a DV still fails on a concurrent commit
instead of replaying. This is the case you raised, with a peer deleting rows.
- A MoR whole-file drop plus a concurrent DV on that file fails validation
instead of leaving an orphan.
- The existing DV-replaced-while-file-stays-live regression stays green.
I'll put up a PR and tag you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
