dramaticlly commented on code in PR #17359:
URL: https://github.com/apache/iceberg/pull/17359#discussion_r3654339506
##########
core/src/test/java/org/apache/iceberg/TestRewriteFiles.java:
##########
@@ -172,6 +172,22 @@ public void testDeleteOnly() {
.hasMessage("Files to delete cannot be empty");
}
+ @TestTemplate
+ public void addingAndDeletingSameFileDisallowed() {
Review Comment:
I think we might also need to fix a few other tests, namely
`testReadStreamWithSnapshotTypeRewriteDataFilesIgnoresReplaceMaxFilesAndRows`
because of
https://github.com/apache/iceberg/blob/6bfb510bbfbd1dbb1ae6c6655db5849125e4b836/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestStructuredStreamingRead3.java#L1041-L1051
##########
core/src/main/java/org/apache/iceberg/BaseRewriteFiles.java:
##########
@@ -152,5 +152,12 @@ private void validateReplacedAndAddedFiles() {
Preconditions.checkArgument(
deletesDeleteFiles() || !addsDeleteFiles(),
"Delete files to add must be empty because there's no delete file to
be rewritten");
+
+ for (DataFile added : addedDataFiles()) {
+ Preconditions.checkArgument(
+ !replacedDataFiles.contains(added),
+ "Cannot add and delete the same file in the same rewrite: %s",
+ added.location());
+ }
Review Comment:
I am wondering if this is the right place to place the guard, if the goal is
to prevent the same file being added and removed in the same snapshot? I think
both overwrite-files and row-delta can be abused in a way to achieve the same
goal.
On a separate note, knowing this produce incorrect result for the change
tracking detection, sometimes such surgical operation can be helpful for stats
backfill as to rectify a previously written file with incorrect column level
stats, want to see if we shall move ask to use dedicated repair such as in
proposed #10784
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
