aokolnychyi commented on code in PR #4703:
URL: https://github.com/apache/iceberg/pull/4703#discussion_r877234494
##########
api/src/main/java/org/apache/iceberg/RewriteFiles.java:
##########
@@ -84,4 +84,12 @@ RewriteFiles rewriteFiles(Set<DataFile> dataFilesToReplace,
Set<DeleteFile> dele
* @return this for method chaining
*/
RewriteFiles validateFromSnapshot(long snapshotId);
+
+ /**
+ * Ignore the position deletes in rewrite validation. Flink upsert job only
generates position deletes in the
+ * ongoing transaction, so it is not necessary to validate position deletes
when rewriting.
+ *
+ * @return this for method chaining
+ */
+ RewriteFiles ignorePosDeletesInValidation();
Review Comment:
I haven't seen #4748. I'll try to check it out later today. From a quick
look, I think it is similar yet a bit different from what I was proposing here.
I was thinking about a new method in `DeleteFileIndex` that would only match
position delete files added in later snapshots (ignoring position deletes added
in the same snapshot) and then having an option to use that from the
`RewriteFiles` API.
I think our assumption should be that multiple engines can write to the
table (we are about to have such use cases, in fact).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
