aokolnychyi commented on code in PR #4703:
URL: https://github.com/apache/iceberg/pull/4703#discussion_r885934904
##########
api/src/main/java/org/apache/iceberg/RewriteFiles.java:
##########
@@ -84,4 +84,12 @@ RewriteFiles rewriteFiles(Set<DataFile> dataFilesToReplace,
Set<DeleteFile> dele
* @return this for method chaining
*/
RewriteFiles validateFromSnapshot(long snapshotId);
+
+ /**
+ * Ignore the position deletes in rewrite validation. Flink upsert job only
generates position deletes in the
+ * ongoing transaction, so it is not necessary to validate position deletes
when rewriting.
+ *
+ * @return this for method chaining
+ */
+ RewriteFiles ignorePosDeletesInValidation();
Review Comment:
Well, I am no longer sure we are talking about the same use case. Let me
give an example of what I meant.
```
snapshot 1 (upsert):
f1, f2, eq1, pos1
```
Suppose `pos1` file references records in `f1`. If I understand correctly,
trying to rewrite `f1` and `f2` would fail right now as there is a matching
position delete. I was proposing to add a method to `RewriteFiles` that would
ignore delete files for data files added in the same snapshot. This should
unlock rewriting upserts and will be safe as long as `pos1` is applied whenever
`f1` and `f2` are rewritten (that's how compaction works right now).
Now what should happen if a concurrent operation is committed while we are
rewriting? Suppose we have another concurrent upsert.
```
snapshot 2 (upsert):
f1, f2, eq1, pos1 (seq = 0)
f3, f4, eq2, pos2 (seq = 1)
```
With the flag I was talking about, we will discard `pos1` but we will still
validate `pos2` does not match `f1` and `f2`. Right now, we will use partition
info and min/max file path to see if `pos2` matches `f1` or `f2`. It can still
give us false positives but we also plan to have bloom filters, which should
improve filtering.
There is also another scenario which we have to avoid.
```
snapshot 2 (Spark merge-on-read on top Flink upsert):
f1, f2, eq1, pos1 (seq = 0)
f3, f4, pos2 (seq = 1)
```
In this case, `pos2` can indeed reference `f1` and `f2`. If that happens, we
can't commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
