mxm commented on code in PR #18303:
URL: https://github.com/apache/iceberg/pull/18303#discussion_r4413655447
##########
flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/DynamicCommitter.java:
##########
@@ -224,6 +244,7 @@ private void commitPendingRequests(
String branch,
NavigableMap<Long, List<CommitRequest<DynamicCommittable>>>
commitRequestMap,
String newFlinkJobId,
+ @Nullable String sinkId,
Review Comment:
I think we want this to be never null. Could we set the sinkId to the jobId
when reading V1/V2 committables? Then the sinkId is never null and the dedup
only needs (sinkId, operatorId). Snapshots without flink.sink-id can still
match on flink.job-id.
##########
flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/DynamicCommitter.java:
##########
@@ -118,9 +120,8 @@ public void
commit(Collection<CommitRequest<DynamicCommittable>> commitRequests)
| | | |
| | | +-- commit
requests at that checkpoint
| | +-- checkpointId, sorted
ascending so older commits go first
- | +-- (jobId, operatorId) of the producing
aggregator; deduplication against the
- | table's snapshot summaries is per (jobId,
operatorId), so each group is
- | walked against the ancestor chain
independently
+ | +-- (jobId, sinkId, operatorId) of the producing
aggregator; each group is
Review Comment:
```suggestion
| +-- (sinkId, operatorId) of the producing
aggregator; each group is
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
