stevenzwu commented on a change in pull request #2109:
URL: https://github.com/apache/iceberg/pull/2109#discussion_r561086752
##########
File path:
flink/src/main/java/org/apache/iceberg/flink/sink/IcebergFilesCommitter.java
##########
@@ -296,6 +318,13 @@ private void commitOperation(SnapshotUpdate<?> operation,
int numDataFiles, int
long start = System.currentTimeMillis();
operation.commit(); // abort is automatically called if this fails.
+
+ Long watermarkForCheckpoint = watermarkPerCheckpoint.get(checkpointId);
Review comment:
We actually [set it as table properties
too](https://github.com/Netflix-Skunkworks/nfflink-connector-iceberg/blob/master/nfflink-connector-iceberg/src/main/java/com/netflix/spaas/nfflink/connector/iceberg/sink/IcebergCommitter.java#L555).
I think table properties is easier for the workflow scheduler (in the batch
system) to query. Otherwise, they have to iterate the snapshots and find out
the latest watermarks for all 3 regions. cc @rdblue
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
