dixingxing0 commented on a change in pull request #2109:
URL: https://github.com/apache/iceberg/pull/2109#discussion_r560947820
##########
File path:
flink/src/main/java/org/apache/iceberg/flink/sink/IcebergFilesCommitter.java
##########
@@ -305,6 +334,14 @@ public void processElement(StreamRecord<WriteResult>
element) {
this.writeResultsOfCurrentCkpt.add(element.getValue());
}
+ @Override
+ public void processWatermark(Watermark mark) throws Exception {
+ super.processWatermark(mark);
+ if (mark.getTimestamp() != Watermark.MAX_WATERMARK.getTimestamp()) {
Review comment:
As you said before, we use watermark to indicate the data completeness
on the ingestion path, i think we do not need to store `MAX_WATERMARK` when
flink job run in streaming-mode.
If flink job run in batch-mode, even we store one `MAX_WATERMARK`, we still
can't know which partition is completed, i think in batch-mode, we can just
simply rely on the scheduling system.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
