Re: Watermark in global commit

2023-02-15 Thread Tobias Fröhlich
Dear Jan, thank you for your answer! The logic that ensures consistency should already be implemented in the TwoPhaseCommitSink and the WithPostCommitTopology. So I would rather like to use these well-tested classes than implement my own logic for this. Best regards Tobias Hi, I'm not

Re: Watermark in global commit

2023-02-14 Thread Jan Lukavský
Hi, I'm not expert on Flink specifially, but your approach might be easier solve when broken down into two steps - create a "stable" input to downstream processing, this might include a specific watermark. In Flink, the "stability" of input for downstream processing is ensured by a

Watermark in global commit

2023-02-14 Thread Tobias Fröhlich
Dear flink team, I am facing the following problem: I would need to write events to parquet files using the FileSink. Subsequently, I want to do something else in a global commit where I need the corresponding watermark. However, the org.apache.flink.connector.file.sink.FileSink forces the