Hi Nick, Are you using EXACTLY_ONCE semantics ? If so the sink would use transactions, and only commit the transaction on checkpoint complete to ensure end-to-end exactly-once. A detailed description could be find in [1]
Best, Yun [1] https://flink.apache.org/features/2018/03/01/end-to-end-exactly-once-apache-flink.html ------------------------------------------------------------------ Sender:nick toker<nick.toker....@gmail.com> Date:2020/12/21 23:52:34 Recipient:user<user@flink.apache.org> Theme:checkpoint delay consume message Hello, We noticed the following behavior: If we enable the flink checkpoints, we saw that there is a delay between the time we write a message to the KAFKA topic and the time the flink kafka connector consumes this message. The delay is closely related to checkpointInterval and/or minPauseBetweenCheckpoints meening that the MAX delay when consuming a message from KAFKA will be one of these parameters. Could you please advise how we can remove/control this delay? we use flink 1.11.2 BR nick