Hi Prabhu, I'm pretty sure that the Kafka 09 consumer commits offsets to Kafka when checkpointing is turned on.
In the FlinkKafkaConsumerBase.notifyCheckpointComplete(), we call fetcher.commitSpecificOffsetsToKafka(checkpointOffsets);, which calls this.consumer.commitSync(offsetsToCommit); in Kafka09Fetcher.commitSpecificOffsetsToKafka(). On Mon, Aug 8, 2016 at 8:24 PM, vpra...@gmail.com <vpra...@gmail.com> wrote: > Hi Stephan, > > The flink kafka 09 connector does not do offset commits to kafka when > checkpointing is turned on. Is there a way to monitor the offset lag in > this > case, > > I am turning on a flink job that reads data from kafka (has about a week > data - around 7 TB) , currently the approximate way that I use is the > number > of records read shown in the flink-UI and the last offset in kafka. > > Thanks, > Prabhu > > > > -- > View this message in context: http://apache-flink-user- > mailing-list-archive.2336050.n4.nabble.com/Flink-Kafka-Consumer-Behaviour- > tp8257p8375.html > Sent from the Apache Flink User Mailing List archive. mailing list archive > at Nabble.com. >
