Re: Flink Kafka connector08 not updating the offsets with the zookeeper

2016-10-13 Thread Robert Metzger
Okay, I see. According to this document, we need to set a consumer id for each groupid and topic: https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structures+in+Zookeeper I created a JIRA for fixing this issue: https://issues.apache.org/jira/browse/FLINK-4822 On Wed, Oct 12, 2016

Re: Flink Kafka connector08 not updating the offsets with the zookeeper

2016-10-12 Thread Anchit Jatana
Hi Robert, Thanks for your response. I just figured out what the issue is. The reason why- bin/kafka-consumer-groups.sh --zookeeper --describe --group is not showing any result is because of the absence of the /consumers//owners/ in the zookeeper. The flink application is creating and

Re: Flink Kafka connector08 not updating the offsets with the zookeeper

2016-10-12 Thread Robert Metzger
Hi Anchit, Can you re-run your job with the debug level for Flink set to DEBUG? Then, you should see the following log message every time the offset is committed of Zookeeper: "Committing offsets to Kafka/ZooKeeper for checkpoint" Alternatively, can you check whether the offsets are available