Thanks for your answer. Does that means that flink does not rely on the offset in written to zookeeper anymore, but relies on the snapshots data, implying that it’s crucial to keep the same snapshot folder before and after the migration to Flink 1.2 ? We were also wondering if the flink consumer was able to restore it’s offset from Zookeeper. Another question : is there an expiration to the snapshots ? We’ve been having issues with an app that we forgot to restart. We did it after a couple of days, but it looks like it did not restore correctly the offset and it started consuming from the oldest offset, creating duplicated data (the kafka queue has over a week of buffer). B.R.
From: Tzu-Li (Gordon) Tai [mailto:tzuli...@apache.org] Sent: lundi 17 avril 2017 07:40 To: user@flink.apache.org Subject: Re: Kafka offset commits Hi, The FlinkKafkaConsumer in 1.2 is able to restore from older version state snapshots and bridge the migration, so there should be no problem in reading the offsets from older state. The smallest or highest offsets will only be used if the offset no longer exists due to Kafka data retention settings. Besides this, there was once fix related to the Kafka 0.8 offsets for Flink 1.2.0 [1]. Shortly put, before the fix, the committed offsets to ZK was off by 1 (wrt to how Kafka itself defines the committed offsets). However, this should not affect the behavior of restoring from offsets in savepoints, so it should be fine. Cheers, Gordon [1] https://issues.apache.org/jira/browse/FLINK-4723 On 13 April 2017 at 10:55:40 PM, Gwenhael Pasquiers (gwenhael.pasqui...@ericsson.com<mailto:gwenhael.pasqui...@ericsson.com>) wrote: Hello, We’re going to migrate some applications that consume data from a Kafka 0.8 from Flink 1.0 to Flink 1.2. We are wondering if the offset commitment system changed between those two versions: is there a risk that the Flink 1.2-based application will start with no offset (thus will use either the smallest or highest one) ? Or can we assume that the Flink 1.2 app will resume its work at the same offset than the Flink 1.0 app stopped (if they use the same consumer group id) ? B.R.
