RE: Kafka offset commits

2017-04-21 Thread Gwenhael Pasquiers
restarting. From: Tzu-Li (Gordon) Tai [mailto:tzuli...@apache.org] Sent: mercredi 19 avril 2017 11:39 To: user@flink.apache.org Subject: Re: Kafka offset commits Thanks for the clarification Aljoscha! Yes, you cannot restore from a 1.0 savepoint in Flink 1.2 (sorry, I missed the “1.0” part on my

Re: Kafka offset commits

2017-04-19 Thread Tzu-Li (Gordon) Tai
don) Tai [mailto:tzuli...@apache.org]  Sent: lundi 17 avril 2017 07:40 To: user@flink.apache.org Subject: Re: Kafka offset commits   Hi,   The FlinkKafkaConsumer in 1.2 is able to restore from older version state snapshots and bridge the migration, so there should be no problem in reading the offs

Re: Kafka offset commits

2017-04-19 Thread Aljoscha Krettek
17 07:40 > To: user@flink.apache.org > Subject: Re: Kafka offset commits > > Hi, > > The FlinkKafkaConsumer in 1.2 is able to restore from older version state > snapshots and bridge the migration, so there should be no problem in reading > the offsets from older state. Th

RE: Kafka offset commits

2017-04-18 Thread Gwenhael Pasquiers
the oldest offset, creating duplicated data (the kafka queue has over a week of buffer). B.R. From: Tzu-Li (Gordon) Tai [mailto:tzuli...@apache.org] Sent: lundi 17 avril 2017 07:40 To: user@flink.apache.org Subject: Re: Kafka offset commits Hi, The FlinkKafkaConsumer in 1.2 is able to restore

Re: Kafka offset commits

2017-04-16 Thread Tzu-Li (Gordon) Tai
Hi, The FlinkKafkaConsumer in 1.2 is able to restore from older version state snapshots and bridge the migration, so there should be no problem in reading the offsets from older state. The smallest or highest offsets will only be used if the offset no longer exists due to Kafka data retention s

Kafka offset commits

2017-04-13 Thread Gwenhael Pasquiers
Hello, We're going to migrate some applications that consume data from a Kafka 0.8 from Flink 1.0 to Flink 1.2. We are wondering if the offset commitment system changed between those two versions: is there a risk that the Flink 1.2-based application will start with no offset (thus will use eit