Re: Read Kafka topic from the beginning

Hi Vladimir, Does current Kafka Consumer implementation allow to read all messages in a > topic from the beginning or from a specific offset. For reading from the beginning, setting "auto.offset.reset" to "smallest" will do the job. Reading from a specific offset is not yet supported yet, but

Re: Read Kafka topic from the beginning

As far as I know "auto.offset.reset" what to do if offset it not available or out of bound? Vladimir On Thursday, December 3, 2015 5:58 PM, Maximilian Michels wrote: Hi Vladimir, You may supply Kafka consumer properties when you create the FlinkKafkaConsumer. Properties

Read Kafka topic from the beginning

I see that Flink 0.10.1 now supports Keyed Schemas which allows us to rely on Kafka topics set to "compact" retention for data persistence. In our topology we wanted to set some topics with Log Compactions enabled and read topic from the beginning when the topology starts or component recovers.

Re: Read Kafka topic from the beginning

Hi Vladimir! The Kafka Consumer can start from any offset internally (it does that for example when recovering a failure). Should be fairly straightforward to set that offset field initially from a parameter. The FlinkKafkaConsumer is part of the user jars anyways. If you want, you can give it a

Re: Read Kafka topic from the beginning

Hi Vladimir, Did you pass the properties to the FlinkKafkaConsumer? Cheers, Max On Thu, Dec 3, 2015 at 7:06 PM, Vladimir Stoyak wrote: > Gave it a try, but does not seem to help. Is it working for you? > > Thanks > > Sent from my iPhone > >> On Dec 3, 2015, at 6:11 PM,

Re: Read Kafka topic from the beginning

Gave it a try, but does not seem to help. Is it working for you? Thanks Sent from my iPhone > On Dec 3, 2015, at 6:11 PM, Vladimir Stoyak wrote: > > As far as I know "auto.offset.reset" what to do if offset it not available or > out of bound? > > Vladimir > > > On