Hi, Are you using Flink savepoints [1] when restoring your application? If you use this the Kafka offset should be stored in state and it should restart from the correct position.
Best, Aljoscha [1] https://ci.apache.org/projects/flink/flink-docs-release-1.3/setup/savepoints.html > On 21 Mar 2017, at 01:50, Jins George <[email protected]> wrote: > > Hello, > > I am writing a Beam pipeline(streaming) with Flink runner to consume data > from Kafka and apply some transformations and persist to Hbase. > > If I restart the application ( due to failure/manual restart), consumer does > not resume from the offset where it was prior to restart. It always resume > from the latest offset. > > If I enable Flink checkpionting with hdfs state back-end, system appears to > be resuming from the earliest offset > > Is there a recommended way to resume from the offset where it was stopped ? > > Thanks, > Jins George
