Cody Koeninger commented on FLINK-3037:

As a user, I want to be able to start low-value, high-volume streams at the 
latest offset, because it's not worth the time/money to start at the beginning.

I don't think silent data loss should be the default, but it should be 

> Make the behavior of the Kafka consumer configurable if the offsets to 
> restore from are not available
> -----------------------------------------------------------------------------------------------------
>                 Key: FLINK-3037
>                 URL: https://issues.apache.org/jira/browse/FLINK-3037
>             Project: Flink
>          Issue Type: Improvement
>          Components: Kafka Connector
>            Reporter: Robert Metzger
> Currently, if the {{FlinkKafkaConsumer}} is restoring a checkpoint and the 
> offset is not available anymore in Kafka, its restoring according to 
> {{auto.offset.reset}}.
> This leads to inconsistent behavior (not exactly-once anymore) because the 
> operators will not receive data in sync with the checkpoint.
> With this pull request, I would like to make the behavior controllable, using 
> a flag. The simplest approach would be to let the consumer fail in that case.

This message was sent by Atlassian JIRA

Reply via email to