Tzu-Li (Gordon) Tai updated FLINK-5704:
    Priority: Major  (was: Blocker)

> Deprecate FlinkKafkaConsumer constructors in favor of improvements to 
> decoupling from Kafka offset committing
> -------------------------------------------------------------------------------------------------------------
>                 Key: FLINK-5704
>                 URL: https://issues.apache.org/jira/browse/FLINK-5704
>             Project: Flink
>          Issue Type: Improvement
>          Components: Kafka Connector, Streaming Connectors
>            Reporter: Tzu-Li (Gordon) Tai
>            Assignee: Tzu-Li (Gordon) Tai
>            Priority: Major
> With FLINK-3398 and FLINK-4280, the {{FlinkKafkaConsumer}} will be able to 
> completely operate independently of committed offsets in Kafka.
> I.e.,
> (1) *Starting position*: when starting, the consumer can choose to not use 
> any committed offsets in Kafka as the starting position
> (2) *Committing offsets back to Kafka*: the consumer can completely opt-out 
> of committing offsets back to Kafka
> However, our current default behaviour for (1) is to respect committed 
> offsets, and (2) is to always have offset committing. Users still have to 
> call the respective setter configuration methods to change this.
> I think we should deprecate the current constructors in favor of new ones 
> with default behaviours (1) start from the latest record, without respecting 
> Kafka offsets, and (2) don't commit offsets.
> With this change, users explicitly call the config methods of FLINK-3398 and 
> FLINK-4280 to *enable* respecting committed offsets for Kafka, instead of 
> _disabling_ it. They would want to / need to enable it, only when perhaps to 
> migrate from a non-Flink consuming application, or they wish to expose the 
> internal checkpointed offsets to measure consumer lag using Kafka toolings.
> The main advantage for this change is that the API of {{FlinkKafkaConsumer}} 
> can speak for itself that it does not depend on committed offsets in Kafka 
> (this is a misconception that users frequently have), and that exactly-once 
> depends solely on offsets checkpointed internally using Flink's checkpointing 
> mechanics.

This message was sent by Atlassian JIRA

Reply via email to