[ 
https://issues.apache.org/jira/browse/FLINK-5704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16361895#comment-16361895
 ] 

Tzu-Li (Gordon) Tai commented on FLINK-5704:
--------------------------------------------

There was some discussion on the mailing list [1] that we do this as part of a 
major rework of the Kafka / Kinesis connectors in Flink 1.6. I'll downgrade the 
priority and reopen this in a Kafka / Kinesis connector rework umbrella issue.

 

[1] 
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Timestamp-watermark-support-in-Kinesis-consumer-td20910.html

> Deprecate FlinkKafkaConsumer constructors in favor of improvements to 
> decoupling from Kafka offset committing
> -------------------------------------------------------------------------------------------------------------
>
>                 Key: FLINK-5704
>                 URL: https://issues.apache.org/jira/browse/FLINK-5704
>             Project: Flink
>          Issue Type: Improvement
>          Components: Kafka Connector, Streaming Connectors
>            Reporter: Tzu-Li (Gordon) Tai
>            Assignee: Tzu-Li (Gordon) Tai
>            Priority: Blocker
>             Fix For: 1.5.0
>
>
> With FLINK-3398 and FLINK-4280, the {{FlinkKafkaConsumer}} will be able to 
> completely operate independently of committed offsets in Kafka.
> I.e.,
> (1) *Starting position*: when starting, the consumer can choose to not use 
> any committed offsets in Kafka as the starting position
> (2) *Committing offsets back to Kafka*: the consumer can completely opt-out 
> of committing offsets back to Kafka
> However, our current default behaviour for (1) is to respect committed 
> offsets, and (2) is to always have offset committing. Users still have to 
> call the respective setter configuration methods to change this.
> I think we should deprecate the current constructors in favor of new ones 
> with default behaviours (1) start from the latest record, without respecting 
> Kafka offsets, and (2) don't commit offsets.
> With this change, users explicitly call the config methods of FLINK-3398 and 
> FLINK-4280 to *enable* respecting committed offsets for Kafka, instead of 
> _disabling_ it. They would want to / need to enable it, only when perhaps to 
> migrate from a non-Flink consuming application, or they wish to expose the 
> internal checkpointed offsets to measure consumer lag using Kafka toolings.
> The main advantage for this change is that the API of {{FlinkKafkaConsumer}} 
> can speak for itself that it does not depend on committed offsets in Kafka 
> (this is a misconception that users frequently have), and that exactly-once 
> depends solely on offsets checkpointed internally using Flink's checkpointing 
> mechanics.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to