[
https://issues.apache.org/jira/browse/FLINK-35630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17856378#comment-17856378
]
tanjialiang commented on FLINK-35630:
-------------------------------------
[~martijnvisser] Your point makes a lot of sense.
The Kafka consumer will theoretically only consume data that is smaller than
the high watermark, and the current problem I'm having is that after changing
partition leaders, the consume offset goes out of range.
Currently our Kafka cluster version is 2.3.0, and after much troubleshooting to
confirm, we think we are experiencing this problem
https://issues.apache.org/jira/browse/KAFKA-9835.
> Kafka source may reset the consume offset to earliest when the partition
> leader changes
> ---------------------------------------------------------------------------------------
>
> Key: FLINK-35630
> URL: https://issues.apache.org/jira/browse/FLINK-35630
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
> Affects Versions: kafka-3.2.0
> Reporter: tanjialiang
> Priority: Major
>
> Kafka producer using the ack=1 option to write data to a topic.
> Flink Kafka source startup with the *scan.startup.mode=earliest-offset*
> option to consume (the Kafka *auto.offset.reset* option will be force
> override to earliest).
> If a partition leader is not available, a follower may become the new leader
> and this may trigger log truncation. It may cause consumers to consume offset
> out of range and use the *auto.offset.reset* strategy to reset the offset.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
