[
https://issues.apache.org/jira/browse/KAFKA-20035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18060342#comment-18060342
]
Ken Huang edited comment on KAFKA-20035 at 2/23/26 1:27 PM:
------------------------------------------------------------
{quote}Interesting case... If people really care about this, why would they use
`auto.offset.reset = latest` in the first place? In this case, it actually does
what it is supposed to do
{quote}
>From a user’s perspective, they typically think in terms of topics rather than
>individual partitions. As a result, this behavior would be confusing for users.
{quote}Have we explored improving the `auto.offset.reset` part instead of
changing the protocol
{quote}
I’m not sure I fully understand this suggestion in this context. Could you
explain more on this thought?
was (Author: JIRAUSER305187):
> Interesting case... If people really care about this, why would they use
> `auto.offset.reset = latest` in the first place? In this case, it actually
> does what it is supposed to do
>From a user’s perspective, they typically think in terms of topics rather than
>individual partitions. As a result, this behavior would be confusing for users.
> Have we explored improving the `auto.offset.reset` part instead of changing
> the protocol
I’m not sure I fully understand this suggestion in this context. Could you
explain more on this thought?
> Prevent data loss during partition expansion by enforcing "earliest" offset
> reset for dynamically added partitions
> ------------------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-20035
> URL: https://issues.apache.org/jira/browse/KAFKA-20035
> Project: Kafka
> Issue Type: Bug
> Components: clients, consumer, core, group-coordinator
> Reporter: Chia-Ping Tsai
> Assignee: Ken Huang
> Priority: Critical
> Labels: kip
>
> Currently, when a consumer group is configured with {{{}auto.offset.reset =
> latest{}}}, dynamically adding new partitions to a subscribed topic can lead
> to data loss due to a race condition.
> The scenario is as follows:
> # A group subscribes to a topic with {{{}auto.offset.reset = latest{}}}.
> # The topic is expanded (e.g., from 3 to 4 partitions).
> # Producers immediately start writing data to the new partition (Partition
> 3).
> # The Group Coordinator detects the change and assigns Partition 3 to a
> member.
> # The member initializes the partition. Since there is no committed offset,
> it applies the
> # *Result: Any messages written to Partition 3 between step 3 and step 5 are
> skipped and lost.*
> From a user's perspective, {{latest}} should mean "start consuming from the
> point of subscription," not "skip data from newly created infrastructure."
> KIP-1282:
> [https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=406619800]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
