hi Jun Yes, updating the docs requires minimal changes to the codebase, but it is not an out-of-the-box solution. From my perspective, this issue can be addressed by setting the console consumer's policy back to latest.
Let's see what Jiunn thinks Best, Chia-Ping On 2026/07/06 15:59:37 Jun Rao via dev wrote: > Hi, Chia-Ping, > > One potential downside of changing the default is that some existing tools > (e.g. console consumer) may depend on this behavior. For example, a user > testing the system may expect to see only new messages and could be > surprised by the change. Another option is to leave the default unchanged, > but document clearly how to prevent losing messages on new partitions > through by_duration. The users can then make the corresponding config > changes if needed. > > Thanks, > > Jun > > On Wed, Jul 1, 2026 at 9:39 AM Chia-Ping Tsai <[email protected]> wrote: > > > Hi Jun, > > > > Thanks for the feedback. I agree with your conclusion. It would be better > > to "make haste" with a new KIP to address this. We can focus the discussion > > on the following points: > > > > 1) Should we change the default reset policy to by_duration for all > > consumers? > > 2) What is the optimal duration to prevent data loss while avoiding large > > backlogs? Note that the classic consumer has a 3-second heartbeat interval, > > whereas the async consumer uses 5 seconds. > > 3) What are the potential side effects of replacing latest with > > by_duration as the default behavior? > > > > Best, > > Chia-Ping > > > > On 2026/06/30 20:39:24 Jun Rao via dev wrote: > > > Hi, Chia-Ping, > > > > > > I agree that using by_duration with a value of 5s to 10s addresses most > > > common issues with expanded partitions, given the default > > > heartbeat.interval.ms is 3 seconds. We can consider setting that as the > > new > > > default. The current proposal may address some other rarer cases, but it > > > feels a bit like over engineering. > > > > > > Thanks, > > > > > > Jun > > > > > > On Tue, Jun 30, 2026 at 5:58 AM Chia-Ping Tsai <[email protected]> > > wrote: > > > > > > > - Short durations fail during extended downtimes, leading to the exact > > > > silent data > > > > loss this KIP aims to prevent. > > > > > > > > Losing the records produced to expanded partitions during a downtime is > > > > actually the expected behavior for the latest policy. This is exactly > > why, > > > > in the current KIP proposal, we prefer by_duration over earliest for > > the > > > > expanded partitions—we want to avoid forcing users to consume a massive > > > > accumulated backlog just because a partition is technically "new". > > > > > > > > To quote my previous comment from May 15: > > > > "...if a consumer is offline for a while and a new partition is created > > > > during that downtime, the user might actually want to skip to latest > > when > > > > resuming, rather than reading from earliest just because the partition > > is > > > > technically 'new' to the group." > > > > > > > > On 2026/06/30 12:12:47 黃竣陽 wrote: > > > > > Hello all, > > > > > > > > > > The metadata blindness and consumer downtime windows are inherently > > > > > unpredictable, making a fixed by_duration approach unviable. No > > static > > > > > duration can accommodate both extremes: > > > > > > > > > > - Short durations fail during extended downtimes, leading to the > > exact > > > > silent data > > > > > loss this KIP aims to prevent. > > > > > - Long durations (calibrated for worst-case scenarios) cause routine > > > > restarts to > > > > > trigger unnecessary mass replays, undermining the latest offset reset > > > > policy. > > > > > > > > > > This is why the proposed solution uses creation time. It provides a > > > > discrete, > > > > > deterministic signal—verifying whether a partition is actually newer > > > > than the group, > > > > > rather than relying on an arbitrary time window that has no > > correlation > > > > with the consumer’s > > > > > actual downtime. > > > > > > > > > > Best Regards, > > > > > Jiunn-Yang > > > > > > > > > > > Chia-Ping Tsai <[email protected]> 於 2026年6月30日 下午3:05 寫道： > > > > > > > > > > > > Hi all, > > > > > > > > > > > > I'd like to revisit Jun previous comment about using > > by_duration=5s. I > > > > realize now that having a small backlog is actually an acceptable > > > > trade-off, especially since the latest policy inherently brings a tiny > > > > backlog to starting consumers anyway due to the time gap between > > fetching > > > > the offset and actually consuming. > > > > > > > > > > > > If we bring the simple by_duration=5s solution back to the KIP, I'd > > > > like to discuss a further step: Should we just change the default > > policy > > > > from latest to by_duration=5s specifically for the new consumer and > > share > > > > consumer? > > > > > > > > > > > > Best, > > > > > > Chia-Ping > > > > > > > > > > > > On 2026/06/15 17:39:19 Chia-Ping Tsai wrote: > > > > > >> hi Jun > > > > > >> > > > > > >> The most important part of this story is how users should expect > > the > > > > data > > > > > >> they can see when using the latest or by_duration policy with > > expanded > > > > > >> partitions. > > > > > >> > > > > > >> Yes, the by_duration policy can minimize data loss, but it is > > > > > >> non-deterministic, which means users will either read too many > > > > historical > > > > > >> records from existing partitions or lose some records from > > expanded > > > > > >> partitions. > > > > > >> > > > > > >> Also, I agree that auto.offset.reset.max.age.ms is a bit hard to > > > > > >> understand, and that is why I preferred having a whole new policy > > > > based > > > > > >> entirely on group creation time (KIP-1282) > > > > > >> > > > > > >> Best, > > > > > >> Chia-Ping > > > > > >> > > > > > >> Jun Rao via dev <[email protected]> 於 2026年6月16日週二 上午1:08寫道： > > > > > >> > > > > > >>> Hi, Chia-Ping and Jiunn-Yang, > > > > > >>> > > > > > >>> Thanks for the reply. I am still trying to understand the value > > of > > > > the new > > > > > >>> configs with the KIP. > > > > > >>> > > > > > >>> The motivation of the KIP is that a user doesn't want to miss the > > > > data if > > > > > >>> the backlog is small. The backlog of the existing partition is > > easy > > > > to > > > > > >>> understand because it relates to retention time. The backlog for > > the > > > > new > > > > > >>> partition is a bit subtle to understand since it depends on the > > > > metadata > > > > > >>> refresh delay. To set auto.offset.reset.max.age.ms, the user > > needs > > > > to > > > > > >>> understand the metadata refresh delay on the consumer side and > > use > > > > it to > > > > > >>> set the config. > > > > > >>> > > > > > >>> Now, let's consider the alternative: setting the same value for > > the > > > > > >>> existing by_duration policy. The KIP lists three issues with this > > > > approach. > > > > > >>> 1. It computes the seek target client-side as now() - duration, > > which > > > > > >>> introduces clock skew across consumers and forces operators to > > choose > > > > > >>> overly large durations, causing unnecessary reprocessing. > > > > > >>> 2. The target timestamp is recomputed on each retry, so failed > > > > > >>> ListOffsetsRequest retries can shift the target forward and > > > > potentially > > > > > >>> miss records produced between attempts. > > > > > >>> 3. It applies uniformly to all partitions without committed > > offsets, > > > > and > > > > > >>> cannot distinguish newly expanded partitions from long-existing > > > > partitions > > > > > >>> newly assigned to the group, leading to unnecessary replay. > > > > > >>> > > > > > >>> Issues 1 and 2 are uncommon and can be mitigated by adding a bit > > > > buffer to > > > > > >>> the metadata refresh delay. We could also consider improving the > > > > > >>> implementation. For issue 3, the metadata refresh delay is > > typically > > > > low > > > > > >>> (in the order of minutes with the classic consumer and tens of > > > > seconds with > > > > > >>> the new consumer). If a user is ok with reading that much backlog > > > > for new > > > > > >>> partitions, it seems they will be ok doing the same for existing > > > > > >>> partitions. > > > > > >>> > > > > > >>> So, instead of introducing a new config, could we just reuse the > > > > existing > > > > > >>> config with better documentation and/or implementation? > > > > > >>> > > > > > >>> Jun > > > > > >>> > > > > > >>> > > > > > >>> On Sat, Jun 13, 2026 at 12:19 AM 黃竣陽 <[email protected]> wrote: > > > > > >>> > > > > > >>>> Hello Jun, > > > > > >>>> > > > > > >>>> You're right that group creation time is the more intuitive > > answer > > > > at > > > > > >>>> first glance, > > > > > >>>> the KIP's own motivation talks about partitions that "predate > > the > > > > group" > > > > > >>>> vs partitions > > > > > >>>> "created during group runtime," which directly points to a > > > > > >>> group-lifecycle > > > > > >>>> classifier. > > > > > >>>> I'd like to walk through why we landed on partition age, and the > > > > > >>>> trade-offs we considered. > > > > > >>>> > > > > > >>>> We evaluated three candidate signals: > > > > > >>>> > > > > > >>>> 1. `by_duration:5secs` > > > > > >>>> > > > > > >>>> This covers the metadata blindness window, but has issues the > > KIP > > > > > >>>> currently documents > > > > > >>>> under "Why not use `by_duration`?": > > > > > >>>> > > > > > >>>> - Client-side `now() - duration` introduces clock skew across > > > > consumers. > > > > > >>>> - `ListOffsets` retries shift the target forward, potentially > > > > missing > > > > > >>>> records produced between > > > > > >>>> attempts. > > > > > >>>> - It applies uniformly to all partitions without committed > > offsets, > > > > > >>>> including pre-existing partitions > > > > > >>>> newly assigned to the group, causing unnecessary replay. > > > > > >>>> > > > > > >>>> 2. Group creation time as classifier > > > > > >>>> > > > > > >>>> This works cleanly when the consumer is actively running. Our > > > > concern > > > > > >>>> is the idle / late-rejoin case: > > > > > >>>> > > > > > >>>> T=0: Group created. > > > > > >>>> T=1..T=100: Consumer idle (down, disconnected, etc.). > > > > > >>>> T=50: Partition added during the idle window. > > > > > >>>> T=100: Consumer resumes. > > > > > >>>> > > > > > >>>> Under group creation time, the new partition is classified as > > new > > > > > >>>> (`50 > 0`) and reset to `earliest`, replaying everything from > > T=50. > > > > > >>>> But during `[T=1, T=100]`, base partitions also accumulated data > > > > that > > > > > >>>> the consumer accepts as lost — that is precisely the contract of > > > > > >>>> `auto.offset.reset=latest`. There is no principled reason to > > treat > > > > > >>>> the new partition differently; both contain backlog accumulated > > > > during > > > > > >>>> the same idle window. > > > > > >>>> > > > > > >>>> This aligns with the "backlog is backlog” principle you raised > > in > > > > > >>>> the KIP-1282 thread: a `latest` user has tolerated some backlog > > on > > > > > >>>> every other partition during the same idle period; forcing > > 0-backlog > > > > > >>>> tolerance only on new partitions would be inconsistent with that > > > > > >>>> tolerance. > > > > > >>>> > > > > > >>>> 3. Partition age vs threshold > > > > > >>>> > > > > > >>>> Partition age corresponds to the actual silent data loss window, > > > > > >>>> the gap between partition creation and the consumer’s metadata > > > > > >>>> refresh. Within this window, data loss is genuinely silent: the > > > > > >>>> consumer had no opportunity to know about the partition. Outside > > > > this > > > > > >>>> window, missing data reflects either: > > > > > >>>> > > > > > >>>> - (a) the user’s tolerated cost of running with idle consumers, > > or > > > > > >>>> - (b) an operational issue to surface via monitoring, not via > > reset > > > > > >>> policy. > > > > > >>>> > > > > > >>>> We did not choose partition age because it is more elegant than > > > > group > > > > > >>>> creation time — we chose it because its failure mode (requires a > > > > > >>>> threshold) is > > > > > >>>> less invasive than the failure mode of group creation time > > > > (overrides > > > > > >>>> user-stated > > > > > >>>> `latest` intent during idle periods). > > > > > >>>> > > > > > >>>> Best Regards, > > > > > >>>> Jiunn-Yang > > > > > >>>> > > > > > >>>>> Chia-Ping Tsai <[email protected]> 於 2026年6月13日 上午11:52 寫道： > > > > > >>>>> > > > > > >>>>> Hi Jun, > > > > > >>>>> > > > > > >>>>> Relying on both creation times will create an inconsistent > > > > scenario. A > > > > > >>>>> consumer that lost all offsets due to a long sleep will seek > > to the > > > > > >>>>> beginning for the partitions created later than the group. > > > > > >>>>> > > > > > >>>>> That is why we initially proposed KIP-1282 to fix the > > inconsistency > > > > > >>>> using a > > > > > >>>>> whole new policy. Since KIP-1282 couldn't reach a consensus, > > > > KIP-1327 > > > > > >>>> goes > > > > > >>>>> back to using flexible configurations to prevent users from > > falling > > > > > >>> into > > > > > >>>>> that pitfall. > > > > > >>>>> > > > > > >>>>> Best, Chia-Ping > > > > > >>>>> > > > > > >>>>> Jun Rao via dev <[email protected]> 於 2026年6月13日週六 > > 上午6:49寫道： > > > > > >>>>> > > > > > >>>>>> Hi, Jiunn-Yang, > > > > > >>>>>> > > > > > >>>>>> Thanks for the reply and sorry for the late reply. > > > > > >>>>>> > > > > > >>>>>> JR1. The design of auto.offset.reset.max.age.ms still feels > > > > weird to > > > > > >>>> me. > > > > > >>>>>> It > > > > > >>>>>> categorizes partitions as new or existing based on the > > partition > > > > > >>>> creation > > > > > >>>>>> time. Intuitively, the categorization should be based on the > > group > > > > > >>>> creation > > > > > >>>>>> time: all partitions existing when the group is created are > > > > existing > > > > > >>> and > > > > > >>>>>> all partitions created after the group creation are new > > > > partitions. > > > > > >>>>>> > > > > > >>>>>> Jun > > > > > >>>>>> > > > > > >>>>>> > > > > > >>>>>> > > > > > >>>>>> On Tue, Jun 9, 2026 at 8:51 AM 黃竣陽 <[email protected]> > > wrote: > > > > > >>>>>> > > > > > >>>>>>> Hi all, > > > > > >>>>>>> > > > > > >>>>>>> Manually bumping this thread. If there is no further > > > > > >>>>>>> discussion, I will close the vote. > > > > > >>>>>>> > > > > > >>>>>>> Best Regards, > > > > > >>>>>>> Jiunn-Yang > > > > > >>>>>>> > > > > > >>>>>>>> 黃竣陽 <[email protected]> 於 2026年6月1日 晚上7:16 寫道： > > > > > >>>>>>>> > > > > > >>>>>>>> Hello Jian, > > > > > >>>>>>>> > > > > > >>>>>>>> Thanks for your feedback, > > > > > >>>>>>>> > > > > > >>>>>>>> Agreed, partition expansion is a common operational task, > > not an > > > > > >>> edge > > > > > >>>>>>>> case. I've updated the Motivation section accordingly. > > > > > >>>>>>>> > > > > > >>>>>>>> Best Regards, > > > > > >>>>>>>> Jiunn-Yang > > > > > >>>>>>>> > > > > > >>>>>>>>> jian fu <[email protected]> 於 2026年6月1日 下午5:49 寫道： > > > > > >>>>>>>>> > > > > > >>>>>>>>> Hi Jiunn-Yang: > > > > > >>>>>>>>> > > > > > >>>>>>>>> Thanks for the KIP. I think it would be useful to clarify > > that > > > > this > > > > > >>>>>> is a > > > > > >>>>>>>>> common scenario rather than an edge case, which further > > > > > >>> demonstrates > > > > > >>>>>> the > > > > > >>>>>>>>> need for this optimization. For example: > > > > > >>>>>>>>> A partition expansion is a common operational task in > > Kafka: To > > > > > >>>>>> balance > > > > > >>>>>>>>> resource utilization and cost, topics are typically created > > > > with a > > > > > >>>>>>> moderate > > > > > >>>>>>>>> default partition count. However, as traffic grows over > > time, > > > > it is > > > > > >>>>>>> often > > > > > >>>>>>>>> necessary to increase the number of partitions to > > accommodate > > > > the > > > > > >>>>>> higher > > > > > >>>>>>>>> workload. > > > > > >>>>>>>>> > > > > > >>>>>>>>> Regards > > > > > >>>>>>>>> Jian > > > > > >>>>>>>>> > > > > > >>>>>>>>> 黃竣陽 <[email protected]> 于2026年5月30日周六 22:31写道： > > > > > >>>>>>>>> > > > > > >>>>>>>>>> Hello chia, > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> Thanks for the comments, I have updated the KIP! > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> Best Regards, > > > > > >>>>>>>>>> Jiunn-Yang > > > > > >>>>>>>>>> > > > > > >>>>>>>>>>> Chia-Ping Tsai <[email protected]> 於 2026年5月30日 晚上8:29 > > 寫道： > > > > > >>>>>>>>>>> > > > > > >>>>>>>>>>> Hi Jiunn-Yang, > > > > > >>>>>>>>>>> > > > > > >>>>>>>>>>> Would you mind removing the terms "hot" and "cold" when > > > > > >>> describing > > > > > >>>>>>>>>>> partitions in the KIP? I understand you are using them to > > > > > >>> describe > > > > > >>>>>> the > > > > > >>>>>>>>>>> "freshness" or the users' need for the records, but > > applying > > > > > >>> these > > > > > >>>>>>> terms > > > > > >>>>>>>>>> to > > > > > >>>>>>>>>>> the partition itself feels a bit unnatural. > > > > > >>>>>>>>>>> > > > > > >>>>>>>>>>> After all, in this scenario, users don't really care > > whether > > > > a > > > > > >>>>>>> partition > > > > > >>>>>>>>>> is > > > > > >>>>>>>>>>> newly expanded or not. Their only expectation is that > > they > > > > won't > > > > > >>>>>>> silently > > > > > >>>>>>>>>>> lose any live records produced to the topic during their > > > > active > > > > > >>>>>>>>>> consumption. > > > > > >>>>>>>>>>> > > > > > >>>>>>>>>>> Best, Chia-Ping > > > > > >>>>>>>>>>> > > > > > >>>>>>>>>>> > > > > > >>>>>>>>>>> > > > > > >>>>>>>>>>> 黃竣陽 <[email protected]> 於 2026年5月30日週六 下午12:30寫道： > > > > > >>>>>>>>>>> > > > > > >>>>>>>>>>>> Hello Jun, > > > > > >>>>>>>>>>>> > > > > > >>>>>>>>>>>> Thanks for the feedback, I have updated the KIP > > motivation > > > > > >>>> section. > > > > > >>>>>>>>>>>> > > > > > >>>>>>>>>>>> Best Regards, > > > > > >>>>>>>>>>>> Jiunn-Yang > > > > > >>>>>>>>>>>> > > > > > >>>>>>>>>>>>> Jun Rao via dev <[email protected]> 於 2026年5月30日 > > > > 凌晨1:12 寫道： > > > > > >>>>>>>>>>>>> > > > > > >>>>>>>>>>>>> Hi, Jiunn-Yang, > > > > > >>>>>>>>>>>>> > > > > > >>>>>>>>>>>>> Thanks for the reply. I think we need a stronger > > > > motivation for > > > > > >>>>>> the > > > > > >>>>>>>>>> KIP. > > > > > >>>>>>>>>>>>> > > > > > >>>>>>>>>>>>> The KIP says "The core insight is that not all > > partitions > > > > > >>> without > > > > > >>>>>> a > > > > > >>>>>>>>>>>>> committed offset are the same. A newly expanded > > partition > > > > (hot) > > > > > >>>> is > > > > > >>>>>>>>>>>>> fundamentally different from a partition the consumer > > has > > > > never > > > > > >>>>>> seen > > > > > >>>>>>>>>>>>> because it predates the group (cold)." Why is the hot > > > > partition > > > > > >>>>>>>>>>>>> fundamentally different from the cold? > > > > > >>>>>>>>>>>>> > > > > > >>>>>>>>>>>>> The KIP says "The existing by_duration policy is also > > > > > >>>> insufficient > > > > > >>>>>>>>>>>> because: > > > > > >>>>>>>>>>>>> > > > > > >>>>>>>>>>>>> - The calculated seek time (now() - duration) varies > > across > > > > > >>> nodes > > > > > >>>>>>> due > > > > > >>>>>>>>>>>> to > > > > > >>>>>>>>>>>>> clock skew. To be safe, users must set an overly large > > > > > >>> duration, > > > > > >>>>>>>>>>>> causing > > > > > >>>>>>>>>>>>> unnecessary reprocessing. > > > > > >>>>>>>>>>>>> - On network errors, the client recalculates the seek > > time > > > > on > > > > > >>>>>> retry, > > > > > >>>>>>>>>>>>> shifting the target timestamp forward and risking data > > > > loss." > > > > > >>>>>>>>>>>>> > > > > > >>>>>>>>>>>>> However, both of these situations are rare. If these > > issues > > > > > >>>>>> persist, > > > > > >>>>>>>>>> more > > > > > >>>>>>>>>>>>> severe problems likely exist elsewhere. Rare situations > > > > don't > > > > > >>>>>> need a > > > > > >>>>>>>>>>>> common > > > > > >>>>>>>>>>>>> solution. If users care about those rare situations, > > they > > > > can > > > > > >>>>>>> implement > > > > > >>>>>>>>>>>>> customized logic using > > > > > >>>>>>>>>> ConsumerRebalanceListener.onPartitionsAssigned(). > > > > > >>>>>>>>>>>>> > > > > > >>>>>>>>>>>>> Jun > > > > > >>>>>>>>>>>>> > > > > > >>>>>>>>>>>>> > > > > > >>>>>>>>>>>>> On Sun, May 17, 2026 at 6:50 AM 黃竣陽 < > > [email protected]> > > > > > >>> wrote: > > > > > >>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>> Hello chia, > > > > > >>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>> Thanks for the feedback, > > > > > >>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>> If the creation time exists, the returned value > > should > > > > always > > > > > >>>> be > > > > > >>>>>>>>>>>> greater > > > > > >>>>>>>>>>>>>> than or equal to zero, right? > > > > > >>>>>>>>>>>>>> I have explicitly mentioned this in the KIP. > > > > > >>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>> New Old (MetadataResponse v0–13) positive > > any > > > > > >>>>>>> field > > > > > >>>>>>>>>>>>>> absent UnsupportedVersionException > > > > > >>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>> The earliest point at which we can detect the version > > > > mismatch > > > > > >>>> is > > > > > >>>>>>>>>> during > > > > > >>>>>>>>>>>>>> the > > > > > >>>>>>>>>>>>>> first metadata fetch after assignment, which occurs > > inside > > > > > >>>>>> poll(). > > > > > >>>>>>>>>>>>>> Therefore, the > > > > > >>>>>>>>>>>>>> user would encounter an UnsupportedVersionException > > from > > > > > >>> poll(). > > > > > >>>>>>> I’ll > > > > > >>>>>>>>>>>>>> clarify this in the KIP. > > > > > >>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>> Best Regards, > > > > > >>>>>>>>>>>>>> Jiunn-Yang > > > > > >>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>> Chia-Ping Tsai <[email protected]> 於 2026年5月17日 > > > > 下午4:50 寫道： > > > > > >>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>> hi Jiunn > > > > > >>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>> PartitionAgeMs (int64, default -1): The age of this > > > > > >>> partition > > > > > >>>>>> in > > > > > >>>>>>>>>>>>>> milliseconds, computed server-side by the broker as > > > > > >>>>>>>>>> broker_current_time > > > > > >>>>>>>>>>>> - > > > > > >>>>>>>>>>>>>> partition_creation_time. Returns -1 if the broker > > does not > > > > > >>>>>> support > > > > > >>>>>>>>>> this > > > > > >>>>>>>>>>>>>> feature or the partition creation time is unknown. > > > > > >>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>> If the creation time exists, the returned value > > should > > > > always > > > > > >>>> be > > > > > >>>>>>>>>>>> greater > > > > > >>>>>>>>>>>>>> than or equal to zero, right? > > > > > >>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>> New Old (MetadataResponse v0–13) positive > > any > > > > > >>>>>>> field > > > > > >>>>>>>>>>>>>> absent UnsupportedVersionException > > > > > >>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>> Will user encounter UnsupportedVersionException when > > > > calling > > > > > >>>>>>>>>> `poll()`? > > > > > >>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>> Best, > > > > > >>>>>>>>>>>>>>> Chia-Ping > > > > > >>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>> On 2026/05/16 04:30:49 黃竣陽 wrote: > > > > > >>>>>>>>>>>>>>>> Hello Jun, chia, > > > > > >>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>> I've updated KIP-1327 with a design change based on > > the > > > > > >>>>>>> discussion > > > > > >>>>>>>>>>>>>>>> feedback. > > > > > >>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>> The updated design decouples the new-partition reset > > > > > >>> behavior > > > > > >>>>>>> from > > > > > >>>>>>>>>>>>>>>> the base auto.offset.reset policy: > > > > > >>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>> - auto.offset.reset.max.age.ms now applies to all > > > > > >>>>>>> auto.offset.reset > > > > > >>>>>>>>>>>>>> values > > > > > >>>>>>>>>>>>>>>> (latest, earliest, by_duration, none). > > > > > >>>>>>>>>>>>>>>> - For new ("hot") partitions, the consumer resets to > > > > > >>>>>>>>>>>>>> auto.offset.reset.new.partitions > > > > > >>>>>>>>>>>>>>>> config setting > > > > > >>>>>>>>>>>>>>>> - For existing ("cold") partitions, the base > > > > > >>> auto.offset.reset > > > > > >>>>>>>>>> policy > > > > > >>>>>>>>>>>>>> continues > > > > > >>>>>>>>>>>>>>>> to apply unchanged. > > > > > >>>>>>>>>>>>>>>> - The new-partition reset behavior is represented > > by a > > > > > >>>> separate > > > > > >>>>>>>>>>>>>> internal config > > > > > >>>>>>>>>>>>>>>> (auto.offset.reset.new.partitions, currently fixed > > to > > > > > >>>>>> earliest). > > > > > >>>>>>>>>> This > > > > > >>>>>>>>>>>>>> decoupled design makes > > > > > >>>>>>>>>>>>>>>> it straightforward to promote the behavior to a > > public > > > > > >>>>>>> user-facing > > > > > >>>>>>>>>>>>>> configuration in a future KIP. > > > > > >>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>> Best Regards, > > > > > >>>>>>>>>>>>>>>> Jiunn-Yang > > > > > >>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>> Chia-Ping Tsai <[email protected]> 於 2026年5月16日 > > > > 清晨7:46 > > > > > >>> 寫道： > > > > > >>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>> hi Jun > > > > > >>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>> I see what you mean now. The proposal from me is > > listed > > > > > >>>> below: > > > > > >>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>> 1) Add auto.offset.reset.new.partitions with a > > default > > > > > >>> value > > > > > >>>>>> of > > > > > >>>>>>>>>>>>>> earliest. It fixes the data loss from both > > by_duration and > > > > > >>>>>> latest, > > > > > >>>>>>> and > > > > > >>>>>>>>>>>> it > > > > > >>>>>>>>>>>>>> does not change the logic of > > auto.offset.reset=earliest. > > > > > >>>>>>>>>>>>>>>>> 2) Mark auto.offset.reset.new.partitions as an > > > > internal > > > > > >>>>>>>>>>>>>> configuration. auto.offset.reset.new > > .partitions=earliest > > > > > >>>> already > > > > > >>>>>>>>>>>>>> addresses the issue, and we can discuss the use cases > > of > > > > other > > > > > >>>>>>> values > > > > > >>>>>>>>>>>> in a > > > > > >>>>>>>>>>>>>> separate KIP. > > > > > >>>>>>>>>>>>>>>>> 3) Both configs, auto.offset.reset.new.partitions > > and > > > > > >>>>>>>>>>>>>> auto.offset.reset.latest.max.age.ms, will be applied > > to > > > > all > > > > > >>> for > > > > > >>>>>>>>>>>>>> consistency. > > > > > >>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>> WDYT? > > > > > >>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>> On 2026/05/15 20:53:20 Jun Rao via dev wrote: > > > > > >>>>>>>>>>>>>>>>>> Hi, Chia-Ping, > > > > > >>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>> Thanks for the reply. > > > > > >>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>> 1. In the motivation section, the KIP says "When a > > > > Kafka > > > > > >>>>>> topic > > > > > >>>>>>> is > > > > > >>>>>>>>>>>>>> expanded > > > > > >>>>>>>>>>>>>>>>>> with new partitions, consumers using the latest > > auto > > > > > >>> offset > > > > > >>>>>>> reset > > > > > >>>>>>>>>>>>>> policy > > > > > >>>>>>>>>>>>>>>>>> will silently miss all records produced to those > > > > > >>> partitions > > > > > >>>>>>> before > > > > > >>>>>>>>>>>> the > > > > > >>>>>>>>>>>>>>>>>> consumer discovers them.". If a user sets > > > > > >>>>>>>>>>>>>>>>>> auto.offset.reset=by_duration=1sec, the same > > record > > > > loss > > > > > >>>>>> issue > > > > > >>>>>>>>>> could > > > > > >>>>>>>>>>>>>> also > > > > > >>>>>>>>>>>>>>>>>> happen, right? > > > > > >>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>> 2. I was thinking auto.offset.reset.new > > .partitions > > > > will > > > > > >>>> take > > > > > >>>>>>> the > > > > > >>>>>>>>>>>> same > > > > > >>>>>>>>>>>>>>>>>> values as auto.offset.reset. So a user could set > > it > > > > > >>>>>>> by_duration if > > > > > >>>>>>>>>>>>>> needed. > > > > > >>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>> Jun > > > > > >>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>> On Thu, May 14, 2026 at 4:06 PM Chia-Ping Tsai < > > > > > >>>>>>>>>> [email protected] > > > > > >>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>> wrote: > > > > > >>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>> hi Jun > > > > > >>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>> Thanks for the feedback. I might be missing > > something > > > > > >>>>>>> important > > > > > >>>>>>>>>>>> from > > > > > >>>>>>>>>>>>>> your > > > > > >>>>>>>>>>>>>>>>>>> suggestion, so please bear with me as I try to > > > > clarify > > > > > >>> with > > > > > >>>>>> a > > > > > >>>>>>> few > > > > > >>>>>>>>>>>>>> questions: > > > > > >>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>> 1. Is there a strong use case for extending this > > > > logic to > > > > > >>>>>>> other > > > > > >>>>>>>>>>>> reset > > > > > >>>>>>>>>>>>>>>>>>> policies? Unlike latest, policies like earliest > > or > > > > > >>>>>> by_duration > > > > > >>>>>>>>>>>> don't > > > > > >>>>>>>>>>>>>> seem > > > > > >>>>>>>>>>>>>>>>>>> to suffer from the same silent data loss issue > > when a > > > > > >>>>>>> partition > > > > > >>>>>>>>>> is > > > > > >>>>>>>>>>>>>> expanded. > > > > > >>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>> 2. What values would we expect users to > > configure for > > > > > >>>>>>>>>>>>>>>>>>> auto.offset.reset.new.partitions? If they set > > it to > > > > > >>>>>> earliest > > > > > >>>>>>> or > > > > > >>>>>>>>>>>>>> latest, > > > > > >>>>>>>>>>>>>>>>>>> we might run into the exact same edge cases. For > > > > example, > > > > > >>>>>> if a > > > > > >>>>>>>>>>>>>> consumer is > > > > > >>>>>>>>>>>>>>>>>>> offline for a while and a new partition is > > created > > > > during > > > > > >>>>>> that > > > > > >>>>>>>>>>>>>> downtime, > > > > > >>>>>>>>>>>>>>>>>>> the user might actually want to skip to latest > > when > > > > > >>>>>> resuming, > > > > > >>>>>>>>>>>> rather > > > > > >>>>>>>>>>>>>> than > > > > > >>>>>>>>>>>>>>>>>>> reading from earliest just because the partition > > is > > > > > >>>>>>> technically > > > > > >>>>>>>>>>>>>> "new" to > > > > > >>>>>>>>>>>>>>>>>>> the group. > > > > > >>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>> This is exactly why we opted for introducing a > > > > max.age > > > > > >>>>>>> threshold. > > > > > >>>>>>>>>>>> It > > > > > >>>>>>>>>>>>>> gives > > > > > >>>>>>>>>>>>>>>>>>> users a time-bound way to define what is > > genuinely > > > > > >>>> "hot/new" > > > > > >>>>>>> and > > > > > >>>>>>>>>>>>>> what is > > > > > >>>>>>>>>>>>>>>>>>> just an old partition they haven't seen yet. > > > > > >>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>> Best, > > > > > >>>>>>>>>>>>>>>>>>> Chia-Ping > > > > > >>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>> On 2026/05/14 20:48:09 Jun Rao via dev wrote: > > > > > >>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang, > > > > > >>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>> Thanks for the KIP. > > > > > >>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>> I find auto.offset.reset.latest.max.age a bit > > > > weird. It > > > > > >>>>>> only > > > > > >>>>>>>>>>>>>> applies when > > > > > >>>>>>>>>>>>>>>>>>>> auto.offset.reset is latest. However, it seems > > that > > > > the > > > > > >>>>>>>>>> motivation > > > > > >>>>>>>>>>>>>>>>>>> equally > > > > > >>>>>>>>>>>>>>>>>>>> applies when auto.offset.reset is set to other > > > > values > > > > > >>> like > > > > > >>>>>>>>>>>>>> by_duration. > > > > > >>>>>>>>>>>>>>>>>>> The > > > > > >>>>>>>>>>>>>>>>>>>> intention is that we want to have a separate > > way to > > > > > >>>> control > > > > > >>>>>>>>>> newly > > > > > >>>>>>>>>>>>>> created > > > > > >>>>>>>>>>>>>>>>>>>> partitions vs existing partitions when the group > > > > starts. > > > > > >>>>>>> Have we > > > > > >>>>>>>>>>>>>>>>>>> considered > > > > > >>>>>>>>>>>>>>>>>>>> adding a new config like auto.offset.reset.new > > > > > >>>> .partitions? > > > > > >>>>>>> If > > > > > >>>>>>>>>>>> this > > > > > >>>>>>>>>>>>>> new > > > > > >>>>>>>>>>>>>>>>>>>> config is not set, the offset reset policy > > defaults > > > > to > > > > > >>> the > > > > > >>>>>>>>>> policy > > > > > >>>>>>>>>>>>>> used > > > > > >>>>>>>>>>>>>>>>>>> for > > > > > >>>>>>>>>>>>>>>>>>>> existing partitions. The user could set it > > > > explicitly to > > > > > >>>>>>>>>> customize > > > > > >>>>>>>>>>>>>> the > > > > > >>>>>>>>>>>>>>>>>>>> behavior for new partitions. > > > > > >>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>> Jun > > > > > >>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>> On Thu, May 7, 2026 at 5:07 AM 黃竣陽 < > > > > [email protected]> > > > > > >>>>>>> wrote: > > > > > >>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>> Hi all, > > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>> I’d like to manually bump this thread. > > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>> Best Regards, > > > > > >>>>>>>>>>>>>>>>>>>>> Jiunn-Yang > > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>> 黃竣陽 <[email protected]> 於 2026年5月1日 晚上10:37 > > 寫道： > > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>> Hello all, > > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback. > > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>> DJ01/DJ02: > > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>> MetadataResponse bumps from v13 to v14. The > > > > > >>>>>>> PartitionMetadata > > > > > >>>>>>>>>>>>>> struct > > > > > >>>>>>>>>>>>>>>>>>>>> gains a new > > > > > >>>>>>>>>>>>>>>>>>>>>> field PartitionAgeMs (int64, default -1), > > computed > > > > > >>>>>>> server-side > > > > > >>>>>>>>>>>> by > > > > > >>>>>>>>>>>>>> the > > > > > >>>>>>>>>>>>>>>>>>>>> broker as > > > > > >>>>>>>>>>>>>>>>>>>>>> broker_current_time - partition_creation_time. > > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>> Also add the consumer heartbeat flow. when > > > > > >>>>>>> MembershipManager > > > > > >>>>>>>>>>>>>> detects > > > > > >>>>>>>>>>>>>>>>>>> a > > > > > >>>>>>>>>>>>>>>>>>>>> newly assigned > > > > > >>>>>>>>>>>>>>>>>>>>>> partition, it explicitly invalidates the > > metadata > > > > for > > > > > >>>> the > > > > > >>>>>>>>>>>> affected > > > > > >>>>>>>>>>>>>>>>>>> topic > > > > > >>>>>>>>>>>>>>>>>>>>> and forces a fresh MetadataRequest > > > > > >>>>>>>>>>>>>>>>>>>>>> before making the offset reset decision, even > > if > > > > the > > > > > >>>>>> topic > > > > > >>>>>>> ID > > > > > >>>>>>>>>> is > > > > > >>>>>>>>>>>>>>>>>>> already > > > > > >>>>>>>>>>>>>>>>>>>>> in the cache. > > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>> MB0: > > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>> The consumer learns the broker's maximum > > supported > > > > > >>>>>>>>>>>>>> MetadataResponse > > > > > >>>>>>>>>>>>>>>>>>>>> version via the > > > > > >>>>>>>>>>>>>>>>>>>>>> ApiVersions negotiation at connection time. > > If the > > > > > >>>>>>> negotiated > > > > > >>>>>>>>>>>>>>>>>>> version is > > > > > >>>>>>>>>>>>>>>>>>>>> unsupported, the consumer > > > > > >>>>>>>>>>>>>>>>>>>>>> knows the broker does not support > > PartitionAgeMs > > > > at > > > > > >>> all > > > > > >>>>>> and > > > > > >>>>>>>>>> can > > > > > >>>>>>>>>>>>>>>>>>> throw an > > > > > >>>>>>>>>>>>>>>>>>>>> UnsupportedVersionException > > > > > >>>>>>>>>>>>>>>>>>>>>> immediately, rather than silently falling > > back to > > > > > >>> latest > > > > > >>>>>>> and > > > > > >>>>>>>>>>>>>> risking > > > > > >>>>>>>>>>>>>>>>>>>>> data loss without any operator-visible signal. > > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>> MB1/MB2/MB3: > > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>> I have addressed these changes in the KIP. > > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>> Best Regards, > > > > > >>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang > > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <[email protected]> 於 > > 2026年4月29日 > > > > > >>>> 下午4:04 > > > > > >>>>>>> 寫道： > > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> hi David > > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> I agree with the direction of moving the > > 'age' > > > > > >>>>>> resolution > > > > > >>>>>>>>>> from > > > > > >>>>>>>>>>>>>> the > > > > > >>>>>>>>>>>>>>>>>>>>> Heartbeat API to the Metadata API to keep the > > > > control > > > > > >>>>>> plane > > > > > >>>>>>>>>>>> clean. > > > > > >>>>>>>>>>>>>> The > > > > > >>>>>>>>>>>>>>>>>>> main > > > > > >>>>>>>>>>>>>>>>>>>>> trade-off, as we noted before, is introducing > > > > > >>>> inter-broker > > > > > >>>>>>>>>> clock > > > > > >>>>>>>>>>>>>> skew. > > > > > >>>>>>>>>>>>>>>>>>> The > > > > > >>>>>>>>>>>>>>>>>>>>> Group Coordinator approach provided a single > > > > source of > > > > > >>>>>> truth > > > > > >>>>>>>>>> for > > > > > >>>>>>>>>>>>>> time. > > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> However, realistically, this time skew > > should be > > > > > >>>>>>> negligible. > > > > > >>>>>>>>>>>>>> Given > > > > > >>>>>>>>>>>>>>>>>>> that > > > > > >>>>>>>>>>>>>>>>>>>>> the max.age threshold will likely be > > configured in > > > > > >>>> minutes > > > > > >>>>>>> or > > > > > >>>>>>>>>>>>>> hours, a > > > > > >>>>>>>>>>>>>>>>>>>>> typical NTP skew (in milliseconds) between > > brokers > > > > > >>> won't > > > > > >>>>>>> impact > > > > > >>>>>>>>>>>> the > > > > > >>>>>>>>>>>>>>>>>>>>> fallback decision. > > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> Best, > > > > > >>>>>>>>>>>>>>>>>>>>>>> Chia-Ping > > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> David Jacot via dev <[email protected]> > > 於 > > > > > >>>>>> 2026年4月29日 > > > > > >>>>>>>>>>>> 下午3:29 > > > > > >>>>>>>>>>>>>> 寫道： > > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> Hi all, > > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP! > > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> Sorry, I haven't really followed the > > previous > > > > > >>>>>>> conversation > > > > > >>>>>>>>>>>> but I > > > > > >>>>>>>>>>>>>>>>>>> took a > > > > > >>>>>>>>>>>>>>>>>>>>>>>> quick look at this one. > > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> DJ01: I don't clearly understand the flow > > with > > > > the > > > > > >>>>>>>>>>>>>>>>>>>>> ConsumerGroupHeartbeat > > > > > >>>>>>>>>>>>>>>>>>>>>>>> API after reading the KIP. There is a new > > > > boolean; > > > > > >>> the > > > > > >>>>>>> KIP > > > > > >>>>>>>>>>>>>> states > > > > > >>>>>>>>>>>>>>>>>>> that > > > > > >>>>>>>>>>>>>>>>>>>>>>>> partition ages are returned only when this > > > > boolean > > > > > >>> is > > > > > >>>>>>> set. > > > > > >>>>>>>>>>>>>>>>>>> Implicitly, > > > > > >>>>>>>>>>>>>>>>>>>>> this > > > > > >>>>>>>>>>>>>>>>>>>>>>>> means that when the consumer receives a new > > > > > >>> partition, > > > > > >>>>>> it > > > > > >>>>>>>>>> will > > > > > >>>>>>>>>>>>>>>>>>> issue a > > > > > >>>>>>>>>>>>>>>>>>>>> new > > > > > >>>>>>>>>>>>>>>>>>>>>>>> HB request with the boolean set to receive > > the > > > > ages. > > > > > >>>> Is > > > > > >>>>>>> my > > > > > >>>>>>>>>>>>>>>>>>>>> understanding > > > > > >>>>>>>>>>>>>>>>>>>>>>>> correct? We should perhaps clarify the flow > > and > > > > also > > > > > >>>>>>> explain > > > > > >>>>>>>>>>>>>> how it > > > > > >>>>>>>>>>>>>>>>>>>>> fits > > > > > >>>>>>>>>>>>>>>>>>>>>>>> into the existing flow (e.g. list offsets, > > fetch > > > > > >>>>>> offsets, > > > > > >>>>>>>>>>>> etc.). > > > > > >>>>>>>>>>>>>>>>>>>>>>>> DJ02: It my understanding is correct, I > > wonder > > > > if > > > > > >>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API is the right > > > > place > > > > > >>> for > > > > > >>>>>>> this > > > > > >>>>>>>>>>>> given > > > > > >>>>>>>>>>>>>>>>>>> that > > > > > >>>>>>>>>>>>>>>>>>>>> a new > > > > > >>>>>>>>>>>>>>>>>>>>>>>> round trip is done anyway. Alternatively, it > > > > could > > > > > >>>>>> simply > > > > > >>>>>>>>>>>>>> include > > > > > >>>>>>>>>>>>>>>>>>> the > > > > > >>>>>>>>>>>>>>>>>>>>>>>> metadata. Generally, we should be rather > > > > cautious > > > > > >>>> about > > > > > >>>>>>> not > > > > > >>>>>>>>>>>>>>>>>>> overloading > > > > > >>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API with > > unrelated > > > > > >>>> concepts. > > > > > >>>>>>> The > > > > > >>>>>>>>>>>> API > > > > > >>>>>>>>>>>>>> is > > > > > >>>>>>>>>>>>>>>>>>> a > > > > > >>>>>>>>>>>>>>>>>>>>>>>> control plane API for assigning or revoking > > > > > >>>> partitions. > > > > > >>>>>>> The > > > > > >>>>>>>>>>>> fact > > > > > >>>>>>>>>>>>>>>>>>> that > > > > > >>>>>>>>>>>>>>>>>>>>> we > > > > > >>>>>>>>>>>>>>>>>>>>>>>> don't want to add it to the corresponding > > > > Streams > > > > > >>> API > > > > > >>>>>>> also > > > > > >>>>>>>>>>>>>> suggests > > > > > >>>>>>>>>>>>>>>>>>>>>>>> something is not quite right. What would we > > do > > > > if we > > > > > >>>>>>> want to > > > > > >>>>>>>>>>>>>>>>>>> support > > > > > >>>>>>>>>>>>>>>>>>>>>>>> Streams in the future? > > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> Best, > > > > > >>>>>>>>>>>>>>>>>>>>>>>> David > > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 29, 2026 at 12:28 AM Muralidhar > > > > Basani > > > > > >>>> via > > > > > >>>>>>> dev > > > > > >>>>>>>>>> < > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> [email protected]> wrote: > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Hi Jiunn, > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Thank you for this great kip. Good to know > > > > about > > > > > >>> the > > > > > >>>>>>> gap. > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> mb-0 - why a new v2 version bump for > > > > > >>>>>>> RequestPartitionAges > > > > > >>>>>>>>>>>>>> field. > > > > > >>>>>>>>>>>>>>>>>>> Can a > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> tagged field (for ex: on response, > > > > PartitionAges on > > > > > >>>>>>>>>>>>>>>>>>> TopicPartitions) > > > > > >>>>>>>>>>>>>>>>>>>>> be > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> used here and avoid version bump? > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> mb-1 - For the new config, is there a > > > > recommended > > > > > >>>>>> value > > > > > >>>>>>> or > > > > > >>>>>>>>>> a > > > > > >>>>>>>>>>>>>>>>>>> ConfigDef > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> validator? Probably it should based on the > > > > > >>>>>>>>>>>> metadata.max.age.ms > > > > > >>>>>>>>>>>>>> ? > > > > > >>>>>>>>>>>>>>>>>>>>> Sizing > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> instructions can be part of javadocs I > > guess. > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> mb-2 - (minor) As there are no changes to > > Kafka > > > > > >>>>>> Streams, > > > > > >>>>>>>>>>>> would > > > > > >>>>>>>>>>>>>> it > > > > > >>>>>>>>>>>>>>>>>>> be > > > > > >>>>>>>>>>>>>>>>>>>>> better > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> to add this new config > > > > > >>>>>> auto.offset.reset.latest.max.age > > > > > >>>>>>> to > > > > > >>>>>>>>>>>> the > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> StreamsConfig block list > > > > > >>>>>>>>>>>>>>>>>>> (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS) > > > > > >>>>>>>>>>>>>>>>>>>>> for a > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> clear warning, incase users configure it? > > This > > > > is > > > > > >>> the > > > > > >>>>>>> most > > > > > >>>>>>>>>>>>>>>>>>> familiar > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> consumer config and users might easily > > > > mistakenly > > > > > >>>>>>> configure > > > > > >>>>>>>>>>>>>> it. Or > > > > > >>>>>>>>>>>>>>>>>>>>> may be > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> it's not worth it to add. > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> mb-3 - (minor) The phrasing "the consumer > > falls > > > > > >>> back > > > > > >>>>>> to > > > > > >>>>>>>>>>>>>> earliest" > > > > > >>>>>>>>>>>>>>>>>>>>> reads as > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> if the config were being changed > > per-partition > > > > > >>> which > > > > > >>>>>>> isn't > > > > > >>>>>>>>>>>>>>>>>>> supported. > > > > > >>>>>>>>>>>>>>>>>>>>> May > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> be rephrasing to something like "consumer > > > > resolves > > > > > >>>> the > > > > > >>>>>>>>>>>> initial > > > > > >>>>>>>>>>>>>>>>>>>>> position to > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> start offset for that partition" as if > > > > earliest was > > > > > >>>>>>> applied > > > > > >>>>>>>>>>>> to > > > > > >>>>>>>>>>>>>>>>>>> that > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> partition only and auto.offset.reset > > config is > > > > > >>>>>>> unchanged. > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Thanks, > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Murali > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 28, 2026 at 2:48 PM 黃竣陽 < > > > > > >>>>>>> [email protected]> > > > > > >>>>>>>>>>>>>> wrote: > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Hi chia, > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> I have updated the KIP to include this > > change. > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards, > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <[email protected]> 於 > > > > > >>> 2026年4月28日 > > > > > >>>>>>> 晚上8:03 > > > > > >>>>>>>>>>>> 寫道： > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> chia_0: Should we expose the partition > > > > creation > > > > > >>>> time > > > > > >>>>>>> via > > > > > >>>>>>>>>>>> the > > > > > >>>>>>>>>>>>>>>>>>> Admin > > > > > >>>>>>>>>>>>>>>>>>>>> API? > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> I assume it would be valuable for users to > > > > > >>> diagnose > > > > > >>>>>> and > > > > > >>>>>>>>>>>>>>>>>>> troubleshoot > > > > > >>>>>>>>>>>>>>>>>>>>> the > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> behavior of > > auto.offset.reset.latest.max.age > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Best, > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/04/28 10:47:58 黃竣陽 wrote: > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone, > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a discussion on > > > > KIP-1327 > > > > > >>>>>>> Prevent > > > > > >>>>>>>>>> Hot > > > > > >>>>>>>>>>>>>> Data > > > > > >>>>>>>>>>>>>>>>>>>>> Loss > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> on > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Partition Expansion for Latest Policy > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> < > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>> > > > > > >>>>>>>>>>>> > > > > > >>>>>>>>>> > > > > > >>>>>>> > > > > > >>>>>> > > > > > >>>> > > > > > >>> > > > > > > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$ > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to introduces > > > > > >>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age, > > > > > >>>>>>>>>>>>>>>>>>>>> a > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> consumer config that lets the > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> latest reset policy distinguish newly > > > > expanded > > > > > >>>>>> (hot) > > > > > >>>>>>>>>>>>>> partitions > > > > > >>>>>>>>>>>>>>>>>>>>> from > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> long-existing (cold) ones. Partitions > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> younger than the configured threshold > > > > > >>>> automatically > > > > > >>>>>>> fall > > > > > >>>>>>>>>>>>>> back > > > > > >>>>>>>>>>>>>>>>>>> to > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> earliest, preventing silent data loss > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> during topic expansion without forcing a > > > > full > > > > > >>>>>>> historical > > > > > >>>>>>>>>>>>>>>>>>> reprocess. > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards, > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>> > > > > > >>>>>>>>>>>> > > > > > >>>>>>>>>>>> > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> > > > > > >>>>>>>> > > > > > >>>>>>> > > > > > >>>>>>> > > > > > >>>>>> > > > > > >>>> > > > > > >>>> > > > > > >>> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > >
