hi all, While I agree we shouldn't surprise the majority of users, relying solely on documentation means some users will still fall into this trap. Realizing this limitation only when data is silently lost during a partition expansion is a truly frustrating experience.
Since we've reached the KIP freeze for 4.4.0, I recommend we focus on updating the docs first. We can put this KIP on hold for now and revisit it after Community Over Code. IIRC, Jiunn-Yang is going to share this specific issue at the event with other Kafka users and providers. Hopefully, we can gather some fresh perspectives and inspiration from the community there. Best, Chia-Ping On 2026/07/07 21:38:22 Jun Rao via dev wrote: > Hi, Jiunn-Yang, > > Thanks for the reply. > > Changing the default value only makes sense if it benefits most users. > Most users > probably never increase the number of partitions. Changing the default > config value gives them no benefit and only introduces a probably > surprising behavior > change. > > Jun > > On Tue, Jul 7, 2026 at 1:22 PM Chia-Ping Tsai <[email protected]> wrote: > > > hi Jiunn > > > > +1 to Approach A :) > > > > Best, > > Chia-Ping > > > > On 2026/07/07 11:55:14 黃竣陽 wrote: > > > Hi Jun, Chia-Ping, > > > > > > I'd like to first explain why I believe changing the default is > > worthwhile. > > > > > > Documentation can only help users who already know about the problem. > > > Because the failure mode is silent, most users discover it only after > > data has > > > already been missed. A safer default such as by_duration:PT5S protects > > users > > > automatically, without requiring prior awareness of the issue. > > > > > > Given that, the question is not whether the issue exists, but how best > > to introduce > > > a safer default. > > > > > > Approach A — Follow a deprecation cycle and land the change in 5.0 > > > - Keep the current default (`latest`) in upcoming 4.x releases, but emit > > a warning that > > > the default will change to `by_duration:PT5S` in 5.0 and explain how > > users can retain > > > the current behavior explicitly. > > > - Change the default to `by_duration:PT5S` in 5.0. > > > - Update shell tools (for example, the console consumer) to explicitly > > use `latest`, preserving > > > the expected "show only new messages" behavior. > > > > > > Approach B — Change the default in a 4.4 minor release > > > - Change the default to `by_duration:PT5S` in a 4.x minor release. > > > - Leave shell tools unchanged. > > > - Emit a warning explaining the change and how users can explicitly > > restore `latest`. > > > > > > I favor Approach B: > > > > > > Silent data loss during partition expansion is a correctness issue, not > > a usability issue. > > > Delaying the safer default until 5.0 leaves users relying on the > > implicit default exposed unnecessarily. > > > Users who need latest can simply pin it explicitly. > > > I don't think shell tools need special handling. by_duration:PT5S only > > replays a few seconds of data, > > > which is negligible in practice and preferable to silently missing > > records. Keeping the behavior consistent > > > is simpler. > > > > > > Best Regards, > > > Jiunn-Yang > > > > > > > Jun Rao via dev <[email protected]> 於 2026年7月7日 清晨5:49 寫道： > > > > > > > > Hi, Chia-Ping, > > > > > > > > I am not sure that we need an out-of-the-box solution. The proposal in > > the > > > > KIP is also not out-of-the-box and requires users to opt-in. > > > > > > > > Thanks, > > > > > > > > Jun > > > > > > > > On Mon, Jul 6, 2026 at 11:34 AM Chia-Ping Tsai <[email protected]> > > wrote: > > > > > > > >> hi Jun > > > >> > > > >> Yes, updating the docs requires minimal changes to the codebase, but > > it is > > > >> not an out-of-the-box solution. From my perspective, this issue can be > > > >> addressed by setting the console consumer's policy back to latest. > > > >> > > > >> Let's see what Jiunn thinks > > > >> > > > >> Best, > > > >> Chia-Ping > > > >> > > > >> On 2026/07/06 15:59:37 Jun Rao via dev wrote: > > > >>> Hi, Chia-Ping, > > > >>> > > > >>> One potential downside of changing the default is that some existing > > > >> tools > > > >>> (e.g. console consumer) may depend on this behavior. For example, a > > user > > > >>> testing the system may expect to see only new messages and could be > > > >>> surprised by the change. Another option is to leave the default > > > >> unchanged, > > > >>> but document clearly how to prevent losing messages on new partitions > > > >>> through by_duration. The users can then make the corresponding config > > > >>> changes if needed. > > > >>> > > > >>> Thanks, > > > >>> > > > >>> Jun > > > >>> > > > >>> On Wed, Jul 1, 2026 at 9:39 AM Chia-Ping Tsai <[email protected]> > > > >> wrote: > > > >>> > > > >>>> Hi Jun, > > > >>>> > > > >>>> Thanks for the feedback. I agree with your conclusion. It would be > > > >> better > > > >>>> to "make haste" with a new KIP to address this. We can focus the > > > >> discussion > > > >>>> on the following points: > > > >>>> > > > >>>> 1) Should we change the default reset policy to by_duration for all > > > >>>> consumers? > > > >>>> 2) What is the optimal duration to prevent data loss while avoiding > > > >> large > > > >>>> backlogs? Note that the classic consumer has a 3-second heartbeat > > > >> interval, > > > >>>> whereas the async consumer uses 5 seconds. > > > >>>> 3) What are the potential side effects of replacing latest with > > > >>>> by_duration as the default behavior? > > > >>>> > > > >>>> Best, > > > >>>> Chia-Ping > > > >>>> > > > >>>> On 2026/06/30 20:39:24 Jun Rao via dev wrote: > > > >>>>> Hi, Chia-Ping, > > > >>>>> > > > >>>>> I agree that using by_duration with a value of 5s to 10s addresses > > > >> most > > > >>>>> common issues with expanded partitions, given the default > > > >>>>> heartbeat.interval.ms is 3 seconds. We can consider setting that > > as > > > >> the > > > >>>> new > > > >>>>> default. The current proposal may address some other rarer cases, > > > >> but it > > > >>>>> feels a bit like over engineering. > > > >>>>> > > > >>>>> Thanks, > > > >>>>> > > > >>>>> Jun > > > >>>>> > > > >>>>> On Tue, Jun 30, 2026 at 5:58 AM Chia-Ping Tsai < > > [email protected]> > > > >>>> wrote: > > > >>>>> > > > >>>>>> - Short durations fail during extended downtimes, leading to the > > > >> exact > > > >>>>>> silent data > > > >>>>>> loss this KIP aims to prevent. > > > >>>>>> > > > >>>>>> Losing the records produced to expanded partitions during a > > > >> downtime is > > > >>>>>> actually the expected behavior for the latest policy. This is > > > >> exactly > > > >>>> why, > > > >>>>>> in the current KIP proposal, we prefer by_duration over earliest > > > >> for > > > >>>> the > > > >>>>>> expanded partitions—we want to avoid forcing users to consume a > > > >> massive > > > >>>>>> accumulated backlog just because a partition is technically "new". > > > >>>>>> > > > >>>>>> To quote my previous comment from May 15: > > > >>>>>> "...if a consumer is offline for a while and a new partition is > > > >> created > > > >>>>>> during that downtime, the user might actually want to skip to > > > >> latest > > > >>>> when > > > >>>>>> resuming, rather than reading from earliest just because the > > > >> partition > > > >>>> is > > > >>>>>> technically 'new' to the group." > > > >>>>>> > > > >>>>>> On 2026/06/30 12:12:47 黃竣陽 wrote: > > > >>>>>>> Hello all, > > > >>>>>>> > > > >>>>>>> The metadata blindness and consumer downtime windows are > > > >> inherently > > > >>>>>>> unpredictable, making a fixed by_duration approach unviable. No > > > >>>> static > > > >>>>>>> duration can accommodate both extremes: > > > >>>>>>> > > > >>>>>>> - Short durations fail during extended downtimes, leading to the > > > >>>> exact > > > >>>>>> silent data > > > >>>>>>> loss this KIP aims to prevent. > > > >>>>>>> - Long durations (calibrated for worst-case scenarios) cause > > > >> routine > > > >>>>>> restarts to > > > >>>>>>> trigger unnecessary mass replays, undermining the latest offset > > > >> reset > > > >>>>>> policy. > > > >>>>>>> > > > >>>>>>> This is why the proposed solution uses creation time. It > > > >> provides a > > > >>>>>> discrete, > > > >>>>>>> deterministic signal—verifying whether a partition is actually > > > >> newer > > > >>>>>> than the group, > > > >>>>>>> rather than relying on an arbitrary time window that has no > > > >>>> correlation > > > >>>>>> with the consumer’s > > > >>>>>>> actual downtime. > > > >>>>>>> > > > >>>>>>> Best Regards, > > > >>>>>>> Jiunn-Yang > > > >>>>>>> > > > >>>>>>>> Chia-Ping Tsai <[email protected]> 於 2026年6月30日 下午3:05 寫道： > > > >>>>>>>> > > > >>>>>>>> Hi all, > > > >>>>>>>> > > > >>>>>>>> I'd like to revisit Jun previous comment about using > > > >>>> by_duration=5s. I > > > >>>>>> realize now that having a small backlog is actually an acceptable > > > >>>>>> trade-off, especially since the latest policy inherently brings a > > > >> tiny > > > >>>>>> backlog to starting consumers anyway due to the time gap between > > > >>>> fetching > > > >>>>>> the offset and actually consuming. > > > >>>>>>>> > > > >>>>>>>> If we bring the simple by_duration=5s solution back to the > > > >> KIP, I'd > > > >>>>>> like to discuss a further step: Should we just change the default > > > >>>> policy > > > >>>>>> from latest to by_duration=5s specifically for the new consumer > > and > > > >>>> share > > > >>>>>> consumer? > > > >>>>>>>> > > > >>>>>>>> Best, > > > >>>>>>>> Chia-Ping > > > >>>>>>>> > > > >>>>>>>> On 2026/06/15 17:39:19 Chia-Ping Tsai wrote: > > > >>>>>>>>> hi Jun > > > >>>>>>>>> > > > >>>>>>>>> The most important part of this story is how users should > > > >> expect > > > >>>> the > > > >>>>>> data > > > >>>>>>>>> they can see when using the latest or by_duration policy with > > > >>>> expanded > > > >>>>>>>>> partitions. > > > >>>>>>>>> > > > >>>>>>>>> Yes, the by_duration policy can minimize data loss, but it is > > > >>>>>>>>> non-deterministic, which means users will either read too many > > > >>>>>> historical > > > >>>>>>>>> records from existing partitions or lose some records from > > > >>>> expanded > > > >>>>>>>>> partitions. > > > >>>>>>>>> > > > >>>>>>>>> Also, I agree that auto.offset.reset.max.age.ms is a bit > > > >> hard to > > > >>>>>>>>> understand, and that is why I preferred having a whole new > > > >> policy > > > >>>>>> based > > > >>>>>>>>> entirely on group creation time (KIP-1282) > > > >>>>>>>>> > > > >>>>>>>>> Best, > > > >>>>>>>>> Chia-Ping > > > >>>>>>>>> > > > >>>>>>>>> Jun Rao via dev <[email protected]> 於 2026年6月16日週二 > > > >> 上午1:08寫道： > > > >>>>>>>>> > > > >>>>>>>>>> Hi, Chia-Ping and Jiunn-Yang, > > > >>>>>>>>>> > > > >>>>>>>>>> Thanks for the reply. I am still trying to understand the > > > >> value > > > >>>> of > > > >>>>>> the new > > > >>>>>>>>>> configs with the KIP. > > > >>>>>>>>>> > > > >>>>>>>>>> The motivation of the KIP is that a user doesn't want to > > > >> miss the > > > >>>>>> data if > > > >>>>>>>>>> the backlog is small. The backlog of the existing partition > > > >> is > > > >>>> easy > > > >>>>>> to > > > >>>>>>>>>> understand because it relates to retention time. The backlog > > > >> for > > > >>>> the > > > >>>>>> new > > > >>>>>>>>>> partition is a bit subtle to understand since it depends on > > > >> the > > > >>>>>> metadata > > > >>>>>>>>>> refresh delay. To set auto.offset.reset.max.age.ms, the user > > > >>>> needs > > > >>>>>> to > > > >>>>>>>>>> understand the metadata refresh delay on the consumer side > > > >> and > > > >>>> use > > > >>>>>> it to > > > >>>>>>>>>> set the config. > > > >>>>>>>>>> > > > >>>>>>>>>> Now, let's consider the alternative: setting the same value > > > >> for > > > >>>> the > > > >>>>>>>>>> existing by_duration policy. The KIP lists three issues with > > > >> this > > > >>>>>> approach. > > > >>>>>>>>>> 1. It computes the seek target client-side as now() - > > > >> duration, > > > >>>> which > > > >>>>>>>>>> introduces clock skew across consumers and forces operators > > > >> to > > > >>>> choose > > > >>>>>>>>>> overly large durations, causing unnecessary reprocessing. > > > >>>>>>>>>> 2. The target timestamp is recomputed on each retry, so > > > >> failed > > > >>>>>>>>>> ListOffsetsRequest retries can shift the target forward and > > > >>>>>> potentially > > > >>>>>>>>>> miss records produced between attempts. > > > >>>>>>>>>> 3. It applies uniformly to all partitions without committed > > > >>>> offsets, > > > >>>>>> and > > > >>>>>>>>>> cannot distinguish newly expanded partitions from > > > >> long-existing > > > >>>>>> partitions > > > >>>>>>>>>> newly assigned to the group, leading to unnecessary replay. > > > >>>>>>>>>> > > > >>>>>>>>>> Issues 1 and 2 are uncommon and can be mitigated by adding a > > > >> bit > > > >>>>>> buffer to > > > >>>>>>>>>> the metadata refresh delay. We could also consider improving > > > >> the > > > >>>>>>>>>> implementation. For issue 3, the metadata refresh delay is > > > >>>> typically > > > >>>>>> low > > > >>>>>>>>>> (in the order of minutes with the classic consumer and tens > > > >> of > > > >>>>>> seconds with > > > >>>>>>>>>> the new consumer). If a user is ok with reading that much > > > >> backlog > > > >>>>>> for new > > > >>>>>>>>>> partitions, it seems they will be ok doing the same for > > > >> existing > > > >>>>>>>>>> partitions. > > > >>>>>>>>>> > > > >>>>>>>>>> So, instead of introducing a new config, could we just reuse > > > >> the > > > >>>>>> existing > > > >>>>>>>>>> config with better documentation and/or implementation? > > > >>>>>>>>>> > > > >>>>>>>>>> Jun > > > >>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>>>>>> On Sat, Jun 13, 2026 at 12:19 AM 黃竣陽 <[email protected]> > > > >> wrote: > > > >>>>>>>>>> > > > >>>>>>>>>>> Hello Jun, > > > >>>>>>>>>>> > > > >>>>>>>>>>> You're right that group creation time is the more intuitive > > > >>>> answer > > > >>>>>> at > > > >>>>>>>>>>> first glance, > > > >>>>>>>>>>> the KIP's own motivation talks about partitions that > > > >> "predate > > > >>>> the > > > >>>>>> group" > > > >>>>>>>>>>> vs partitions > > > >>>>>>>>>>> "created during group runtime," which directly points to a > > > >>>>>>>>>> group-lifecycle > > > >>>>>>>>>>> classifier. > > > >>>>>>>>>>> I'd like to walk through why we landed on partition age, > > > >> and the > > > >>>>>>>>>>> trade-offs we considered. > > > >>>>>>>>>>> > > > >>>>>>>>>>> We evaluated three candidate signals: > > > >>>>>>>>>>> > > > >>>>>>>>>>> 1. `by_duration:5secs` > > > >>>>>>>>>>> > > > >>>>>>>>>>> This covers the metadata blindness window, but has issues > > > >> the > > > >>>> KIP > > > >>>>>>>>>>> currently documents > > > >>>>>>>>>>> under "Why not use `by_duration`?": > > > >>>>>>>>>>> > > > >>>>>>>>>>> - Client-side `now() - duration` introduces clock skew > > > >> across > > > >>>>>> consumers. > > > >>>>>>>>>>> - `ListOffsets` retries shift the target forward, > > > >> potentially > > > >>>>>> missing > > > >>>>>>>>>>> records produced between > > > >>>>>>>>>>> attempts. > > > >>>>>>>>>>> - It applies uniformly to all partitions without committed > > > >>>> offsets, > > > >>>>>>>>>>> including pre-existing partitions > > > >>>>>>>>>>> newly assigned to the group, causing unnecessary replay. > > > >>>>>>>>>>> > > > >>>>>>>>>>> 2. Group creation time as classifier > > > >>>>>>>>>>> > > > >>>>>>>>>>> This works cleanly when the consumer is actively running. > > > >> Our > > > >>>>>> concern > > > >>>>>>>>>>> is the idle / late-rejoin case: > > > >>>>>>>>>>> > > > >>>>>>>>>>> T=0: Group created. > > > >>>>>>>>>>> T=1..T=100: Consumer idle (down, disconnected, etc.). > > > >>>>>>>>>>> T=50: Partition added during the idle window. > > > >>>>>>>>>>> T=100: Consumer resumes. > > > >>>>>>>>>>> > > > >>>>>>>>>>> Under group creation time, the new partition is classified > > > >> as > > > >>>> new > > > >>>>>>>>>>> (`50 > 0`) and reset to `earliest`, replaying everything > > > >> from > > > >>>> T=50. > > > >>>>>>>>>>> But during `[T=1, T=100]`, base partitions also accumulated > > > >> data > > > >>>>>> that > > > >>>>>>>>>>> the consumer accepts as lost — that is precisely the > > > >> contract of > > > >>>>>>>>>>> `auto.offset.reset=latest`. There is no principled reason to > > > >>>> treat > > > >>>>>>>>>>> the new partition differently; both contain backlog > > > >> accumulated > > > >>>>>> during > > > >>>>>>>>>>> the same idle window. > > > >>>>>>>>>>> > > > >>>>>>>>>>> This aligns with the "backlog is backlog” principle you > > > >> raised > > > >>>> in > > > >>>>>>>>>>> the KIP-1282 thread: a `latest` user has tolerated some > > > >> backlog > > > >>>> on > > > >>>>>>>>>>> every other partition during the same idle period; forcing > > > >>>> 0-backlog > > > >>>>>>>>>>> tolerance only on new partitions would be inconsistent with > > > >> that > > > >>>>>>>>>>> tolerance. > > > >>>>>>>>>>> > > > >>>>>>>>>>> 3. Partition age vs threshold > > > >>>>>>>>>>> > > > >>>>>>>>>>> Partition age corresponds to the actual silent data loss > > > >> window, > > > >>>>>>>>>>> the gap between partition creation and the consumer’s > > > >> metadata > > > >>>>>>>>>>> refresh. Within this window, data loss is genuinely silent: > > > >> the > > > >>>>>>>>>>> consumer had no opportunity to know about the partition. > > > >> Outside > > > >>>>>> this > > > >>>>>>>>>>> window, missing data reflects either: > > > >>>>>>>>>>> > > > >>>>>>>>>>> - (a) the user’s tolerated cost of running with idle > > > >> consumers, > > > >>>> or > > > >>>>>>>>>>> - (b) an operational issue to surface via monitoring, not > > > >> via > > > >>>> reset > > > >>>>>>>>>> policy. > > > >>>>>>>>>>> > > > >>>>>>>>>>> We did not choose partition age because it is more elegant > > > >> than > > > >>>>>> group > > > >>>>>>>>>>> creation time — we chose it because its failure mode > > > >> (requires a > > > >>>>>>>>>>> threshold) is > > > >>>>>>>>>>> less invasive than the failure mode of group creation time > > > >>>>>> (overrides > > > >>>>>>>>>>> user-stated > > > >>>>>>>>>>> `latest` intent during idle periods). > > > >>>>>>>>>>> > > > >>>>>>>>>>> Best Regards, > > > >>>>>>>>>>> Jiunn-Yang > > > >>>>>>>>>>> > > > >>>>>>>>>>>> Chia-Ping Tsai <[email protected]> 於 2026年6月13日 上午11:52 > > > >> 寫道： > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> Hi Jun, > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> Relying on both creation times will create an inconsistent > > > >>>>>> scenario. A > > > >>>>>>>>>>>> consumer that lost all offsets due to a long sleep will > > > >> seek > > > >>>> to the > > > >>>>>>>>>>>> beginning for the partitions created later than the group. > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> That is why we initially proposed KIP-1282 to fix the > > > >>>> inconsistency > > > >>>>>>>>>>> using a > > > >>>>>>>>>>>> whole new policy. Since KIP-1282 couldn't reach a > > > >> consensus, > > > >>>>>> KIP-1327 > > > >>>>>>>>>>> goes > > > >>>>>>>>>>>> back to using flexible configurations to prevent users from > > > >>>> falling > > > >>>>>>>>>> into > > > >>>>>>>>>>>> that pitfall. > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> Best, Chia-Ping > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> Jun Rao via dev <[email protected]> 於 2026年6月13日週六 > > > >>>> 上午6:49寫道： > > > >>>>>>>>>>>> > > > >>>>>>>>>>>>> Hi, Jiunn-Yang, > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> Thanks for the reply and sorry for the late reply. > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> JR1. The design of auto.offset.reset.max.age.ms still > > > >> feels > > > >>>>>> weird to > > > >>>>>>>>>>> me. > > > >>>>>>>>>>>>> It > > > >>>>>>>>>>>>> categorizes partitions as new or existing based on the > > > >>>> partition > > > >>>>>>>>>>> creation > > > >>>>>>>>>>>>> time. Intuitively, the categorization should be based on > > > >> the > > > >>>> group > > > >>>>>>>>>>> creation > > > >>>>>>>>>>>>> time: all partitions existing when the group is created > > > >> are > > > >>>>>> existing > > > >>>>>>>>>> and > > > >>>>>>>>>>>>> all partitions created after the group creation are new > > > >>>>>> partitions. > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> Jun > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> On Tue, Jun 9, 2026 at 8:51 AM 黃竣陽 <[email protected]> > > > >>>> wrote: > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>>> Hi all, > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> Manually bumping this thread. If there is no further > > > >>>>>>>>>>>>>> discussion, I will close the vote. > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> Best Regards, > > > >>>>>>>>>>>>>> Jiunn-Yang > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> 黃竣陽 <[email protected]> 於 2026年6月1日 晚上7:16 寫道： > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> Hello Jian, > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> Thanks for your feedback, > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> Agreed, partition expansion is a common operational > > > >> task, > > > >>>> not an > > > >>>>>>>>>> edge > > > >>>>>>>>>>>>>>> case. I've updated the Motivation section accordingly. > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> Best Regards, > > > >>>>>>>>>>>>>>> Jiunn-Yang > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> jian fu <[email protected]> 於 2026年6月1日 下午5:49 寫道： > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> Hi Jiunn-Yang: > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> Thanks for the KIP. I think it would be useful to > > > >> clarify > > > >>>> that > > > >>>>>> this > > > >>>>>>>>>>>>> is a > > > >>>>>>>>>>>>>>>> common scenario rather than an edge case, which further > > > >>>>>>>>>> demonstrates > > > >>>>>>>>>>>>> the > > > >>>>>>>>>>>>>>>> need for this optimization. For example: > > > >>>>>>>>>>>>>>>> A partition expansion is a common operational task in > > > >>>> Kafka: To > > > >>>>>>>>>>>>> balance > > > >>>>>>>>>>>>>>>> resource utilization and cost, topics are typically > > > >> created > > > >>>>>> with a > > > >>>>>>>>>>>>>> moderate > > > >>>>>>>>>>>>>>>> default partition count. However, as traffic grows over > > > >>>> time, > > > >>>>>> it is > > > >>>>>>>>>>>>>> often > > > >>>>>>>>>>>>>>>> necessary to increase the number of partitions to > > > >>>> accommodate > > > >>>>>> the > > > >>>>>>>>>>>>> higher > > > >>>>>>>>>>>>>>>> workload. > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> Regards > > > >>>>>>>>>>>>>>>> Jian > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> 黃竣陽 <[email protected]> 于2026年5月30日周六 22:31写道： > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>> Hello chia, > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>> Thanks for the comments, I have updated the KIP! > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>> Best Regards, > > > >>>>>>>>>>>>>>>>> Jiunn-Yang > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>> Chia-Ping Tsai <[email protected]> 於 2026年5月30日 > > > >> 晚上8:29 > > > >>>> 寫道： > > > >>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>> Hi Jiunn-Yang, > > > >>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>> Would you mind removing the terms "hot" and "cold" > > > >> when > > > >>>>>>>>>> describing > > > >>>>>>>>>>>>>>>>>> partitions in the KIP? I understand you are using > > > >> them to > > > >>>>>>>>>> describe > > > >>>>>>>>>>>>> the > > > >>>>>>>>>>>>>>>>>> "freshness" or the users' need for the records, but > > > >>>> applying > > > >>>>>>>>>> these > > > >>>>>>>>>>>>>> terms > > > >>>>>>>>>>>>>>>>> to > > > >>>>>>>>>>>>>>>>>> the partition itself feels a bit unnatural. > > > >>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>> After all, in this scenario, users don't really care > > > >>>> whether > > > >>>>>> a > > > >>>>>>>>>>>>>> partition > > > >>>>>>>>>>>>>>>>> is > > > >>>>>>>>>>>>>>>>>> newly expanded or not. Their only expectation is that > > > >>>> they > > > >>>>>> won't > > > >>>>>>>>>>>>>> silently > > > >>>>>>>>>>>>>>>>>> lose any live records produced to the topic during > > > >> their > > > >>>>>> active > > > >>>>>>>>>>>>>>>>> consumption. > > > >>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>> Best, Chia-Ping > > > >>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>> 黃竣陽 <[email protected]> 於 2026年5月30日週六 下午12:30寫道： > > > >>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>> Hello Jun, > > > >>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>> Thanks for the feedback, I have updated the KIP > > > >>>> motivation > > > >>>>>>>>>>> section. > > > >>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>> Best Regards, > > > >>>>>>>>>>>>>>>>>>> Jiunn-Yang > > > >>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>> Jun Rao via dev <[email protected]> 於 > > > >> 2026年5月30日 > > > >>>>>> 凌晨1:12 寫道： > > > >>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang, > > > >>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>> Thanks for the reply. I think we need a stronger > > > >>>>>> motivation for > > > >>>>>>>>>>>>> the > > > >>>>>>>>>>>>>>>>> KIP. > > > >>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>> The KIP says "The core insight is that not all > > > >>>> partitions > > > >>>>>>>>>> without > > > >>>>>>>>>>>>> a > > > >>>>>>>>>>>>>>>>>>>> committed offset are the same. A newly expanded > > > >>>> partition > > > >>>>>> (hot) > > > >>>>>>>>>>> is > > > >>>>>>>>>>>>>>>>>>>> fundamentally different from a partition the > > > >> consumer > > > >>>> has > > > >>>>>> never > > > >>>>>>>>>>>>> seen > > > >>>>>>>>>>>>>>>>>>>> because it predates the group (cold)." Why is the > > > >> hot > > > >>>>>> partition > > > >>>>>>>>>>>>>>>>>>>> fundamentally different from the cold? > > > >>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>> The KIP says "The existing by_duration policy is > > > >> also > > > >>>>>>>>>>> insufficient > > > >>>>>>>>>>>>>>>>>>> because: > > > >>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>> - The calculated seek time (now() - duration) > > > >> varies > > > >>>> across > > > >>>>>>>>>> nodes > > > >>>>>>>>>>>>>> due > > > >>>>>>>>>>>>>>>>>>> to > > > >>>>>>>>>>>>>>>>>>>> clock skew. To be safe, users must set an overly > > > >> large > > > >>>>>>>>>> duration, > > > >>>>>>>>>>>>>>>>>>> causing > > > >>>>>>>>>>>>>>>>>>>> unnecessary reprocessing. > > > >>>>>>>>>>>>>>>>>>>> - On network errors, the client recalculates the > > > >> seek > > > >>>> time > > > >>>>>> on > > > >>>>>>>>>>>>> retry, > > > >>>>>>>>>>>>>>>>>>>> shifting the target timestamp forward and risking > > > >> data > > > >>>>>> loss." > > > >>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>> However, both of these situations are rare. If > > > >> these > > > >>>> issues > > > >>>>>>>>>>>>> persist, > > > >>>>>>>>>>>>>>>>> more > > > >>>>>>>>>>>>>>>>>>>> severe problems likely exist elsewhere. Rare > > > >> situations > > > >>>>>> don't > > > >>>>>>>>>>>>> need a > > > >>>>>>>>>>>>>>>>>>> common > > > >>>>>>>>>>>>>>>>>>>> solution. If users care about those rare > > > >> situations, > > > >>>> they > > > >>>>>> can > > > >>>>>>>>>>>>>> implement > > > >>>>>>>>>>>>>>>>>>>> customized logic using > > > >>>>>>>>>>>>>>>>> ConsumerRebalanceListener.onPartitionsAssigned(). > > > >>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>> Jun > > > >>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>> On Sun, May 17, 2026 at 6:50 AM 黃竣陽 < > > > >>>> [email protected]> > > > >>>>>>>>>> wrote: > > > >>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>> Hello chia, > > > >>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>> Thanks for the feedback, > > > >>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>> If the creation time exists, the returned value > > > >>>> should > > > >>>>>> always > > > >>>>>>>>>>> be > > > >>>>>>>>>>>>>>>>>>> greater > > > >>>>>>>>>>>>>>>>>>>>> than or equal to zero, right? > > > >>>>>>>>>>>>>>>>>>>>> I have explicitly mentioned this in the KIP. > > > >>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>> New Old (MetadataResponse v0–13) positive > > > >>>> any > > > >>>>>>>>>>>>>> field > > > >>>>>>>>>>>>>>>>>>>>> absent UnsupportedVersionException > > > >>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>> The earliest point at which we can detect the > > > >> version > > > >>>>>> mismatch > > > >>>>>>>>>>> is > > > >>>>>>>>>>>>>>>>> during > > > >>>>>>>>>>>>>>>>>>>>> the > > > >>>>>>>>>>>>>>>>>>>>> first metadata fetch after assignment, which > > > >> occurs > > > >>>> inside > > > >>>>>>>>>>>>> poll(). > > > >>>>>>>>>>>>>>>>>>>>> Therefore, the > > > >>>>>>>>>>>>>>>>>>>>> user would encounter an > > > >> UnsupportedVersionException > > > >>>> from > > > >>>>>>>>>> poll(). > > > >>>>>>>>>>>>>> I’ll > > > >>>>>>>>>>>>>>>>>>>>> clarify this in the KIP. > > > >>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>> Best Regards, > > > >>>>>>>>>>>>>>>>>>>>> Jiunn-Yang > > > >>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <[email protected]> 於 > > > >> 2026年5月17日 > > > >>>>>> 下午4:50 寫道： > > > >>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>> hi Jiunn > > > >>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>> PartitionAgeMs (int64, default -1): The age of > > > >> this > > > >>>>>>>>>> partition > > > >>>>>>>>>>>>> in > > > >>>>>>>>>>>>>>>>>>>>> milliseconds, computed server-side by the broker > > > >> as > > > >>>>>>>>>>>>>>>>> broker_current_time > > > >>>>>>>>>>>>>>>>>>> - > > > >>>>>>>>>>>>>>>>>>>>> partition_creation_time. Returns -1 if the broker > > > >>>> does not > > > >>>>>>>>>>>>> support > > > >>>>>>>>>>>>>>>>> this > > > >>>>>>>>>>>>>>>>>>>>> feature or the partition creation time is unknown. > > > >>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>> If the creation time exists, the returned value > > > >>>> should > > > >>>>>> always > > > >>>>>>>>>>> be > > > >>>>>>>>>>>>>>>>>>> greater > > > >>>>>>>>>>>>>>>>>>>>> than or equal to zero, right? > > > >>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>> New Old (MetadataResponse v0–13) positive > > > >>>> any > > > >>>>>>>>>>>>>> field > > > >>>>>>>>>>>>>>>>>>>>> absent UnsupportedVersionException > > > >>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>> Will user encounter UnsupportedVersionException > > > >> when > > > >>>>>> calling > > > >>>>>>>>>>>>>>>>> `poll()`? > > > >>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>> Best, > > > >>>>>>>>>>>>>>>>>>>>>> Chia-Ping > > > >>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>> On 2026/05/16 04:30:49 黃竣陽 wrote: > > > >>>>>>>>>>>>>>>>>>>>>>> Hello Jun, chia, > > > >>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>> I've updated KIP-1327 with a design change > > > >> based on > > > >>>> the > > > >>>>>>>>>>>>>> discussion > > > >>>>>>>>>>>>>>>>>>>>>>> feedback. > > > >>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>> The updated design decouples the new-partition > > > >> reset > > > >>>>>>>>>> behavior > > > >>>>>>>>>>>>>> from > > > >>>>>>>>>>>>>>>>>>>>>>> the base auto.offset.reset policy: > > > >>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>> - auto.offset.reset.max.age.ms now applies to > > > >> all > > > >>>>>>>>>>>>>> auto.offset.reset > > > >>>>>>>>>>>>>>>>>>>>> values > > > >>>>>>>>>>>>>>>>>>>>>>> (latest, earliest, by_duration, none). > > > >>>>>>>>>>>>>>>>>>>>>>> - For new ("hot") partitions, the consumer > > > >> resets to > > > >>>>>>>>>>>>>>>>>>>>> auto.offset.reset.new.partitions > > > >>>>>>>>>>>>>>>>>>>>>>> config setting > > > >>>>>>>>>>>>>>>>>>>>>>> - For existing ("cold") partitions, the base > > > >>>>>>>>>> auto.offset.reset > > > >>>>>>>>>>>>>>>>> policy > > > >>>>>>>>>>>>>>>>>>>>> continues > > > >>>>>>>>>>>>>>>>>>>>>>> to apply unchanged. > > > >>>>>>>>>>>>>>>>>>>>>>> - The new-partition reset behavior is > > > >> represented > > > >>>> by a > > > >>>>>>>>>>> separate > > > >>>>>>>>>>>>>>>>>>>>> internal config > > > >>>>>>>>>>>>>>>>>>>>>>> (auto.offset.reset.new.partitions, currently > > > >> fixed > > > >>>> to > > > >>>>>>>>>>>>> earliest). > > > >>>>>>>>>>>>>>>>> This > > > >>>>>>>>>>>>>>>>>>>>> decoupled design makes > > > >>>>>>>>>>>>>>>>>>>>>>> it straightforward to promote the behavior to a > > > >>>> public > > > >>>>>>>>>>>>>> user-facing > > > >>>>>>>>>>>>>>>>>>>>> configuration in a future KIP. > > > >>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>> Best Regards, > > > >>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang > > > >>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <[email protected]> 於 > > > >> 2026年5月16日 > > > >>>>>> 清晨7:46 > > > >>>>>>>>>> 寫道： > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>> hi Jun > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>> I see what you mean now. The proposal from me > > > >> is > > > >>>> listed > > > >>>>>>>>>>> below: > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>> 1) Add auto.offset.reset.new.partitions with a > > > >>>> default > > > >>>>>>>>>> value > > > >>>>>>>>>>>>> of > > > >>>>>>>>>>>>>>>>>>>>> earliest. It fixes the data loss from both > > > >>>> by_duration and > > > >>>>>>>>>>>>> latest, > > > >>>>>>>>>>>>>> and > > > >>>>>>>>>>>>>>>>>>> it > > > >>>>>>>>>>>>>>>>>>>>> does not change the logic of > > > >>>> auto.offset.reset=earliest. > > > >>>>>>>>>>>>>>>>>>>>>>>> 2) Mark auto.offset.reset.new.partitions as an > > > >>>>>> internal > > > >>>>>>>>>>>>>>>>>>>>> configuration. auto.offset.reset.new > > > >>>> .partitions=earliest > > > >>>>>>>>>>> already > > > >>>>>>>>>>>>>>>>>>>>> addresses the issue, and we can discuss the use > > > >> cases > > > >>>> of > > > >>>>>> other > > > >>>>>>>>>>>>>> values > > > >>>>>>>>>>>>>>>>>>> in a > > > >>>>>>>>>>>>>>>>>>>>> separate KIP. > > > >>>>>>>>>>>>>>>>>>>>>>>> 3) Both configs, auto.offset.reset.new > > > >> .partitions > > > >>>> and > > > >>>>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age.ms, will be > > > >> applied > > > >>>> to > > > >>>>>> all > > > >>>>>>>>>> for > > > >>>>>>>>>>>>>>>>>>>>> consistency. > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>> WDYT? > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>> On 2026/05/15 20:53:20 Jun Rao via dev wrote: > > > >>>>>>>>>>>>>>>>>>>>>>>>> Hi, Chia-Ping, > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply. > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>> 1. In the motivation section, the KIP says > > > >> "When a > > > >>>>>> Kafka > > > >>>>>>>>>>>>> topic > > > >>>>>>>>>>>>>> is > > > >>>>>>>>>>>>>>>>>>>>> expanded > > > >>>>>>>>>>>>>>>>>>>>>>>>> with new partitions, consumers using the > > > >> latest > > > >>>> auto > > > >>>>>>>>>> offset > > > >>>>>>>>>>>>>> reset > > > >>>>>>>>>>>>>>>>>>>>> policy > > > >>>>>>>>>>>>>>>>>>>>>>>>> will silently miss all records produced to > > > >> those > > > >>>>>>>>>> partitions > > > >>>>>>>>>>>>>> before > > > >>>>>>>>>>>>>>>>>>> the > > > >>>>>>>>>>>>>>>>>>>>>>>>> consumer discovers them.". If a user sets > > > >>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset=by_duration=1sec, the same > > > >>>> record > > > >>>>>> loss > > > >>>>>>>>>>>>> issue > > > >>>>>>>>>>>>>>>>> could > > > >>>>>>>>>>>>>>>>>>>>> also > > > >>>>>>>>>>>>>>>>>>>>>>>>> happen, right? > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>> 2. I was thinking auto.offset.reset.new > > > >>>> .partitions > > > >>>>>> will > > > >>>>>>>>>>> take > > > >>>>>>>>>>>>>> the > > > >>>>>>>>>>>>>>>>>>> same > > > >>>>>>>>>>>>>>>>>>>>>>>>> values as auto.offset.reset. So a user could > > > >> set > > > >>>> it > > > >>>>>>>>>>>>>> by_duration if > > > >>>>>>>>>>>>>>>>>>>>> needed. > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>> Jun > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 14, 2026 at 4:06 PM Chia-Ping > > > >> Tsai < > > > >>>>>>>>>>>>>>>>> [email protected] > > > >>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>> wrote: > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> hi Jun > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback. I might be missing > > > >>>> something > > > >>>>>>>>>>>>>> important > > > >>>>>>>>>>>>>>>>>>> from > > > >>>>>>>>>>>>>>>>>>>>> your > > > >>>>>>>>>>>>>>>>>>>>>>>>>> suggestion, so please bear with me as I try > > > >> to > > > >>>>>> clarify > > > >>>>>>>>>> with > > > >>>>>>>>>>>>> a > > > >>>>>>>>>>>>>> few > > > >>>>>>>>>>>>>>>>>>>>> questions: > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> 1. Is there a strong use case for extending > > > >> this > > > >>>>>> logic to > > > >>>>>>>>>>>>>> other > > > >>>>>>>>>>>>>>>>>>> reset > > > >>>>>>>>>>>>>>>>>>>>>>>>>> policies? Unlike latest, policies like > > > >> earliest > > > >>>> or > > > >>>>>>>>>>>>> by_duration > > > >>>>>>>>>>>>>>>>>>> don't > > > >>>>>>>>>>>>>>>>>>>>> seem > > > >>>>>>>>>>>>>>>>>>>>>>>>>> to suffer from the same silent data loss > > > >> issue > > > >>>> when a > > > >>>>>>>>>>>>>> partition > > > >>>>>>>>>>>>>>>>> is > > > >>>>>>>>>>>>>>>>>>>>> expanded. > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> 2. What values would we expect users to > > > >>>> configure for > > > >>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.new.partitions? If they > > > >> set > > > >>>> it to > > > >>>>>>>>>>>>> earliest > > > >>>>>>>>>>>>>> or > > > >>>>>>>>>>>>>>>>>>>>> latest, > > > >>>>>>>>>>>>>>>>>>>>>>>>>> we might run into the exact same edge cases. > > > >> For > > > >>>>>> example, > > > >>>>>>>>>>>>> if a > > > >>>>>>>>>>>>>>>>>>>>> consumer is > > > >>>>>>>>>>>>>>>>>>>>>>>>>> offline for a while and a new partition is > > > >>>> created > > > >>>>>> during > > > >>>>>>>>>>>>> that > > > >>>>>>>>>>>>>>>>>>>>> downtime, > > > >>>>>>>>>>>>>>>>>>>>>>>>>> the user might actually want to skip to > > > >> latest > > > >>>> when > > > >>>>>>>>>>>>> resuming, > > > >>>>>>>>>>>>>>>>>>> rather > > > >>>>>>>>>>>>>>>>>>>>> than > > > >>>>>>>>>>>>>>>>>>>>>>>>>> reading from earliest just because the > > > >> partition > > > >>>> is > > > >>>>>>>>>>>>>> technically > > > >>>>>>>>>>>>>>>>>>>>> "new" to > > > >>>>>>>>>>>>>>>>>>>>>>>>>> the group. > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> This is exactly why we opted for introducing > > > >> a > > > >>>>>> max.age > > > >>>>>>>>>>>>>> threshold. > > > >>>>>>>>>>>>>>>>>>> It > > > >>>>>>>>>>>>>>>>>>>>> gives > > > >>>>>>>>>>>>>>>>>>>>>>>>>> users a time-bound way to define what is > > > >>>> genuinely > > > >>>>>>>>>>> "hot/new" > > > >>>>>>>>>>>>>> and > > > >>>>>>>>>>>>>>>>>>>>> what is > > > >>>>>>>>>>>>>>>>>>>>>>>>>> just an old partition they haven't seen yet. > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Best, > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/05/14 20:48:09 Jun Rao via dev wrote: > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang, > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP. > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> I find auto.offset.reset.latest.max.age a > > > >> bit > > > >>>>>> weird. It > > > >>>>>>>>>>>>> only > > > >>>>>>>>>>>>>>>>>>>>> applies when > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset is latest. However, it > > > >> seems > > > >>>> that > > > >>>>>> the > > > >>>>>>>>>>>>>>>>> motivation > > > >>>>>>>>>>>>>>>>>>>>>>>>>> equally > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> applies when auto.offset.reset is set to > > > >> other > > > >>>>>> values > > > >>>>>>>>>> like > > > >>>>>>>>>>>>>>>>>>>>> by_duration. > > > >>>>>>>>>>>>>>>>>>>>>>>>>> The > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> intention is that we want to have a separate > > > >>>> way to > > > >>>>>>>>>>> control > > > >>>>>>>>>>>>>>>>> newly > > > >>>>>>>>>>>>>>>>>>>>> created > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> partitions vs existing partitions when the > > > >> group > > > >>>>>> starts. > > > >>>>>>>>>>>>>> Have we > > > >>>>>>>>>>>>>>>>>>>>>>>>>> considered > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> adding a new config like > > > >> auto.offset.reset.new > > > >>>>>>>>>>> .partitions? > > > >>>>>>>>>>>>>> If > > > >>>>>>>>>>>>>>>>>>> this > > > >>>>>>>>>>>>>>>>>>>>> new > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> config is not set, the offset reset policy > > > >>>> defaults > > > >>>>>> to > > > >>>>>>>>>> the > > > >>>>>>>>>>>>>>>>> policy > > > >>>>>>>>>>>>>>>>>>>>> used > > > >>>>>>>>>>>>>>>>>>>>>>>>>> for > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> existing partitions. The user could set it > > > >>>>>> explicitly to > > > >>>>>>>>>>>>>>>>> customize > > > >>>>>>>>>>>>>>>>>>>>> the > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> behavior for new partitions. > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Jun > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 7, 2026 at 5:07 AM 黃竣陽 < > > > >>>>>> [email protected]> > > > >>>>>>>>>>>>>> wrote: > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all, > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I’d like to manually bump this thread. > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards, > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 黃竣陽 <[email protected]> 於 2026年5月1日 > > > >> 晚上10:37 > > > >>>> 寫道： > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello all, > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback. > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ01/DJ02: > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> MetadataResponse bumps from v13 to v14. > > > >> The > > > >>>>>>>>>>>>>> PartitionMetadata > > > >>>>>>>>>>>>>>>>>>>>> struct > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> gains a new > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> field PartitionAgeMs (int64, default -1), > > > >>>> computed > > > >>>>>>>>>>>>>> server-side > > > >>>>>>>>>>>>>>>>>>> by > > > >>>>>>>>>>>>>>>>>>>>> the > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> broker as > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> broker_current_time - > > > >> partition_creation_time. > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Also add the consumer heartbeat flow. when > > > >>>>>>>>>>>>>> MembershipManager > > > >>>>>>>>>>>>>>>>>>>>> detects > > > >>>>>>>>>>>>>>>>>>>>>>>>>> a > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> newly assigned > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition, it explicitly invalidates the > > > >>>> metadata > > > >>>>>> for > > > >>>>>>>>>>> the > > > >>>>>>>>>>>>>>>>>>> affected > > > >>>>>>>>>>>>>>>>>>>>>>>>>> topic > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> and forces a fresh MetadataRequest > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> before making the offset reset decision, > > > >> even > > > >>>> if > > > >>>>>> the > > > >>>>>>>>>>>>> topic > > > >>>>>>>>>>>>>> ID > > > >>>>>>>>>>>>>>>>> is > > > >>>>>>>>>>>>>>>>>>>>>>>>>> already > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> in the cache. > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> MB0: > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> The consumer learns the broker's maximum > > > >>>> supported > > > >>>>>>>>>>>>>>>>>>>>> MetadataResponse > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> version via the > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> ApiVersions negotiation at connection > > > >> time. > > > >>>> If the > > > >>>>>>>>>>>>>> negotiated > > > >>>>>>>>>>>>>>>>>>>>>>>>>> version is > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> unsupported, the consumer > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> knows the broker does not support > > > >>>> PartitionAgeMs > > > >>>>>> at > > > >>>>>>>>>> all > > > >>>>>>>>>>>>> and > > > >>>>>>>>>>>>>>>>> can > > > >>>>>>>>>>>>>>>>>>>>>>>>>> throw an > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> UnsupportedVersionException > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> immediately, rather than silently falling > > > >>>> back to > > > >>>>>>>>>> latest > > > >>>>>>>>>>>>>> and > > > >>>>>>>>>>>>>>>>>>>>> risking > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> data loss without any operator-visible > > > >> signal. > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> MB1/MB2/MB3: > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have addressed these changes in the KIP. > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards, > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <[email protected]> 於 > > > >>>> 2026年4月29日 > > > >>>>>>>>>>> 下午4:04 > > > >>>>>>>>>>>>>> 寫道： > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi David > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I agree with the direction of moving the > > > >>>> 'age' > > > >>>>>>>>>>>>> resolution > > > >>>>>>>>>>>>>>>>> from > > > >>>>>>>>>>>>>>>>>>>>> the > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Heartbeat API to the Metadata API to keep > > > >> the > > > >>>>>> control > > > >>>>>>>>>>>>> plane > > > >>>>>>>>>>>>>>>>>>> clean. > > > >>>>>>>>>>>>>>>>>>>>> The > > > >>>>>>>>>>>>>>>>>>>>>>>>>> main > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> trade-off, as we noted before, is > > > >> introducing > > > >>>>>>>>>>> inter-broker > > > >>>>>>>>>>>>>>>>> clock > > > >>>>>>>>>>>>>>>>>>>>> skew. > > > >>>>>>>>>>>>>>>>>>>>>>>>>> The > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Group Coordinator approach provided a > > > >> single > > > >>>>>> source of > > > >>>>>>>>>>>>> truth > > > >>>>>>>>>>>>>>>>> for > > > >>>>>>>>>>>>>>>>>>>>> time. > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> However, realistically, this time skew > > > >>>> should be > > > >>>>>>>>>>>>>> negligible. > > > >>>>>>>>>>>>>>>>>>>>> Given > > > >>>>>>>>>>>>>>>>>>>>>>>>>> that > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> the max.age threshold will likely be > > > >>>> configured in > > > >>>>>>>>>>> minutes > > > >>>>>>>>>>>>>> or > > > >>>>>>>>>>>>>>>>>>>>> hours, a > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> typical NTP skew (in milliseconds) between > > > >>>> brokers > > > >>>>>>>>>> won't > > > >>>>>>>>>>>>>> impact > > > >>>>>>>>>>>>>>>>>>> the > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> fallback decision. > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> David Jacot via dev < > > > >> [email protected]> > > > >>>> 於 > > > >>>>>>>>>>>>> 2026年4月29日 > > > >>>>>>>>>>>>>>>>>>> 下午3:29 > > > >>>>>>>>>>>>>>>>>>>>> 寫道： > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all, > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP! > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry, I haven't really followed the > > > >>>> previous > > > >>>>>>>>>>>>>> conversation > > > >>>>>>>>>>>>>>>>>>> but I > > > >>>>>>>>>>>>>>>>>>>>>>>>>> took a > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> quick look at this one. > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ01: I don't clearly understand the > > > >> flow > > > >>>> with > > > >>>>>> the > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> ConsumerGroupHeartbeat > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> API after reading the KIP. There is a > > > >> new > > > >>>>>> boolean; > > > >>>>>>>>>> the > > > >>>>>>>>>>>>>> KIP > > > >>>>>>>>>>>>>>>>>>>>> states > > > >>>>>>>>>>>>>>>>>>>>>>>>>> that > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition ages are returned only when > > > >> this > > > >>>>>> boolean > > > >>>>>>>>>> is > > > >>>>>>>>>>>>>> set. > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Implicitly, > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> this > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> means that when the consumer receives a > > > >> new > > > >>>>>>>>>> partition, > > > >>>>>>>>>>>>> it > > > >>>>>>>>>>>>>>>>> will > > > >>>>>>>>>>>>>>>>>>>>>>>>>> issue a > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> new > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> HB request with the boolean set to > > > >> receive > > > >>>> the > > > >>>>>> ages. > > > >>>>>>>>>>> Is > > > >>>>>>>>>>>>>> my > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> understanding > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> correct? We should perhaps clarify the > > > >> flow > > > >>>> and > > > >>>>>> also > > > >>>>>>>>>>>>>> explain > > > >>>>>>>>>>>>>>>>>>>>> how it > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> fits > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> into the existing flow (e.g. list > > > >> offsets, > > > >>>> fetch > > > >>>>>>>>>>>>> offsets, > > > >>>>>>>>>>>>>>>>>>> etc.). > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ02: It my understanding is correct, I > > > >>>> wonder > > > >>>>>> if > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API is the > > > >> right > > > >>>>>> place > > > >>>>>>>>>> for > > > >>>>>>>>>>>>>> this > > > >>>>>>>>>>>>>>>>>>> given > > > >>>>>>>>>>>>>>>>>>>>>>>>>> that > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> a new > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> round trip is done anyway. > > > >> Alternatively, it > > > >>>>>> could > > > >>>>>>>>>>>>> simply > > > >>>>>>>>>>>>>>>>>>>>> include > > > >>>>>>>>>>>>>>>>>>>>>>>>>> the > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> metadata. Generally, we should be rather > > > >>>>>> cautious > > > >>>>>>>>>>> about > > > >>>>>>>>>>>>>> not > > > >>>>>>>>>>>>>>>>>>>>>>>>>> overloading > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API with > > > >>>> unrelated > > > >>>>>>>>>>> concepts. > > > >>>>>>>>>>>>>> The > > > >>>>>>>>>>>>>>>>>>> API > > > >>>>>>>>>>>>>>>>>>>>> is > > > >>>>>>>>>>>>>>>>>>>>>>>>>> a > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> control plane API for assigning or > > > >> revoking > > > >>>>>>>>>>> partitions. > > > >>>>>>>>>>>>>> The > > > >>>>>>>>>>>>>>>>>>> fact > > > >>>>>>>>>>>>>>>>>>>>>>>>>> that > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> we > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> don't want to add it to the > > > >> corresponding > > > >>>>>> Streams > > > >>>>>>>>>> API > > > >>>>>>>>>>>>>> also > > > >>>>>>>>>>>>>>>>>>>>> suggests > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> something is not quite right. What > > > >> would we > > > >>>> do > > > >>>>>> if we > > > >>>>>>>>>>>>>> want to > > > >>>>>>>>>>>>>>>>>>>>>>>>>> support > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Streams in the future? > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> David > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 29, 2026 at 12:28 AM > > > >> Muralidhar > > > >>>>>> Basani > > > >>>>>>>>>>> via > > > >>>>>>>>>>>>>> dev > > > >>>>>>>>>>>>>>>>> < > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [email protected]> wrote: > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jiunn, > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for this great kip. Good to > > > >> know > > > >>>>>> about > > > >>>>>>>>>> the > > > >>>>>>>>>>>>>> gap. > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-0 - why a new v2 version bump for > > > >>>>>>>>>>>>>> RequestPartitionAges > > > >>>>>>>>>>>>>>>>>>>>> field. > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Can a > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tagged field (for ex: on response, > > > >>>>>> PartitionAges on > > > >>>>>>>>>>>>>>>>>>>>>>>>>> TopicPartitions) > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> be > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> used here and avoid version bump? > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-1 - For the new config, is there a > > > >>>>>> recommended > > > >>>>>>>>>>>>> value > > > >>>>>>>>>>>>>> or > > > >>>>>>>>>>>>>>>>> a > > > >>>>>>>>>>>>>>>>>>>>>>>>>> ConfigDef > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> validator? Probably it should based on > > > >> the > > > >>>>>>>>>>>>>>>>>>> metadata.max.age.ms > > > >>>>>>>>>>>>>>>>>>>>> ? > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Sizing > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instructions can be part of javadocs I > > > >>>> guess. > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-2 - (minor) As there are no changes > > > >> to > > > >>>> Kafka > > > >>>>>>>>>>>>> Streams, > > > >>>>>>>>>>>>>>>>>>> would > > > >>>>>>>>>>>>>>>>>>>>> it > > > >>>>>>>>>>>>>>>>>>>>>>>>>> be > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> better > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to add this new config > > > >>>>>>>>>>>>> auto.offset.reset.latest.max.age > > > >>>>>>>>>>>>>> to > > > >>>>>>>>>>>>>>>>>>> the > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> StreamsConfig block list > > > >>>>>>>>>>>>>>>>>>>>>>>>>> (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS) > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> for a > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clear warning, incase users configure > > > >> it? > > > >>>> This > > > >>>>>> is > > > >>>>>>>>>> the > > > >>>>>>>>>>>>>> most > > > >>>>>>>>>>>>>>>>>>>>>>>>>> familiar > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config and users might easily > > > >>>>>> mistakenly > > > >>>>>>>>>>>>>> configure > > > >>>>>>>>>>>>>>>>>>>>> it. Or > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> may be > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it's not worth it to add. > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-3 - (minor) The phrasing "the > > > >> consumer > > > >>>> falls > > > >>>>>>>>>> back > > > >>>>>>>>>>>>> to > > > >>>>>>>>>>>>>>>>>>>>> earliest" > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> reads as > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> if the config were being changed > > > >>>> per-partition > > > >>>>>>>>>> which > > > >>>>>>>>>>>>>> isn't > > > >>>>>>>>>>>>>>>>>>>>>>>>>> supported. > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> May > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be rephrasing to something like > > > >> "consumer > > > >>>>>> resolves > > > >>>>>>>>>>> the > > > >>>>>>>>>>>>>>>>>>> initial > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> position to > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> start offset for that partition" as if > > > >>>>>> earliest was > > > >>>>>>>>>>>>>> applied > > > >>>>>>>>>>>>>>>>>>> to > > > >>>>>>>>>>>>>>>>>>>>>>>>>> that > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition only and auto.offset.reset > > > >>>> config is > > > >>>>>>>>>>>>>> unchanged. > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks, > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Murali > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 28, 2026 at 2:48 PM 黃竣陽 < > > > >>>>>>>>>>>>>> [email protected]> > > > >>>>>>>>>>>>>>>>>>>>> wrote: > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi chia, > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have updated the KIP to include this > > > >>>> change. > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards, > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <[email protected]> > > > >> 於 > > > >>>>>>>>>> 2026年4月28日 > > > >>>>>>>>>>>>>> 晚上8:03 > > > >>>>>>>>>>>>>>>>>>> 寫道： > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> chia_0: Should we expose the > > > >> partition > > > >>>>>> creation > > > >>>>>>>>>>> time > > > >>>>>>>>>>>>>> via > > > >>>>>>>>>>>>>>>>>>> the > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Admin > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> API? > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I assume it would be valuable for > > > >> users to > > > >>>>>>>>>> diagnose > > > >>>>>>>>>>>>> and > > > >>>>>>>>>>>>>>>>>>>>>>>>>> troubleshoot > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> the > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> behavior of > > > >>>> auto.offset.reset.latest.max.age > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/04/28 10:47:58 黃竣陽 wrote: > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone, > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a discussion > > > >> on > > > >>>>>> KIP-1327 > > > >>>>>>>>>>>>>> Prevent > > > >>>>>>>>>>>>>>>>> Hot > > > >>>>>>>>>>>>>>>>>>>>> Data > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Loss > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Partition Expansion for Latest Policy > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> < > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>> > > > >>>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>> > > > >>>> > > > >> > > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$ > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to introduces > > > >>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age, > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> a > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config that lets the > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> latest reset policy distinguish > > > >> newly > > > >>>>>> expanded > > > >>>>>>>>>>>>> (hot) > > > >>>>>>>>>>>>>>>>>>>>> partitions > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> from > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> long-existing (cold) ones. Partitions > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> younger than the configured > > > >> threshold > > > >>>>>>>>>>> automatically > > > >>>>>>>>>>>>>> fall > > > >>>>>>>>>>>>>>>>>>>>> back > > > >>>>>>>>>>>>>>>>>>>>>>>>>> to > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> earliest, preventing silent data loss > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during topic expansion without > > > >> forcing a > > > >>>>>> full > > > >>>>>>>>>>>>>> historical > > > >>>>>>>>>>>>>>>>>>>>>>>>>> reprocess. > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards, > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>> > > > >>>>>>>>>>> > > > >>>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>> > > > >>>>> > > > >>>> > > > >>> > > > >> > > > > > > > > >
