Hi, I'd like to confirm what is happening with this KIP for AK 4.4. I think the answer is that we've decided to improve the docs (I see a PR) and leave it at that for now. Do I understand correctly?
Thanks, Andrew On 2026/07/08 01:32:41 Chia-Ping Tsai wrote: > hi all, > > While I agree we shouldn't surprise the majority of users, relying solely on > documentation means some users will still fall into this trap. Realizing this > limitation only when data is silently lost during a partition expansion is a > truly frustrating experience. > > Since we've reached the KIP freeze for 4.4.0, I recommend we focus on > updating the docs first. > > We can put this KIP on hold for now and revisit it after Community Over Code. > IIRC, Jiunn-Yang is going to share this specific issue at the event with > other Kafka users and providers. Hopefully, we can gather some fresh > perspectives and inspiration from the community there. > > Best, > Chia-Ping > > On 2026/07/07 21:38:22 Jun Rao via dev wrote: > > Hi, Jiunn-Yang, > > > > Thanks for the reply. > > > > Changing the default value only makes sense if it benefits most users. > > Most users > > probably never increase the number of partitions. Changing the default > > config value gives them no benefit and only introduces a probably > > surprising behavior > > change. > > > > Jun > > > > On Tue, Jul 7, 2026 at 1:22 PM Chia-Ping Tsai <[email protected]> wrote: > > > > > hi Jiunn > > > > > > +1 to Approach A :) > > > > > > Best, > > > Chia-Ping > > > > > > On 2026/07/07 11:55:14 黃竣陽 wrote: > > > > Hi Jun, Chia-Ping, > > > > > > > > I'd like to first explain why I believe changing the default is > > > worthwhile. > > > > > > > > Documentation can only help users who already know about the problem. > > > > Because the failure mode is silent, most users discover it only after > > > data has > > > > already been missed. A safer default such as by_duration:PT5S protects > > > users > > > > automatically, without requiring prior awareness of the issue. > > > > > > > > Given that, the question is not whether the issue exists, but how best > > > to introduce > > > > a safer default. > > > > > > > > Approach A — Follow a deprecation cycle and land the change in 5.0 > > > > - Keep the current default (`latest`) in upcoming 4.x releases, but emit > > > a warning that > > > > the default will change to `by_duration:PT5S` in 5.0 and explain how > > > users can retain > > > > the current behavior explicitly. > > > > - Change the default to `by_duration:PT5S` in 5.0. > > > > - Update shell tools (for example, the console consumer) to explicitly > > > use `latest`, preserving > > > > the expected "show only new messages" behavior. > > > > > > > > Approach B — Change the default in a 4.4 minor release > > > > - Change the default to `by_duration:PT5S` in a 4.x minor release. > > > > - Leave shell tools unchanged. > > > > - Emit a warning explaining the change and how users can explicitly > > > restore `latest`. > > > > > > > > I favor Approach B: > > > > > > > > Silent data loss during partition expansion is a correctness issue, not > > > a usability issue. > > > > Delaying the safer default until 5.0 leaves users relying on the > > > implicit default exposed unnecessarily. > > > > Users who need latest can simply pin it explicitly. > > > > I don't think shell tools need special handling. by_duration:PT5S only > > > replays a few seconds of data, > > > > which is negligible in practice and preferable to silently missing > > > records. Keeping the behavior consistent > > > > is simpler. > > > > > > > > Best Regards, > > > > Jiunn-Yang > > > > > > > > > Jun Rao via dev <[email protected]> 於 2026年7月7日 清晨5:49 寫道： > > > > > > > > > > Hi, Chia-Ping, > > > > > > > > > > I am not sure that we need an out-of-the-box solution. The proposal in > > > the > > > > > KIP is also not out-of-the-box and requires users to opt-in. > > > > > > > > > > Thanks, > > > > > > > > > > Jun > > > > > > > > > > On Mon, Jul 6, 2026 at 11:34 AM Chia-Ping Tsai <[email protected]> > > > wrote: > > > > > > > > > >> hi Jun > > > > >> > > > > >> Yes, updating the docs requires minimal changes to the codebase, but > > > it is > > > > >> not an out-of-the-box solution. From my perspective, this issue can > > > > >> be > > > > >> addressed by setting the console consumer's policy back to latest. > > > > >> > > > > >> Let's see what Jiunn thinks > > > > >> > > > > >> Best, > > > > >> Chia-Ping > > > > >> > > > > >> On 2026/07/06 15:59:37 Jun Rao via dev wrote: > > > > >>> Hi, Chia-Ping, > > > > >>> > > > > >>> One potential downside of changing the default is that some existing > > > > >> tools > > > > >>> (e.g. console consumer) may depend on this behavior. For example, a > > > user > > > > >>> testing the system may expect to see only new messages and could be > > > > >>> surprised by the change. Another option is to leave the default > > > > >> unchanged, > > > > >>> but document clearly how to prevent losing messages on new > > > > >>> partitions > > > > >>> through by_duration. The users can then make the corresponding > > > > >>> config > > > > >>> changes if needed. > > > > >>> > > > > >>> Thanks, > > > > >>> > > > > >>> Jun > > > > >>> > > > > >>> On Wed, Jul 1, 2026 at 9:39 AM Chia-Ping Tsai <[email protected]> > > > > >> wrote: > > > > >>> > > > > >>>> Hi Jun, > > > > >>>> > > > > >>>> Thanks for the feedback. I agree with your conclusion. It would be > > > > >> better > > > > >>>> to "make haste" with a new KIP to address this. We can focus the > > > > >> discussion > > > > >>>> on the following points: > > > > >>>> > > > > >>>> 1) Should we change the default reset policy to by_duration for all > > > > >>>> consumers? > > > > >>>> 2) What is the optimal duration to prevent data loss while avoiding > > > > >> large > > > > >>>> backlogs? Note that the classic consumer has a 3-second heartbeat > > > > >> interval, > > > > >>>> whereas the async consumer uses 5 seconds. > > > > >>>> 3) What are the potential side effects of replacing latest with > > > > >>>> by_duration as the default behavior? > > > > >>>> > > > > >>>> Best, > > > > >>>> Chia-Ping > > > > >>>> > > > > >>>> On 2026/06/30 20:39:24 Jun Rao via dev wrote: > > > > >>>>> Hi, Chia-Ping, > > > > >>>>> > > > > >>>>> I agree that using by_duration with a value of 5s to 10s addresses > > > > >> most > > > > >>>>> common issues with expanded partitions, given the default > > > > >>>>> heartbeat.interval.ms is 3 seconds. We can consider setting that > > > as > > > > >> the > > > > >>>> new > > > > >>>>> default. The current proposal may address some other rarer cases, > > > > >> but it > > > > >>>>> feels a bit like over engineering. > > > > >>>>> > > > > >>>>> Thanks, > > > > >>>>> > > > > >>>>> Jun > > > > >>>>> > > > > >>>>> On Tue, Jun 30, 2026 at 5:58 AM Chia-Ping Tsai < > > > [email protected]> > > > > >>>> wrote: > > > > >>>>> > > > > >>>>>> - Short durations fail during extended downtimes, leading to the > > > > >> exact > > > > >>>>>> silent data > > > > >>>>>> loss this KIP aims to prevent. > > > > >>>>>> > > > > >>>>>> Losing the records produced to expanded partitions during a > > > > >> downtime is > > > > >>>>>> actually the expected behavior for the latest policy. This is > > > > >> exactly > > > > >>>> why, > > > > >>>>>> in the current KIP proposal, we prefer by_duration over earliest > > > > >> for > > > > >>>> the > > > > >>>>>> expanded partitions—we want to avoid forcing users to consume a > > > > >> massive > > > > >>>>>> accumulated backlog just because a partition is technically > > > > >>>>>> "new". > > > > >>>>>> > > > > >>>>>> To quote my previous comment from May 15: > > > > >>>>>> "...if a consumer is offline for a while and a new partition is > > > > >> created > > > > >>>>>> during that downtime, the user might actually want to skip to > > > > >> latest > > > > >>>> when > > > > >>>>>> resuming, rather than reading from earliest just because the > > > > >> partition > > > > >>>> is > > > > >>>>>> technically 'new' to the group." > > > > >>>>>> > > > > >>>>>> On 2026/06/30 12:12:47 黃竣陽 wrote: > > > > >>>>>>> Hello all, > > > > >>>>>>> > > > > >>>>>>> The metadata blindness and consumer downtime windows are > > > > >> inherently > > > > >>>>>>> unpredictable, making a fixed by_duration approach unviable. No > > > > >>>> static > > > > >>>>>>> duration can accommodate both extremes: > > > > >>>>>>> > > > > >>>>>>> - Short durations fail during extended downtimes, leading to the > > > > >>>> exact > > > > >>>>>> silent data > > > > >>>>>>> loss this KIP aims to prevent. > > > > >>>>>>> - Long durations (calibrated for worst-case scenarios) cause > > > > >> routine > > > > >>>>>> restarts to > > > > >>>>>>> trigger unnecessary mass replays, undermining the latest offset > > > > >> reset > > > > >>>>>> policy. > > > > >>>>>>> > > > > >>>>>>> This is why the proposed solution uses creation time. It > > > > >> provides a > > > > >>>>>> discrete, > > > > >>>>>>> deterministic signal—verifying whether a partition is actually > > > > >> newer > > > > >>>>>> than the group, > > > > >>>>>>> rather than relying on an arbitrary time window that has no > > > > >>>> correlation > > > > >>>>>> with the consumer’s > > > > >>>>>>> actual downtime. > > > > >>>>>>> > > > > >>>>>>> Best Regards, > > > > >>>>>>> Jiunn-Yang > > > > >>>>>>> > > > > >>>>>>>> Chia-Ping Tsai <[email protected]> 於 2026年6月30日 下午3:05 寫道： > > > > >>>>>>>> > > > > >>>>>>>> Hi all, > > > > >>>>>>>> > > > > >>>>>>>> I'd like to revisit Jun previous comment about using > > > > >>>> by_duration=5s. I > > > > >>>>>> realize now that having a small backlog is actually an acceptable > > > > >>>>>> trade-off, especially since the latest policy inherently brings a > > > > >> tiny > > > > >>>>>> backlog to starting consumers anyway due to the time gap between > > > > >>>> fetching > > > > >>>>>> the offset and actually consuming. > > > > >>>>>>>> > > > > >>>>>>>> If we bring the simple by_duration=5s solution back to the > > > > >> KIP, I'd > > > > >>>>>> like to discuss a further step: Should we just change the default > > > > >>>> policy > > > > >>>>>> from latest to by_duration=5s specifically for the new consumer > > > and > > > > >>>> share > > > > >>>>>> consumer? > > > > >>>>>>>> > > > > >>>>>>>> Best, > > > > >>>>>>>> Chia-Ping > > > > >>>>>>>> > > > > >>>>>>>> On 2026/06/15 17:39:19 Chia-Ping Tsai wrote: > > > > >>>>>>>>> hi Jun > > > > >>>>>>>>> > > > > >>>>>>>>> The most important part of this story is how users should > > > > >> expect > > > > >>>> the > > > > >>>>>> data > > > > >>>>>>>>> they can see when using the latest or by_duration policy with > > > > >>>> expanded > > > > >>>>>>>>> partitions. > > > > >>>>>>>>> > > > > >>>>>>>>> Yes, the by_duration policy can minimize data loss, but it is > > > > >>>>>>>>> non-deterministic, which means users will either read too many > > > > >>>>>> historical > > > > >>>>>>>>> records from existing partitions or lose some records from > > > > >>>> expanded > > > > >>>>>>>>> partitions. > > > > >>>>>>>>> > > > > >>>>>>>>> Also, I agree that auto.offset.reset.max.age.ms is a bit > > > > >> hard to > > > > >>>>>>>>> understand, and that is why I preferred having a whole new > > > > >> policy > > > > >>>>>> based > > > > >>>>>>>>> entirely on group creation time (KIP-1282) > > > > >>>>>>>>> > > > > >>>>>>>>> Best, > > > > >>>>>>>>> Chia-Ping > > > > >>>>>>>>> > > > > >>>>>>>>> Jun Rao via dev <[email protected]> 於 2026年6月16日週二 > > > > >> 上午1:08寫道： > > > > >>>>>>>>> > > > > >>>>>>>>>> Hi, Chia-Ping and Jiunn-Yang, > > > > >>>>>>>>>> > > > > >>>>>>>>>> Thanks for the reply. I am still trying to understand the > > > > >> value > > > > >>>> of > > > > >>>>>> the new > > > > >>>>>>>>>> configs with the KIP. > > > > >>>>>>>>>> > > > > >>>>>>>>>> The motivation of the KIP is that a user doesn't want to > > > > >> miss the > > > > >>>>>> data if > > > > >>>>>>>>>> the backlog is small. The backlog of the existing partition > > > > >> is > > > > >>>> easy > > > > >>>>>> to > > > > >>>>>>>>>> understand because it relates to retention time. The backlog > > > > >> for > > > > >>>> the > > > > >>>>>> new > > > > >>>>>>>>>> partition is a bit subtle to understand since it depends on > > > > >> the > > > > >>>>>> metadata > > > > >>>>>>>>>> refresh delay. To set auto.offset.reset.max.age.ms, the user > > > > >>>> needs > > > > >>>>>> to > > > > >>>>>>>>>> understand the metadata refresh delay on the consumer side > > > > >> and > > > > >>>> use > > > > >>>>>> it to > > > > >>>>>>>>>> set the config. > > > > >>>>>>>>>> > > > > >>>>>>>>>> Now, let's consider the alternative: setting the same value > > > > >> for > > > > >>>> the > > > > >>>>>>>>>> existing by_duration policy. The KIP lists three issues with > > > > >> this > > > > >>>>>> approach. > > > > >>>>>>>>>> 1. It computes the seek target client-side as now() - > > > > >> duration, > > > > >>>> which > > > > >>>>>>>>>> introduces clock skew across consumers and forces operators > > > > >> to > > > > >>>> choose > > > > >>>>>>>>>> overly large durations, causing unnecessary reprocessing. > > > > >>>>>>>>>> 2. The target timestamp is recomputed on each retry, so > > > > >> failed > > > > >>>>>>>>>> ListOffsetsRequest retries can shift the target forward and > > > > >>>>>> potentially > > > > >>>>>>>>>> miss records produced between attempts. > > > > >>>>>>>>>> 3. It applies uniformly to all partitions without committed > > > > >>>> offsets, > > > > >>>>>> and > > > > >>>>>>>>>> cannot distinguish newly expanded partitions from > > > > >> long-existing > > > > >>>>>> partitions > > > > >>>>>>>>>> newly assigned to the group, leading to unnecessary replay. > > > > >>>>>>>>>> > > > > >>>>>>>>>> Issues 1 and 2 are uncommon and can be mitigated by adding a > > > > >> bit > > > > >>>>>> buffer to > > > > >>>>>>>>>> the metadata refresh delay. We could also consider improving > > > > >> the > > > > >>>>>>>>>> implementation. For issue 3, the metadata refresh delay is > > > > >>>> typically > > > > >>>>>> low > > > > >>>>>>>>>> (in the order of minutes with the classic consumer and tens > > > > >> of > > > > >>>>>> seconds with > > > > >>>>>>>>>> the new consumer). If a user is ok with reading that much > > > > >> backlog > > > > >>>>>> for new > > > > >>>>>>>>>> partitions, it seems they will be ok doing the same for > > > > >> existing > > > > >>>>>>>>>> partitions. > > > > >>>>>>>>>> > > > > >>>>>>>>>> So, instead of introducing a new config, could we just reuse > > > > >> the > > > > >>>>>> existing > > > > >>>>>>>>>> config with better documentation and/or implementation? > > > > >>>>>>>>>> > > > > >>>>>>>>>> Jun > > > > >>>>>>>>>> > > > > >>>>>>>>>> > > > > >>>>>>>>>> On Sat, Jun 13, 2026 at 12:19 AM 黃竣陽 <[email protected]> > > > > >> wrote: > > > > >>>>>>>>>> > > > > >>>>>>>>>>> Hello Jun, > > > > >>>>>>>>>>> > > > > >>>>>>>>>>> You're right that group creation time is the more intuitive > > > > >>>> answer > > > > >>>>>> at > > > > >>>>>>>>>>> first glance, > > > > >>>>>>>>>>> the KIP's own motivation talks about partitions that > > > > >> "predate > > > > >>>> the > > > > >>>>>> group" > > > > >>>>>>>>>>> vs partitions > > > > >>>>>>>>>>> "created during group runtime," which directly points to a > > > > >>>>>>>>>> group-lifecycle > > > > >>>>>>>>>>> classifier. > > > > >>>>>>>>>>> I'd like to walk through why we landed on partition age, > > > > >> and the > > > > >>>>>>>>>>> trade-offs we considered. > > > > >>>>>>>>>>> > > > > >>>>>>>>>>> We evaluated three candidate signals: > > > > >>>>>>>>>>> > > > > >>>>>>>>>>> 1. `by_duration:5secs` > > > > >>>>>>>>>>> > > > > >>>>>>>>>>> This covers the metadata blindness window, but has issues > > > > >> the > > > > >>>> KIP > > > > >>>>>>>>>>> currently documents > > > > >>>>>>>>>>> under "Why not use `by_duration`?": > > > > >>>>>>>>>>> > > > > >>>>>>>>>>> - Client-side `now() - duration` introduces clock skew > > > > >> across > > > > >>>>>> consumers. > > > > >>>>>>>>>>> - `ListOffsets` retries shift the target forward, > > > > >> potentially > > > > >>>>>> missing > > > > >>>>>>>>>>> records produced between > > > > >>>>>>>>>>> attempts. > > > > >>>>>>>>>>> - It applies uniformly to all partitions without committed > > > > >>>> offsets, > > > > >>>>>>>>>>> including pre-existing partitions > > > > >>>>>>>>>>> newly assigned to the group, causing unnecessary replay. > > > > >>>>>>>>>>> > > > > >>>>>>>>>>> 2. Group creation time as classifier > > > > >>>>>>>>>>> > > > > >>>>>>>>>>> This works cleanly when the consumer is actively running. > > > > >> Our > > > > >>>>>> concern > > > > >>>>>>>>>>> is the idle / late-rejoin case: > > > > >>>>>>>>>>> > > > > >>>>>>>>>>> T=0: Group created. > > > > >>>>>>>>>>> T=1..T=100: Consumer idle (down, disconnected, etc.). > > > > >>>>>>>>>>> T=50: Partition added during the idle window. > > > > >>>>>>>>>>> T=100: Consumer resumes. > > > > >>>>>>>>>>> > > > > >>>>>>>>>>> Under group creation time, the new partition is classified > > > > >> as > > > > >>>> new > > > > >>>>>>>>>>> (`50 > 0`) and reset to `earliest`, replaying everything > > > > >> from > > > > >>>> T=50. > > > > >>>>>>>>>>> But during `[T=1, T=100]`, base partitions also accumulated > > > > >> data > > > > >>>>>> that > > > > >>>>>>>>>>> the consumer accepts as lost — that is precisely the > > > > >> contract of > > > > >>>>>>>>>>> `auto.offset.reset=latest`. There is no principled reason to > > > > >>>> treat > > > > >>>>>>>>>>> the new partition differently; both contain backlog > > > > >> accumulated > > > > >>>>>> during > > > > >>>>>>>>>>> the same idle window. > > > > >>>>>>>>>>> > > > > >>>>>>>>>>> This aligns with the "backlog is backlog” principle you > > > > >> raised > > > > >>>> in > > > > >>>>>>>>>>> the KIP-1282 thread: a `latest` user has tolerated some > > > > >> backlog > > > > >>>> on > > > > >>>>>>>>>>> every other partition during the same idle period; forcing > > > > >>>> 0-backlog > > > > >>>>>>>>>>> tolerance only on new partitions would be inconsistent with > > > > >> that > > > > >>>>>>>>>>> tolerance. > > > > >>>>>>>>>>> > > > > >>>>>>>>>>> 3. Partition age vs threshold > > > > >>>>>>>>>>> > > > > >>>>>>>>>>> Partition age corresponds to the actual silent data loss > > > > >> window, > > > > >>>>>>>>>>> the gap between partition creation and the consumer’s > > > > >> metadata > > > > >>>>>>>>>>> refresh. Within this window, data loss is genuinely silent: > > > > >> the > > > > >>>>>>>>>>> consumer had no opportunity to know about the partition. > > > > >> Outside > > > > >>>>>> this > > > > >>>>>>>>>>> window, missing data reflects either: > > > > >>>>>>>>>>> > > > > >>>>>>>>>>> - (a) the user’s tolerated cost of running with idle > > > > >> consumers, > > > > >>>> or > > > > >>>>>>>>>>> - (b) an operational issue to surface via monitoring, not > > > > >> via > > > > >>>> reset > > > > >>>>>>>>>> policy. > > > > >>>>>>>>>>> > > > > >>>>>>>>>>> We did not choose partition age because it is more elegant > > > > >> than > > > > >>>>>> group > > > > >>>>>>>>>>> creation time — we chose it because its failure mode > > > > >> (requires a > > > > >>>>>>>>>>> threshold) is > > > > >>>>>>>>>>> less invasive than the failure mode of group creation time > > > > >>>>>> (overrides > > > > >>>>>>>>>>> user-stated > > > > >>>>>>>>>>> `latest` intent during idle periods). > > > > >>>>>>>>>>> > > > > >>>>>>>>>>> Best Regards, > > > > >>>>>>>>>>> Jiunn-Yang > > > > >>>>>>>>>>> > > > > >>>>>>>>>>>> Chia-Ping Tsai <[email protected]> 於 2026年6月13日 上午11:52 > > > > >> 寫道： > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> Hi Jun, > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> Relying on both creation times will create an inconsistent > > > > >>>>>> scenario. A > > > > >>>>>>>>>>>> consumer that lost all offsets due to a long sleep will > > > > >> seek > > > > >>>> to the > > > > >>>>>>>>>>>> beginning for the partitions created later than the group. > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> That is why we initially proposed KIP-1282 to fix the > > > > >>>> inconsistency > > > > >>>>>>>>>>> using a > > > > >>>>>>>>>>>> whole new policy. Since KIP-1282 couldn't reach a > > > > >> consensus, > > > > >>>>>> KIP-1327 > > > > >>>>>>>>>>> goes > > > > >>>>>>>>>>>> back to using flexible configurations to prevent users from > > > > >>>> falling > > > > >>>>>>>>>> into > > > > >>>>>>>>>>>> that pitfall. > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> Best, Chia-Ping > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> Jun Rao via dev <[email protected]> 於 2026年6月13日週六 > > > > >>>> 上午6:49寫道： > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>>> Hi, Jiunn-Yang, > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>> Thanks for the reply and sorry for the late reply. > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>> JR1. The design of auto.offset.reset.max.age.ms still > > > > >> feels > > > > >>>>>> weird to > > > > >>>>>>>>>>> me. > > > > >>>>>>>>>>>>> It > > > > >>>>>>>>>>>>> categorizes partitions as new or existing based on the > > > > >>>> partition > > > > >>>>>>>>>>> creation > > > > >>>>>>>>>>>>> time. Intuitively, the categorization should be based on > > > > >> the > > > > >>>> group > > > > >>>>>>>>>>> creation > > > > >>>>>>>>>>>>> time: all partitions existing when the group is created > > > > >> are > > > > >>>>>> existing > > > > >>>>>>>>>> and > > > > >>>>>>>>>>>>> all partitions created after the group creation are new > > > > >>>>>> partitions. > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>> Jun > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>> On Tue, Jun 9, 2026 at 8:51 AM 黃竣陽 <[email protected]> > > > > >>>> wrote: > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> Hi all, > > > > >>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> Manually bumping this thread. If there is no further > > > > >>>>>>>>>>>>>> discussion, I will close the vote. > > > > >>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> Best Regards, > > > > >>>>>>>>>>>>>> Jiunn-Yang > > > > >>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> 黃竣陽 <[email protected]> 於 2026年6月1日 晚上7:16 寫道： > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> Hello Jian, > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> Thanks for your feedback, > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> Agreed, partition expansion is a common operational > > > > >> task, > > > > >>>> not an > > > > >>>>>>>>>> edge > > > > >>>>>>>>>>>>>>> case. I've updated the Motivation section accordingly. > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> Best Regards, > > > > >>>>>>>>>>>>>>> Jiunn-Yang > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>> jian fu <[email protected]> 於 2026年6月1日 下午5:49 寫道： > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>> Hi Jiunn-Yang: > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>> Thanks for the KIP. I think it would be useful to > > > > >> clarify > > > > >>>> that > > > > >>>>>> this > > > > >>>>>>>>>>>>> is a > > > > >>>>>>>>>>>>>>>> common scenario rather than an edge case, which further > > > > >>>>>>>>>> demonstrates > > > > >>>>>>>>>>>>> the > > > > >>>>>>>>>>>>>>>> need for this optimization. For example: > > > > >>>>>>>>>>>>>>>> A partition expansion is a common operational task in > > > > >>>> Kafka: To > > > > >>>>>>>>>>>>> balance > > > > >>>>>>>>>>>>>>>> resource utilization and cost, topics are typically > > > > >> created > > > > >>>>>> with a > > > > >>>>>>>>>>>>>> moderate > > > > >>>>>>>>>>>>>>>> default partition count. However, as traffic grows over > > > > >>>> time, > > > > >>>>>> it is > > > > >>>>>>>>>>>>>> often > > > > >>>>>>>>>>>>>>>> necessary to increase the number of partitions to > > > > >>>> accommodate > > > > >>>>>> the > > > > >>>>>>>>>>>>> higher > > > > >>>>>>>>>>>>>>>> workload. > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>> Regards > > > > >>>>>>>>>>>>>>>> Jian > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>> 黃竣陽 <[email protected]> 于2026年5月30日周六 22:31写道： > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> Hello chia, > > > > >>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> Thanks for the comments, I have updated the KIP! > > > > >>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> Best Regards, > > > > >>>>>>>>>>>>>>>>> Jiunn-Yang > > > > >>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>> Chia-Ping Tsai <[email protected]> 於 2026年5月30日 > > > > >> 晚上8:29 > > > > >>>> 寫道： > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>> Hi Jiunn-Yang, > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>> Would you mind removing the terms "hot" and "cold" > > > > >> when > > > > >>>>>>>>>> describing > > > > >>>>>>>>>>>>>>>>>> partitions in the KIP? I understand you are using > > > > >> them to > > > > >>>>>>>>>> describe > > > > >>>>>>>>>>>>> the > > > > >>>>>>>>>>>>>>>>>> "freshness" or the users' need for the records, but > > > > >>>> applying > > > > >>>>>>>>>> these > > > > >>>>>>>>>>>>>> terms > > > > >>>>>>>>>>>>>>>>> to > > > > >>>>>>>>>>>>>>>>>> the partition itself feels a bit unnatural. > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>> After all, in this scenario, users don't really care > > > > >>>> whether > > > > >>>>>> a > > > > >>>>>>>>>>>>>> partition > > > > >>>>>>>>>>>>>>>>> is > > > > >>>>>>>>>>>>>>>>>> newly expanded or not. Their only expectation is that > > > > >>>> they > > > > >>>>>> won't > > > > >>>>>>>>>>>>>> silently > > > > >>>>>>>>>>>>>>>>>> lose any live records produced to the topic during > > > > >> their > > > > >>>>>> active > > > > >>>>>>>>>>>>>>>>> consumption. > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>> Best, Chia-Ping > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>> 黃竣陽 <[email protected]> 於 2026年5月30日週六 下午12:30寫道： > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>> Hello Jun, > > > > >>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>> Thanks for the feedback, I have updated the KIP > > > > >>>> motivation > > > > >>>>>>>>>>> section. > > > > >>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>> Best Regards, > > > > >>>>>>>>>>>>>>>>>>> Jiunn-Yang > > > > >>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>> Jun Rao via dev <[email protected]> 於 > > > > >> 2026年5月30日 > > > > >>>>>> 凌晨1:12 寫道： > > > > >>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang, > > > > >>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>> Thanks for the reply. I think we need a stronger > > > > >>>>>> motivation for > > > > >>>>>>>>>>>>> the > > > > >>>>>>>>>>>>>>>>> KIP. > > > > >>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>> The KIP says "The core insight is that not all > > > > >>>> partitions > > > > >>>>>>>>>> without > > > > >>>>>>>>>>>>> a > > > > >>>>>>>>>>>>>>>>>>>> committed offset are the same. A newly expanded > > > > >>>> partition > > > > >>>>>> (hot) > > > > >>>>>>>>>>> is > > > > >>>>>>>>>>>>>>>>>>>> fundamentally different from a partition the > > > > >> consumer > > > > >>>> has > > > > >>>>>> never > > > > >>>>>>>>>>>>> seen > > > > >>>>>>>>>>>>>>>>>>>> because it predates the group (cold)." Why is the > > > > >> hot > > > > >>>>>> partition > > > > >>>>>>>>>>>>>>>>>>>> fundamentally different from the cold? > > > > >>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>> The KIP says "The existing by_duration policy is > > > > >> also > > > > >>>>>>>>>>> insufficient > > > > >>>>>>>>>>>>>>>>>>> because: > > > > >>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>> - The calculated seek time (now() - duration) > > > > >> varies > > > > >>>> across > > > > >>>>>>>>>> nodes > > > > >>>>>>>>>>>>>> due > > > > >>>>>>>>>>>>>>>>>>> to > > > > >>>>>>>>>>>>>>>>>>>> clock skew. To be safe, users must set an overly > > > > >> large > > > > >>>>>>>>>> duration, > > > > >>>>>>>>>>>>>>>>>>> causing > > > > >>>>>>>>>>>>>>>>>>>> unnecessary reprocessing. > > > > >>>>>>>>>>>>>>>>>>>> - On network errors, the client recalculates the > > > > >> seek > > > > >>>> time > > > > >>>>>> on > > > > >>>>>>>>>>>>> retry, > > > > >>>>>>>>>>>>>>>>>>>> shifting the target timestamp forward and risking > > > > >> data > > > > >>>>>> loss." > > > > >>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>> However, both of these situations are rare. If > > > > >> these > > > > >>>> issues > > > > >>>>>>>>>>>>> persist, > > > > >>>>>>>>>>>>>>>>> more > > > > >>>>>>>>>>>>>>>>>>>> severe problems likely exist elsewhere. Rare > > > > >> situations > > > > >>>>>> don't > > > > >>>>>>>>>>>>> need a > > > > >>>>>>>>>>>>>>>>>>> common > > > > >>>>>>>>>>>>>>>>>>>> solution. If users care about those rare > > > > >> situations, > > > > >>>> they > > > > >>>>>> can > > > > >>>>>>>>>>>>>> implement > > > > >>>>>>>>>>>>>>>>>>>> customized logic using > > > > >>>>>>>>>>>>>>>>> ConsumerRebalanceListener.onPartitionsAssigned(). > > > > >>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>> Jun > > > > >>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>> On Sun, May 17, 2026 at 6:50 AM 黃竣陽 < > > > > >>>> [email protected]> > > > > >>>>>>>>>> wrote: > > > > >>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>> Hello chia, > > > > >>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>> Thanks for the feedback, > > > > >>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>> If the creation time exists, the returned value > > > > >>>> should > > > > >>>>>> always > > > > >>>>>>>>>>> be > > > > >>>>>>>>>>>>>>>>>>> greater > > > > >>>>>>>>>>>>>>>>>>>>> than or equal to zero, right? > > > > >>>>>>>>>>>>>>>>>>>>> I have explicitly mentioned this in the KIP. > > > > >>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>> New Old (MetadataResponse v0–13) positive > > > > >>>> any > > > > >>>>>>>>>>>>>> field > > > > >>>>>>>>>>>>>>>>>>>>> absent UnsupportedVersionException > > > > >>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>> The earliest point at which we can detect the > > > > >> version > > > > >>>>>> mismatch > > > > >>>>>>>>>>> is > > > > >>>>>>>>>>>>>>>>> during > > > > >>>>>>>>>>>>>>>>>>>>> the > > > > >>>>>>>>>>>>>>>>>>>>> first metadata fetch after assignment, which > > > > >> occurs > > > > >>>> inside > > > > >>>>>>>>>>>>> poll(). > > > > >>>>>>>>>>>>>>>>>>>>> Therefore, the > > > > >>>>>>>>>>>>>>>>>>>>> user would encounter an > > > > >> UnsupportedVersionException > > > > >>>> from > > > > >>>>>>>>>> poll(). > > > > >>>>>>>>>>>>>> I’ll > > > > >>>>>>>>>>>>>>>>>>>>> clarify this in the KIP. > > > > >>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>> Best Regards, > > > > >>>>>>>>>>>>>>>>>>>>> Jiunn-Yang > > > > >>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <[email protected]> 於 > > > > >> 2026年5月17日 > > > > >>>>>> 下午4:50 寫道： > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>> hi Jiunn > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>> PartitionAgeMs (int64, default -1): The age of > > > > >> this > > > > >>>>>>>>>> partition > > > > >>>>>>>>>>>>> in > > > > >>>>>>>>>>>>>>>>>>>>> milliseconds, computed server-side by the broker > > > > >> as > > > > >>>>>>>>>>>>>>>>> broker_current_time > > > > >>>>>>>>>>>>>>>>>>> - > > > > >>>>>>>>>>>>>>>>>>>>> partition_creation_time. Returns -1 if the broker > > > > >>>> does not > > > > >>>>>>>>>>>>> support > > > > >>>>>>>>>>>>>>>>> this > > > > >>>>>>>>>>>>>>>>>>>>> feature or the partition creation time is unknown. > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>> If the creation time exists, the returned value > > > > >>>> should > > > > >>>>>> always > > > > >>>>>>>>>>> be > > > > >>>>>>>>>>>>>>>>>>> greater > > > > >>>>>>>>>>>>>>>>>>>>> than or equal to zero, right? > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>> New Old (MetadataResponse v0–13) positive > > > > >>>> any > > > > >>>>>>>>>>>>>> field > > > > >>>>>>>>>>>>>>>>>>>>> absent UnsupportedVersionException > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>> Will user encounter UnsupportedVersionException > > > > >> when > > > > >>>>>> calling > > > > >>>>>>>>>>>>>>>>> `poll()`? > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>> Best, > > > > >>>>>>>>>>>>>>>>>>>>>> Chia-Ping > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>> On 2026/05/16 04:30:49 黃竣陽 wrote: > > > > >>>>>>>>>>>>>>>>>>>>>>> Hello Jun, chia, > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>> I've updated KIP-1327 with a design change > > > > >> based on > > > > >>>> the > > > > >>>>>>>>>>>>>> discussion > > > > >>>>>>>>>>>>>>>>>>>>>>> feedback. > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>> The updated design decouples the new-partition > > > > >> reset > > > > >>>>>>>>>> behavior > > > > >>>>>>>>>>>>>> from > > > > >>>>>>>>>>>>>>>>>>>>>>> the base auto.offset.reset policy: > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>> - auto.offset.reset.max.age.ms now applies to > > > > >> all > > > > >>>>>>>>>>>>>> auto.offset.reset > > > > >>>>>>>>>>>>>>>>>>>>> values > > > > >>>>>>>>>>>>>>>>>>>>>>> (latest, earliest, by_duration, none). > > > > >>>>>>>>>>>>>>>>>>>>>>> - For new ("hot") partitions, the consumer > > > > >> resets to > > > > >>>>>>>>>>>>>>>>>>>>> auto.offset.reset.new.partitions > > > > >>>>>>>>>>>>>>>>>>>>>>> config setting > > > > >>>>>>>>>>>>>>>>>>>>>>> - For existing ("cold") partitions, the base > > > > >>>>>>>>>> auto.offset.reset > > > > >>>>>>>>>>>>>>>>> policy > > > > >>>>>>>>>>>>>>>>>>>>> continues > > > > >>>>>>>>>>>>>>>>>>>>>>> to apply unchanged. > > > > >>>>>>>>>>>>>>>>>>>>>>> - The new-partition reset behavior is > > > > >> represented > > > > >>>> by a > > > > >>>>>>>>>>> separate > > > > >>>>>>>>>>>>>>>>>>>>> internal config > > > > >>>>>>>>>>>>>>>>>>>>>>> (auto.offset.reset.new.partitions, currently > > > > >> fixed > > > > >>>> to > > > > >>>>>>>>>>>>> earliest). > > > > >>>>>>>>>>>>>>>>> This > > > > >>>>>>>>>>>>>>>>>>>>> decoupled design makes > > > > >>>>>>>>>>>>>>>>>>>>>>> it straightforward to promote the behavior to a > > > > >>>> public > > > > >>>>>>>>>>>>>> user-facing > > > > >>>>>>>>>>>>>>>>>>>>> configuration in a future KIP. > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>> Best Regards, > > > > >>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <[email protected]> 於 > > > > >> 2026年5月16日 > > > > >>>>>> 清晨7:46 > > > > >>>>>>>>>> 寫道： > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>> hi Jun > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>> I see what you mean now. The proposal from me > > > > >> is > > > > >>>> listed > > > > >>>>>>>>>>> below: > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>> 1) Add auto.offset.reset.new.partitions with a > > > > >>>> default > > > > >>>>>>>>>> value > > > > >>>>>>>>>>>>> of > > > > >>>>>>>>>>>>>>>>>>>>> earliest. It fixes the data loss from both > > > > >>>> by_duration and > > > > >>>>>>>>>>>>> latest, > > > > >>>>>>>>>>>>>> and > > > > >>>>>>>>>>>>>>>>>>> it > > > > >>>>>>>>>>>>>>>>>>>>> does not change the logic of > > > > >>>> auto.offset.reset=earliest. > > > > >>>>>>>>>>>>>>>>>>>>>>>> 2) Mark auto.offset.reset.new.partitions as an > > > > >>>>>> internal > > > > >>>>>>>>>>>>>>>>>>>>> configuration. auto.offset.reset.new > > > > >>>> .partitions=earliest > > > > >>>>>>>>>>> already > > > > >>>>>>>>>>>>>>>>>>>>> addresses the issue, and we can discuss the use > > > > >> cases > > > > >>>> of > > > > >>>>>> other > > > > >>>>>>>>>>>>>> values > > > > >>>>>>>>>>>>>>>>>>> in a > > > > >>>>>>>>>>>>>>>>>>>>> separate KIP. > > > > >>>>>>>>>>>>>>>>>>>>>>>> 3) Both configs, auto.offset.reset.new > > > > >> .partitions > > > > >>>> and > > > > >>>>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age.ms, will be > > > > >> applied > > > > >>>> to > > > > >>>>>> all > > > > >>>>>>>>>> for > > > > >>>>>>>>>>>>>>>>>>>>> consistency. > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>> WDYT? > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>> On 2026/05/15 20:53:20 Jun Rao via dev wrote: > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Hi, Chia-Ping, > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply. > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> 1. In the motivation section, the KIP says > > > > >> "When a > > > > >>>>>> Kafka > > > > >>>>>>>>>>>>> topic > > > > >>>>>>>>>>>>>> is > > > > >>>>>>>>>>>>>>>>>>>>> expanded > > > > >>>>>>>>>>>>>>>>>>>>>>>>> with new partitions, consumers using the > > > > >> latest > > > > >>>> auto > > > > >>>>>>>>>> offset > > > > >>>>>>>>>>>>>> reset > > > > >>>>>>>>>>>>>>>>>>>>> policy > > > > >>>>>>>>>>>>>>>>>>>>>>>>> will silently miss all records produced to > > > > >> those > > > > >>>>>>>>>> partitions > > > > >>>>>>>>>>>>>> before > > > > >>>>>>>>>>>>>>>>>>> the > > > > >>>>>>>>>>>>>>>>>>>>>>>>> consumer discovers them.". If a user sets > > > > >>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset=by_duration=1sec, the same > > > > >>>> record > > > > >>>>>> loss > > > > >>>>>>>>>>>>> issue > > > > >>>>>>>>>>>>>>>>> could > > > > >>>>>>>>>>>>>>>>>>>>> also > > > > >>>>>>>>>>>>>>>>>>>>>>>>> happen, right? > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> 2. I was thinking auto.offset.reset.new > > > > >>>> .partitions > > > > >>>>>> will > > > > >>>>>>>>>>> take > > > > >>>>>>>>>>>>>> the > > > > >>>>>>>>>>>>>>>>>>> same > > > > >>>>>>>>>>>>>>>>>>>>>>>>> values as auto.offset.reset. So a user could > > > > >> set > > > > >>>> it > > > > >>>>>>>>>>>>>> by_duration if > > > > >>>>>>>>>>>>>>>>>>>>> needed. > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Jun > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 14, 2026 at 4:06 PM Chia-Ping > > > > >> Tsai < > > > > >>>>>>>>>>>>>>>>> [email protected] > > > > >>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>> wrote: > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> hi Jun > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback. I might be missing > > > > >>>> something > > > > >>>>>>>>>>>>>> important > > > > >>>>>>>>>>>>>>>>>>> from > > > > >>>>>>>>>>>>>>>>>>>>> your > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> suggestion, so please bear with me as I try > > > > >> to > > > > >>>>>> clarify > > > > >>>>>>>>>> with > > > > >>>>>>>>>>>>> a > > > > >>>>>>>>>>>>>> few > > > > >>>>>>>>>>>>>>>>>>>>> questions: > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> 1. Is there a strong use case for extending > > > > >> this > > > > >>>>>> logic to > > > > >>>>>>>>>>>>>> other > > > > >>>>>>>>>>>>>>>>>>> reset > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> policies? Unlike latest, policies like > > > > >> earliest > > > > >>>> or > > > > >>>>>>>>>>>>> by_duration > > > > >>>>>>>>>>>>>>>>>>> don't > > > > >>>>>>>>>>>>>>>>>>>>> seem > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> to suffer from the same silent data loss > > > > >> issue > > > > >>>> when a > > > > >>>>>>>>>>>>>> partition > > > > >>>>>>>>>>>>>>>>> is > > > > >>>>>>>>>>>>>>>>>>>>> expanded. > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> 2. What values would we expect users to > > > > >>>> configure for > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.new.partitions? If they > > > > >> set > > > > >>>> it to > > > > >>>>>>>>>>>>> earliest > > > > >>>>>>>>>>>>>> or > > > > >>>>>>>>>>>>>>>>>>>>> latest, > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> we might run into the exact same edge cases. > > > > >> For > > > > >>>>>> example, > > > > >>>>>>>>>>>>> if a > > > > >>>>>>>>>>>>>>>>>>>>> consumer is > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> offline for a while and a new partition is > > > > >>>> created > > > > >>>>>> during > > > > >>>>>>>>>>>>> that > > > > >>>>>>>>>>>>>>>>>>>>> downtime, > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> the user might actually want to skip to > > > > >> latest > > > > >>>> when > > > > >>>>>>>>>>>>> resuming, > > > > >>>>>>>>>>>>>>>>>>> rather > > > > >>>>>>>>>>>>>>>>>>>>> than > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> reading from earliest just because the > > > > >> partition > > > > >>>> is > > > > >>>>>>>>>>>>>> technically > > > > >>>>>>>>>>>>>>>>>>>>> "new" to > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> the group. > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> This is exactly why we opted for introducing > > > > >> a > > > > >>>>>> max.age > > > > >>>>>>>>>>>>>> threshold. > > > > >>>>>>>>>>>>>>>>>>> It > > > > >>>>>>>>>>>>>>>>>>>>> gives > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> users a time-bound way to define what is > > > > >>>> genuinely > > > > >>>>>>>>>>> "hot/new" > > > > >>>>>>>>>>>>>> and > > > > >>>>>>>>>>>>>>>>>>>>> what is > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> just an old partition they haven't seen yet. > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Best, > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/05/14 20:48:09 Jun Rao via dev wrote: > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang, > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP. > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> I find auto.offset.reset.latest.max.age a > > > > >> bit > > > > >>>>>> weird. It > > > > >>>>>>>>>>>>> only > > > > >>>>>>>>>>>>>>>>>>>>> applies when > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset is latest. However, it > > > > >> seems > > > > >>>> that > > > > >>>>>> the > > > > >>>>>>>>>>>>>>>>> motivation > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> equally > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> applies when auto.offset.reset is set to > > > > >> other > > > > >>>>>> values > > > > >>>>>>>>>> like > > > > >>>>>>>>>>>>>>>>>>>>> by_duration. > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> The > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> intention is that we want to have a separate > > > > >>>> way to > > > > >>>>>>>>>>> control > > > > >>>>>>>>>>>>>>>>> newly > > > > >>>>>>>>>>>>>>>>>>>>> created > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> partitions vs existing partitions when the > > > > >> group > > > > >>>>>> starts. > > > > >>>>>>>>>>>>>> Have we > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> considered > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> adding a new config like > > > > >> auto.offset.reset.new > > > > >>>>>>>>>>> .partitions? > > > > >>>>>>>>>>>>>> If > > > > >>>>>>>>>>>>>>>>>>> this > > > > >>>>>>>>>>>>>>>>>>>>> new > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> config is not set, the offset reset policy > > > > >>>> defaults > > > > >>>>>> to > > > > >>>>>>>>>> the > > > > >>>>>>>>>>>>>>>>> policy > > > > >>>>>>>>>>>>>>>>>>>>> used > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> for > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> existing partitions. The user could set it > > > > >>>>>> explicitly to > > > > >>>>>>>>>>>>>>>>> customize > > > > >>>>>>>>>>>>>>>>>>>>> the > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> behavior for new partitions. > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Jun > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 7, 2026 at 5:07 AM 黃竣陽 < > > > > >>>>>> [email protected]> > > > > >>>>>>>>>>>>>> wrote: > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all, > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I’d like to manually bump this thread. > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards, > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 黃竣陽 <[email protected]> 於 2026年5月1日 > > > > >> 晚上10:37 > > > > >>>> 寫道： > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello all, > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback. > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ01/DJ02: > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> MetadataResponse bumps from v13 to v14. > > > > >> The > > > > >>>>>>>>>>>>>> PartitionMetadata > > > > >>>>>>>>>>>>>>>>>>>>> struct > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> gains a new > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> field PartitionAgeMs (int64, default -1), > > > > >>>> computed > > > > >>>>>>>>>>>>>> server-side > > > > >>>>>>>>>>>>>>>>>>> by > > > > >>>>>>>>>>>>>>>>>>>>> the > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> broker as > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> broker_current_time - > > > > >> partition_creation_time. > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Also add the consumer heartbeat flow. when > > > > >>>>>>>>>>>>>> MembershipManager > > > > >>>>>>>>>>>>>>>>>>>>> detects > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> a > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> newly assigned > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition, it explicitly invalidates the > > > > >>>> metadata > > > > >>>>>> for > > > > >>>>>>>>>>> the > > > > >>>>>>>>>>>>>>>>>>> affected > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> topic > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> and forces a fresh MetadataRequest > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> before making the offset reset decision, > > > > >> even > > > > >>>> if > > > > >>>>>> the > > > > >>>>>>>>>>>>> topic > > > > >>>>>>>>>>>>>> ID > > > > >>>>>>>>>>>>>>>>> is > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> already > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> in the cache. > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> MB0: > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> The consumer learns the broker's maximum > > > > >>>> supported > > > > >>>>>>>>>>>>>>>>>>>>> MetadataResponse > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> version via the > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> ApiVersions negotiation at connection > > > > >> time. > > > > >>>> If the > > > > >>>>>>>>>>>>>> negotiated > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> version is > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> unsupported, the consumer > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> knows the broker does not support > > > > >>>> PartitionAgeMs > > > > >>>>>> at > > > > >>>>>>>>>> all > > > > >>>>>>>>>>>>> and > > > > >>>>>>>>>>>>>>>>> can > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> throw an > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> UnsupportedVersionException > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> immediately, rather than silently falling > > > > >>>> back to > > > > >>>>>>>>>> latest > > > > >>>>>>>>>>>>>> and > > > > >>>>>>>>>>>>>>>>>>>>> risking > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> data loss without any operator-visible > > > > >> signal. > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> MB1/MB2/MB3: > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have addressed these changes in the KIP. > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards, > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <[email protected]> 於 > > > > >>>> 2026年4月29日 > > > > >>>>>>>>>>> 下午4:04 > > > > >>>>>>>>>>>>>> 寫道： > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi David > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I agree with the direction of moving the > > > > >>>> 'age' > > > > >>>>>>>>>>>>> resolution > > > > >>>>>>>>>>>>>>>>> from > > > > >>>>>>>>>>>>>>>>>>>>> the > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Heartbeat API to the Metadata API to keep > > > > >> the > > > > >>>>>> control > > > > >>>>>>>>>>>>> plane > > > > >>>>>>>>>>>>>>>>>>> clean. > > > > >>>>>>>>>>>>>>>>>>>>> The > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> main > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> trade-off, as we noted before, is > > > > >> introducing > > > > >>>>>>>>>>> inter-broker > > > > >>>>>>>>>>>>>>>>> clock > > > > >>>>>>>>>>>>>>>>>>>>> skew. > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> The > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Group Coordinator approach provided a > > > > >> single > > > > >>>>>> source of > > > > >>>>>>>>>>>>> truth > > > > >>>>>>>>>>>>>>>>> for > > > > >>>>>>>>>>>>>>>>>>>>> time. > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> However, realistically, this time skew > > > > >>>> should be > > > > >>>>>>>>>>>>>> negligible. > > > > >>>>>>>>>>>>>>>>>>>>> Given > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> that > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> the max.age threshold will likely be > > > > >>>> configured in > > > > >>>>>>>>>>> minutes > > > > >>>>>>>>>>>>>> or > > > > >>>>>>>>>>>>>>>>>>>>> hours, a > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> typical NTP skew (in milliseconds) between > > > > >>>> brokers > > > > >>>>>>>>>> won't > > > > >>>>>>>>>>>>>> impact > > > > >>>>>>>>>>>>>>>>>>> the > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> fallback decision. > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> David Jacot via dev < > > > > >> [email protected]> > > > > >>>> 於 > > > > >>>>>>>>>>>>> 2026年4月29日 > > > > >>>>>>>>>>>>>>>>>>> 下午3:29 > > > > >>>>>>>>>>>>>>>>>>>>> 寫道： > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all, > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP! > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry, I haven't really followed the > > > > >>>> previous > > > > >>>>>>>>>>>>>> conversation > > > > >>>>>>>>>>>>>>>>>>> but I > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> took a > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> quick look at this one. > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ01: I don't clearly understand the > > > > >> flow > > > > >>>> with > > > > >>>>>> the > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> ConsumerGroupHeartbeat > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> API after reading the KIP. There is a > > > > >> new > > > > >>>>>> boolean; > > > > >>>>>>>>>> the > > > > >>>>>>>>>>>>>> KIP > > > > >>>>>>>>>>>>>>>>>>>>> states > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> that > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition ages are returned only when > > > > >> this > > > > >>>>>> boolean > > > > >>>>>>>>>> is > > > > >>>>>>>>>>>>>> set. > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Implicitly, > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> this > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> means that when the consumer receives a > > > > >> new > > > > >>>>>>>>>> partition, > > > > >>>>>>>>>>>>> it > > > > >>>>>>>>>>>>>>>>> will > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> issue a > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> new > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> HB request with the boolean set to > > > > >> receive > > > > >>>> the > > > > >>>>>> ages. > > > > >>>>>>>>>>> Is > > > > >>>>>>>>>>>>>> my > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> understanding > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> correct? We should perhaps clarify the > > > > >> flow > > > > >>>> and > > > > >>>>>> also > > > > >>>>>>>>>>>>>> explain > > > > >>>>>>>>>>>>>>>>>>>>> how it > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> fits > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> into the existing flow (e.g. list > > > > >> offsets, > > > > >>>> fetch > > > > >>>>>>>>>>>>> offsets, > > > > >>>>>>>>>>>>>>>>>>> etc.). > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ02: It my understanding is correct, I > > > > >>>> wonder > > > > >>>>>> if > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API is the > > > > >> right > > > > >>>>>> place > > > > >>>>>>>>>> for > > > > >>>>>>>>>>>>>> this > > > > >>>>>>>>>>>>>>>>>>> given > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> that > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> a new > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> round trip is done anyway. > > > > >> Alternatively, it > > > > >>>>>> could > > > > >>>>>>>>>>>>> simply > > > > >>>>>>>>>>>>>>>>>>>>> include > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> the > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> metadata. Generally, we should be rather > > > > >>>>>> cautious > > > > >>>>>>>>>>> about > > > > >>>>>>>>>>>>>> not > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> overloading > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API with > > > > >>>> unrelated > > > > >>>>>>>>>>> concepts. > > > > >>>>>>>>>>>>>> The > > > > >>>>>>>>>>>>>>>>>>> API > > > > >>>>>>>>>>>>>>>>>>>>> is > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> a > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> control plane API for assigning or > > > > >> revoking > > > > >>>>>>>>>>> partitions. > > > > >>>>>>>>>>>>>> The > > > > >>>>>>>>>>>>>>>>>>> fact > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> that > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> we > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> don't want to add it to the > > > > >> corresponding > > > > >>>>>> Streams > > > > >>>>>>>>>> API > > > > >>>>>>>>>>>>>> also > > > > >>>>>>>>>>>>>>>>>>>>> suggests > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> something is not quite right. What > > > > >> would we > > > > >>>> do > > > > >>>>>> if we > > > > >>>>>>>>>>>>>> want to > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> support > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Streams in the future? > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> David > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 29, 2026 at 12:28 AM > > > > >> Muralidhar > > > > >>>>>> Basani > > > > >>>>>>>>>>> via > > > > >>>>>>>>>>>>>> dev > > > > >>>>>>>>>>>>>>>>> < > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [email protected]> wrote: > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jiunn, > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for this great kip. Good to > > > > >> know > > > > >>>>>> about > > > > >>>>>>>>>> the > > > > >>>>>>>>>>>>>> gap. > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-0 - why a new v2 version bump for > > > > >>>>>>>>>>>>>> RequestPartitionAges > > > > >>>>>>>>>>>>>>>>>>>>> field. > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Can a > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tagged field (for ex: on response, > > > > >>>>>> PartitionAges on > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> TopicPartitions) > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> be > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> used here and avoid version bump? > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-1 - For the new config, is there a > > > > >>>>>> recommended > > > > >>>>>>>>>>>>> value > > > > >>>>>>>>>>>>>> or > > > > >>>>>>>>>>>>>>>>> a > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> ConfigDef > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> validator? Probably it should based on > > > > >> the > > > > >>>>>>>>>>>>>>>>>>> metadata.max.age.ms > > > > >>>>>>>>>>>>>>>>>>>>> ? > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Sizing > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instructions can be part of javadocs I > > > > >>>> guess. > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-2 - (minor) As there are no changes > > > > >> to > > > > >>>> Kafka > > > > >>>>>>>>>>>>> Streams, > > > > >>>>>>>>>>>>>>>>>>> would > > > > >>>>>>>>>>>>>>>>>>>>> it > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> be > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> better > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to add this new config > > > > >>>>>>>>>>>>> auto.offset.reset.latest.max.age > > > > >>>>>>>>>>>>>> to > > > > >>>>>>>>>>>>>>>>>>> the > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> StreamsConfig block list > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS) > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> for a > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clear warning, incase users configure > > > > >> it? > > > > >>>> This > > > > >>>>>> is > > > > >>>>>>>>>> the > > > > >>>>>>>>>>>>>> most > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> familiar > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config and users might easily > > > > >>>>>> mistakenly > > > > >>>>>>>>>>>>>> configure > > > > >>>>>>>>>>>>>>>>>>>>> it. Or > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> may be > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it's not worth it to add. > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-3 - (minor) The phrasing "the > > > > >> consumer > > > > >>>> falls > > > > >>>>>>>>>> back > > > > >>>>>>>>>>>>> to > > > > >>>>>>>>>>>>>>>>>>>>> earliest" > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> reads as > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> if the config were being changed > > > > >>>> per-partition > > > > >>>>>>>>>> which > > > > >>>>>>>>>>>>>> isn't > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> supported. > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> May > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be rephrasing to something like > > > > >> "consumer > > > > >>>>>> resolves > > > > >>>>>>>>>>> the > > > > >>>>>>>>>>>>>>>>>>> initial > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> position to > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> start offset for that partition" as if > > > > >>>>>> earliest was > > > > >>>>>>>>>>>>>> applied > > > > >>>>>>>>>>>>>>>>>>> to > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> that > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition only and auto.offset.reset > > > > >>>> config is > > > > >>>>>>>>>>>>>> unchanged. > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks, > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Murali > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 28, 2026 at 2:48 PM 黃竣陽 < > > > > >>>>>>>>>>>>>> [email protected]> > > > > >>>>>>>>>>>>>>>>>>>>> wrote: > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi chia, > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have updated the KIP to include this > > > > >>>> change. > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards, > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <[email protected]> > > > > >> 於 > > > > >>>>>>>>>> 2026年4月28日 > > > > >>>>>>>>>>>>>> 晚上8:03 > > > > >>>>>>>>>>>>>>>>>>> 寫道： > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> chia_0: Should we expose the > > > > >> partition > > > > >>>>>> creation > > > > >>>>>>>>>>> time > > > > >>>>>>>>>>>>>> via > > > > >>>>>>>>>>>>>>>>>>> the > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Admin > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> API? > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I assume it would be valuable for > > > > >> users to > > > > >>>>>>>>>> diagnose > > > > >>>>>>>>>>>>> and > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> troubleshoot > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> the > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> behavior of > > > > >>>> auto.offset.reset.latest.max.age > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/04/28 10:47:58 黃竣陽 wrote: > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone, > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a discussion > > > > >> on > > > > >>>>>> KIP-1327 > > > > >>>>>>>>>>>>>> Prevent > > > > >>>>>>>>>>>>>>>>> Hot > > > > >>>>>>>>>>>>>>>>>>>>> Data > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Loss > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Partition Expansion for Latest Policy > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> < > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>> > > > > >>>>>>>>>> > > > > >>>>>> > > > > >>>> > > > > >> > > > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$ > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to introduces > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age, > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> a > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config that lets the > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> latest reset policy distinguish > > > > >> newly > > > > >>>>>> expanded > > > > >>>>>>>>>>>>> (hot) > > > > >>>>>>>>>>>>>>>>>>>>> partitions > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> from > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> long-existing (cold) ones. Partitions > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> younger than the configured > > > > >> threshold > > > > >>>>>>>>>>> automatically > > > > >>>>>>>>>>>>>> fall > > > > >>>>>>>>>>>>>>>>>>>>> back > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> to > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> earliest, preventing silent data loss > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during topic expansion without > > > > >> forcing a > > > > >>>>>> full > > > > >>>>>>>>>>>>>> historical > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> reprocess. > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards, > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>> > > > > >>>>>>>>>>> > > > > >>>>>>>>>> > > > > >>>>>>>>> > > > > >>>>>>> > > > > >>>>>>> > > > > >>>>>>> > > > > >>>>>> > > > > >>>>> > > > > >>>> > > > > >>> > > > > >> > > > > > > > > > > > > > >
