Hi Jiunn-Yang, I've just been re-reading this KIP to ensure that share group support is not needed. I've discovered that we made a mistake implementing KIP-932 in this area, so we'll rectify that and check whether it's really tight enough not to require KIP-1327.
One trivial comments for consistency. AS1: In kafka-consumer-groups.sh, unknown column data is represented by "-", not "N/A". In kafka-topics.sh, we do use "N/A". Thanks, Andrew On 2026/06/23 23:23:38 黃竣陽 wrote: > Hi all, > > Manually bumping this thread. > > Best Regards, > Jiunn-Yang > > > 黃竣陽 <[email protected]> 於 2026年6月17日 晚上9:17 寫道： > > > > Hello chia, > > > > Thanks for the feedback, I have updated the KIP. > > > > Best Regards, > > Jiunn-Yang > > > >> Chia-Ping Tsai <[email protected]> 於 2026年6月17日 凌晨12:47 寫道： > >> > >> hi Jiunn-Yang > >> > >>> When the config is set on a cluster that has not yet been upgraded... > >>> classification cannot occur... the consumer falls back to the base > >>> auto.offset.reset for the affected partitions. No exception is thrown, > >>> and no operational disruption results. > >> > >> Existing group can't take advantage of this excellent new configuration. > >> Allowing users to modify the group creation time might be overkill. > >> Instead, we could print a useful warning message to guide users. For > >> example, we can suggest that they re-create the group with their existing > >> committed offsets > >> > >>> Protocol changes > >> > >> Would you mind listing those RPC changes in a table format? > >> > >>> The full interaction matrix between the base policy and the new-partition > >>> policy is: > >> > >> Please add a filed to describe the target scenario when using these > >> policies > >> > >> Best, > >> Chia-Ping > >> > >> > >> On 2026/06/16 16:14:49 黃竣陽 wrote: > >>> Hello Jun, chia, > >>> > >>> Thanks for the feedback, I have updated the KIP for the new > >>> approach, PTAL > >>> > >>> Best Regards, > >>> Jiunn-Yang > >>> > >>>> Chia-Ping Tsai <[email protected]> 於 2026年6月16日 上午8:23 寫道： > >>>> > >>>> hi Jun > >>>> > >>>> Yes, your approach is great. I think the combination of latest (for > >>>> existing partitions) and by_duration (for new partitions) can address > >>>> 99% of the complaints I have heard regarding this issue. > >>>> > >>>> Also, leveraging the group creation time here opens the door to > >>>> implementing a new policy based on timestamp seek in the future, should > >>>> the community want to pursue that. > >>>> > >>>> Thanks for your patience and constructive feedback. We will update the > >>>> KIP accordingly. > >>>> > >>>> Best, Chia-Ping > >>>> > >>>>> Jun Rao via dev <[email protected]> 於 2026年6月16日 清晨5:11 寫道： > >>>>> > >>>>> Hi, Chia-Ping, > >>>>> > >>>>> Thanks for the reply. > >>>>> > >>>>> I agree that it's probably useful to allow a user to configure a > >>>>> different > >>>>> offset policy for existing partitions vs new partitions. However, using > >>>>> group creation time to capture that seems more intuitive. Here is > >>>>> another > >>>>> proposal: remove auto.offset.reset.max.age.ms and categorize new > >>>>> partitions > >>>>> based on group creation time. Introduce > >>>>> a new config auto.offset.reset.new.partitions whose values can be > >>>>> earliest, > >>>>> latest and by_duration, the same as auto.offset.reset. Users can set > >>>>> `auto.offset.reset.new.partitions` to `earliest` if they want to > >>>>> guarantee > >>>>> no data loss on new partitions. They can also use by_duration to set an > >>>>> upper bound on the backlog replayed, which can be different from that of > >>>>> the existing partitions. This will address your concern about too much > >>>>> backlog being replayed when the offsets are lost. What do you think? > >>>>> > >>>>> Jun > >>>>> > >>>>> > >>>>>> On Mon, Jun 15, 2026 at 10:39 AM Chia-Ping Tsai <[email protected]> > >>>>>> wrote: > >>>>>> > >>>>>> hi Jun > >>>>>> > >>>>>> The most important part of this story is how users should expect the > >>>>>> data > >>>>>> they can see when using the latest or by_duration policy with expanded > >>>>>> partitions. > >>>>>> > >>>>>> Yes, the by_duration policy can minimize data loss, but it is > >>>>>> non-deterministic, which means users will either read too many > >>>>>> historical > >>>>>> records from existing partitions or lose some records from expanded > >>>>>> partitions. > >>>>>> > >>>>>> Also, I agree that auto.offset.reset.max.age.ms > >>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$> > >>>>>> is a bit hard to understand, and that is why I preferred having a > >>>>>> whole new > >>>>>> policy based entirely on group creation time (KIP-1282) > >>>>>> > >>>>>> Best, > >>>>>> Chia-Ping > >>>>>> > >>>>>> Jun Rao via dev <[email protected]> 於 2026年6月16日週二 上午1:08寫道： > >>>>>> > >>>>>>> Hi, Chia-Ping and Jiunn-Yang, > >>>>>>> > >>>>>>> Thanks for the reply. I am still trying to understand the value of > >>>>>>> the new > >>>>>>> configs with the KIP. > >>>>>>> > >>>>>>> The motivation of the KIP is that a user doesn't want to miss the > >>>>>>> data if > >>>>>>> the backlog is small. The backlog of the existing partition is easy to > >>>>>>> understand because it relates to retention time. The backlog for the > >>>>>>> new > >>>>>>> partition is a bit subtle to understand since it depends on the > >>>>>>> metadata > >>>>>>> refresh delay. To set auto.offset.reset.max.age.ms > >>>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$>, > >>>>>>> the user needs to > >>>>>>> understand the metadata refresh delay on the consumer side and use it > >>>>>>> to > >>>>>>> set the config. > >>>>>>> > >>>>>>> Now, let's consider the alternative: setting the same value for the > >>>>>>> existing by_duration policy. The KIP lists three issues with this > >>>>>>> approach. > >>>>>>> 1. It computes the seek target client-side as now() - duration, which > >>>>>>> introduces clock skew across consumers and forces operators to choose > >>>>>>> overly large durations, causing unnecessary reprocessing. > >>>>>>> 2. The target timestamp is recomputed on each retry, so failed > >>>>>>> ListOffsetsRequest retries can shift the target forward and > >>>>>>> potentially > >>>>>>> miss records produced between attempts. > >>>>>>> 3. It applies uniformly to all partitions without committed offsets, > >>>>>>> and > >>>>>>> cannot distinguish newly expanded partitions from long-existing > >>>>>>> partitions > >>>>>>> newly assigned to the group, leading to unnecessary replay. > >>>>>>> > >>>>>>> Issues 1 and 2 are uncommon and can be mitigated by adding a bit > >>>>>>> buffer to > >>>>>>> the metadata refresh delay. We could also consider improving the > >>>>>>> implementation. For issue 3, the metadata refresh delay is typically > >>>>>>> low > >>>>>>> (in the order of minutes with the classic consumer and tens of seconds > >>>>>>> with > >>>>>>> the new consumer). If a user is ok with reading that much backlog for > >>>>>>> new > >>>>>>> partitions, it seems they will be ok doing the same for existing > >>>>>>> partitions. > >>>>>>> > >>>>>>> So, instead of introducing a new config, could we just reuse the > >>>>>>> existing > >>>>>>> config with better documentation and/or implementation? > >>>>>>> > >>>>>>> Jun > >>>>>>> > >>>>>>> > >>>>>>>> On Sat, Jun 13, 2026 at 12:19 AM 黃竣陽 <[email protected]> wrote: > >>>>>>> > >>>>>>>> Hello Jun, > >>>>>>>> > >>>>>>>> You're right that group creation time is the more intuitive answer at > >>>>>>>> first glance, > >>>>>>>> the KIP's own motivation talks about partitions that "predate the > >>>>>>>> group" > >>>>>>>> vs partitions > >>>>>>>> "created during group runtime," which directly points to a > >>>>>>> group-lifecycle > >>>>>>>> classifier. > >>>>>>>> I'd like to walk through why we landed on partition age, and the > >>>>>>>> trade-offs we considered. > >>>>>>>> > >>>>>>>> We evaluated three candidate signals: > >>>>>>>> > >>>>>>>> 1. `by_duration:5secs` > >>>>>>>> > >>>>>>>> This covers the metadata blindness window, but has issues the KIP > >>>>>>>> currently documents > >>>>>>>> under "Why not use `by_duration`?": > >>>>>>>> > >>>>>>>> - Client-side `now() - duration` introduces clock skew across > >>>>>>>> consumers. > >>>>>>>> - `ListOffsets` retries shift the target forward, potentially missing > >>>>>>>> records produced between > >>>>>>>> attempts. > >>>>>>>> - It applies uniformly to all partitions without committed offsets, > >>>>>>>> including pre-existing partitions > >>>>>>>> newly assigned to the group, causing unnecessary replay. > >>>>>>>> > >>>>>>>> 2. Group creation time as classifier > >>>>>>>> > >>>>>>>> This works cleanly when the consumer is actively running. Our concern > >>>>>>>> is the idle / late-rejoin case: > >>>>>>>> > >>>>>>>> T=0: Group created. > >>>>>>>> T=1..T=100: Consumer idle (down, disconnected, etc.). > >>>>>>>> T=50: Partition added during the idle window. > >>>>>>>> T=100: Consumer resumes. > >>>>>>>> > >>>>>>>> Under group creation time, the new partition is classified as new > >>>>>>>> (`50 > 0`) and reset to `earliest`, replaying everything from T=50. > >>>>>>>> But during `[T=1, T=100]`, base partitions also accumulated data that > >>>>>>>> the consumer accepts as lost — that is precisely the contract of > >>>>>>>> `auto.offset.reset=latest`. There is no principled reason to treat > >>>>>>>> the new partition differently; both contain backlog accumulated > >>>>>>>> during > >>>>>>>> the same idle window. > >>>>>>>> > >>>>>>>> This aligns with the "backlog is backlog” principle you raised in > >>>>>>>> the KIP-1282 thread: a `latest` user has tolerated some backlog on > >>>>>>>> every other partition during the same idle period; forcing 0-backlog > >>>>>>>> tolerance only on new partitions would be inconsistent with that > >>>>>>>> tolerance. > >>>>>>>> > >>>>>>>> 3. Partition age vs threshold > >>>>>>>> > >>>>>>>> Partition age corresponds to the actual silent data loss window, > >>>>>>>> the gap between partition creation and the consumer’s metadata > >>>>>>>> refresh. Within this window, data loss is genuinely silent: the > >>>>>>>> consumer had no opportunity to know about the partition. Outside this > >>>>>>>> window, missing data reflects either: > >>>>>>>> > >>>>>>>> - (a) the user’s tolerated cost of running with idle consumers, or > >>>>>>>> - (b) an operational issue to surface via monitoring, not via reset > >>>>>>> policy. > >>>>>>>> > >>>>>>>> We did not choose partition age because it is more elegant than group > >>>>>>>> creation time — we chose it because its failure mode (requires a > >>>>>>>> threshold) is > >>>>>>>> less invasive than the failure mode of group creation time (overrides > >>>>>>>> user-stated > >>>>>>>> `latest` intent during idle periods). > >>>>>>>> > >>>>>>>> Best Regards, > >>>>>>>> Jiunn-Yang > >>>>>>>> > >>>>>>>>> Chia-Ping Tsai <[email protected]> 於 2026年6月13日 上午11:52 寫道： > >>>>>>>>> > >>>>>>>>> Hi Jun, > >>>>>>>>> > >>>>>>>>> Relying on both creation times will create an inconsistent > >>>>>>>>> scenario. A > >>>>>>>>> consumer that lost all offsets due to a long sleep will seek to the > >>>>>>>>> beginning for the partitions created later than the group. > >>>>>>>>> > >>>>>>>>> That is why we initially proposed KIP-1282 to fix the inconsistency > >>>>>>>> using a > >>>>>>>>> whole new policy. Since KIP-1282 couldn't reach a consensus, > >>>>>>>>> KIP-1327 > >>>>>>>> goes > >>>>>>>>> back to using flexible configurations to prevent users from falling > >>>>>>> into > >>>>>>>>> that pitfall. > >>>>>>>>> > >>>>>>>>> Best, Chia-Ping > >>>>>>>>> > >>>>>>>>> Jun Rao via dev <[email protected]> 於 2026年6月13日週六 上午6:49寫道： > >>>>>>>>> > >>>>>>>>>> Hi, Jiunn-Yang, > >>>>>>>>>> > >>>>>>>>>> Thanks for the reply and sorry for the late reply. > >>>>>>>>>> > >>>>>>>>>> JR1. The design of auto.offset.reset.max.age.ms > >>>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$> > >>>>>>> still feels weird to > >>>>>>>> me. > >>>>>>>>>> It > >>>>>>>>>> categorizes partitions as new or existing based on the partition > >>>>>>>> creation > >>>>>>>>>> time. Intuitively, the categorization should be based on the group > >>>>>>>> creation > >>>>>>>>>> time: all partitions existing when the group is created are > >>>>>>>>>> existing > >>>>>>> and > >>>>>>>>>> all partitions created after the group creation are new partitions. > >>>>>>>>>> > >>>>>>>>>> Jun > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> On Tue, Jun 9, 2026 at 8:51 AM 黃竣陽 <[email protected]> wrote: > >>>>>>>>>> > >>>>>>>>>>> Hi all, > >>>>>>>>>>> > >>>>>>>>>>> Manually bumping this thread. If there is no further > >>>>>>>>>>> discussion, I will close the vote. > >>>>>>>>>>> > >>>>>>>>>>> Best Regards, > >>>>>>>>>>> Jiunn-Yang > >>>>>>>>>>> > >>>>>>>>>>>> 黃竣陽 <[email protected]> 於 2026年6月1日 晚上7:16 寫道： > >>>>>>>>>>>> > >>>>>>>>>>>> Hello Jian, > >>>>>>>>>>>> > >>>>>>>>>>>> Thanks for your feedback, > >>>>>>>>>>>> > >>>>>>>>>>>> Agreed, partition expansion is a common operational task, not an > >>>>>>> edge > >>>>>>>>>>>> case. I've updated the Motivation section accordingly. > >>>>>>>>>>>> > >>>>>>>>>>>> Best Regards, > >>>>>>>>>>>> Jiunn-Yang > >>>>>>>>>>>> > >>>>>>>>>>>>> jian fu <[email protected]> 於 2026年6月1日 下午5:49 寫道： > >>>>>>>>>>>>> > >>>>>>>>>>>>> Hi Jiunn-Yang: > >>>>>>>>>>>>> > >>>>>>>>>>>>> Thanks for the KIP. I think it would be useful to clarify that > >>>>>>> this > >>>>>>>>>> is a > >>>>>>>>>>>>> common scenario rather than an edge case, which further > >>>>>>> demonstrates > >>>>>>>>>> the > >>>>>>>>>>>>> need for this optimization. For example: > >>>>>>>>>>>>> A partition expansion is a common operational task in Kafka: To > >>>>>>>>>> balance > >>>>>>>>>>>>> resource utilization and cost, topics are typically created > >>>>>>>>>>>>> with a > >>>>>>>>>>> moderate > >>>>>>>>>>>>> default partition count. However, as traffic grows over time, it > >>>>>>> is > >>>>>>>>>>> often > >>>>>>>>>>>>> necessary to increase the number of partitions to accommodate > >>>>>>>>>>>>> the > >>>>>>>>>> higher > >>>>>>>>>>>>> workload. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Regards > >>>>>>>>>>>>> Jian > >>>>>>>>>>>>> > >>>>>>>>>>>>> 黃竣陽 <[email protected]> 于2026年5月30日周六 22:31写道： > >>>>>>>>>>>>> > >>>>>>>>>>>>>> Hello chia, > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Thanks for the comments, I have updated the KIP! > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Best Regards, > >>>>>>>>>>>>>> Jiunn-Yang > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Chia-Ping Tsai <[email protected]> 於 2026年5月30日 晚上8:29 寫道： > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Hi Jiunn-Yang, > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Would you mind removing the terms "hot" and "cold" when > >>>>>>> describing > >>>>>>>>>>>>>>> partitions in the KIP? I understand you are using them to > >>>>>>> describe > >>>>>>>>>> the > >>>>>>>>>>>>>>> "freshness" or the users' need for the records, but applying > >>>>>>> these > >>>>>>>>>>> terms > >>>>>>>>>>>>>> to > >>>>>>>>>>>>>>> the partition itself feels a bit unnatural. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> After all, in this scenario, users don't really care whether a > >>>>>>>>>>> partition > >>>>>>>>>>>>>> is > >>>>>>>>>>>>>>> newly expanded or not. Their only expectation is that they > >>>>>>>>>>>>>>> won't > >>>>>>>>>>> silently > >>>>>>>>>>>>>>> lose any live records produced to the topic during their > >>>>>>>>>>>>>>> active > >>>>>>>>>>>>>> consumption. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Best, Chia-Ping > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> 黃竣陽 <[email protected]> 於 2026年5月30日週六 下午12:30寫道： > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Hello Jun, > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Thanks for the feedback, I have updated the KIP motivation > >>>>>>>> section. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Best Regards, > >>>>>>>>>>>>>>>> Jiunn-Yang > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Jun Rao via dev <[email protected]> 於 2026年5月30日 凌晨1:12 > >>>>>>> 寫道： > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Hi, Jiunn-Yang, > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Thanks for the reply. I think we need a stronger motivation > >>>>>>> for > >>>>>>>>>> the > >>>>>>>>>>>>>> KIP. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> The KIP says "The core insight is that not all partitions > >>>>>>> without > >>>>>>>>>> a > >>>>>>>>>>>>>>>>> committed offset are the same. A newly expanded partition > >>>>>>> (hot) > >>>>>>>> is > >>>>>>>>>>>>>>>>> fundamentally different from a partition the consumer has > >>>>>>> never > >>>>>>>>>> seen > >>>>>>>>>>>>>>>>> because it predates the group (cold)." Why is the hot > >>>>>>> partition > >>>>>>>>>>>>>>>>> fundamentally different from the cold? > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> The KIP says "The existing by_duration policy is also > >>>>>>>> insufficient > >>>>>>>>>>>>>>>> because: > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> - The calculated seek time (now() - duration) varies across > >>>>>>> nodes > >>>>>>>>>>> due > >>>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>>> clock skew. To be safe, users must set an overly large > >>>>>>> duration, > >>>>>>>>>>>>>>>> causing > >>>>>>>>>>>>>>>>> unnecessary reprocessing. > >>>>>>>>>>>>>>>>> - On network errors, the client recalculates the seek time > >>>>>>>>>>>>>>>>> on > >>>>>>>>>> retry, > >>>>>>>>>>>>>>>>> shifting the target timestamp forward and risking data > >>>>>>>>>>>>>>>>> loss." > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> However, both of these situations are rare. If these issues > >>>>>>>>>> persist, > >>>>>>>>>>>>>> more > >>>>>>>>>>>>>>>>> severe problems likely exist elsewhere. Rare situations > >>>>>>>>>>>>>>>>> don't > >>>>>>>>>> need a > >>>>>>>>>>>>>>>> common > >>>>>>>>>>>>>>>>> solution. If users care about those rare situations, they > >>>>>>>>>>>>>>>>> can > >>>>>>>>>>> implement > >>>>>>>>>>>>>>>>> customized logic using > >>>>>>>>>>>>>> ConsumerRebalanceListener.onPartitionsAssigned(). > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Jun > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> On Sun, May 17, 2026 at 6:50 AM 黃竣陽 <[email protected]> > >>>>>>> wrote: > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Hello chia, > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Thanks for the feedback, > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> If the creation time exists, the returned value should > >>>>>>> always > >>>>>>>> be > >>>>>>>>>>>>>>>> greater > >>>>>>>>>>>>>>>>>> than or equal to zero, right? > >>>>>>>>>>>>>>>>>> I have explicitly mentioned this in the KIP. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> New Old (MetadataResponse v0–13) positive any > >>>>>>>>>>> field > >>>>>>>>>>>>>>>>>> absent UnsupportedVersionException > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> The earliest point at which we can detect the version > >>>>>>> mismatch > >>>>>>>> is > >>>>>>>>>>>>>> during > >>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>> first metadata fetch after assignment, which occurs inside > >>>>>>>>>> poll(). > >>>>>>>>>>>>>>>>>> Therefore, the > >>>>>>>>>>>>>>>>>> user would encounter an UnsupportedVersionException from > >>>>>>> poll(). > >>>>>>>>>>> I’ll > >>>>>>>>>>>>>>>>>> clarify this in the KIP. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Best Regards, > >>>>>>>>>>>>>>>>>> Jiunn-Yang > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <[email protected]> 於 2026年5月17日 下午4:50 > >>>>>>> 寫道： > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> hi Jiunn > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> PartitionAgeMs (int64, default -1): The age of this > >>>>>>> partition > >>>>>>>>>> in > >>>>>>>>>>>>>>>>>> milliseconds, computed server-side by the broker as > >>>>>>>>>>>>>> broker_current_time > >>>>>>>>>>>>>>>> - > >>>>>>>>>>>>>>>>>> partition_creation_time. Returns -1 if the broker does not > >>>>>>>>>> support > >>>>>>>>>>>>>> this > >>>>>>>>>>>>>>>>>> feature or the partition creation time is unknown. > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> If the creation time exists, the returned value should > >>>>>>> always > >>>>>>>> be > >>>>>>>>>>>>>>>> greater > >>>>>>>>>>>>>>>>>> than or equal to zero, right? > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> New Old (MetadataResponse v0–13) positive any > >>>>>>>>>>> field > >>>>>>>>>>>>>>>>>> absent UnsupportedVersionException > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Will user encounter UnsupportedVersionException when > >>>>>>>>>>>>>>>>>>> calling > >>>>>>>>>>>>>> `poll()`? > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Best, > >>>>>>>>>>>>>>>>>>> Chia-Ping > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> On 2026/05/16 04:30:49 黃竣陽 wrote: > >>>>>>>>>>>>>>>>>>>> Hello Jun, chia, > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> I've updated KIP-1327 with a design change based on the > >>>>>>>>>>> discussion > >>>>>>>>>>>>>>>>>>>> feedback. > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> The updated design decouples the new-partition reset > >>>>>>> behavior > >>>>>>>>>>> from > >>>>>>>>>>>>>>>>>>>> the base auto.offset.reset policy: > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> - auto.offset.reset.max.age.ms > >>>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$> > >>>>>>> now applies to all > >>>>>>>>>>> auto.offset.reset > >>>>>>>>>>>>>>>>>> values > >>>>>>>>>>>>>>>>>>>> (latest, earliest, by_duration, none). > >>>>>>>>>>>>>>>>>>>> - For new ("hot") partitions, the consumer resets to > >>>>>>>>>>>>>>>>>> auto.offset.reset.new > >>>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$> > >>>>>>> .partitions > >>>>>>>>>>>>>>>>>>>> config setting > >>>>>>>>>>>>>>>>>>>> - For existing ("cold") partitions, the base > >>>>>>> auto.offset.reset > >>>>>>>>>>>>>> policy > >>>>>>>>>>>>>>>>>> continues > >>>>>>>>>>>>>>>>>>>> to apply unchanged. > >>>>>>>>>>>>>>>>>>>> - The new-partition reset behavior is represented by a > >>>>>>>> separate > >>>>>>>>>>>>>>>>>> internal config > >>>>>>>>>>>>>>>>>>>> (auto.offset.reset.new > >>>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>.partitions, > >>>>>>> currently fixed to > >>>>>>>>>> earliest). > >>>>>>>>>>>>>> This > >>>>>>>>>>>>>>>>>> decoupled design makes > >>>>>>>>>>>>>>>>>>>> it straightforward to promote the behavior to a public > >>>>>>>>>>> user-facing > >>>>>>>>>>>>>>>>>> configuration in a future KIP. > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Best Regards, > >>>>>>>>>>>>>>>>>>>> Jiunn-Yang > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <[email protected]> 於 2026年5月16日 清晨7:46 > >>>>>>> 寫道： > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> hi Jun > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> I see what you mean now. The proposal from me is listed > >>>>>>>> below: > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> 1) Add auto.offset.reset.new > >>>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>.partitions > >>>>>>> with a default value > >>>>>>>>>> of > >>>>>>>>>>>>>>>>>> earliest. It fixes the data loss from both by_duration and > >>>>>>>>>> latest, > >>>>>>>>>>> and > >>>>>>>>>>>>>>>> it > >>>>>>>>>>>>>>>>>> does not change the logic of auto.offset.reset=earliest. > >>>>>>>>>>>>>>>>>>>>> 2) Mark auto.offset.reset.new > >>>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>.partitions > >>>>>>> as an internal > >>>>>>>>>>>>>>>>>> configuration. auto.offset.reset.new > >>>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$> > >>>>>>> .partitions=earliest > >>>>>>>> already > >>>>>>>>>>>>>>>>>> addresses the issue, and we can discuss the use cases of > >>>>>>> other > >>>>>>>>>>> values > >>>>>>>>>>>>>>>> in a > >>>>>>>>>>>>>>>>>> separate KIP. > >>>>>>>>>>>>>>>>>>>>> 3) Both configs, auto.offset.reset.new > >>>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>.partitions > >>>>>>> and > >>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age.ms > >>>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.latest.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfu9JSP4l$>, > >>>>>>> will be applied to all for > >>>>>>>>>>>>>>>>>> consistency. > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> WDYT? > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> On 2026/05/15 20:53:20 Jun Rao via dev wrote: > >>>>>>>>>>>>>>>>>>>>>> Hi, Chia-Ping, > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> Thanks for the reply. > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> 1. In the motivation section, the KIP says "When a > >>>>>>>>>>>>>>>>>>>>>> Kafka > >>>>>>>>>> topic > >>>>>>>>>>> is > >>>>>>>>>>>>>>>>>> expanded > >>>>>>>>>>>>>>>>>>>>>> with new partitions, consumers using the latest auto > >>>>>>> offset > >>>>>>>>>>> reset > >>>>>>>>>>>>>>>>>> policy > >>>>>>>>>>>>>>>>>>>>>> will silently miss all records produced to those > >>>>>>> partitions > >>>>>>>>>>> before > >>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>>> consumer discovers them.". If a user sets > >>>>>>>>>>>>>>>>>>>>>> auto.offset.reset=by_duration=1sec, the same record > >>>>>>>>>>>>>>>>>>>>>> loss > >>>>>>>>>> issue > >>>>>>>>>>>>>> could > >>>>>>>>>>>>>>>>>> also > >>>>>>>>>>>>>>>>>>>>>> happen, right? > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> 2. I was thinking auto.offset.reset.new > >>>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>.partitions > >>>>>>> will > >>>>>>>> take > >>>>>>>>>>> the > >>>>>>>>>>>>>>>> same > >>>>>>>>>>>>>>>>>>>>>> values as auto.offset.reset. So a user could set it > >>>>>>>>>>> by_duration if > >>>>>>>>>>>>>>>>>> needed. > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> Jun > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> On Thu, May 14, 2026 at 4:06 PM Chia-Ping Tsai < > >>>>>>>>>>>>>> [email protected] > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> hi Jun > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback. I might be missing something > >>>>>>>>>>> important > >>>>>>>>>>>>>>>> from > >>>>>>>>>>>>>>>>>> your > >>>>>>>>>>>>>>>>>>>>>>> suggestion, so please bear with me as I try to clarify > >>>>>>> with > >>>>>>>>>> a > >>>>>>>>>>> few > >>>>>>>>>>>>>>>>>> questions: > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> 1. Is there a strong use case for extending this logic > >>>>>>> to > >>>>>>>>>>> other > >>>>>>>>>>>>>>>> reset > >>>>>>>>>>>>>>>>>>>>>>> policies? Unlike latest, policies like earliest or > >>>>>>>>>> by_duration > >>>>>>>>>>>>>>>> don't > >>>>>>>>>>>>>>>>>> seem > >>>>>>>>>>>>>>>>>>>>>>> to suffer from the same silent data loss issue when a > >>>>>>>>>>> partition > >>>>>>>>>>>>>> is > >>>>>>>>>>>>>>>>>> expanded. > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> 2. What values would we expect users to configure for > >>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.new > >>>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>.partitions? > >>>>>>> If they set it to > >>>>>>>>>> earliest > >>>>>>>>>>> or > >>>>>>>>>>>>>>>>>> latest, > >>>>>>>>>>>>>>>>>>>>>>> we might run into the exact same edge cases. For > >>>>>>> example, > >>>>>>>>>> if a > >>>>>>>>>>>>>>>>>> consumer is > >>>>>>>>>>>>>>>>>>>>>>> offline for a while and a new partition is created > >>>>>>> during > >>>>>>>>>> that > >>>>>>>>>>>>>>>>>> downtime, > >>>>>>>>>>>>>>>>>>>>>>> the user might actually want to skip to latest when > >>>>>>>>>> resuming, > >>>>>>>>>>>>>>>> rather > >>>>>>>>>>>>>>>>>> than > >>>>>>>>>>>>>>>>>>>>>>> reading from earliest just because the partition is > >>>>>>>>>>> technically > >>>>>>>>>>>>>>>>>> "new" to > >>>>>>>>>>>>>>>>>>>>>>> the group. > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> This is exactly why we opted for introducing a max.age > >>>>>>>>>>> threshold. > >>>>>>>>>>>>>>>> It > >>>>>>>>>>>>>>>>>> gives > >>>>>>>>>>>>>>>>>>>>>>> users a time-bound way to define what is genuinely > >>>>>>>> "hot/new" > >>>>>>>>>>> and > >>>>>>>>>>>>>>>>>> what is > >>>>>>>>>>>>>>>>>>>>>>> just an old partition they haven't seen yet. > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> Best, > >>>>>>>>>>>>>>>>>>>>>>> Chia-Ping > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> On 2026/05/14 20:48:09 Jun Rao via dev wrote: > >>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang, > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP. > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> I find auto.offset.reset.latest.max.age a bit weird. > >>>>>>>>>>>>>>>>>>>>>>>> It > >>>>>>>>>> only > >>>>>>>>>>>>>>>>>> applies when > >>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset is latest. However, it seems that > >>>>>>>>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>> motivation > >>>>>>>>>>>>>>>>>>>>>>> equally > >>>>>>>>>>>>>>>>>>>>>>>> applies when auto.offset.reset is set to other values > >>>>>>> like > >>>>>>>>>>>>>>>>>> by_duration. > >>>>>>>>>>>>>>>>>>>>>>> The > >>>>>>>>>>>>>>>>>>>>>>>> intention is that we want to have a separate way to > >>>>>>>> control > >>>>>>>>>>>>>> newly > >>>>>>>>>>>>>>>>>> created > >>>>>>>>>>>>>>>>>>>>>>>> partitions vs existing partitions when the group > >>>>>>> starts. > >>>>>>>>>>> Have we > >>>>>>>>>>>>>>>>>>>>>>> considered > >>>>>>>>>>>>>>>>>>>>>>>> adding a new config like auto.offset.reset.new > >>>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$> > >>>>>>>> .partitions? > >>>>>>>>>>> If > >>>>>>>>>>>>>>>> this > >>>>>>>>>>>>>>>>>> new > >>>>>>>>>>>>>>>>>>>>>>>> config is not set, the offset reset policy defaults > >>>>>>>>>>>>>>>>>>>>>>>> to > >>>>>>> the > >>>>>>>>>>>>>> policy > >>>>>>>>>>>>>>>>>> used > >>>>>>>>>>>>>>>>>>>>>>> for > >>>>>>>>>>>>>>>>>>>>>>>> existing partitions. The user could set it explicitly > >>>>>>> to > >>>>>>>>>>>>>> customize > >>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>>>>> behavior for new partitions. > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> Jun > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 7, 2026 at 5:07 AM 黃竣陽 > >>>>>>>>>>>>>>>>>>>>>>>> <[email protected] > >>>>>>>> > >>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> Hi all, > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> I’d like to manually bump this thread. > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> Best Regards, > >>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> 黃竣陽 <[email protected]> 於 2026年5月1日 晚上10:37 寫道： > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> Hello all, > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback. > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> DJ01/DJ02: > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> MetadataResponse bumps from v13 to v14. The > >>>>>>>>>>> PartitionMetadata > >>>>>>>>>>>>>>>>>> struct > >>>>>>>>>>>>>>>>>>>>>>>>> gains a new > >>>>>>>>>>>>>>>>>>>>>>>>>> field PartitionAgeMs (int64, default -1), computed > >>>>>>>>>>> server-side > >>>>>>>>>>>>>>>> by > >>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>>>>>> broker as > >>>>>>>>>>>>>>>>>>>>>>>>>> broker_current_time - partition_creation_time. > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> Also add the consumer heartbeat flow. when > >>>>>>>>>>> MembershipManager > >>>>>>>>>>>>>>>>>> detects > >>>>>>>>>>>>>>>>>>>>>>> a > >>>>>>>>>>>>>>>>>>>>>>>>> newly assigned > >>>>>>>>>>>>>>>>>>>>>>>>>> partition, it explicitly invalidates the metadata > >>>>>>>>>>>>>>>>>>>>>>>>>> for > >>>>>>>> the > >>>>>>>>>>>>>>>> affected > >>>>>>>>>>>>>>>>>>>>>>> topic > >>>>>>>>>>>>>>>>>>>>>>>>> and forces a fresh MetadataRequest > >>>>>>>>>>>>>>>>>>>>>>>>>> before making the offset reset decision, even if > >>>>>>>>>>>>>>>>>>>>>>>>>> the > >>>>>>>>>> topic > >>>>>>>>>>> ID > >>>>>>>>>>>>>> is > >>>>>>>>>>>>>>>>>>>>>>> already > >>>>>>>>>>>>>>>>>>>>>>>>> in the cache. > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> MB0: > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> The consumer learns the broker's maximum supported > >>>>>>>>>>>>>>>>>> MetadataResponse > >>>>>>>>>>>>>>>>>>>>>>>>> version via the > >>>>>>>>>>>>>>>>>>>>>>>>>> ApiVersions negotiation at connection time. If the > >>>>>>>>>>> negotiated > >>>>>>>>>>>>>>>>>>>>>>> version is > >>>>>>>>>>>>>>>>>>>>>>>>> unsupported, the consumer > >>>>>>>>>>>>>>>>>>>>>>>>>> knows the broker does not support PartitionAgeMs at > >>>>>>> all > >>>>>>>>>> and > >>>>>>>>>>>>>> can > >>>>>>>>>>>>>>>>>>>>>>> throw an > >>>>>>>>>>>>>>>>>>>>>>>>> UnsupportedVersionException > >>>>>>>>>>>>>>>>>>>>>>>>>> immediately, rather than silently falling back to > >>>>>>> latest > >>>>>>>>>>> and > >>>>>>>>>>>>>>>>>> risking > >>>>>>>>>>>>>>>>>>>>>>>>> data loss without any operator-visible signal. > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> MB1/MB2/MB3: > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> I have addressed these changes in the KIP. > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards, > >>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <[email protected]> 於 2026年4月29日 > >>>>>>>> 下午4:04 > >>>>>>>>>>> 寫道： > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> hi David > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> I agree with the direction of moving the 'age' > >>>>>>>>>> resolution > >>>>>>>>>>>>>> from > >>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>>>>>> Heartbeat API to the Metadata API to keep the > >>>>>>>>>>>>>>>>>>>>>>>>> control > >>>>>>>>>> plane > >>>>>>>>>>>>>>>> clean. > >>>>>>>>>>>>>>>>>> The > >>>>>>>>>>>>>>>>>>>>>>> main > >>>>>>>>>>>>>>>>>>>>>>>>> trade-off, as we noted before, is introducing > >>>>>>>> inter-broker > >>>>>>>>>>>>>> clock > >>>>>>>>>>>>>>>>>> skew. > >>>>>>>>>>>>>>>>>>>>>>> The > >>>>>>>>>>>>>>>>>>>>>>>>> Group Coordinator approach provided a single source > >>>>>>>>>>>>>>>>>>>>>>>>> of > >>>>>>>>>> truth > >>>>>>>>>>>>>> for > >>>>>>>>>>>>>>>>>> time. > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> However, realistically, this time skew should be > >>>>>>>>>>> negligible. > >>>>>>>>>>>>>>>>>> Given > >>>>>>>>>>>>>>>>>>>>>>> that > >>>>>>>>>>>>>>>>>>>>>>>>> the max.age threshold will likely be configured in > >>>>>>>> minutes > >>>>>>>>>>> or > >>>>>>>>>>>>>>>>>> hours, a > >>>>>>>>>>>>>>>>>>>>>>>>> typical NTP skew (in milliseconds) between brokers > >>>>>>> won't > >>>>>>>>>>> impact > >>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>>>>>> fallback decision. > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Best, > >>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> David Jacot via dev <[email protected]> 於 > >>>>>>>>>> 2026年4月29日 > >>>>>>>>>>>>>>>> 下午3:29 > >>>>>>>>>>>>>>>>>> 寫道： > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all, > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP! > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry, I haven't really followed the previous > >>>>>>>>>>> conversation > >>>>>>>>>>>>>>>> but I > >>>>>>>>>>>>>>>>>>>>>>> took a > >>>>>>>>>>>>>>>>>>>>>>>>>>>> quick look at this one. > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ01: I don't clearly understand the flow with > >>>>>>>>>>>>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>>>>>> ConsumerGroupHeartbeat > >>>>>>>>>>>>>>>>>>>>>>>>>>>> API after reading the KIP. There is a new > >>>>>>>>>>>>>>>>>>>>>>>>>>>> boolean; > >>>>>>> the > >>>>>>>>>>> KIP > >>>>>>>>>>>>>>>>>> states > >>>>>>>>>>>>>>>>>>>>>>> that > >>>>>>>>>>>>>>>>>>>>>>>>>>>> partition ages are returned only when this > >>>>>>>>>>>>>>>>>>>>>>>>>>>> boolean > >>>>>>> is > >>>>>>>>>>> set. > >>>>>>>>>>>>>>>>>>>>>>> Implicitly, > >>>>>>>>>>>>>>>>>>>>>>>>> this > >>>>>>>>>>>>>>>>>>>>>>>>>>>> means that when the consumer receives a new > >>>>>>> partition, > >>>>>>>>>> it > >>>>>>>>>>>>>> will > >>>>>>>>>>>>>>>>>>>>>>> issue a > >>>>>>>>>>>>>>>>>>>>>>>>> new > >>>>>>>>>>>>>>>>>>>>>>>>>>>> HB request with the boolean set to receive the > >>>>>>> ages. > >>>>>>>> Is > >>>>>>>>>>> my > >>>>>>>>>>>>>>>>>>>>>>>>> understanding > >>>>>>>>>>>>>>>>>>>>>>>>>>>> correct? We should perhaps clarify the flow and > >>>>>>> also > >>>>>>>>>>> explain > >>>>>>>>>>>>>>>>>> how it > >>>>>>>>>>>>>>>>>>>>>>>>> fits > >>>>>>>>>>>>>>>>>>>>>>>>>>>> into the existing flow (e.g. list offsets, fetch > >>>>>>>>>> offsets, > >>>>>>>>>>>>>>>> etc.). > >>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ02: It my understanding is correct, I wonder if > >>>>>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API is the right place > >>>>>>> for > >>>>>>>>>>> this > >>>>>>>>>>>>>>>> given > >>>>>>>>>>>>>>>>>>>>>>> that > >>>>>>>>>>>>>>>>>>>>>>>>> a new > >>>>>>>>>>>>>>>>>>>>>>>>>>>> round trip is done anyway. Alternatively, it > >>>>>>>>>>>>>>>>>>>>>>>>>>>> could > >>>>>>>>>> simply > >>>>>>>>>>>>>>>>>> include > >>>>>>>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>>>>>>>>> metadata. Generally, we should be rather cautious > >>>>>>>> about > >>>>>>>>>>> not > >>>>>>>>>>>>>>>>>>>>>>> overloading > >>>>>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API with unrelated > >>>>>>>> concepts. > >>>>>>>>>>> The > >>>>>>>>>>>>>>>> API > >>>>>>>>>>>>>>>>>> is > >>>>>>>>>>>>>>>>>>>>>>> a > >>>>>>>>>>>>>>>>>>>>>>>>>>>> control plane API for assigning or revoking > >>>>>>>> partitions. > >>>>>>>>>>> The > >>>>>>>>>>>>>>>> fact > >>>>>>>>>>>>>>>>>>>>>>> that > >>>>>>>>>>>>>>>>>>>>>>>>> we > >>>>>>>>>>>>>>>>>>>>>>>>>>>> don't want to add it to the corresponding Streams > >>>>>>> API > >>>>>>>>>>> also > >>>>>>>>>>>>>>>>>> suggests > >>>>>>>>>>>>>>>>>>>>>>>>>>>> something is not quite right. What would we do if > >>>>>>> we > >>>>>>>>>>> want to > >>>>>>>>>>>>>>>>>>>>>>> support > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Streams in the future? > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, > >>>>>>>>>>>>>>>>>>>>>>>>>>>> David > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 29, 2026 at 12:28 AM Muralidhar > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Basani > >>>>>>>> via > >>>>>>>>>>> dev > >>>>>>>>>>>>>> < > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> [email protected]> wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jiunn, > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for this great kip. Good to know about > >>>>>>> the > >>>>>>>>>>> gap. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-0 - why a new v2 version bump for > >>>>>>>>>>> RequestPartitionAges > >>>>>>>>>>>>>>>>>> field. > >>>>>>>>>>>>>>>>>>>>>>> Can a > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> tagged field (for ex: on response, PartitionAges > >>>>>>> on > >>>>>>>>>>>>>>>>>>>>>>> TopicPartitions) > >>>>>>>>>>>>>>>>>>>>>>>>> be > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> used here and avoid version bump? > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-1 - For the new config, is there a > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> recommended > >>>>>>>>>> value > >>>>>>>>>>> or > >>>>>>>>>>>>>> a > >>>>>>>>>>>>>>>>>>>>>>> ConfigDef > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> validator? Probably it should based on the > >>>>>>>>>>>>>>>> metadata.max.age.ms > >>>>>>> <https://urldefense.com/v3/__http://metadata.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkflKEb5SK$> > >>>>>>>>>>>>>>>>>> ? > >>>>>>>>>>>>>>>>>>>>>>>>> Sizing > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> instructions can be part of javadocs I guess. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-2 - (minor) As there are no changes to Kafka > >>>>>>>>>> Streams, > >>>>>>>>>>>>>>>> would > >>>>>>>>>>>>>>>>>> it > >>>>>>>>>>>>>>>>>>>>>>> be > >>>>>>>>>>>>>>>>>>>>>>>>> better > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> to add this new config > >>>>>>>>>> auto.offset.reset.latest.max.age > >>>>>>>>>>> to > >>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> StreamsConfig block list > >>>>>>>>>>>>>>>>>>>>>>> (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS) > >>>>>>>>>>>>>>>>>>>>>>>>> for a > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> clear warning, incase users configure it? This > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> is > >>>>>>> the > >>>>>>>>>>> most > >>>>>>>>>>>>>>>>>>>>>>> familiar > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config and users might easily > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> mistakenly > >>>>>>>>>>> configure > >>>>>>>>>>>>>>>>>> it. Or > >>>>>>>>>>>>>>>>>>>>>>>>> may be > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> it's not worth it to add. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-3 - (minor) The phrasing "the consumer falls > >>>>>>> back > >>>>>>>>>> to > >>>>>>>>>>>>>>>>>> earliest" > >>>>>>>>>>>>>>>>>>>>>>>>> reads as > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> if the config were being changed per-partition > >>>>>>> which > >>>>>>>>>>> isn't > >>>>>>>>>>>>>>>>>>>>>>> supported. > >>>>>>>>>>>>>>>>>>>>>>>>> May > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> be rephrasing to something like "consumer > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> resolves > >>>>>>>> the > >>>>>>>>>>>>>>>> initial > >>>>>>>>>>>>>>>>>>>>>>>>> position to > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> start offset for that partition" as if earliest > >>>>>>> was > >>>>>>>>>>> applied > >>>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>>>>>>>>> that > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition only and auto.offset.reset config is > >>>>>>>>>>> unchanged. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Murali > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 28, 2026 at 2:48 PM 黃竣陽 < > >>>>>>>>>>> [email protected]> > >>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi chia, > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have updated the KIP to include this change. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards, > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <[email protected]> 於 > >>>>>>> 2026年4月28日 > >>>>>>>>>>> 晚上8:03 > >>>>>>>>>>>>>>>> 寫道： > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> chia_0: Should we expose the partition > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> creation > >>>>>>>> time > >>>>>>>>>>> via > >>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>>>> Admin > >>>>>>>>>>>>>>>>>>>>>>>>> API? > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I assume it would be valuable for users to > >>>>>>> diagnose > >>>>>>>>>> and > >>>>>>>>>>>>>>>>>>>>>>> troubleshoot > >>>>>>>>>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> behavior of auto.offset.reset.latest.max.age > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/04/28 10:47:58 黃竣陽 wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone, > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a discussion on > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KIP-1327 > >>>>>>>>>>> Prevent > >>>>>>>>>>>>>> Hot > >>>>>>>>>>>>>>>>>> Data > >>>>>>>>>>>>>>>>>>>>>>>>> Loss > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> on > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Partition Expansion for Latest Policy > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> < > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$ > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to introduces > >>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age, > >>>>>>>>>>>>>>>>>>>>>>>>> a > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config that lets the > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> latest reset policy distinguish newly > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> expanded > >>>>>>>>>> (hot) > >>>>>>>>>>>>>>>>>> partitions > >>>>>>>>>>>>>>>>>>>>>>>>> from > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> long-existing (cold) ones. Partitions > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> younger than the configured threshold > >>>>>>>> automatically > >>>>>>>>>>> fall > >>>>>>>>>>>>>>>>>> back > >>>>>>>>>>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> earliest, preventing silent data loss > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during topic expansion without forcing a full > >>>>>>>>>>> historical > >>>>>>>>>>>>>>>>>>>>>>> reprocess. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards, > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>> > >>> > >>> > >>> > > > > >
