> It inspires me to think about mirror clusters. What if the mirror group is > created 'after' the expanded partitions? In this scenario, the new > configuration 'max.age.ms' could handle it if users are aware of the > migration and set a suitable time interval. Of course, this would require > both creation times to be exposed to clients.
Thinking more about the mirror cluster scenario we discussed, I realize it introduces significant complexity. Handling it properly would require a tolerance time window rather than a strict time interval to account for replication lag. To prevent scope creep and keep KIP-1327 focused on its primary goal, I think it would be best to defer the multi-cluster/mirroring scenario and explore those approaches in a separate KIP On 2026/06/24 21:29:19 Chia-Ping Tsai wrote: > hi all, > > > JR3. The KIP exposes the group/partition creation time to the client. I am > > not sure if there are other use cases for those timestamps. An alternative > > is to avoid exposing those timestamps to the client. The group coordinator > > will categorize the partitions using those timestamps and include the > > categorization in the assigned partitions in GroupHeartbeatResponse. > > It inspires me to think about mirror clusters. What if the mirror group is > created 'after' the expanded partitions? In this scenario, the new > configuration 'max.age.ms' could handle it if users are aware of the > migration and set a suitable time interval. Of course, this would require > both creation times to be exposed to clients. > > Best, > Chia-Ping > > On 2026/06/24 20:46:16 Jun Rao via dev wrote: > > Hi, Jiunn-Yang, > > > > Thanks for the updated KIP. > > > > JR2. Share consumer (KIP-932) states: "If the number of partitions is > > increased for a topic with a subscription in a share group, the SPSO for > > the newly created share-partitions is initialized to 0 ". However, this > > part has not been implemented yet. Currently, all partitions without offset > > follows the strategy given by share.auto.offset.reset. So, we have an > > opportunity to see if it's better to align this KIP's changes with the > > share consumer. > > JR2.1 The first part is whether customizing the offset for new partitions > > to something other than 0 is useful in the share consumer. It seems that > > the motivation in this KIP applies to share consumer too. > > JR2.2 The second part is how new partitions are detected. The mechanism > > introduced in this KIP seems quite effective. A group coordinator could > > pick up newly created partitions for an existing topic or newly created > > topics matching the regex. Those partitions are categorized as new > > partitions and will follow the strategy given by > > auto.offset.reset.new.partitions. > > This makes sense since those partitions typically don't have a large > > backlog. A consumer could also additionally subscribe to an existing topic > > in an existing group. Those partitions will be categorized as existing > > partitions. This also makes sense since those partitions could have a large > > backlog and it is better to follow the strategy given by auto.offset.reset. > > It > > will be useful to think through whether the logic for categorizing new > > partitions should also be reused in the share consumer. > > > > JR3. The KIP exposes the group/partition creation time to the client. I am > > not sure if there are other use cases for those timestamps. An alternative > > is to avoid exposing those timestamps to the client. The group coordinator > > will categorize the partitions using those timestamps and include the > > categorization in the assigned partitions in GroupHeartbeatResponse. > > > > JR4. It would be useful to support this feature on existing groups as well. > > For example, any partition without a creation timestamp can always be > > categorized as an existing partition. > > > > Jun > > > > On Wed, Jun 24, 2026 at 6:12 AM 黃竣陽 <[email protected]> wrote: > > > > > Hello Andrew, > > > > > > Thanks for your feedback, > > > >> I've just been re-reading this KIP to ensure that share group support > > > >> is > > > >> not needed. I've discovered that we made a mistake implementing KIP-932 > > > in > > > >> this area, so we'll rectify that and check whether it's really tight > > > enough > > > >> not to require KIP-1327. > > > > > > Could you walk us through the specific area where the mistake was found? > > > That would > > > help us assess whether it naturally fits within the scope of KIP-1327 or > > > should be addressed > > > separately. > > > > > > AS1 I have updated the KIP > > > > > > Best Regards, > > > Jiunn-Yang > > > > > > > Chia-Ping Tsai <[email protected]> 於 2026年6月24日 晚上8:49 寫道： > > > > > > > > hi Andrew > > > > > > > >> I've discovered that we made a mistake implementing KIP-932 in this > > > area, > > > > so we'll rectify that and check whether it's really tight enough not to > > > > require KIP-1327. > > > > > > > > Would you mind sharing more details about that? If it's not anything > > > close > > > > to NP-hard, we could definitely look into covering it within KIP-1327 > > > > > > > > Best, > > > > Chia-Ping > > > > > > > > > > > > Andrew Schofield <[email protected]> 於 2026年6月24日週三 下午7:49寫道： > > > > > > > >> Hi Jiunn-Yang, > > > >> I've just been re-reading this KIP to ensure that share group support > > > >> is > > > >> not needed. I've discovered that we made a mistake implementing KIP-932 > > > in > > > >> this area, so we'll rectify that and check whether it's really tight > > > enough > > > >> not to require KIP-1327. > > > >> > > > >> One trivial comments for consistency. > > > >> > > > >> AS1: In kafka-consumer-groups.sh, unknown column data is represented by > > > >> "-", not "N/A". In kafka-topics.sh, we do use "N/A". > > > >> > > > >> Thanks, > > > >> Andrew > > > >> > > > >> On 2026/06/23 23:23:38 黃竣陽 wrote: > > > >>> Hi all, > > > >>> > > > >>> Manually bumping this thread. > > > >>> > > > >>> Best Regards, > > > >>> Jiunn-Yang > > > >>> > > > >>>> 黃竣陽 <[email protected]> 於 2026年6月17日 晚上9:17 寫道： > > > >>>> > > > >>>> Hello chia, > > > >>>> > > > >>>> Thanks for the feedback, I have updated the KIP. > > > >>>> > > > >>>> Best Regards, > > > >>>> Jiunn-Yang > > > >>>> > > > >>>>> Chia-Ping Tsai <[email protected]> 於 2026年6月17日 凌晨12:47 寫道： > > > >>>>> > > > >>>>> hi Jiunn-Yang > > > >>>>> > > > >>>>>> When the config is set on a cluster that has not yet been > > > >> upgraded... classification cannot occur... the consumer falls back to > > > the > > > >> base auto.offset.reset for the affected partitions. No exception is > > > thrown, > > > >> and no operational disruption results. > > > >>>>> > > > >>>>> Existing group can't take advantage of this excellent new > > > >> configuration. Allowing users to modify the group creation time might > > > >> be > > > >> overkill. Instead, we could print a useful warning message to guide > > > users. > > > >> For example, we can suggest that they re-create the group with their > > > >> existing committed offsets > > > >>>>> > > > >>>>>> Protocol changes > > > >>>>> > > > >>>>> Would you mind listing those RPC changes in a table format? > > > >>>>> > > > >>>>>> The full interaction matrix between the base policy and the > > > >> new-partition policy is: > > > >>>>> > > > >>>>> Please add a filed to describe the target scenario when using these > > > >> policies > > > >>>>> > > > >>>>> Best, > > > >>>>> Chia-Ping > > > >>>>> > > > >>>>> > > > >>>>> On 2026/06/16 16:14:49 黃竣陽 wrote: > > > >>>>>> Hello Jun, chia, > > > >>>>>> > > > >>>>>> Thanks for the feedback, I have updated the KIP for the new > > > >>>>>> approach, PTAL > > > >>>>>> > > > >>>>>> Best Regards, > > > >>>>>> Jiunn-Yang > > > >>>>>> > > > >>>>>>> Chia-Ping Tsai <[email protected]> 於 2026年6月16日 上午8:23 寫道： > > > >>>>>>> > > > >>>>>>> hi Jun > > > >>>>>>> > > > >>>>>>> Yes, your approach is great. I think the combination of latest > > > >>>>>>> (for > > > >> existing partitions) and by_duration (for new partitions) can address > > > 99% > > > >> of the complaints I have heard regarding this issue. > > > >>>>>>> > > > >>>>>>> Also, leveraging the group creation time here opens the door to > > > >> implementing a new policy based on timestamp seek in the future, should > > > the > > > >> community want to pursue that. > > > >>>>>>> > > > >>>>>>> Thanks for your patience and constructive feedback. We will update > > > >> the KIP accordingly. > > > >>>>>>> > > > >>>>>>> Best, Chia-Ping > > > >>>>>>> > > > >>>>>>>> Jun Rao via dev <[email protected]> 於 2026年6月16日 清晨5:11 寫道： > > > >>>>>>>> > > > >>>>>>>> Hi, Chia-Ping, > > > >>>>>>>> > > > >>>>>>>> Thanks for the reply. > > > >>>>>>>> > > > >>>>>>>> I agree that it's probably useful to allow a user to configure a > > > >> different > > > >>>>>>>> offset policy for existing partitions vs new partitions. However, > > > >> using > > > >>>>>>>> group creation time to capture that seems more intuitive. Here is > > > >> another > > > >>>>>>>> proposal: remove auto.offset.reset.max.age.ms and categorize new > > > >> partitions > > > >>>>>>>> based on group creation time. Introduce > > > >>>>>>>> a new config auto.offset.reset.new.partitions whose values can be > > > >> earliest, > > > >>>>>>>> latest and by_duration, the same as auto.offset.reset. Users can > > > >> set > > > >>>>>>>> `auto.offset.reset.new.partitions` to `earliest` if they want to > > > >> guarantee > > > >>>>>>>> no data loss on new partitions. They can also use by_duration to > > > >> set an > > > >>>>>>>> upper bound on the backlog replayed, which can be different from > > > >> that of > > > >>>>>>>> the existing partitions. This will address your concern about too > > > >> much > > > >>>>>>>> backlog being replayed when the offsets are lost. What do you > > > >> think? > > > >>>>>>>> > > > >>>>>>>> Jun > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>>> On Mon, Jun 15, 2026 at 10:39 AM Chia-Ping Tsai < > > > >> [email protected]> wrote: > > > >>>>>>>>> > > > >>>>>>>>> hi Jun > > > >>>>>>>>> > > > >>>>>>>>> The most important part of this story is how users should expect > > > >> the data > > > >>>>>>>>> they can see when using the latest or by_duration policy with > > > >> expanded > > > >>>>>>>>> partitions. > > > >>>>>>>>> > > > >>>>>>>>> Yes, the by_duration policy can minimize data loss, but it is > > > >>>>>>>>> non-deterministic, which means users will either read too many > > > >> historical > > > >>>>>>>>> records from existing partitions or lose some records from > > > >> expanded > > > >>>>>>>>> partitions. > > > >>>>>>>>> > > > >>>>>>>>> Also, I agree that auto.offset.reset.max.age.ms > > > >>>>>>>>> < > > > >> > > > https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$ > > > >>> > > > >>>>>>>>> is a bit hard to understand, and that is why I preferred having > > > >>>>>>>>> a > > > >> whole new > > > >>>>>>>>> policy based entirely on group creation time (KIP-1282) > > > >>>>>>>>> > > > >>>>>>>>> Best, > > > >>>>>>>>> Chia-Ping > > > >>>>>>>>> > > > >>>>>>>>> Jun Rao via dev <[email protected]> 於 2026年6月16日週二 上午1:08寫道： > > > >>>>>>>>> > > > >>>>>>>>>> Hi, Chia-Ping and Jiunn-Yang, > > > >>>>>>>>>> > > > >>>>>>>>>> Thanks for the reply. I am still trying to understand the value > > > >> of the new > > > >>>>>>>>>> configs with the KIP. > > > >>>>>>>>>> > > > >>>>>>>>>> The motivation of the KIP is that a user doesn't want to miss > > > >> the data if > > > >>>>>>>>>> the backlog is small. The backlog of the existing partition is > > > >> easy to > > > >>>>>>>>>> understand because it relates to retention time. The backlog > > > >>>>>>>>>> for > > > >> the new > > > >>>>>>>>>> partition is a bit subtle to understand since it depends on the > > > >> metadata > > > >>>>>>>>>> refresh delay. To set auto.offset.reset.max.age.ms > > > >>>>>>>>>> < > > > >> > > > https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$ > > > >>> , > > > >>>>>>>>>> the user needs to > > > >>>>>>>>>> understand the metadata refresh delay on the consumer side and > > > >> use it to > > > >>>>>>>>>> set the config. > > > >>>>>>>>>> > > > >>>>>>>>>> Now, let's consider the alternative: setting the same value for > > > >> the > > > >>>>>>>>>> existing by_duration policy. The KIP lists three issues with > > > this > > > >>>>>>>>>> approach. > > > >>>>>>>>>> 1. It computes the seek target client-side as now() - duration, > > > >> which > > > >>>>>>>>>> introduces clock skew across consumers and forces operators to > > > >> choose > > > >>>>>>>>>> overly large durations, causing unnecessary reprocessing. > > > >>>>>>>>>> 2. The target timestamp is recomputed on each retry, so failed > > > >>>>>>>>>> ListOffsetsRequest retries can shift the target forward and > > > >> potentially > > > >>>>>>>>>> miss records produced between attempts. > > > >>>>>>>>>> 3. It applies uniformly to all partitions without committed > > > >> offsets, and > > > >>>>>>>>>> cannot distinguish newly expanded partitions from long-existing > > > >> partitions > > > >>>>>>>>>> newly assigned to the group, leading to unnecessary replay. > > > >>>>>>>>>> > > > >>>>>>>>>> Issues 1 and 2 are uncommon and can be mitigated by adding a > > > >>>>>>>>>> bit > > > >> buffer to > > > >>>>>>>>>> the metadata refresh delay. We could also consider improving > > > >>>>>>>>>> the > > > >>>>>>>>>> implementation. For issue 3, the metadata refresh delay is > > > >> typically low > > > >>>>>>>>>> (in the order of minutes with the classic consumer and tens of > > > >> seconds > > > >>>>>>>>>> with > > > >>>>>>>>>> the new consumer). If a user is ok with reading that much > > > >> backlog for new > > > >>>>>>>>>> partitions, it seems they will be ok doing the same for > > > >>>>>>>>>> existing > > > >>>>>>>>>> partitions. > > > >>>>>>>>>> > > > >>>>>>>>>> So, instead of introducing a new config, could we just reuse > > > >>>>>>>>>> the > > > >> existing > > > >>>>>>>>>> config with better documentation and/or implementation? > > > >>>>>>>>>> > > > >>>>>>>>>> Jun > > > >>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>>>>>>> On Sat, Jun 13, 2026 at 12:19 AM 黃竣陽 <[email protected]> > > > >> wrote: > > > >>>>>>>>>> > > > >>>>>>>>>>> Hello Jun, > > > >>>>>>>>>>> > > > >>>>>>>>>>> You're right that group creation time is the more intuitive > > > >> answer at > > > >>>>>>>>>>> first glance, > > > >>>>>>>>>>> the KIP's own motivation talks about partitions that "predate > > > >> the group" > > > >>>>>>>>>>> vs partitions > > > >>>>>>>>>>> "created during group runtime," which directly points to a > > > >>>>>>>>>> group-lifecycle > > > >>>>>>>>>>> classifier. > > > >>>>>>>>>>> I'd like to walk through why we landed on partition age, and > > > the > > > >>>>>>>>>>> trade-offs we considered. > > > >>>>>>>>>>> > > > >>>>>>>>>>> We evaluated three candidate signals: > > > >>>>>>>>>>> > > > >>>>>>>>>>> 1. `by_duration:5secs` > > > >>>>>>>>>>> > > > >>>>>>>>>>> This covers the metadata blindness window, but has issues the > > > >> KIP > > > >>>>>>>>>>> currently documents > > > >>>>>>>>>>> under "Why not use `by_duration`?": > > > >>>>>>>>>>> > > > >>>>>>>>>>> - Client-side `now() - duration` introduces clock skew across > > > >> consumers. > > > >>>>>>>>>>> - `ListOffsets` retries shift the target forward, potentially > > > >> missing > > > >>>>>>>>>>> records produced between > > > >>>>>>>>>>> attempts. > > > >>>>>>>>>>> - It applies uniformly to all partitions without committed > > > >> offsets, > > > >>>>>>>>>>> including pre-existing partitions > > > >>>>>>>>>>> newly assigned to the group, causing unnecessary replay. > > > >>>>>>>>>>> > > > >>>>>>>>>>> 2. Group creation time as classifier > > > >>>>>>>>>>> > > > >>>>>>>>>>> This works cleanly when the consumer is actively running. Our > > > >> concern > > > >>>>>>>>>>> is the idle / late-rejoin case: > > > >>>>>>>>>>> > > > >>>>>>>>>>> T=0: Group created. > > > >>>>>>>>>>> T=1..T=100: Consumer idle (down, disconnected, etc.). > > > >>>>>>>>>>> T=50: Partition added during the idle window. > > > >>>>>>>>>>> T=100: Consumer resumes. > > > >>>>>>>>>>> > > > >>>>>>>>>>> Under group creation time, the new partition is classified as > > > >> new > > > >>>>>>>>>>> (`50 > 0`) and reset to `earliest`, replaying everything from > > > >> T=50. > > > >>>>>>>>>>> But during `[T=1, T=100]`, base partitions also accumulated > > > >> data that > > > >>>>>>>>>>> the consumer accepts as lost — that is precisely the contract > > > of > > > >>>>>>>>>>> `auto.offset.reset=latest`. There is no principled reason to > > > >> treat > > > >>>>>>>>>>> the new partition differently; both contain backlog > > > >>>>>>>>>>> accumulated > > > >> during > > > >>>>>>>>>>> the same idle window. > > > >>>>>>>>>>> > > > >>>>>>>>>>> This aligns with the "backlog is backlog” principle you raised > > > >> in > > > >>>>>>>>>>> the KIP-1282 thread: a `latest` user has tolerated some > > > >>>>>>>>>>> backlog > > > >> on > > > >>>>>>>>>>> every other partition during the same idle period; forcing > > > >> 0-backlog > > > >>>>>>>>>>> tolerance only on new partitions would be inconsistent with > > > that > > > >>>>>>>>>>> tolerance. > > > >>>>>>>>>>> > > > >>>>>>>>>>> 3. Partition age vs threshold > > > >>>>>>>>>>> > > > >>>>>>>>>>> Partition age corresponds to the actual silent data loss > > > window, > > > >>>>>>>>>>> the gap between partition creation and the consumer’s metadata > > > >>>>>>>>>>> refresh. Within this window, data loss is genuinely silent: > > > >>>>>>>>>>> the > > > >>>>>>>>>>> consumer had no opportunity to know about the partition. > > > >> Outside this > > > >>>>>>>>>>> window, missing data reflects either: > > > >>>>>>>>>>> > > > >>>>>>>>>>> - (a) the user’s tolerated cost of running with idle > > > >>>>>>>>>>> consumers, > > > >> or > > > >>>>>>>>>>> - (b) an operational issue to surface via monitoring, not via > > > >> reset > > > >>>>>>>>>> policy. > > > >>>>>>>>>>> > > > >>>>>>>>>>> We did not choose partition age because it is more elegant > > > >>>>>>>>>>> than > > > >> group > > > >>>>>>>>>>> creation time — we chose it because its failure mode (requires > > > a > > > >>>>>>>>>>> threshold) is > > > >>>>>>>>>>> less invasive than the failure mode of group creation time > > > >> (overrides > > > >>>>>>>>>>> user-stated > > > >>>>>>>>>>> `latest` intent during idle periods). > > > >>>>>>>>>>> > > > >>>>>>>>>>> Best Regards, > > > >>>>>>>>>>> Jiunn-Yang > > > >>>>>>>>>>> > > > >>>>>>>>>>>> Chia-Ping Tsai <[email protected]> 於 2026年6月13日 上午11:52 寫道： > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> Hi Jun, > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> Relying on both creation times will create an inconsistent > > > >> scenario. A > > > >>>>>>>>>>>> consumer that lost all offsets due to a long sleep will seek > > > >> to the > > > >>>>>>>>>>>> beginning for the partitions created later than the group. > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> That is why we initially proposed KIP-1282 to fix the > > > >> inconsistency > > > >>>>>>>>>>> using a > > > >>>>>>>>>>>> whole new policy. Since KIP-1282 couldn't reach a consensus, > > > >> KIP-1327 > > > >>>>>>>>>>> goes > > > >>>>>>>>>>>> back to using flexible configurations to prevent users from > > > >> falling > > > >>>>>>>>>> into > > > >>>>>>>>>>>> that pitfall. > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> Best, Chia-Ping > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> Jun Rao via dev <[email protected]> 於 2026年6月13日週六 > > > >> 上午6:49寫道： > > > >>>>>>>>>>>> > > > >>>>>>>>>>>>> Hi, Jiunn-Yang, > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> Thanks for the reply and sorry for the late reply. > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> JR1. The design of auto.offset.reset.max.age.ms > > > >>>>>>>>>> < > > > >> > > > https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$ > > > >>> > > > >>>>>>>>>> still feels weird to > > > >>>>>>>>>>> me. > > > >>>>>>>>>>>>> It > > > >>>>>>>>>>>>> categorizes partitions as new or existing based on the > > > >> partition > > > >>>>>>>>>>> creation > > > >>>>>>>>>>>>> time. Intuitively, the categorization should be based on the > > > >> group > > > >>>>>>>>>>> creation > > > >>>>>>>>>>>>> time: all partitions existing when the group is created are > > > >> existing > > > >>>>>>>>>> and > > > >>>>>>>>>>>>> all partitions created after the group creation are new > > > >> partitions. > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> Jun > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> On Tue, Jun 9, 2026 at 8:51 AM 黃竣陽 <[email protected]> > > > >> wrote: > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>>> Hi all, > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> Manually bumping this thread. If there is no further > > > >>>>>>>>>>>>>> discussion, I will close the vote. > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> Best Regards, > > > >>>>>>>>>>>>>> Jiunn-Yang > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> 黃竣陽 <[email protected]> 於 2026年6月1日 晚上7:16 寫道： > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> Hello Jian, > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> Thanks for your feedback, > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> Agreed, partition expansion is a common operational task, > > > >> not an > > > >>>>>>>>>> edge > > > >>>>>>>>>>>>>>> case. I've updated the Motivation section accordingly. > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> Best Regards, > > > >>>>>>>>>>>>>>> Jiunn-Yang > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> jian fu <[email protected]> 於 2026年6月1日 下午5:49 寫道： > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> Hi Jiunn-Yang: > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> Thanks for the KIP. I think it would be useful to clarify > > > >> that > > > >>>>>>>>>> this > > > >>>>>>>>>>>>> is a > > > >>>>>>>>>>>>>>>> common scenario rather than an edge case, which further > > > >>>>>>>>>> demonstrates > > > >>>>>>>>>>>>> the > > > >>>>>>>>>>>>>>>> need for this optimization. For example: > > > >>>>>>>>>>>>>>>> A partition expansion is a common operational task in > > > >> Kafka: To > > > >>>>>>>>>>>>> balance > > > >>>>>>>>>>>>>>>> resource utilization and cost, topics are typically > > > >> created with a > > > >>>>>>>>>>>>>> moderate > > > >>>>>>>>>>>>>>>> default partition count. However, as traffic grows over > > > >> time, it > > > >>>>>>>>>> is > > > >>>>>>>>>>>>>> often > > > >>>>>>>>>>>>>>>> necessary to increase the number of partitions to > > > >> accommodate the > > > >>>>>>>>>>>>> higher > > > >>>>>>>>>>>>>>>> workload. > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> Regards > > > >>>>>>>>>>>>>>>> Jian > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> 黃竣陽 <[email protected]> 于2026年5月30日周六 22:31写道： > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>> Hello chia, > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>> Thanks for the comments, I have updated the KIP! > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>> Best Regards, > > > >>>>>>>>>>>>>>>>> Jiunn-Yang > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>> Chia-Ping Tsai <[email protected]> 於 2026年5月30日 晚上8:29 > > > >> 寫道： > > > >>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>> Hi Jiunn-Yang, > > > >>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>> Would you mind removing the terms "hot" and "cold" when > > > >>>>>>>>>> describing > > > >>>>>>>>>>>>>>>>>> partitions in the KIP? I understand you are using them > > > to > > > >>>>>>>>>> describe > > > >>>>>>>>>>>>> the > > > >>>>>>>>>>>>>>>>>> "freshness" or the users' need for the records, but > > > >> applying > > > >>>>>>>>>> these > > > >>>>>>>>>>>>>> terms > > > >>>>>>>>>>>>>>>>> to > > > >>>>>>>>>>>>>>>>>> the partition itself feels a bit unnatural. > > > >>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>> After all, in this scenario, users don't really care > > > >> whether a > > > >>>>>>>>>>>>>> partition > > > >>>>>>>>>>>>>>>>> is > > > >>>>>>>>>>>>>>>>>> newly expanded or not. Their only expectation is that > > > >> they won't > > > >>>>>>>>>>>>>> silently > > > >>>>>>>>>>>>>>>>>> lose any live records produced to the topic during > > > >>>>>>>>>>>>>>>>>> their > > > >> active > > > >>>>>>>>>>>>>>>>> consumption. > > > >>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>> Best, Chia-Ping > > > >>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>> 黃竣陽 <[email protected]> 於 2026年5月30日週六 下午12:30寫道： > > > >>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>> Hello Jun, > > > >>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>> Thanks for the feedback, I have updated the KIP > > > >> motivation > > > >>>>>>>>>>> section. > > > >>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>> Best Regards, > > > >>>>>>>>>>>>>>>>>>> Jiunn-Yang > > > >>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>> Jun Rao via dev <[email protected]> 於 2026年5月30日 > > > >> 凌晨1:12 > > > >>>>>>>>>> 寫道： > > > >>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang, > > > >>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>> Thanks for the reply. I think we need a stronger > > > >> motivation > > > >>>>>>>>>> for > > > >>>>>>>>>>>>> the > > > >>>>>>>>>>>>>>>>> KIP. > > > >>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>> The KIP says "The core insight is that not all > > > >> partitions > > > >>>>>>>>>> without > > > >>>>>>>>>>>>> a > > > >>>>>>>>>>>>>>>>>>>> committed offset are the same. A newly expanded > > > >> partition > > > >>>>>>>>>> (hot) > > > >>>>>>>>>>> is > > > >>>>>>>>>>>>>>>>>>>> fundamentally different from a partition the consumer > > > >> has > > > >>>>>>>>>> never > > > >>>>>>>>>>>>> seen > > > >>>>>>>>>>>>>>>>>>>> because it predates the group (cold)." Why is the hot > > > >>>>>>>>>> partition > > > >>>>>>>>>>>>>>>>>>>> fundamentally different from the cold? > > > >>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>> The KIP says "The existing by_duration policy is also > > > >>>>>>>>>>> insufficient > > > >>>>>>>>>>>>>>>>>>> because: > > > >>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>> - The calculated seek time (now() - duration) varies > > > >> across > > > >>>>>>>>>> nodes > > > >>>>>>>>>>>>>> due > > > >>>>>>>>>>>>>>>>>>> to > > > >>>>>>>>>>>>>>>>>>>> clock skew. To be safe, users must set an overly > > > >>>>>>>>>>>>>>>>>>>> large > > > >>>>>>>>>> duration, > > > >>>>>>>>>>>>>>>>>>> causing > > > >>>>>>>>>>>>>>>>>>>> unnecessary reprocessing. > > > >>>>>>>>>>>>>>>>>>>> - On network errors, the client recalculates the seek > > > >> time on > > > >>>>>>>>>>>>> retry, > > > >>>>>>>>>>>>>>>>>>>> shifting the target timestamp forward and risking > > > >>>>>>>>>>>>>>>>>>>> data > > > >> loss." > > > >>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>> However, both of these situations are rare. If these > > > >> issues > > > >>>>>>>>>>>>> persist, > > > >>>>>>>>>>>>>>>>> more > > > >>>>>>>>>>>>>>>>>>>> severe problems likely exist elsewhere. Rare > > > >> situations don't > > > >>>>>>>>>>>>> need a > > > >>>>>>>>>>>>>>>>>>> common > > > >>>>>>>>>>>>>>>>>>>> solution. If users care about those rare situations, > > > >> they can > > > >>>>>>>>>>>>>> implement > > > >>>>>>>>>>>>>>>>>>>> customized logic using > > > >>>>>>>>>>>>>>>>> ConsumerRebalanceListener.onPartitionsAssigned(). > > > >>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>> Jun > > > >>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>> On Sun, May 17, 2026 at 6:50 AM 黃竣陽 < > > > >> [email protected]> > > > >>>>>>>>>> wrote: > > > >>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>> Hello chia, > > > >>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>> Thanks for the feedback, > > > >>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>> If the creation time exists, the returned value > > > >> should > > > >>>>>>>>>> always > > > >>>>>>>>>>> be > > > >>>>>>>>>>>>>>>>>>> greater > > > >>>>>>>>>>>>>>>>>>>>> than or equal to zero, right? > > > >>>>>>>>>>>>>>>>>>>>> I have explicitly mentioned this in the KIP. > > > >>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>> New Old (MetadataResponse v0–13) positive > > > >> any > > > >>>>>>>>>>>>>> field > > > >>>>>>>>>>>>>>>>>>>>> absent UnsupportedVersionException > > > >>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>> The earliest point at which we can detect the > > > >>>>>>>>>>>>>>>>>>>>> version > > > >>>>>>>>>> mismatch > > > >>>>>>>>>>> is > > > >>>>>>>>>>>>>>>>> during > > > >>>>>>>>>>>>>>>>>>>>> the > > > >>>>>>>>>>>>>>>>>>>>> first metadata fetch after assignment, which occurs > > > >> inside > > > >>>>>>>>>>>>> poll(). > > > >>>>>>>>>>>>>>>>>>>>> Therefore, the > > > >>>>>>>>>>>>>>>>>>>>> user would encounter an UnsupportedVersionException > > > >> from > > > >>>>>>>>>> poll(). > > > >>>>>>>>>>>>>> I’ll > > > >>>>>>>>>>>>>>>>>>>>> clarify this in the KIP. > > > >>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>> Best Regards, > > > >>>>>>>>>>>>>>>>>>>>> Jiunn-Yang > > > >>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <[email protected]> 於 2026年5月17日 > > > >> 下午4:50 > > > >>>>>>>>>> 寫道： > > > >>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>> hi Jiunn > > > >>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>> PartitionAgeMs (int64, default -1): The age of > > > >>>>>>>>>>>>>>>>>>>>>>> this > > > >>>>>>>>>> partition > > > >>>>>>>>>>>>> in > > > >>>>>>>>>>>>>>>>>>>>> milliseconds, computed server-side by the broker as > > > >>>>>>>>>>>>>>>>> broker_current_time > > > >>>>>>>>>>>>>>>>>>> - > > > >>>>>>>>>>>>>>>>>>>>> partition_creation_time. Returns -1 if the broker > > > >> does not > > > >>>>>>>>>>>>> support > > > >>>>>>>>>>>>>>>>> this > > > >>>>>>>>>>>>>>>>>>>>> feature or the partition creation time is unknown. > > > >>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>> If the creation time exists, the returned value > > > >> should > > > >>>>>>>>>> always > > > >>>>>>>>>>> be > > > >>>>>>>>>>>>>>>>>>> greater > > > >>>>>>>>>>>>>>>>>>>>> than or equal to zero, right? > > > >>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>> New Old (MetadataResponse v0–13) positive > > > >> any > > > >>>>>>>>>>>>>> field > > > >>>>>>>>>>>>>>>>>>>>> absent UnsupportedVersionException > > > >>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>> Will user encounter UnsupportedVersionException > > > >>>>>>>>>>>>>>>>>>>>>> when > > > >> calling > > > >>>>>>>>>>>>>>>>> `poll()`? > > > >>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>> Best, > > > >>>>>>>>>>>>>>>>>>>>>> Chia-Ping > > > >>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>> On 2026/05/16 04:30:49 黃竣陽 wrote: > > > >>>>>>>>>>>>>>>>>>>>>>> Hello Jun, chia, > > > >>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>> I've updated KIP-1327 with a design change based > > > >>>>>>>>>>>>>>>>>>>>>>> on > > > >> the > > > >>>>>>>>>>>>>> discussion > > > >>>>>>>>>>>>>>>>>>>>>>> feedback. > > > >>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>> The updated design decouples the new-partition > > > reset > > > >>>>>>>>>> behavior > > > >>>>>>>>>>>>>> from > > > >>>>>>>>>>>>>>>>>>>>>>> the base auto.offset.reset policy: > > > >>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>> - auto.offset.reset.max.age.ms > > > >>>>>>>>>> < > > > >> > > > https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$ > > > >>> > > > >>>>>>>>>> now applies to all > > > >>>>>>>>>>>>>> auto.offset.reset > > > >>>>>>>>>>>>>>>>>>>>> values > > > >>>>>>>>>>>>>>>>>>>>>>> (latest, earliest, by_duration, none). > > > >>>>>>>>>>>>>>>>>>>>>>> - For new ("hot") partitions, the consumer resets > > > to > > > >>>>>>>>>>>>>>>>>>>>> auto.offset.reset.new > > > >>>>>>>>>> < > > > >> > > > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$ > > > >>> > > > >>>>>>>>>> .partitions > > > >>>>>>>>>>>>>>>>>>>>>>> config setting > > > >>>>>>>>>>>>>>>>>>>>>>> - For existing ("cold") partitions, the base > > > >>>>>>>>>> auto.offset.reset > > > >>>>>>>>>>>>>>>>> policy > > > >>>>>>>>>>>>>>>>>>>>> continues > > > >>>>>>>>>>>>>>>>>>>>>>> to apply unchanged. > > > >>>>>>>>>>>>>>>>>>>>>>> - The new-partition reset behavior is represented > > > >> by a > > > >>>>>>>>>>> separate > > > >>>>>>>>>>>>>>>>>>>>> internal config > > > >>>>>>>>>>>>>>>>>>>>>>> (auto.offset.reset.new > > > >>>>>>>>>> < > > > >> > > > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$ > > > >>> .partitions, > > > >>>>>>>>>> currently fixed to > > > >>>>>>>>>>>>> earliest). > > > >>>>>>>>>>>>>>>>> This > > > >>>>>>>>>>>>>>>>>>>>> decoupled design makes > > > >>>>>>>>>>>>>>>>>>>>>>> it straightforward to promote the behavior to a > > > >> public > > > >>>>>>>>>>>>>> user-facing > > > >>>>>>>>>>>>>>>>>>>>> configuration in a future KIP. > > > >>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>> Best Regards, > > > >>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang > > > >>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <[email protected]> 於 2026年5月16日 > > > >> 清晨7:46 > > > >>>>>>>>>> 寫道： > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>> hi Jun > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>> I see what you mean now. The proposal from me is > > > >> listed > > > >>>>>>>>>>> below: > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>> 1) Add auto.offset.reset.new > > > >>>>>>>>>> < > > > >> > > > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$ > > > >>> .partitions > > > >>>>>>>>>> with a default value > > > >>>>>>>>>>>>> of > > > >>>>>>>>>>>>>>>>>>>>> earliest. It fixes the data loss from both > > > >> by_duration and > > > >>>>>>>>>>>>> latest, > > > >>>>>>>>>>>>>> and > > > >>>>>>>>>>>>>>>>>>> it > > > >>>>>>>>>>>>>>>>>>>>> does not change the logic of > > > >> auto.offset.reset=earliest. > > > >>>>>>>>>>>>>>>>>>>>>>>> 2) Mark auto.offset.reset.new > > > >>>>>>>>>> < > > > >> > > > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$ > > > >>> .partitions > > > >>>>>>>>>> as an internal > > > >>>>>>>>>>>>>>>>>>>>> configuration. auto.offset.reset.new > > > >>>>>>>>>> < > > > >> > > > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$ > > > >>> > > > >>>>>>>>>> .partitions=earliest > > > >>>>>>>>>>> already > > > >>>>>>>>>>>>>>>>>>>>> addresses the issue, and we can discuss the use > > > >>>>>>>>>>>>>>>>>>>>> cases > > > >> of > > > >>>>>>>>>> other > > > >>>>>>>>>>>>>> values > > > >>>>>>>>>>>>>>>>>>> in a > > > >>>>>>>>>>>>>>>>>>>>> separate KIP. > > > >>>>>>>>>>>>>>>>>>>>>>>> 3) Both configs, auto.offset.reset.new > > > >>>>>>>>>> < > > > >> > > > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$ > > > >>> .partitions > > > >>>>>>>>>> and > > > >>>>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age.ms > > > >>>>>>>>>> < > > > >> > > > https://urldefense.com/v3/__http://auto.offset.reset.latest.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfu9JSP4l$ > > > >>> , > > > >>>>>>>>>> will be applied to all for > > > >>>>>>>>>>>>>>>>>>>>> consistency. > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>> WDYT? > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>> On 2026/05/15 20:53:20 Jun Rao via dev wrote: > > > >>>>>>>>>>>>>>>>>>>>>>>>> Hi, Chia-Ping, > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply. > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>> 1. In the motivation section, the KIP says "When > > > >> a Kafka > > > >>>>>>>>>>>>> topic > > > >>>>>>>>>>>>>> is > > > >>>>>>>>>>>>>>>>>>>>> expanded > > > >>>>>>>>>>>>>>>>>>>>>>>>> with new partitions, consumers using the latest > > > >> auto > > > >>>>>>>>>> offset > > > >>>>>>>>>>>>>> reset > > > >>>>>>>>>>>>>>>>>>>>> policy > > > >>>>>>>>>>>>>>>>>>>>>>>>> will silently miss all records produced to those > > > >>>>>>>>>> partitions > > > >>>>>>>>>>>>>> before > > > >>>>>>>>>>>>>>>>>>> the > > > >>>>>>>>>>>>>>>>>>>>>>>>> consumer discovers them.". If a user sets > > > >>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset=by_duration=1sec, the same > > > >> record loss > > > >>>>>>>>>>>>> issue > > > >>>>>>>>>>>>>>>>> could > > > >>>>>>>>>>>>>>>>>>>>> also > > > >>>>>>>>>>>>>>>>>>>>>>>>> happen, right? > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>> 2. I was thinking auto.offset.reset.new > > > >>>>>>>>>> < > > > >> > > > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$ > > > >>> .partitions > > > >>>>>>>>>> will > > > >>>>>>>>>>> take > > > >>>>>>>>>>>>>> the > > > >>>>>>>>>>>>>>>>>>> same > > > >>>>>>>>>>>>>>>>>>>>>>>>> values as auto.offset.reset. So a user could set > > > >> it > > > >>>>>>>>>>>>>> by_duration if > > > >>>>>>>>>>>>>>>>>>>>> needed. > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>> Jun > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 14, 2026 at 4:06 PM Chia-Ping Tsai < > > > >>>>>>>>>>>>>>>>> [email protected] > > > >>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>> wrote: > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> hi Jun > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback. I might be missing > > > >> something > > > >>>>>>>>>>>>>> important > > > >>>>>>>>>>>>>>>>>>> from > > > >>>>>>>>>>>>>>>>>>>>> your > > > >>>>>>>>>>>>>>>>>>>>>>>>>> suggestion, so please bear with me as I try to > > > >> clarify > > > >>>>>>>>>> with > > > >>>>>>>>>>>>> a > > > >>>>>>>>>>>>>> few > > > >>>>>>>>>>>>>>>>>>>>> questions: > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> 1. Is there a strong use case for extending > > > >>>>>>>>>>>>>>>>>>>>>>>>>> this > > > >> logic > > > >>>>>>>>>> to > > > >>>>>>>>>>>>>> other > > > >>>>>>>>>>>>>>>>>>> reset > > > >>>>>>>>>>>>>>>>>>>>>>>>>> policies? Unlike latest, policies like earliest > > > >> or > > > >>>>>>>>>>>>> by_duration > > > >>>>>>>>>>>>>>>>>>> don't > > > >>>>>>>>>>>>>>>>>>>>> seem > > > >>>>>>>>>>>>>>>>>>>>>>>>>> to suffer from the same silent data loss issue > > > >> when a > > > >>>>>>>>>>>>>> partition > > > >>>>>>>>>>>>>>>>> is > > > >>>>>>>>>>>>>>>>>>>>> expanded. > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> 2. What values would we expect users to > > > >> configure for > > > >>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.new > > > >>>>>>>>>> < > > > >> > > > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$ > > > >>> .partitions? > > > >>>>>>>>>> If they set it to > > > >>>>>>>>>>>>> earliest > > > >>>>>>>>>>>>>> or > > > >>>>>>>>>>>>>>>>>>>>> latest, > > > >>>>>>>>>>>>>>>>>>>>>>>>>> we might run into the exact same edge cases. > > > >>>>>>>>>>>>>>>>>>>>>>>>>> For > > > >>>>>>>>>> example, > > > >>>>>>>>>>>>> if a > > > >>>>>>>>>>>>>>>>>>>>> consumer is > > > >>>>>>>>>>>>>>>>>>>>>>>>>> offline for a while and a new partition is > > > >> created > > > >>>>>>>>>> during > > > >>>>>>>>>>>>> that > > > >>>>>>>>>>>>>>>>>>>>> downtime, > > > >>>>>>>>>>>>>>>>>>>>>>>>>> the user might actually want to skip to latest > > > >> when > > > >>>>>>>>>>>>> resuming, > > > >>>>>>>>>>>>>>>>>>> rather > > > >>>>>>>>>>>>>>>>>>>>> than > > > >>>>>>>>>>>>>>>>>>>>>>>>>> reading from earliest just because the > > > >>>>>>>>>>>>>>>>>>>>>>>>>> partition > > > >> is > > > >>>>>>>>>>>>>> technically > > > >>>>>>>>>>>>>>>>>>>>> "new" to > > > >>>>>>>>>>>>>>>>>>>>>>>>>> the group. > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> This is exactly why we opted for introducing a > > > >> max.age > > > >>>>>>>>>>>>>> threshold. > > > >>>>>>>>>>>>>>>>>>> It > > > >>>>>>>>>>>>>>>>>>>>> gives > > > >>>>>>>>>>>>>>>>>>>>>>>>>> users a time-bound way to define what is > > > >> genuinely > > > >>>>>>>>>>> "hot/new" > > > >>>>>>>>>>>>>> and > > > >>>>>>>>>>>>>>>>>>>>> what is > > > >>>>>>>>>>>>>>>>>>>>>>>>>> just an old partition they haven't seen yet. > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Best, > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/05/14 20:48:09 Jun Rao via dev wrote: > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang, > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP. > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> I find auto.offset.reset.latest.max.age a bit > > > >> weird. It > > > >>>>>>>>>>>>> only > > > >>>>>>>>>>>>>>>>>>>>> applies when > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset is latest. However, it seems > > > >> that the > > > >>>>>>>>>>>>>>>>> motivation > > > >>>>>>>>>>>>>>>>>>>>>>>>>> equally > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> applies when auto.offset.reset is set to other > > > >> values > > > >>>>>>>>>> like > > > >>>>>>>>>>>>>>>>>>>>> by_duration. > > > >>>>>>>>>>>>>>>>>>>>>>>>>> The > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> intention is that we want to have a separate > > > >> way to > > > >>>>>>>>>>> control > > > >>>>>>>>>>>>>>>>> newly > > > >>>>>>>>>>>>>>>>>>>>> created > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> partitions vs existing partitions when the > > > group > > > >>>>>>>>>> starts. > > > >>>>>>>>>>>>>> Have we > > > >>>>>>>>>>>>>>>>>>>>>>>>>> considered > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> adding a new config like auto.offset.reset.new > > > >>>>>>>>>> < > > > >> > > > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$ > > > >>> > > > >>>>>>>>>>> .partitions? > > > >>>>>>>>>>>>>> If > > > >>>>>>>>>>>>>>>>>>> this > > > >>>>>>>>>>>>>>>>>>>>> new > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> config is not set, the offset reset policy > > > >> defaults to > > > >>>>>>>>>> the > > > >>>>>>>>>>>>>>>>> policy > > > >>>>>>>>>>>>>>>>>>>>> used > > > >>>>>>>>>>>>>>>>>>>>>>>>>> for > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> existing partitions. The user could set it > > > >> explicitly > > > >>>>>>>>>> to > > > >>>>>>>>>>>>>>>>> customize > > > >>>>>>>>>>>>>>>>>>>>> the > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> behavior for new partitions. > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Jun > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 7, 2026 at 5:07 AM 黃竣陽 < > > > >> [email protected] > > > >>>>>>>>>>> > > > >>>>>>>>>>>>>> wrote: > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all, > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I’d like to manually bump this thread. > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards, > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 黃竣陽 <[email protected]> 於 2026年5月1日 晚上10:37 > > > >> 寫道： > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello all, > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback. > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ01/DJ02: > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> MetadataResponse bumps from v13 to v14. The > > > >>>>>>>>>>>>>> PartitionMetadata > > > >>>>>>>>>>>>>>>>>>>>> struct > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> gains a new > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> field PartitionAgeMs (int64, default -1), > > > >> computed > > > >>>>>>>>>>>>>> server-side > > > >>>>>>>>>>>>>>>>>>> by > > > >>>>>>>>>>>>>>>>>>>>> the > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> broker as > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> broker_current_time - > > > partition_creation_time. > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Also add the consumer heartbeat flow. when > > > >>>>>>>>>>>>>> MembershipManager > > > >>>>>>>>>>>>>>>>>>>>> detects > > > >>>>>>>>>>>>>>>>>>>>>>>>>> a > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> newly assigned > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition, it explicitly invalidates the > > > >> metadata for > > > >>>>>>>>>>> the > > > >>>>>>>>>>>>>>>>>>> affected > > > >>>>>>>>>>>>>>>>>>>>>>>>>> topic > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> and forces a fresh MetadataRequest > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> before making the offset reset decision, > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> even > > > >> if the > > > >>>>>>>>>>>>> topic > > > >>>>>>>>>>>>>> ID > > > >>>>>>>>>>>>>>>>> is > > > >>>>>>>>>>>>>>>>>>>>>>>>>> already > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> in the cache. > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> MB0: > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> The consumer learns the broker's maximum > > > >> supported > > > >>>>>>>>>>>>>>>>>>>>> MetadataResponse > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> version via the > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> ApiVersions negotiation at connection time. > > > >> If the > > > >>>>>>>>>>>>>> negotiated > > > >>>>>>>>>>>>>>>>>>>>>>>>>> version is > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> unsupported, the consumer > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> knows the broker does not support > > > >> PartitionAgeMs at > > > >>>>>>>>>> all > > > >>>>>>>>>>>>> and > > > >>>>>>>>>>>>>>>>> can > > > >>>>>>>>>>>>>>>>>>>>>>>>>> throw an > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> UnsupportedVersionException > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> immediately, rather than silently falling > > > >> back to > > > >>>>>>>>>> latest > > > >>>>>>>>>>>>>> and > > > >>>>>>>>>>>>>>>>>>>>> risking > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> data loss without any operator-visible > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> signal. > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> MB1/MB2/MB3: > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have addressed these changes in the KIP. > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards, > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <[email protected]> 於 > > > >> 2026年4月29日 > > > >>>>>>>>>>> 下午4:04 > > > >>>>>>>>>>>>>> 寫道： > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi David > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I agree with the direction of moving the > > > >> 'age' > > > >>>>>>>>>>>>> resolution > > > >>>>>>>>>>>>>>>>> from > > > >>>>>>>>>>>>>>>>>>>>> the > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Heartbeat API to the Metadata API to keep the > > > >> control > > > >>>>>>>>>>>>> plane > > > >>>>>>>>>>>>>>>>>>> clean. > > > >>>>>>>>>>>>>>>>>>>>> The > > > >>>>>>>>>>>>>>>>>>>>>>>>>> main > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> trade-off, as we noted before, is introducing > > > >>>>>>>>>>> inter-broker > > > >>>>>>>>>>>>>>>>> clock > > > >>>>>>>>>>>>>>>>>>>>> skew. > > > >>>>>>>>>>>>>>>>>>>>>>>>>> The > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Group Coordinator approach provided a single > > > >> source of > > > >>>>>>>>>>>>> truth > > > >>>>>>>>>>>>>>>>> for > > > >>>>>>>>>>>>>>>>>>>>> time. > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> However, realistically, this time skew > > > >> should be > > > >>>>>>>>>>>>>> negligible. > > > >>>>>>>>>>>>>>>>>>>>> Given > > > >>>>>>>>>>>>>>>>>>>>>>>>>> that > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> the max.age threshold will likely be > > > >> configured in > > > >>>>>>>>>>> minutes > > > >>>>>>>>>>>>>> or > > > >>>>>>>>>>>>>>>>>>>>> hours, a > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> typical NTP skew (in milliseconds) between > > > >> brokers > > > >>>>>>>>>> won't > > > >>>>>>>>>>>>>> impact > > > >>>>>>>>>>>>>>>>>>> the > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> fallback decision. > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> David Jacot via dev <[email protected]> > > > >> 於 > > > >>>>>>>>>>>>> 2026年4月29日 > > > >>>>>>>>>>>>>>>>>>> 下午3:29 > > > >>>>>>>>>>>>>>>>>>>>> 寫道： > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all, > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP! > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry, I haven't really followed the > > > >> previous > > > >>>>>>>>>>>>>> conversation > > > >>>>>>>>>>>>>>>>>>> but I > > > >>>>>>>>>>>>>>>>>>>>>>>>>> took a > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> quick look at this one. > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ01: I don't clearly understand the flow > > > >> with the > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> ConsumerGroupHeartbeat > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> API after reading the KIP. There is a new > > > >> boolean; > > > >>>>>>>>>> the > > > >>>>>>>>>>>>>> KIP > > > >>>>>>>>>>>>>>>>>>>>> states > > > >>>>>>>>>>>>>>>>>>>>>>>>>> that > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition ages are returned only when this > > > >> boolean > > > >>>>>>>>>> is > > > >>>>>>>>>>>>>> set. > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Implicitly, > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> this > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> means that when the consumer receives a > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> new > > > >>>>>>>>>> partition, > > > >>>>>>>>>>>>> it > > > >>>>>>>>>>>>>>>>> will > > > >>>>>>>>>>>>>>>>>>>>>>>>>> issue a > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> new > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> HB request with the boolean set to receive > > > >> the > > > >>>>>>>>>> ages. > > > >>>>>>>>>>> Is > > > >>>>>>>>>>>>>> my > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> understanding > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> correct? We should perhaps clarify the > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> flow > > > >> and > > > >>>>>>>>>> also > > > >>>>>>>>>>>>>> explain > > > >>>>>>>>>>>>>>>>>>>>> how it > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> fits > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> into the existing flow (e.g. list offsets, > > > >> fetch > > > >>>>>>>>>>>>> offsets, > > > >>>>>>>>>>>>>>>>>>> etc.). > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ02: It my understanding is correct, I > > > >> wonder if > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API is the > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> right > > > >> place > > > >>>>>>>>>> for > > > >>>>>>>>>>>>>> this > > > >>>>>>>>>>>>>>>>>>> given > > > >>>>>>>>>>>>>>>>>>>>>>>>>> that > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> a new > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> round trip is done anyway. Alternatively, > > > >> it could > > > >>>>>>>>>>>>> simply > > > >>>>>>>>>>>>>>>>>>>>> include > > > >>>>>>>>>>>>>>>>>>>>>>>>>> the > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> metadata. Generally, we should be rather > > > >> cautious > > > >>>>>>>>>>> about > > > >>>>>>>>>>>>>> not > > > >>>>>>>>>>>>>>>>>>>>>>>>>> overloading > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API with > > > >> unrelated > > > >>>>>>>>>>> concepts. > > > >>>>>>>>>>>>>> The > > > >>>>>>>>>>>>>>>>>>> API > > > >>>>>>>>>>>>>>>>>>>>> is > > > >>>>>>>>>>>>>>>>>>>>>>>>>> a > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> control plane API for assigning or > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> revoking > > > >>>>>>>>>>> partitions. > > > >>>>>>>>>>>>>> The > > > >>>>>>>>>>>>>>>>>>> fact > > > >>>>>>>>>>>>>>>>>>>>>>>>>> that > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> we > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> don't want to add it to the corresponding > > > >> Streams > > > >>>>>>>>>> API > > > >>>>>>>>>>>>>> also > > > >>>>>>>>>>>>>>>>>>>>> suggests > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> something is not quite right. What would > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we > > > >> do if > > > >>>>>>>>>> we > > > >>>>>>>>>>>>>> want to > > > >>>>>>>>>>>>>>>>>>>>>>>>>> support > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Streams in the future? > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> David > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 29, 2026 at 12:28 AM > > > >> Muralidhar Basani > > > >>>>>>>>>>> via > > > >>>>>>>>>>>>>> dev > > > >>>>>>>>>>>>>>>>> < > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [email protected]> wrote: > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jiunn, > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for this great kip. Good to > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> know > > > >> about > > > >>>>>>>>>> the > > > >>>>>>>>>>>>>> gap. > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-0 - why a new v2 version bump for > > > >>>>>>>>>>>>>> RequestPartitionAges > > > >>>>>>>>>>>>>>>>>>>>> field. > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Can a > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tagged field (for ex: on response, > > > >> PartitionAges > > > >>>>>>>>>> on > > > >>>>>>>>>>>>>>>>>>>>>>>>>> TopicPartitions) > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> be > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> used here and avoid version bump? > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-1 - For the new config, is there a > > > >> recommended > > > >>>>>>>>>>>>> value > > > >>>>>>>>>>>>>> or > > > >>>>>>>>>>>>>>>>> a > > > >>>>>>>>>>>>>>>>>>>>>>>>>> ConfigDef > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> validator? Probably it should based on > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the > > > >>>>>>>>>>>>>>>>>>> metadata.max.age.ms > > > >>>>>>>>>> < > > > >> > > > https://urldefense.com/v3/__http://metadata.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkflKEb5SK$ > > > >>> > > > >>>>>>>>>>>>>>>>>>>>> ? > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Sizing > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instructions can be part of javadocs I > > > >> guess. > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-2 - (minor) As there are no changes to > > > >> Kafka > > > >>>>>>>>>>>>> Streams, > > > >>>>>>>>>>>>>>>>>>> would > > > >>>>>>>>>>>>>>>>>>>>> it > > > >>>>>>>>>>>>>>>>>>>>>>>>>> be > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> better > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to add this new config > > > >>>>>>>>>>>>> auto.offset.reset.latest.max.age > > > >>>>>>>>>>>>>> to > > > >>>>>>>>>>>>>>>>>>> the > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> StreamsConfig block list > > > >>>>>>>>>>>>>>>>>>>>>>>>>> (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS) > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> for a > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clear warning, incase users configure it? > > > >> This is > > > >>>>>>>>>> the > > > >>>>>>>>>>>>>> most > > > >>>>>>>>>>>>>>>>>>>>>>>>>> familiar > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config and users might easily > > > >> mistakenly > > > >>>>>>>>>>>>>> configure > > > >>>>>>>>>>>>>>>>>>>>> it. Or > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> may be > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it's not worth it to add. > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-3 - (minor) The phrasing "the consumer > > > >> falls > > > >>>>>>>>>> back > > > >>>>>>>>>>>>> to > > > >>>>>>>>>>>>>>>>>>>>> earliest" > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> reads as > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> if the config were being changed > > > >> per-partition > > > >>>>>>>>>> which > > > >>>>>>>>>>>>>> isn't > > > >>>>>>>>>>>>>>>>>>>>>>>>>> supported. > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> May > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be rephrasing to something like "consumer > > > >> resolves > > > >>>>>>>>>>> the > > > >>>>>>>>>>>>>>>>>>> initial > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> position to > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> start offset for that partition" as if > > > >> earliest > > > >>>>>>>>>> was > > > >>>>>>>>>>>>>> applied > > > >>>>>>>>>>>>>>>>>>> to > > > >>>>>>>>>>>>>>>>>>>>>>>>>> that > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition only and auto.offset.reset > > > >> config is > > > >>>>>>>>>>>>>> unchanged. > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks, > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Murali > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 28, 2026 at 2:48 PM 黃竣陽 < > > > >>>>>>>>>>>>>> [email protected]> > > > >>>>>>>>>>>>>>>>>>>>> wrote: > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi chia, > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have updated the KIP to include this > > > >> change. > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards, > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <[email protected]> 於 > > > >>>>>>>>>> 2026年4月28日 > > > >>>>>>>>>>>>>> 晚上8:03 > > > >>>>>>>>>>>>>>>>>>> 寫道： > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> chia_0: Should we expose the partition > > > >> creation > > > >>>>>>>>>>> time > > > >>>>>>>>>>>>>> via > > > >>>>>>>>>>>>>>>>>>> the > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Admin > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> API? > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I assume it would be valuable for users > > > to > > > >>>>>>>>>> diagnose > > > >>>>>>>>>>>>> and > > > >>>>>>>>>>>>>>>>>>>>>>>>>> troubleshoot > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> the > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> behavior of > > > >> auto.offset.reset.latest.max.age > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/04/28 10:47:58 黃竣陽 wrote: > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone, > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a discussion on > > > >> KIP-1327 > > > >>>>>>>>>>>>>> Prevent > > > >>>>>>>>>>>>>>>>> Hot > > > >>>>>>>>>>>>>>>>>>>>> Data > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Loss > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Partition Expansion for Latest Policy > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> < > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>> > > > >>>>>>>>>>> > > > >>>>>>>>>> > > > >> > > > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$ > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to introduces > > > >>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age, > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> a > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config that lets the > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> latest reset policy distinguish newly > > > >> expanded > > > >>>>>>>>>>>>> (hot) > > > >>>>>>>>>>>>>>>>>>>>> partitions > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> from > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> long-existing (cold) ones. Partitions > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> younger than the configured threshold > > > >>>>>>>>>>> automatically > > > >>>>>>>>>>>>>> fall > > > >>>>>>>>>>>>>>>>>>>>> back > > > >>>>>>>>>>>>>>>>>>>>>>>>>> to > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> earliest, preventing silent data loss > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during topic expansion without forcing > > > >> a full > > > >>>>>>>>>>>>>> historical > > > >>>>>>>>>>>>>>>>>>>>>>>>>> reprocess. > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards, > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>> > > > >>>>>>>>>>> > > > >>>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>> > > > >>> > > > >>> > > > >>> > > > >> > > > > > > > > >
