Hi Jun, When a user adds an existing topic to a share group subscription, we do use the auto.offset.reset policy because those partitions already exist and may well have data on them. We only use 0 in the case where a topic already being consumed gained some additional new partitions. These partitions will be empty at the time of creation, so zero seems like a reasonable starting point. Partitions being consumed in a share group always have a known starting position once assigned. We don't have the equivalent of a consumer group not having a committed offset.
Thanks, Andrew On 2026/06/26 17:36:31 Jun Rao via dev wrote: > Hi, Andrew, > > Thanks for the reply. > > I am not sure it's always a good idea to start new partitions with offset 0 > for the share group. For example, once a share group starts, a user could > add a subscription to an existing topic. That existing topic could have a > large backlog and starting it with offset 0 forces the user to consume that > entire backlog. In KIP-1327, such a partition will be treated as an > existing partition and will follow the auto.offset.reset policy. This seems > more intuitive. It would be useful to consider if the share group should > adopt the same policy. > > Jun > > On Fri, Jun 26, 2026 at 3:42 AM Andrew Schofield <[email protected]> > wrote: > > > Hi Jiunn-Yang et al., > > Some background on the problem I hinted at for the KIP-932 implementation. > > KIP-932 states "If the number of partitions is increased for a topic with a > > subscription in a share group, the SPSO for the newly created > > share-partitions is initialized to 0 (which is of course both the earliest > > and latest offset for an empty topic-partition). This means there is no > > doubt about what happens when the number of partitions is increased." > > > > The problem was that we had not implemented the use of 0, but instead were > > using the share.auto.offset.reset config to choose the initial offset in > > this case. We write a ShareGroupStatePartitionMetadata record when a topic > > is first being initialized for consumption by a share group, and this > > contains a list of the partitions which we are initializing. As a result, I > > believe we can already tell when partitions are added, and we do know when > > to initialize to 0 as it says in the KIP. We will put in a patch that uses > > 0 as intended. > > > > I do not mind whether we want to incorporate share groups into KIP-1327, > > but I do not think that it is necessary to do so. > > > > Thanks, > > Andrew > > > > On 2026/06/26 04:31:28 Chia-Ping Tsai wrote: > > > > JR5. Since this changes the on-disk data format for PartitionRecord, we > > > > need to gate this by a new MV. It also changes the format of > > > > ConsumerGroupMetadataValue, so we need to gate it too. I'm not sure if > > this > > > > requires a new MV or a new GV. > > > > > > That is a good point. If we want to maintain consistency for the > > NewPartitions field when the group coordinator migrates between new and old > > binaries, a version gate is definitely required. For example: > > > > > > GV_2(2, MetadataVersion.IBP_4_4_IV0, Map.of()) > > > > > > The group coordinator should NOT set the NewPartitions field when the gv > > is smaller than gv_2 > > > > > > Best, > > > Chia-Ping > > > > > > On 2026/06/25 20:01:39 Jun Rao via dev wrote: > > > > Hi, Jiunn-Yang, > > > > > > > > JR2. We should try to maintain consistency across our APIs. So, it > > would be > > > > useful to understand if the user-facing part of this KIP is what share > > > > consumer wants too. It would be problematic if we proceed with this > > > > approach only for the share consumer to choose a different one later. > > > > > > > > JR3. I was thinking that if we add the NewPartitions field in > > > > GroupHeartbeatResponse, there is no need to expose the group/partition > > > > creation timestamp to other RPCs and the Cli. > > > > > > > > JR5. Since this changes the on-disk data format for PartitionRecord, we > > > > need to gate this by a new MV. It also changes the format of > > > > ConsumerGroupMetadataValue, so we need to gate it too. I'm not sure if > > this > > > > requires a new MV or a new GV. > > > > > > > > Thanks, > > > > > > > > Jun > > > > > > > > On Thu, Jun 25, 2026 at 5:52 AM 黃竣陽 <[email protected]> wrote: > > > > > > > > > Hello Jun, chia, > > > > > > > > > > Thanks for your feedback. > > > > > > > > > > JR3 JR4 I have updated the KIP. > > > > > > > > > > JR2 The new-partition detection mechanism is reusable. The > > coordinator > > > > > already has > > > > > access to both timestamps required for classification, so the > > > > > determination can be made > > > > > server-side without introducing any new RPCs. The main difference is > > where > > > > > the policy is > > > > > enforced: modern consumer resolves it on the client, whereas a share > > group > > > > > resolves the > > > > > starting offset on the partition leader. As a result, the > > classification > > > > > decision would need to > > > > > be propagated through the existing share-state initialization path. > > We > > > > > would also need to persist > > > > > the group's creation time as part of the share-group state, since > > that > > > > > information is not stored today. > > > > > > > > > > None of this is a blocker, but it does require changes to the > > share-state > > > > > format and introduces > > > > > a share-specific configuration. Given that additional scope, I would > > > > > prefer to keep share groups out > > > > > of scope for this KIP and address them in a follow-up effort, while > > > > > documenting the relevant reuse > > > > > opportunities here. > > > > > > > > > > Best Regards, > > > > > Jiunn-Yang > > > > > > > > > > > Chia-Ping Tsai <[email protected]> 於 2026年6月25日 下午1:31 寫道： > > > > > > > > > > > >> It inspires me to think about mirror clusters. What if the mirror > > group > > > > > is created 'after' the expanded partitions? In this scenario, the new > > > > > configuration 'max.age.ms' could handle it if users are aware of the > > > > > migration and set a suitable time interval. Of course, this would > > require > > > > > both creation times to be exposed to clients. > > > > > > > > > > > > Thinking more about the mirror cluster scenario we discussed, I > > realize > > > > > it introduces significant complexity. Handling it properly would > > require a > > > > > tolerance time window rather than a strict time interval to account > > for > > > > > replication lag. To prevent scope creep and keep KIP-1327 focused on > > its > > > > > primary goal, I think it would be best to defer the > > multi-cluster/mirroring > > > > > scenario and explore those approaches in a separate KIP > > > > > > > > > > > > On 2026/06/24 21:29:19 Chia-Ping Tsai wrote: > > > > > >> hi all, > > > > > >> > > > > > >>> JR3. The KIP exposes the group/partition creation time to the > > client. > > > > > I am > > > > > >>> not sure if there are other use cases for those timestamps. An > > > > > alternative > > > > > >>> is to avoid exposing those timestamps to the client. The group > > > > > coordinator > > > > > >>> will categorize the partitions using those timestamps and > > include the > > > > > >>> categorization in the assigned partitions in > > GroupHeartbeatResponse. > > > > > >> > > > > > >> It inspires me to think about mirror clusters. What if the mirror > > group > > > > > is created 'after' the expanded partitions? In this scenario, the new > > > > > configuration 'max.age.ms' could handle it if users are aware of the > > > > > migration and set a suitable time interval. Of course, this would > > require > > > > > both creation times to be exposed to clients. > > > > > >> > > > > > >> Best, > > > > > >> Chia-Ping > > > > > >> > > > > > >> On 2026/06/24 20:46:16 Jun Rao via dev wrote: > > > > > >>> Hi, Jiunn-Yang, > > > > > >>> > > > > > >>> Thanks for the updated KIP. > > > > > >>> > > > > > >>> JR2. Share consumer (KIP-932) states: "If the number of > > partitions is > > > > > >>> increased for a topic with a subscription in a share group, the > > SPSO > > > > > for > > > > > >>> the newly created share-partitions is initialized to 0 ". > > However, this > > > > > >>> part has not been implemented yet. Currently, all partitions > > without > > > > > offset > > > > > >>> follows the strategy given by share.auto.offset.reset. So, we > > have an > > > > > >>> opportunity to see if it's better to align this KIP's changes > > with the > > > > > >>> share consumer. > > > > > >>> JR2.1 The first part is whether customizing the offset for new > > > > > partitions > > > > > >>> to something other than 0 is useful in the share consumer. It > > seems > > > > > that > > > > > >>> the motivation in this KIP applies to share consumer too. > > > > > >>> JR2.2 The second part is how new partitions are detected. The > > mechanism > > > > > >>> introduced in this KIP seems quite effective. A group > > coordinator could > > > > > >>> pick up newly created partitions for an existing topic or newly > > created > > > > > >>> topics matching the regex. Those partitions are categorized as > > new > > > > > >>> partitions and will follow the strategy given by > > > > > >>> auto.offset.reset.new.partitions. > > > > > >>> This makes sense since those partitions typically don't have a > > large > > > > > >>> backlog. A consumer could also additionally subscribe to an > > existing > > > > > topic > > > > > >>> in an existing group. Those partitions will be categorized as > > existing > > > > > >>> partitions. This also makes sense since those partitions could > > have a > > > > > large > > > > > >>> backlog and it is better to follow the strategy given by > > > > > auto.offset.reset. It > > > > > >>> will be useful to think through whether the logic for > > categorizing new > > > > > >>> partitions should also be reused in the share consumer. > > > > > >>> > > > > > >>> JR3. The KIP exposes the group/partition creation time to the > > client. > > > > > I am > > > > > >>> not sure if there are other use cases for those timestamps. An > > > > > alternative > > > > > >>> is to avoid exposing those timestamps to the client. The group > > > > > coordinator > > > > > >>> will categorize the partitions using those timestamps and > > include the > > > > > >>> categorization in the assigned partitions in > > GroupHeartbeatResponse. > > > > > >>> > > > > > >>> JR4. It would be useful to support this feature on existing > > groups as > > > > > well. > > > > > >>> For example, any partition without a creation timestamp can > > always be > > > > > >>> categorized as an existing partition. > > > > > >>> > > > > > >>> Jun > > > > > >>> > > > > > >>> On Wed, Jun 24, 2026 at 6:12 AM 黃竣陽 <[email protected]> wrote: > > > > > >>> > > > > > >>>> Hello Andrew, > > > > > >>>> > > > > > >>>> Thanks for your feedback, > > > > > >>>>>> I've just been re-reading this KIP to ensure that share group > > > > > support is > > > > > >>>>>> not needed. I've discovered that we made a mistake > > implementing > > > > > KIP-932 > > > > > >>>> in > > > > > >>>>>> this area, so we'll rectify that and check whether it's > > really tight > > > > > >>>> enough > > > > > >>>>>> not to require KIP-1327. > > > > > >>>> > > > > > >>>> Could you walk us through the specific area where the mistake > > was > > > > > found? > > > > > >>>> That would > > > > > >>>> help us assess whether it naturally fits within the scope of > > KIP-1327 > > > > > or > > > > > >>>> should be addressed > > > > > >>>> separately. > > > > > >>>> > > > > > >>>> AS1 I have updated the KIP > > > > > >>>> > > > > > >>>> Best Regards, > > > > > >>>> Jiunn-Yang > > > > > >>>> > > > > > >>>>> Chia-Ping Tsai <[email protected]> 於 2026年6月24日 晚上8:49 寫道： > > > > > >>>>> > > > > > >>>>> hi Andrew > > > > > >>>>> > > > > > >>>>>> I've discovered that we made a mistake implementing KIP-932 > > in this > > > > > >>>> area, > > > > > >>>>> so we'll rectify that and check whether it's really tight > > enough not > > > > > to > > > > > >>>>> require KIP-1327. > > > > > >>>>> > > > > > >>>>> Would you mind sharing more details about that? If it's not > > anything > > > > > >>>> close > > > > > >>>>> to NP-hard, we could definitely look into covering it within > > KIP-1327 > > > > > >>>>> > > > > > >>>>> Best, > > > > > >>>>> Chia-Ping > > > > > >>>>> > > > > > >>>>> > > > > > >>>>> Andrew Schofield <[email protected]> 於 2026年6月24日週三 > > 下午7:49寫道： > > > > > >>>>> > > > > > >>>>>> Hi Jiunn-Yang, > > > > > >>>>>> I've just been re-reading this KIP to ensure that share group > > > > > support is > > > > > >>>>>> not needed. I've discovered that we made a mistake > > implementing > > > > > KIP-932 > > > > > >>>> in > > > > > >>>>>> this area, so we'll rectify that and check whether it's > > really tight > > > > > >>>> enough > > > > > >>>>>> not to require KIP-1327. > > > > > >>>>>> > > > > > >>>>>> One trivial comments for consistency. > > > > > >>>>>> > > > > > >>>>>> AS1: In kafka-consumer-groups.sh, unknown column data is > > > > > represented by > > > > > >>>>>> "-", not "N/A". In kafka-topics.sh, we do use "N/A". > > > > > >>>>>> > > > > > >>>>>> Thanks, > > > > > >>>>>> Andrew > > > > > >>>>>> > > > > > >>>>>> On 2026/06/23 23:23:38 黃竣陽 wrote: > > > > > >>>>>>> Hi all, > > > > > >>>>>>> > > > > > >>>>>>> Manually bumping this thread. > > > > > >>>>>>> > > > > > >>>>>>> Best Regards, > > > > > >>>>>>> Jiunn-Yang > > > > > >>>>>>> > > > > > >>>>>>>> 黃竣陽 <[email protected]> 於 2026年6月17日 晚上9:17 寫道： > > > > > >>>>>>>> > > > > > >>>>>>>> Hello chia, > > > > > >>>>>>>> > > > > > >>>>>>>> Thanks for the feedback, I have updated the KIP. > > > > > >>>>>>>> > > > > > >>>>>>>> Best Regards, > > > > > >>>>>>>> Jiunn-Yang > > > > > >>>>>>>> > > > > > >>>>>>>>> Chia-Ping Tsai <[email protected]> 於 2026年6月17日 凌晨12:47 > > 寫道： > > > > > >>>>>>>>> > > > > > >>>>>>>>> hi Jiunn-Yang > > > > > >>>>>>>>> > > > > > >>>>>>>>>> When the config is set on a cluster that has not yet been > > > > > >>>>>> upgraded... classification cannot occur... the consumer falls > > back > > > > > to > > > > > >>>> the > > > > > >>>>>> base auto.offset.reset for the affected partitions. No > > exception is > > > > > >>>> thrown, > > > > > >>>>>> and no operational disruption results. > > > > > >>>>>>>>> > > > > > >>>>>>>>> Existing group can't take advantage of this excellent new > > > > > >>>>>> configuration. Allowing users to modify the group creation > > time > > > > > might be > > > > > >>>>>> overkill. Instead, we could print a useful warning message to > > guide > > > > > >>>> users. > > > > > >>>>>> For example, we can suggest that they re-create the group > > with their > > > > > >>>>>> existing committed offsets > > > > > >>>>>>>>> > > > > > >>>>>>>>>> Protocol changes > > > > > >>>>>>>>> > > > > > >>>>>>>>> Would you mind listing those RPC changes in a table format? > > > > > >>>>>>>>> > > > > > >>>>>>>>>> The full interaction matrix between the base policy and > > the > > > > > >>>>>> new-partition policy is: > > > > > >>>>>>>>> > > > > > >>>>>>>>> Please add a filed to describe the target scenario when > > using > > > > > these > > > > > >>>>>> policies > > > > > >>>>>>>>> > > > > > >>>>>>>>> Best, > > > > > >>>>>>>>> Chia-Ping > > > > > >>>>>>>>> > > > > > >>>>>>>>> > > > > > >>>>>>>>> On 2026/06/16 16:14:49 黃竣陽 wrote: > > > > > >>>>>>>>>> Hello Jun, chia, > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> Thanks for the feedback, I have updated the KIP for the > > new > > > > > >>>>>>>>>> approach, PTAL > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> Best Regards, > > > > > >>>>>>>>>> Jiunn-Yang > > > > > >>>>>>>>>> > > > > > >>>>>>>>>>> Chia-Ping Tsai <[email protected]> 於 2026年6月16日 上午8:23 > > 寫道： > > > > > >>>>>>>>>>> > > > > > >>>>>>>>>>> hi Jun > > > > > >>>>>>>>>>> > > > > > >>>>>>>>>>> Yes, your approach is great. I think the combination of > > latest > > > > > (for > > > > > >>>>>> existing partitions) and by_duration (for new partitions) can > > > > > address > > > > > >>>> 99% > > > > > >>>>>> of the complaints I have heard regarding this issue. > > > > > >>>>>>>>>>> > > > > > >>>>>>>>>>> Also, leveraging the group creation time here opens the > > door to > > > > > >>>>>> implementing a new policy based on timestamp seek in the > > future, > > > > > should > > > > > >>>> the > > > > > >>>>>> community want to pursue that. > > > > > >>>>>>>>>>> > > > > > >>>>>>>>>>> Thanks for your patience and constructive feedback. We > > will > > > > > update > > > > > >>>>>> the KIP accordingly. > > > > > >>>>>>>>>>> > > > > > >>>>>>>>>>> Best, Chia-Ping > > > > > >>>>>>>>>>> > > > > > >>>>>>>>>>>> Jun Rao via dev <[email protected]> 於 2026年6月16日 > > 清晨5:11 > > > > > 寫道： > > > > > >>>>>>>>>>>> > > > > > >>>>>>>>>>>> Hi, Chia-Ping, > > > > > >>>>>>>>>>>> > > > > > >>>>>>>>>>>> Thanks for the reply. > > > > > >>>>>>>>>>>> > > > > > >>>>>>>>>>>> I agree that it's probably useful to allow a user to > > > > > configure a > > > > > >>>>>> different > > > > > >>>>>>>>>>>> offset policy for existing partitions vs new partitions. > > > > > However, > > > > > >>>>>> using > > > > > >>>>>>>>>>>> group creation time to capture that seems more > > intuitive. > > > > > Here is > > > > > >>>>>> another > > > > > >>>>>>>>>>>> proposal: remove auto.offset.reset.max.age.ms and > > categorize > > > > > new > > > > > >>>>>> partitions > > > > > >>>>>>>>>>>> based on group creation time. Introduce > > > > > >>>>>>>>>>>> a new config auto.offset.reset.new.partitions whose > > values > > > > > can be > > > > > >>>>>> earliest, > > > > > >>>>>>>>>>>> latest and by_duration, the same as auto.offset.reset. > > Users > > > > > can > > > > > >>>>>> set > > > > > >>>>>>>>>>>> `auto.offset.reset.new.partitions` to `earliest` if > > they > > > > > want to > > > > > >>>>>> guarantee > > > > > >>>>>>>>>>>> no data loss on new partitions. They can also use > > by_duration > > > > > to > > > > > >>>>>> set an > > > > > >>>>>>>>>>>> upper bound on the backlog replayed, which can be > > different > > > > > from > > > > > >>>>>> that of > > > > > >>>>>>>>>>>> the existing partitions. This will address your concern > > about > > > > > too > > > > > >>>>>> much > > > > > >>>>>>>>>>>> backlog being replayed when the offsets are lost. What > > do you > > > > > >>>>>> think? > > > > > >>>>>>>>>>>> > > > > > >>>>>>>>>>>> Jun > > > > > >>>>>>>>>>>> > > > > > >>>>>>>>>>>> > > > > > >>>>>>>>>>>>> On Mon, Jun 15, 2026 at 10:39 AM Chia-Ping Tsai < > > > > > >>>>>> [email protected]> wrote: > > > > > >>>>>>>>>>>>> > > > > > >>>>>>>>>>>>> hi Jun > > > > > >>>>>>>>>>>>> > > > > > >>>>>>>>>>>>> The most important part of this story is how users > > should > > > > > expect > > > > > >>>>>> the data > > > > > >>>>>>>>>>>>> they can see when using the latest or by_duration > > policy with > > > > > >>>>>> expanded > > > > > >>>>>>>>>>>>> partitions. > > > > > >>>>>>>>>>>>> > > > > > >>>>>>>>>>>>> Yes, the by_duration policy can minimize data loss, > > but it is > > > > > >>>>>>>>>>>>> non-deterministic, which means users will either read > > too > > > > > many > > > > > >>>>>> historical > > > > > >>>>>>>>>>>>> records from existing partitions or lose some records > > from > > > > > >>>>>> expanded > > > > > >>>>>>>>>>>>> partitions. > > > > > >>>>>>>>>>>>> > > > > > >>>>>>>>>>>>> Also, I agree that auto.offset.reset.max.age.ms > > > > > >>>>>>>>>>>>> < > > > > > >>>>>> > > > > > >>>> > > > > > > > https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$ > > > > > >>>>>>> > > > > > >>>>>>>>>>>>> is a bit hard to understand, and that is why I > > preferred > > > > > having a > > > > > >>>>>> whole new > > > > > >>>>>>>>>>>>> policy based entirely on group creation time (KIP-1282) > > > > > >>>>>>>>>>>>> > > > > > >>>>>>>>>>>>> Best, > > > > > >>>>>>>>>>>>> Chia-Ping > > > > > >>>>>>>>>>>>> > > > > > >>>>>>>>>>>>> Jun Rao via dev <[email protected]> 於 2026年6月16日週二 > > > > > 上午1:08寫道： > > > > > >>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>> Hi, Chia-Ping and Jiunn-Yang, > > > > > >>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>> Thanks for the reply. I am still trying to understand > > the > > > > > value > > > > > >>>>>> of the new > > > > > >>>>>>>>>>>>>> configs with the KIP. > > > > > >>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>> The motivation of the KIP is that a user doesn't want > > to > > > > > miss > > > > > >>>>>> the data if > > > > > >>>>>>>>>>>>>> the backlog is small. The backlog of the existing > > partition > > > > > is > > > > > >>>>>> easy to > > > > > >>>>>>>>>>>>>> understand because it relates to retention time. The > > > > > backlog for > > > > > >>>>>> the new > > > > > >>>>>>>>>>>>>> partition is a bit subtle to understand since it > > depends on > > > > > the > > > > > >>>>>> metadata > > > > > >>>>>>>>>>>>>> refresh delay. To set auto.offset.reset.max.age.ms > > > > > >>>>>>>>>>>>>> < > > > > > >>>>>> > > > > > >>>> > > > > > > > https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$ > > > > > >>>>>>> , > > > > > >>>>>>>>>>>>>> the user needs to > > > > > >>>>>>>>>>>>>> understand the metadata refresh delay on the consumer > > side > > > > > and > > > > > >>>>>> use it to > > > > > >>>>>>>>>>>>>> set the config. > > > > > >>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>> Now, let's consider the alternative: setting the same > > value > > > > > for > > > > > >>>>>> the > > > > > >>>>>>>>>>>>>> existing by_duration policy. The KIP lists three > > issues with > > > > > >>>> this > > > > > >>>>>>>>>>>>>> approach. > > > > > >>>>>>>>>>>>>> 1. It computes the seek target client-side as now() - > > > > > duration, > > > > > >>>>>> which > > > > > >>>>>>>>>>>>>> introduces clock skew across consumers and forces > > operators > > > > > to > > > > > >>>>>> choose > > > > > >>>>>>>>>>>>>> overly large durations, causing unnecessary > > reprocessing. > > > > > >>>>>>>>>>>>>> 2. The target timestamp is recomputed on each retry, > > so > > > > > failed > > > > > >>>>>>>>>>>>>> ListOffsetsRequest retries can shift the target > > forward and > > > > > >>>>>> potentially > > > > > >>>>>>>>>>>>>> miss records produced between attempts. > > > > > >>>>>>>>>>>>>> 3. It applies uniformly to all partitions without > > committed > > > > > >>>>>> offsets, and > > > > > >>>>>>>>>>>>>> cannot distinguish newly expanded partitions from > > > > > long-existing > > > > > >>>>>> partitions > > > > > >>>>>>>>>>>>>> newly assigned to the group, leading to unnecessary > > replay. > > > > > >>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>> Issues 1 and 2 are uncommon and can be mitigated by > > adding > > > > > a bit > > > > > >>>>>> buffer to > > > > > >>>>>>>>>>>>>> the metadata refresh delay. We could also consider > > > > > improving the > > > > > >>>>>>>>>>>>>> implementation. For issue 3, the metadata refresh > > delay is > > > > > >>>>>> typically low > > > > > >>>>>>>>>>>>>> (in the order of minutes with the classic consumer > > and tens > > > > > of > > > > > >>>>>> seconds > > > > > >>>>>>>>>>>>>> with > > > > > >>>>>>>>>>>>>> the new consumer). If a user is ok with reading that > > much > > > > > >>>>>> backlog for new > > > > > >>>>>>>>>>>>>> partitions, it seems they will be ok doing the same > > for > > > > > existing > > > > > >>>>>>>>>>>>>> partitions. > > > > > >>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>> So, instead of introducing a new config, could we just > > > > > reuse the > > > > > >>>>>> existing > > > > > >>>>>>>>>>>>>> config with better documentation and/or > > implementation? > > > > > >>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>> Jun > > > > > >>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>> On Sat, Jun 13, 2026 at 12:19 AM 黃竣陽 < > > [email protected]> > > > > > >>>>>> wrote: > > > > > >>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>> Hello Jun, > > > > > >>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>> You're right that group creation time is the more > > intuitive > > > > > >>>>>> answer at > > > > > >>>>>>>>>>>>>>> first glance, > > > > > >>>>>>>>>>>>>>> the KIP's own motivation talks about partitions that > > > > > "predate > > > > > >>>>>> the group" > > > > > >>>>>>>>>>>>>>> vs partitions > > > > > >>>>>>>>>>>>>>> "created during group runtime," which directly > > points to a > > > > > >>>>>>>>>>>>>> group-lifecycle > > > > > >>>>>>>>>>>>>>> classifier. > > > > > >>>>>>>>>>>>>>> I'd like to walk through why we landed on partition > > age, > > > > > and > > > > > >>>> the > > > > > >>>>>>>>>>>>>>> trade-offs we considered. > > > > > >>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>> We evaluated three candidate signals: > > > > > >>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>> 1. `by_duration:5secs` > > > > > >>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>> This covers the metadata blindness window, but has > > issues > > > > > the > > > > > >>>>>> KIP > > > > > >>>>>>>>>>>>>>> currently documents > > > > > >>>>>>>>>>>>>>> under "Why not use `by_duration`?": > > > > > >>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>> - Client-side `now() - duration` introduces clock > > skew > > > > > across > > > > > >>>>>> consumers. > > > > > >>>>>>>>>>>>>>> - `ListOffsets` retries shift the target forward, > > > > > potentially > > > > > >>>>>> missing > > > > > >>>>>>>>>>>>>>> records produced between > > > > > >>>>>>>>>>>>>>> attempts. > > > > > >>>>>>>>>>>>>>> - It applies uniformly to all partitions without > > committed > > > > > >>>>>> offsets, > > > > > >>>>>>>>>>>>>>> including pre-existing partitions > > > > > >>>>>>>>>>>>>>> newly assigned to the group, causing unnecessary > > replay. > > > > > >>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>> 2. Group creation time as classifier > > > > > >>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>> This works cleanly when the consumer is actively > > running. > > > > > Our > > > > > >>>>>> concern > > > > > >>>>>>>>>>>>>>> is the idle / late-rejoin case: > > > > > >>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>> T=0: Group created. > > > > > >>>>>>>>>>>>>>> T=1..T=100: Consumer idle (down, disconnected, > > etc.). > > > > > >>>>>>>>>>>>>>> T=50: Partition added during the idle window. > > > > > >>>>>>>>>>>>>>> T=100: Consumer resumes. > > > > > >>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>> Under group creation time, the new partition is > > classified > > > > > as > > > > > >>>>>> new > > > > > >>>>>>>>>>>>>>> (`50 > 0`) and reset to `earliest`, replaying > > everything > > > > > from > > > > > >>>>>> T=50. > > > > > >>>>>>>>>>>>>>> But during `[T=1, T=100]`, base partitions also > > accumulated > > > > > >>>>>> data that > > > > > >>>>>>>>>>>>>>> the consumer accepts as lost — that is precisely the > > > > > contract > > > > > >>>> of > > > > > >>>>>>>>>>>>>>> `auto.offset.reset=latest`. There is no principled > > reason > > > > > to > > > > > >>>>>> treat > > > > > >>>>>>>>>>>>>>> the new partition differently; both contain backlog > > > > > accumulated > > > > > >>>>>> during > > > > > >>>>>>>>>>>>>>> the same idle window. > > > > > >>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>> This aligns with the "backlog is backlog” principle > > you > > > > > raised > > > > > >>>>>> in > > > > > >>>>>>>>>>>>>>> the KIP-1282 thread: a `latest` user has tolerated > > some > > > > > backlog > > > > > >>>>>> on > > > > > >>>>>>>>>>>>>>> every other partition during the same idle period; > > forcing > > > > > >>>>>> 0-backlog > > > > > >>>>>>>>>>>>>>> tolerance only on new partitions would be > > inconsistent with > > > > > >>>> that > > > > > >>>>>>>>>>>>>>> tolerance. > > > > > >>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>> 3. Partition age vs threshold > > > > > >>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>> Partition age corresponds to the actual silent data > > loss > > > > > >>>> window, > > > > > >>>>>>>>>>>>>>> the gap between partition creation and the consumer’s > > > > > metadata > > > > > >>>>>>>>>>>>>>> refresh. Within this window, data loss is genuinely > > > > > silent: the > > > > > >>>>>>>>>>>>>>> consumer had no opportunity to know about the > > partition. > > > > > >>>>>> Outside this > > > > > >>>>>>>>>>>>>>> window, missing data reflects either: > > > > > >>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>> - (a) the user’s tolerated cost of running with idle > > > > > consumers, > > > > > >>>>>> or > > > > > >>>>>>>>>>>>>>> - (b) an operational issue to surface via > > monitoring, not > > > > > via > > > > > >>>>>> reset > > > > > >>>>>>>>>>>>>> policy. > > > > > >>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>> We did not choose partition age because it is more > > elegant > > > > > than > > > > > >>>>>> group > > > > > >>>>>>>>>>>>>>> creation time — we chose it because its failure mode > > > > > (requires > > > > > >>>> a > > > > > >>>>>>>>>>>>>>> threshold) is > > > > > >>>>>>>>>>>>>>> less invasive than the failure mode of group > > creation time > > > > > >>>>>> (overrides > > > > > >>>>>>>>>>>>>>> user-stated > > > > > >>>>>>>>>>>>>>> `latest` intent during idle periods). > > > > > >>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>> Best Regards, > > > > > >>>>>>>>>>>>>>> Jiunn-Yang > > > > > >>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>> Chia-Ping Tsai <[email protected]> 於 2026年6月13日 > > 上午11:52 > > > > > 寫道： > > > > > >>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>> Hi Jun, > > > > > >>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>> Relying on both creation times will create an > > inconsistent > > > > > >>>>>> scenario. A > > > > > >>>>>>>>>>>>>>>> consumer that lost all offsets due to a long sleep > > will > > > > > seek > > > > > >>>>>> to the > > > > > >>>>>>>>>>>>>>>> beginning for the partitions created later than the > > group. > > > > > >>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>> That is why we initially proposed KIP-1282 to fix > > the > > > > > >>>>>> inconsistency > > > > > >>>>>>>>>>>>>>> using a > > > > > >>>>>>>>>>>>>>>> whole new policy. Since KIP-1282 couldn't reach a > > > > > consensus, > > > > > >>>>>> KIP-1327 > > > > > >>>>>>>>>>>>>>> goes > > > > > >>>>>>>>>>>>>>>> back to using flexible configurations to prevent > > users > > > > > from > > > > > >>>>>> falling > > > > > >>>>>>>>>>>>>> into > > > > > >>>>>>>>>>>>>>>> that pitfall. > > > > > >>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>> Best, Chia-Ping > > > > > >>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>> Jun Rao via dev <[email protected]> 於 > > 2026年6月13日週六 > > > > > >>>>>> 上午6:49寫道： > > > > > >>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>> Hi, Jiunn-Yang, > > > > > >>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>> Thanks for the reply and sorry for the late reply. > > > > > >>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>> JR1. The design of auto.offset.reset.max.age.ms > > > > > >>>>>>>>>>>>>> < > > > > > >>>>>> > > > > > >>>> > > > > > > > https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$ > > > > > >>>>>>> > > > > > >>>>>>>>>>>>>> still feels weird to > > > > > >>>>>>>>>>>>>>> me. > > > > > >>>>>>>>>>>>>>>>> It > > > > > >>>>>>>>>>>>>>>>> categorizes partitions as new or existing based on > > the > > > > > >>>>>> partition > > > > > >>>>>>>>>>>>>>> creation > > > > > >>>>>>>>>>>>>>>>> time. Intuitively, the categorization should be > > based on > > > > > the > > > > > >>>>>> group > > > > > >>>>>>>>>>>>>>> creation > > > > > >>>>>>>>>>>>>>>>> time: all partitions existing when the group is > > created > > > > > are > > > > > >>>>>> existing > > > > > >>>>>>>>>>>>>> and > > > > > >>>>>>>>>>>>>>>>> all partitions created after the group creation > > are new > > > > > >>>>>> partitions. > > > > > >>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>> Jun > > > > > >>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>> On Tue, Jun 9, 2026 at 8:51 AM 黃竣陽 < > > [email protected]> > > > > > >>>>>> wrote: > > > > > >>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>> Hi all, > > > > > >>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>> Manually bumping this thread. If there is no > > further > > > > > >>>>>>>>>>>>>>>>>> discussion, I will close the vote. > > > > > >>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>> Best Regards, > > > > > >>>>>>>>>>>>>>>>>> Jiunn-Yang > > > > > >>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>> 黃竣陽 <[email protected]> 於 2026年6月1日 晚上7:16 寫道： > > > > > >>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>> Hello Jian, > > > > > >>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>> Thanks for your feedback, > > > > > >>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>> Agreed, partition expansion is a common > > operational > > > > > task, > > > > > >>>>>> not an > > > > > >>>>>>>>>>>>>> edge > > > > > >>>>>>>>>>>>>>>>>>> case. I've updated the Motivation section > > accordingly. > > > > > >>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>> Best Regards, > > > > > >>>>>>>>>>>>>>>>>>> Jiunn-Yang > > > > > >>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>> jian fu <[email protected]> 於 2026年6月1日 > > 下午5:49 寫道： > > > > > >>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>> Hi Jiunn-Yang: > > > > > >>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>> Thanks for the KIP. I think it would be useful > > to > > > > > clarify > > > > > >>>>>> that > > > > > >>>>>>>>>>>>>> this > > > > > >>>>>>>>>>>>>>>>> is a > > > > > >>>>>>>>>>>>>>>>>>>> common scenario rather than an edge case, which > > > > > further > > > > > >>>>>>>>>>>>>> demonstrates > > > > > >>>>>>>>>>>>>>>>> the > > > > > >>>>>>>>>>>>>>>>>>>> need for this optimization. For example: > > > > > >>>>>>>>>>>>>>>>>>>> A partition expansion is a common operational > > task in > > > > > >>>>>> Kafka: To > > > > > >>>>>>>>>>>>>>>>> balance > > > > > >>>>>>>>>>>>>>>>>>>> resource utilization and cost, topics are > > typically > > > > > >>>>>> created with a > > > > > >>>>>>>>>>>>>>>>>> moderate > > > > > >>>>>>>>>>>>>>>>>>>> default partition count. However, as traffic > > grows > > > > > over > > > > > >>>>>> time, it > > > > > >>>>>>>>>>>>>> is > > > > > >>>>>>>>>>>>>>>>>> often > > > > > >>>>>>>>>>>>>>>>>>>> necessary to increase the number of partitions > > to > > > > > >>>>>> accommodate the > > > > > >>>>>>>>>>>>>>>>> higher > > > > > >>>>>>>>>>>>>>>>>>>> workload. > > > > > >>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>> Regards > > > > > >>>>>>>>>>>>>>>>>>>> Jian > > > > > >>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>> 黃竣陽 <[email protected]> 于2026年5月30日周六 22:31写道： > > > > > >>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>> Hello chia, > > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>> Thanks for the comments, I have updated the > > KIP! > > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>> Best Regards, > > > > > >>>>>>>>>>>>>>>>>>>>> Jiunn-Yang > > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <[email protected]> 於 > > 2026年5月30日 > > > > > 晚上8:29 > > > > > >>>>>> 寫道： > > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>> Hi Jiunn-Yang, > > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>> Would you mind removing the terms "hot" and > > "cold" > > > > > when > > > > > >>>>>>>>>>>>>> describing > > > > > >>>>>>>>>>>>>>>>>>>>>> partitions in the KIP? I understand you are > > using > > > > > them > > > > > >>>> to > > > > > >>>>>>>>>>>>>> describe > > > > > >>>>>>>>>>>>>>>>> the > > > > > >>>>>>>>>>>>>>>>>>>>>> "freshness" or the users' need for the > > records, but > > > > > >>>>>> applying > > > > > >>>>>>>>>>>>>> these > > > > > >>>>>>>>>>>>>>>>>> terms > > > > > >>>>>>>>>>>>>>>>>>>>> to > > > > > >>>>>>>>>>>>>>>>>>>>>> the partition itself feels a bit unnatural. > > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>> After all, in this scenario, users don't > > really care > > > > > >>>>>> whether a > > > > > >>>>>>>>>>>>>>>>>> partition > > > > > >>>>>>>>>>>>>>>>>>>>> is > > > > > >>>>>>>>>>>>>>>>>>>>>> newly expanded or not. Their only expectation > > is > > > > > that > > > > > >>>>>> they won't > > > > > >>>>>>>>>>>>>>>>>> silently > > > > > >>>>>>>>>>>>>>>>>>>>>> lose any live records produced to the topic > > during > > > > > their > > > > > >>>>>> active > > > > > >>>>>>>>>>>>>>>>>>>>> consumption. > > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>> Best, Chia-Ping > > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>> 黃竣陽 <[email protected]> 於 2026年5月30日週六 > > 下午12:30寫道： > > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> Hello Jun, > > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback, I have updated the > > KIP > > > > > >>>>>> motivation > > > > > >>>>>>>>>>>>>>> section. > > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> Best Regards, > > > > > >>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang > > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> Jun Rao via dev <[email protected]> 於 > > > > > 2026年5月30日 > > > > > >>>>>> 凌晨1:12 > > > > > >>>>>>>>>>>>>> 寫道： > > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang, > > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply. I think we need a > > stronger > > > > > >>>>>> motivation > > > > > >>>>>>>>>>>>>> for > > > > > >>>>>>>>>>>>>>>>> the > > > > > >>>>>>>>>>>>>>>>>>>>> KIP. > > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> The KIP says "The core insight is that not > > all > > > > > >>>>>> partitions > > > > > >>>>>>>>>>>>>> without > > > > > >>>>>>>>>>>>>>>>> a > > > > > >>>>>>>>>>>>>>>>>>>>>>>> committed offset are the same. A newly > > expanded > > > > > >>>>>> partition > > > > > >>>>>>>>>>>>>> (hot) > > > > > >>>>>>>>>>>>>>> is > > > > > >>>>>>>>>>>>>>>>>>>>>>>> fundamentally different from a partition the > > > > > consumer > > > > > >>>>>> has > > > > > >>>>>>>>>>>>>> never > > > > > >>>>>>>>>>>>>>>>> seen > > > > > >>>>>>>>>>>>>>>>>>>>>>>> because it predates the group (cold)." Why > > is the > > > > > hot > > > > > >>>>>>>>>>>>>> partition > > > > > >>>>>>>>>>>>>>>>>>>>>>>> fundamentally different from the cold? > > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> The KIP says "The existing by_duration > > policy is > > > > > also > > > > > >>>>>>>>>>>>>>> insufficient > > > > > >>>>>>>>>>>>>>>>>>>>>>> because: > > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> - The calculated seek time (now() - > > duration) > > > > > varies > > > > > >>>>>> across > > > > > >>>>>>>>>>>>>> nodes > > > > > >>>>>>>>>>>>>>>>>> due > > > > > >>>>>>>>>>>>>>>>>>>>>>> to > > > > > >>>>>>>>>>>>>>>>>>>>>>>> clock skew. To be safe, users must set an > > overly > > > > > large > > > > > >>>>>>>>>>>>>> duration, > > > > > >>>>>>>>>>>>>>>>>>>>>>> causing > > > > > >>>>>>>>>>>>>>>>>>>>>>>> unnecessary reprocessing. > > > > > >>>>>>>>>>>>>>>>>>>>>>>> - On network errors, the client > > recalculates the > > > > > seek > > > > > >>>>>> time on > > > > > >>>>>>>>>>>>>>>>> retry, > > > > > >>>>>>>>>>>>>>>>>>>>>>>> shifting the target timestamp forward and > > risking > > > > > data > > > > > >>>>>> loss." > > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> However, both of these situations are rare. > > If > > > > > these > > > > > >>>>>> issues > > > > > >>>>>>>>>>>>>>>>> persist, > > > > > >>>>>>>>>>>>>>>>>>>>> more > > > > > >>>>>>>>>>>>>>>>>>>>>>>> severe problems likely exist elsewhere. Rare > > > > > >>>>>> situations don't > > > > > >>>>>>>>>>>>>>>>> need a > > > > > >>>>>>>>>>>>>>>>>>>>>>> common > > > > > >>>>>>>>>>>>>>>>>>>>>>>> solution. If users care about those rare > > > > > situations, > > > > > >>>>>> they can > > > > > >>>>>>>>>>>>>>>>>> implement > > > > > >>>>>>>>>>>>>>>>>>>>>>>> customized logic using > > > > > >>>>>>>>>>>>>>>>>>>>> > > ConsumerRebalanceListener.onPartitionsAssigned(). > > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> Jun > > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> On Sun, May 17, 2026 at 6:50 AM 黃竣陽 < > > > > > >>>>>> [email protected]> > > > > > >>>>>>>>>>>>>> wrote: > > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Hello chia, > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback, > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> If the creation time exists, the returned > > value > > > > > >>>>>> should > > > > > >>>>>>>>>>>>>> always > > > > > >>>>>>>>>>>>>>> be > > > > > >>>>>>>>>>>>>>>>>>>>>>> greater > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> than or equal to zero, right? > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> I have explicitly mentioned this in the > > KIP. > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> New Old (MetadataResponse v0–13) > > positive > > > > > >>>>>> any > > > > > >>>>>>>>>>>>>>>>>> field > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> absent UnsupportedVersionException > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> The earliest point at which we can detect > > the > > > > > version > > > > > >>>>>>>>>>>>>> mismatch > > > > > >>>>>>>>>>>>>>> is > > > > > >>>>>>>>>>>>>>>>>>>>> during > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> the > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> first metadata fetch after assignment, > > which > > > > > occurs > > > > > >>>>>> inside > > > > > >>>>>>>>>>>>>>>>> poll(). > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Therefore, the > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> user would encounter an > > > > > UnsupportedVersionException > > > > > >>>>>> from > > > > > >>>>>>>>>>>>>> poll(). > > > > > >>>>>>>>>>>>>>>>>> I’ll > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> clarify this in the KIP. > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Best Regards, > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <[email protected]> 於 > > > > > 2026年5月17日 > > > > > >>>>>> 下午4:50 > > > > > >>>>>>>>>>>>>> 寫道： > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> PartitionAgeMs (int64, default -1): The > > age of > > > > > this > > > > > >>>>>>>>>>>>>> partition > > > > > >>>>>>>>>>>>>>>>> in > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> milliseconds, computed server-side by the > > broker > > > > > as > > > > > >>>>>>>>>>>>>>>>>>>>> broker_current_time > > > > > >>>>>>>>>>>>>>>>>>>>>>> - > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> partition_creation_time. Returns -1 if the > > broker > > > > > >>>>>> does not > > > > > >>>>>>>>>>>>>>>>> support > > > > > >>>>>>>>>>>>>>>>>>>>> this > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> feature or the partition creation time is > > > > > unknown. > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> If the creation time exists, the returned > > value > > > > > >>>>>> should > > > > > >>>>>>>>>>>>>> always > > > > > >>>>>>>>>>>>>>> be > > > > > >>>>>>>>>>>>>>>>>>>>>>> greater > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> than or equal to zero, right? > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> New Old (MetadataResponse v0–13) > > positive > > > > > >>>>>> any > > > > > >>>>>>>>>>>>>>>>>> field > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> absent UnsupportedVersionException > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Will user encounter > > UnsupportedVersionException > > > > > when > > > > > >>>>>> calling > > > > > >>>>>>>>>>>>>>>>>>>>> `poll()`? > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Best, > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/05/16 04:30:49 黃竣陽 wrote: > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Hello Jun, chia, > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> I've updated KIP-1327 with a design > > change > > > > > based on > > > > > >>>>>> the > > > > > >>>>>>>>>>>>>>>>>> discussion > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> feedback. > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> The updated design decouples the > > new-partition > > > > > >>>> reset > > > > > >>>>>>>>>>>>>> behavior > > > > > >>>>>>>>>>>>>>>>>> from > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> the base auto.offset.reset policy: > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> - auto.offset.reset.max.age.ms > > > > > >>>>>>>>>>>>>> < > > > > > >>>>>> > > > > > >>>> > > > > > > > https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$ > > > > > >>>>>>> > > > > > >>>>>>>>>>>>>> now applies to all > > > > > >>>>>>>>>>>>>>>>>> auto.offset.reset > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> values > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> (latest, earliest, by_duration, none). > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> - For new ("hot") partitions, the > > consumer > > > > > resets > > > > > >>>> to > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.new > > > > > >>>>>>>>>>>>>> < > > > > > >>>>>> > > > > > >>>> > > > > > > > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$ > > > > > >>>>>>> > > > > > >>>>>>>>>>>>>> .partitions > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> config setting > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> - For existing ("cold") partitions, the > > base > > > > > >>>>>>>>>>>>>> auto.offset.reset > > > > > >>>>>>>>>>>>>>>>>>>>> policy > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> continues > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> to apply unchanged. > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> - The new-partition reset behavior is > > > > > represented > > > > > >>>>>> by a > > > > > >>>>>>>>>>>>>>> separate > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> internal config > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> (auto.offset.reset.new > > > > > >>>>>>>>>>>>>> < > > > > > >>>>>> > > > > > >>>> > > > > > > > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$ > > > > > >>>>>>> .partitions, > > > > > >>>>>>>>>>>>>> currently fixed to > > > > > >>>>>>>>>>>>>>>>> earliest). > > > > > >>>>>>>>>>>>>>>>>>>>> This > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> decoupled design makes > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> it straightforward to promote the > > behavior to a > > > > > >>>>>> public > > > > > >>>>>>>>>>>>>>>>>> user-facing > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> configuration in a future KIP. > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards, > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <[email protected]> 於 > > > > > 2026年5月16日 > > > > > >>>>>> 清晨7:46 > > > > > >>>>>>>>>>>>>> 寫道： > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jun > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I see what you mean now. The proposal > > from me > > > > > is > > > > > >>>>>> listed > > > > > >>>>>>>>>>>>>>> below: > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 1) Add auto.offset.reset.new > > > > > >>>>>>>>>>>>>> < > > > > > >>>>>> > > > > > >>>> > > > > > > > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$ > > > > > >>>>>>> .partitions > > > > > >>>>>>>>>>>>>> with a default value > > > > > >>>>>>>>>>>>>>>>> of > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> earliest. It fixes the data loss from both > > > > > >>>>>> by_duration and > > > > > >>>>>>>>>>>>>>>>> latest, > > > > > >>>>>>>>>>>>>>>>>> and > > > > > >>>>>>>>>>>>>>>>>>>>>>> it > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> does not change the logic of > > > > > >>>>>> auto.offset.reset=earliest. > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 2) Mark auto.offset.reset.new > > > > > >>>>>>>>>>>>>> < > > > > > >>>>>> > > > > > >>>> > > > > > > > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$ > > > > > >>>>>>> .partitions > > > > > >>>>>>>>>>>>>> as an internal > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> configuration. auto.offset.reset.new > > > > > >>>>>>>>>>>>>> < > > > > > >>>>>> > > > > > >>>> > > > > > > > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$ > > > > > >>>>>>> > > > > > >>>>>>>>>>>>>> .partitions=earliest > > > > > >>>>>>>>>>>>>>> already > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> addresses the issue, and we can discuss > > the use > > > > > cases > > > > > >>>>>> of > > > > > >>>>>>>>>>>>>> other > > > > > >>>>>>>>>>>>>>>>>> values > > > > > >>>>>>>>>>>>>>>>>>>>>>> in a > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> separate KIP. > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 3) Both configs, auto.offset.reset.new > > > > > >>>>>>>>>>>>>> < > > > > > >>>>>> > > > > > >>>> > > > > > > > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$ > > > > > >>>>>>> .partitions > > > > > >>>>>>>>>>>>>> and > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age.ms > > > > > >>>>>>>>>>>>>> < > > > > > >>>>>> > > > > > >>>> > > > > > > > https://urldefense.com/v3/__http://auto.offset.reset.latest.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfu9JSP4l$ > > > > > >>>>>>> , > > > > > >>>>>>>>>>>>>> will be applied to all for > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> consistency. > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> WDYT? > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/05/15 20:53:20 Jun Rao via dev > > wrote: > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Chia-Ping, > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply. > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. In the motivation section, the KIP > > says > > > > > "When > > > > > >>>>>> a Kafka > > > > > >>>>>>>>>>>>>>>>> topic > > > > > >>>>>>>>>>>>>>>>>> is > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> expanded > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> with new partitions, consumers using > > the > > > > > latest > > > > > >>>>>> auto > > > > > >>>>>>>>>>>>>> offset > > > > > >>>>>>>>>>>>>>>>>> reset > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> policy > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> will silently miss all records > > produced to > > > > > those > > > > > >>>>>>>>>>>>>> partitions > > > > > >>>>>>>>>>>>>>>>>> before > > > > > >>>>>>>>>>>>>>>>>>>>>>> the > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer discovers them.". If a user > > sets > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset=by_duration=1sec, > > the same > > > > > >>>>>> record loss > > > > > >>>>>>>>>>>>>>>>> issue > > > > > >>>>>>>>>>>>>>>>>>>>> could > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> also > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> happen, right? > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. I was thinking > > auto.offset.reset.new > > > > > >>>>>>>>>>>>>> < > > > > > >>>>>> > > > > > >>>> > > > > > > > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$ > > > > > >>>>>>> .partitions > > > > > >>>>>>>>>>>>>> will > > > > > >>>>>>>>>>>>>>> take > > > > > >>>>>>>>>>>>>>>>>> the > > > > > >>>>>>>>>>>>>>>>>>>>>>> same > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> values as auto.offset.reset. So a user > > could > > > > > set > > > > > >>>>>> it > > > > > >>>>>>>>>>>>>>>>>> by_duration if > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> needed. > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 14, 2026 at 4:06 PM > > Chia-Ping > > > > > Tsai < > > > > > >>>>>>>>>>>>>>>>>>>>> [email protected] > > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> wrote: > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jun > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback. I might be > > missing > > > > > >>>>>> something > > > > > >>>>>>>>>>>>>>>>>> important > > > > > >>>>>>>>>>>>>>>>>>>>>>> from > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> your > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> suggestion, so please bear with me as > > I try > > > > > to > > > > > >>>>>> clarify > > > > > >>>>>>>>>>>>>> with > > > > > >>>>>>>>>>>>>>>>> a > > > > > >>>>>>>>>>>>>>>>>> few > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> questions: > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. Is there a strong use case for > > extending > > > > > this > > > > > >>>>>> logic > > > > > >>>>>>>>>>>>>> to > > > > > >>>>>>>>>>>>>>>>>> other > > > > > >>>>>>>>>>>>>>>>>>>>>>> reset > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> policies? Unlike latest, policies like > > > > > earliest > > > > > >>>>>> or > > > > > >>>>>>>>>>>>>>>>> by_duration > > > > > >>>>>>>>>>>>>>>>>>>>>>> don't > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> seem > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to suffer from the same silent data > > loss > > > > > issue > > > > > >>>>>> when a > > > > > >>>>>>>>>>>>>>>>>> partition > > > > > >>>>>>>>>>>>>>>>>>>>> is > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> expanded. > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. What values would we expect users > > to > > > > > >>>>>> configure for > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.new > > > > > >>>>>>>>>>>>>> < > > > > > >>>>>> > > > > > >>>> > > > > > > > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$ > > > > > >>>>>>> .partitions? > > > > > >>>>>>>>>>>>>> If they set it to > > > > > >>>>>>>>>>>>>>>>> earliest > > > > > >>>>>>>>>>>>>>>>>> or > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> latest, > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we might run into the exact same edge > > > > > cases. For > > > > > >>>>>>>>>>>>>> example, > > > > > >>>>>>>>>>>>>>>>> if a > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> consumer is > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> offline for a while and a new > > partition is > > > > > >>>>>> created > > > > > >>>>>>>>>>>>>> during > > > > > >>>>>>>>>>>>>>>>> that > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> downtime, > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the user might actually want to skip > > to > > > > > latest > > > > > >>>>>> when > > > > > >>>>>>>>>>>>>>>>> resuming, > > > > > >>>>>>>>>>>>>>>>>>>>>>> rather > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> than > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> reading from earliest just because the > > > > > partition > > > > > >>>>>> is > > > > > >>>>>>>>>>>>>>>>>> technically > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> "new" to > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the group. > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This is exactly why we opted for > > > > > introducing a > > > > > >>>>>> max.age > > > > > >>>>>>>>>>>>>>>>>> threshold. > > > > > >>>>>>>>>>>>>>>>>>>>>>> It > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> gives > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> users a time-bound way to define what > > is > > > > > >>>>>> genuinely > > > > > >>>>>>>>>>>>>>> "hot/new" > > > > > >>>>>>>>>>>>>>>>>> and > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> what is > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> just an old partition they haven't > > seen yet. > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/05/14 20:48:09 Jun Rao via dev > > > > > wrote: > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang, > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP. > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I find > > auto.offset.reset.latest.max.age a > > > > > bit > > > > > >>>>>> weird. It > > > > > >>>>>>>>>>>>>>>>> only > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> applies when > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset is latest. > > However, it > > > > > seems > > > > > >>>>>> that the > > > > > >>>>>>>>>>>>>>>>>>>>> motivation > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> equally > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> applies when auto.offset.reset is > > set to > > > > > other > > > > > >>>>>> values > > > > > >>>>>>>>>>>>>> like > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> by_duration. > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> intention is that we want to have a > > > > > separate > > > > > >>>>>> way to > > > > > >>>>>>>>>>>>>>> control > > > > > >>>>>>>>>>>>>>>>>>>>> newly > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> created > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partitions vs existing partitions > > when the > > > > > >>>> group > > > > > >>>>>>>>>>>>>> starts. > > > > > >>>>>>>>>>>>>>>>>> Have we > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> considered > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> adding a new config like > > > > > auto.offset.reset.new > > > > > >>>>>>>>>>>>>> < > > > > > >>>>>> > > > > > >>>> > > > > > > > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$ > > > > > >>>>>>> > > > > > >>>>>>>>>>>>>>> .partitions? > > > > > >>>>>>>>>>>>>>>>>> If > > > > > >>>>>>>>>>>>>>>>>>>>>>> this > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> new > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> config is not set, the offset reset > > policy > > > > > >>>>>> defaults to > > > > > >>>>>>>>>>>>>> the > > > > > >>>>>>>>>>>>>>>>>>>>> policy > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> used > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> existing partitions. The user could > > set it > > > > > >>>>>> explicitly > > > > > >>>>>>>>>>>>>> to > > > > > >>>>>>>>>>>>>>>>>>>>> customize > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> the > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> behavior for new partitions. > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 7, 2026 at 5:07 AM 黃竣陽 < > > > > > >>>>>> [email protected] > > > > > >>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>> wrote: > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all, > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I’d like to manually bump this > > thread. > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards, > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 黃竣陽 <[email protected]> 於 > > 2026年5月1日 > > > > > 晚上10:37 > > > > > >>>>>> 寫道： > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello all, > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback. > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ01/DJ02: > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> MetadataResponse bumps from v13 to > > v14. > > > > > The > > > > > >>>>>>>>>>>>>>>>>> PartitionMetadata > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> struct > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> gains a new > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> field PartitionAgeMs (int64, > > default -1), > > > > > >>>>>> computed > > > > > >>>>>>>>>>>>>>>>>> server-side > > > > > >>>>>>>>>>>>>>>>>>>>>>> by > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> the > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> broker as > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> broker_current_time - > > > > > >>>> partition_creation_time. > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Also add the consumer heartbeat > > flow. > > > > > when > > > > > >>>>>>>>>>>>>>>>>> MembershipManager > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> detects > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> newly assigned > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition, it explicitly > > invalidates the > > > > > >>>>>> metadata for > > > > > >>>>>>>>>>>>>>> the > > > > > >>>>>>>>>>>>>>>>>>>>>>> affected > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> topic > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and forces a fresh MetadataRequest > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> before making the offset reset > > decision, > > > > > even > > > > > >>>>>> if the > > > > > >>>>>>>>>>>>>>>>> topic > > > > > >>>>>>>>>>>>>>>>>> ID > > > > > >>>>>>>>>>>>>>>>>>>>> is > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> already > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in the cache. > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> MB0: > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The consumer learns the broker's > > maximum > > > > > >>>>>> supported > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> MetadataResponse > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> version via the > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ApiVersions negotiation at > > connection > > > > > time. > > > > > >>>>>> If the > > > > > >>>>>>>>>>>>>>>>>> negotiated > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> version is > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> unsupported, the consumer > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> knows the broker does not support > > > > > >>>>>> PartitionAgeMs at > > > > > >>>>>>>>>>>>>> all > > > > > >>>>>>>>>>>>>>>>> and > > > > > >>>>>>>>>>>>>>>>>>>>> can > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> throw an > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> UnsupportedVersionException > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> immediately, rather than silently > > falling > > > > > >>>>>> back to > > > > > >>>>>>>>>>>>>> latest > > > > > >>>>>>>>>>>>>>>>>> and > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> risking > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> data loss without any > > operator-visible > > > > > signal. > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> MB1/MB2/MB3: > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have addressed these changes in > > the > > > > > KIP. > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards, > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai < > > [email protected]> 於 > > > > > >>>>>> 2026年4月29日 > > > > > >>>>>>>>>>>>>>> 下午4:04 > > > > > >>>>>>>>>>>>>>>>>> 寫道： > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi David > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I agree with the direction of > > moving the > > > > > >>>>>> 'age' > > > > > >>>>>>>>>>>>>>>>> resolution > > > > > >>>>>>>>>>>>>>>>>>>>> from > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> the > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Heartbeat API to the Metadata API > > to keep > > > > > the > > > > > >>>>>> control > > > > > >>>>>>>>>>>>>>>>> plane > > > > > >>>>>>>>>>>>>>>>>>>>>>> clean. > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> The > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> main > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> trade-off, as we noted before, is > > > > > introducing > > > > > >>>>>>>>>>>>>>> inter-broker > > > > > >>>>>>>>>>>>>>>>>>>>> clock > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> skew. > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Group Coordinator approach provided > > a > > > > > single > > > > > >>>>>> source of > > > > > >>>>>>>>>>>>>>>>> truth > > > > > >>>>>>>>>>>>>>>>>>>>> for > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> time. > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> However, realistically, this time > > skew > > > > > >>>>>> should be > > > > > >>>>>>>>>>>>>>>>>> negligible. > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Given > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the max.age threshold will likely be > > > > > >>>>>> configured in > > > > > >>>>>>>>>>>>>>> minutes > > > > > >>>>>>>>>>>>>>>>>> or > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> hours, a > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> typical NTP skew (in milliseconds) > > between > > > > > >>>>>> brokers > > > > > >>>>>>>>>>>>>> won't > > > > > >>>>>>>>>>>>>>>>>> impact > > > > > >>>>>>>>>>>>>>>>>>>>>>> the > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fallback decision. > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> David Jacot via dev < > > > > > [email protected]> > > > > > >>>>>> 於 > > > > > >>>>>>>>>>>>>>>>> 2026年4月29日 > > > > > >>>>>>>>>>>>>>>>>>>>>>> 下午3:29 > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> 寫道： > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all, > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP! > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry, I haven't really followed > > the > > > > > >>>>>> previous > > > > > >>>>>>>>>>>>>>>>>> conversation > > > > > >>>>>>>>>>>>>>>>>>>>>>> but I > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> took a > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> quick look at this one. > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ01: I don't clearly understand > > the > > > > > flow > > > > > >>>>>> with the > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ConsumerGroupHeartbeat > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> API after reading the KIP. There > > is a > > > > > new > > > > > >>>>>> boolean; > > > > > >>>>>>>>>>>>>> the > > > > > >>>>>>>>>>>>>>>>>> KIP > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> states > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition ages are returned only > > when > > > > > this > > > > > >>>>>> boolean > > > > > >>>>>>>>>>>>>> is > > > > > >>>>>>>>>>>>>>>>>> set. > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Implicitly, > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> means that when the consumer > > receives > > > > > a new > > > > > >>>>>>>>>>>>>> partition, > > > > > >>>>>>>>>>>>>>>>> it > > > > > >>>>>>>>>>>>>>>>>>>>> will > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> issue a > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> new > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> HB request with the boolean set > > to > > > > > receive > > > > > >>>>>> the > > > > > >>>>>>>>>>>>>> ages. > > > > > >>>>>>>>>>>>>>> Is > > > > > >>>>>>>>>>>>>>>>>> my > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> understanding > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> correct? We should perhaps > > clarify the > > > > > flow > > > > > >>>>>> and > > > > > >>>>>>>>>>>>>> also > > > > > >>>>>>>>>>>>>>>>>> explain > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> how it > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fits > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> into the existing flow (e.g. list > > > > > offsets, > > > > > >>>>>> fetch > > > > > >>>>>>>>>>>>>>>>> offsets, > > > > > >>>>>>>>>>>>>>>>>>>>>>> etc.). > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ02: It my understanding is > > correct, I > > > > > >>>>>> wonder if > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API > > is the > > > > > right > > > > > >>>>>> place > > > > > >>>>>>>>>>>>>> for > > > > > >>>>>>>>>>>>>>>>>> this > > > > > >>>>>>>>>>>>>>>>>>>>>>> given > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a new > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> round trip is done anyway. > > > > > Alternatively, > > > > > >>>>>> it could > > > > > >>>>>>>>>>>>>>>>> simply > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> include > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> metadata. Generally, we should be > > > > > rather > > > > > >>>>>> cautious > > > > > >>>>>>>>>>>>>>> about > > > > > >>>>>>>>>>>>>>>>>> not > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> overloading > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API > > with > > > > > >>>>>> unrelated > > > > > >>>>>>>>>>>>>>> concepts. > > > > > >>>>>>>>>>>>>>>>>> The > > > > > >>>>>>>>>>>>>>>>>>>>>>> API > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> is > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> control plane API for assigning > > or > > > > > revoking > > > > > >>>>>>>>>>>>>>> partitions. > > > > > >>>>>>>>>>>>>>>>>> The > > > > > >>>>>>>>>>>>>>>>>>>>>>> fact > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> don't want to add it to the > > > > > corresponding > > > > > >>>>>> Streams > > > > > >>>>>>>>>>>>>> API > > > > > >>>>>>>>>>>>>>>>>> also > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> suggests > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> something is not quite right. > > What > > > > > would we > > > > > >>>>>> do if > > > > > >>>>>>>>>>>>>> we > > > > > >>>>>>>>>>>>>>>>>> want to > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> support > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Streams in the future? > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> David > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 29, 2026 at 12:28 AM > > > > > >>>>>> Muralidhar Basani > > > > > >>>>>>>>>>>>>>> via > > > > > >>>>>>>>>>>>>>>>>> dev > > > > > >>>>>>>>>>>>>>>>>>>>> < > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [email protected]> wrote: > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jiunn, > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for this great kip. > > Good to > > > > > know > > > > > >>>>>> about > > > > > >>>>>>>>>>>>>> the > > > > > >>>>>>>>>>>>>>>>>> gap. > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-0 - why a new v2 version > > bump for > > > > > >>>>>>>>>>>>>>>>>> RequestPartitionAges > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> field. > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Can a > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tagged field (for ex: on > > response, > > > > > >>>>>> PartitionAges > > > > > >>>>>>>>>>>>>> on > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> TopicPartitions) > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> used here and avoid version > > bump? > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-1 - For the new config, is > > there a > > > > > >>>>>> recommended > > > > > >>>>>>>>>>>>>>>>> value > > > > > >>>>>>>>>>>>>>>>>> or > > > > > >>>>>>>>>>>>>>>>>>>>> a > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ConfigDef > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> validator? Probably it should > > based > > > > > on the > > > > > >>>>>>>>>>>>>>>>>>>>>>> metadata.max.age.ms > > > > > >>>>>>>>>>>>>> < > > > > > >>>>>> > > > > > >>>> > > > > > > > https://urldefense.com/v3/__http://metadata.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkflKEb5SK$ > > > > > >>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> ? > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sizing > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instructions can be part of > > javadocs I > > > > > >>>>>> guess. > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-2 - (minor) As there are no > > > > > changes to > > > > > >>>>>> Kafka > > > > > >>>>>>>>>>>>>>>>> Streams, > > > > > >>>>>>>>>>>>>>>>>>>>>>> would > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> it > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> better > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to add this new config > > > > > >>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age > > > > > >>>>>>>>>>>>>>>>>> to > > > > > >>>>>>>>>>>>>>>>>>>>>>> the > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> StreamsConfig block list > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS) > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for a > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clear warning, incase users > > configure > > > > > it? > > > > > >>>>>> This is > > > > > >>>>>>>>>>>>>> the > > > > > >>>>>>>>>>>>>>>>>> most > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> familiar > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config and users might > > easily > > > > > >>>>>> mistakenly > > > > > >>>>>>>>>>>>>>>>>> configure > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> it. Or > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> may be > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it's not worth it to add. > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-3 - (minor) The phrasing "the > > > > > consumer > > > > > >>>>>> falls > > > > > >>>>>>>>>>>>>> back > > > > > >>>>>>>>>>>>>>>>> to > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> earliest" > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> reads as > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> if the config were being changed > > > > > >>>>>> per-partition > > > > > >>>>>>>>>>>>>> which > > > > > >>>>>>>>>>>>>>>>>> isn't > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> supported. > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> May > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be rephrasing to something like > > > > > "consumer > > > > > >>>>>> resolves > > > > > >>>>>>>>>>>>>>> the > > > > > >>>>>>>>>>>>>>>>>>>>>>> initial > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> position to > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> start offset for that > > partition" as if > > > > > >>>>>> earliest > > > > > >>>>>>>>>>>>>> was > > > > > >>>>>>>>>>>>>>>>>> applied > > > > > >>>>>>>>>>>>>>>>>>>>>>> to > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition only and > > auto.offset.reset > > > > > >>>>>> config is > > > > > >>>>>>>>>>>>>>>>>> unchanged. > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks, > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Murali > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 28, 2026 at > > 2:48 PM 黃竣陽 < > > > > > >>>>>>>>>>>>>>>>>> [email protected]> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> wrote: > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi chia, > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have updated the KIP to > > include > > > > > this > > > > > >>>>>> change. > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards, > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai < > > [email protected]> > > > > > 於 > > > > > >>>>>>>>>>>>>> 2026年4月28日 > > > > > >>>>>>>>>>>>>>>>>> 晚上8:03 > > > > > >>>>>>>>>>>>>>>>>>>>>>> 寫道： > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> chia_0: Should we expose the > > > > > partition > > > > > >>>>>> creation > > > > > >>>>>>>>>>>>>>> time > > > > > >>>>>>>>>>>>>>>>>> via > > > > > >>>>>>>>>>>>>>>>>>>>>>> the > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Admin > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> API? > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I assume it would be valuable > > for > > > > > users > > > > > >>>> to > > > > > >>>>>>>>>>>>>> diagnose > > > > > >>>>>>>>>>>>>>>>> and > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> troubleshoot > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> behavior of > > > > > >>>>>> auto.offset.reset.latest.max.age > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/04/28 10:47:58 黃竣陽 > > wrote: > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone, > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a > > discussion > > > > > on > > > > > >>>>>> KIP-1327 > > > > > >>>>>>>>>>>>>>>>>> Prevent > > > > > >>>>>>>>>>>>>>>>>>>>> Hot > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Data > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Loss > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Partition Expansion for Latest > > Policy > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> < > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>> > > > > > >>>>>> > > > > > >>>> > > > > > > > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$ > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to > > introduces > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age, > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config that lets the > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> latest reset policy > > distinguish > > > > > newly > > > > > >>>>>> expanded > > > > > >>>>>>>>>>>>>>>>> (hot) > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> partitions > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> from > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> long-existing (cold) ones. > > Partitions > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> younger than the configured > > > > > threshold > > > > > >>>>>>>>>>>>>>> automatically > > > > > >>>>>>>>>>>>>>>>>> fall > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> back > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> earliest, preventing silent > > data loss > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during topic expansion > > without > > > > > forcing > > > > > >>>>>> a full > > > > > >>>>>>>>>>>>>>>>>> historical > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> reprocess. > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards, > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>> > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> > > > > > >>>>>>>> > > > > > >>>>>>> > > > > > >>>>>>> > > > > > >>>>>>> > > > > > >>>>>> > > > > > >>>> > > > > > >>>> > > > > > >>> > > > > > >> > > > > > > > > > > > > > > > > > > > >
