Hi all, Manually bumping this thread.
Best Regards, Jiunn-Yang > 黃竣陽 <[email protected]> 於 2026年6月17日 晚上9:17 寫道： > > Hello chia, > > Thanks for the feedback, I have updated the KIP. > > Best Regards, > Jiunn-Yang > >> Chia-Ping Tsai <[email protected]> 於 2026年6月17日 凌晨12:47 寫道： >> >> hi Jiunn-Yang >> >>> When the config is set on a cluster that has not yet been upgraded... >>> classification cannot occur... the consumer falls back to the base >>> auto.offset.reset for the affected partitions. No exception is thrown, and >>> no operational disruption results. >> >> Existing group can't take advantage of this excellent new configuration. >> Allowing users to modify the group creation time might be overkill. Instead, >> we could print a useful warning message to guide users. For example, we can >> suggest that they re-create the group with their existing committed offsets >> >>> Protocol changes >> >> Would you mind listing those RPC changes in a table format? >> >>> The full interaction matrix between the base policy and the new-partition >>> policy is: >> >> Please add a filed to describe the target scenario when using these policies >> >> Best, >> Chia-Ping >> >> >> On 2026/06/16 16:14:49 黃竣陽 wrote: >>> Hello Jun, chia, >>> >>> Thanks for the feedback, I have updated the KIP for the new >>> approach, PTAL >>> >>> Best Regards, >>> Jiunn-Yang >>> >>>> Chia-Ping Tsai <[email protected]> 於 2026年6月16日 上午8:23 寫道： >>>> >>>> hi Jun >>>> >>>> Yes, your approach is great. I think the combination of latest (for >>>> existing partitions) and by_duration (for new partitions) can address 99% >>>> of the complaints I have heard regarding this issue. >>>> >>>> Also, leveraging the group creation time here opens the door to >>>> implementing a new policy based on timestamp seek in the future, should >>>> the community want to pursue that. >>>> >>>> Thanks for your patience and constructive feedback. We will update the KIP >>>> accordingly. >>>> >>>> Best, Chia-Ping >>>> >>>>> Jun Rao via dev <[email protected]> 於 2026年6月16日 清晨5:11 寫道： >>>>> >>>>> Hi, Chia-Ping, >>>>> >>>>> Thanks for the reply. >>>>> >>>>> I agree that it's probably useful to allow a user to configure a different >>>>> offset policy for existing partitions vs new partitions. However, using >>>>> group creation time to capture that seems more intuitive. Here is another >>>>> proposal: remove auto.offset.reset.max.age.ms and categorize new >>>>> partitions >>>>> based on group creation time. Introduce >>>>> a new config auto.offset.reset.new.partitions whose values can be >>>>> earliest, >>>>> latest and by_duration, the same as auto.offset.reset. Users can set >>>>> `auto.offset.reset.new.partitions` to `earliest` if they want to guarantee >>>>> no data loss on new partitions. They can also use by_duration to set an >>>>> upper bound on the backlog replayed, which can be different from that of >>>>> the existing partitions. This will address your concern about too much >>>>> backlog being replayed when the offsets are lost. What do you think? >>>>> >>>>> Jun >>>>> >>>>> >>>>>> On Mon, Jun 15, 2026 at 10:39 AM Chia-Ping Tsai <[email protected]> >>>>>> wrote: >>>>>> >>>>>> hi Jun >>>>>> >>>>>> The most important part of this story is how users should expect the data >>>>>> they can see when using the latest or by_duration policy with expanded >>>>>> partitions. >>>>>> >>>>>> Yes, the by_duration policy can minimize data loss, but it is >>>>>> non-deterministic, which means users will either read too many historical >>>>>> records from existing partitions or lose some records from expanded >>>>>> partitions. >>>>>> >>>>>> Also, I agree that auto.offset.reset.max.age.ms >>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$> >>>>>> is a bit hard to understand, and that is why I preferred having a whole >>>>>> new >>>>>> policy based entirely on group creation time (KIP-1282) >>>>>> >>>>>> Best, >>>>>> Chia-Ping >>>>>> >>>>>> Jun Rao via dev <[email protected]> 於 2026年6月16日週二 上午1:08寫道： >>>>>> >>>>>>> Hi, Chia-Ping and Jiunn-Yang, >>>>>>> >>>>>>> Thanks for the reply. I am still trying to understand the value of the >>>>>>> new >>>>>>> configs with the KIP. >>>>>>> >>>>>>> The motivation of the KIP is that a user doesn't want to miss the data >>>>>>> if >>>>>>> the backlog is small. The backlog of the existing partition is easy to >>>>>>> understand because it relates to retention time. The backlog for the new >>>>>>> partition is a bit subtle to understand since it depends on the metadata >>>>>>> refresh delay. To set auto.offset.reset.max.age.ms >>>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$>, >>>>>>> the user needs to >>>>>>> understand the metadata refresh delay on the consumer side and use it to >>>>>>> set the config. >>>>>>> >>>>>>> Now, let's consider the alternative: setting the same value for the >>>>>>> existing by_duration policy. The KIP lists three issues with this >>>>>>> approach. >>>>>>> 1. It computes the seek target client-side as now() - duration, which >>>>>>> introduces clock skew across consumers and forces operators to choose >>>>>>> overly large durations, causing unnecessary reprocessing. >>>>>>> 2. The target timestamp is recomputed on each retry, so failed >>>>>>> ListOffsetsRequest retries can shift the target forward and potentially >>>>>>> miss records produced between attempts. >>>>>>> 3. It applies uniformly to all partitions without committed offsets, and >>>>>>> cannot distinguish newly expanded partitions from long-existing >>>>>>> partitions >>>>>>> newly assigned to the group, leading to unnecessary replay. >>>>>>> >>>>>>> Issues 1 and 2 are uncommon and can be mitigated by adding a bit buffer >>>>>>> to >>>>>>> the metadata refresh delay. We could also consider improving the >>>>>>> implementation. For issue 3, the metadata refresh delay is typically low >>>>>>> (in the order of minutes with the classic consumer and tens of seconds >>>>>>> with >>>>>>> the new consumer). If a user is ok with reading that much backlog for >>>>>>> new >>>>>>> partitions, it seems they will be ok doing the same for existing >>>>>>> partitions. >>>>>>> >>>>>>> So, instead of introducing a new config, could we just reuse the >>>>>>> existing >>>>>>> config with better documentation and/or implementation? >>>>>>> >>>>>>> Jun >>>>>>> >>>>>>> >>>>>>>> On Sat, Jun 13, 2026 at 12:19 AM 黃竣陽 <[email protected]> wrote: >>>>>>> >>>>>>>> Hello Jun, >>>>>>>> >>>>>>>> You're right that group creation time is the more intuitive answer at >>>>>>>> first glance, >>>>>>>> the KIP's own motivation talks about partitions that "predate the >>>>>>>> group" >>>>>>>> vs partitions >>>>>>>> "created during group runtime," which directly points to a >>>>>>> group-lifecycle >>>>>>>> classifier. >>>>>>>> I'd like to walk through why we landed on partition age, and the >>>>>>>> trade-offs we considered. >>>>>>>> >>>>>>>> We evaluated three candidate signals: >>>>>>>> >>>>>>>> 1. `by_duration:5secs` >>>>>>>> >>>>>>>> This covers the metadata blindness window, but has issues the KIP >>>>>>>> currently documents >>>>>>>> under "Why not use `by_duration`?": >>>>>>>> >>>>>>>> - Client-side `now() - duration` introduces clock skew across >>>>>>>> consumers. >>>>>>>> - `ListOffsets` retries shift the target forward, potentially missing >>>>>>>> records produced between >>>>>>>> attempts. >>>>>>>> - It applies uniformly to all partitions without committed offsets, >>>>>>>> including pre-existing partitions >>>>>>>> newly assigned to the group, causing unnecessary replay. >>>>>>>> >>>>>>>> 2. Group creation time as classifier >>>>>>>> >>>>>>>> This works cleanly when the consumer is actively running. Our concern >>>>>>>> is the idle / late-rejoin case: >>>>>>>> >>>>>>>> T=0: Group created. >>>>>>>> T=1..T=100: Consumer idle (down, disconnected, etc.). >>>>>>>> T=50: Partition added during the idle window. >>>>>>>> T=100: Consumer resumes. >>>>>>>> >>>>>>>> Under group creation time, the new partition is classified as new >>>>>>>> (`50 > 0`) and reset to `earliest`, replaying everything from T=50. >>>>>>>> But during `[T=1, T=100]`, base partitions also accumulated data that >>>>>>>> the consumer accepts as lost — that is precisely the contract of >>>>>>>> `auto.offset.reset=latest`. There is no principled reason to treat >>>>>>>> the new partition differently; both contain backlog accumulated during >>>>>>>> the same idle window. >>>>>>>> >>>>>>>> This aligns with the "backlog is backlog” principle you raised in >>>>>>>> the KIP-1282 thread: a `latest` user has tolerated some backlog on >>>>>>>> every other partition during the same idle period; forcing 0-backlog >>>>>>>> tolerance only on new partitions would be inconsistent with that >>>>>>>> tolerance. >>>>>>>> >>>>>>>> 3. Partition age vs threshold >>>>>>>> >>>>>>>> Partition age corresponds to the actual silent data loss window, >>>>>>>> the gap between partition creation and the consumer’s metadata >>>>>>>> refresh. Within this window, data loss is genuinely silent: the >>>>>>>> consumer had no opportunity to know about the partition. Outside this >>>>>>>> window, missing data reflects either: >>>>>>>> >>>>>>>> - (a) the user’s tolerated cost of running with idle consumers, or >>>>>>>> - (b) an operational issue to surface via monitoring, not via reset >>>>>>> policy. >>>>>>>> >>>>>>>> We did not choose partition age because it is more elegant than group >>>>>>>> creation time — we chose it because its failure mode (requires a >>>>>>>> threshold) is >>>>>>>> less invasive than the failure mode of group creation time (overrides >>>>>>>> user-stated >>>>>>>> `latest` intent during idle periods). >>>>>>>> >>>>>>>> Best Regards, >>>>>>>> Jiunn-Yang >>>>>>>> >>>>>>>>> Chia-Ping Tsai <[email protected]> 於 2026年6月13日 上午11:52 寫道： >>>>>>>>> >>>>>>>>> Hi Jun, >>>>>>>>> >>>>>>>>> Relying on both creation times will create an inconsistent scenario. A >>>>>>>>> consumer that lost all offsets due to a long sleep will seek to the >>>>>>>>> beginning for the partitions created later than the group. >>>>>>>>> >>>>>>>>> That is why we initially proposed KIP-1282 to fix the inconsistency >>>>>>>> using a >>>>>>>>> whole new policy. Since KIP-1282 couldn't reach a consensus, KIP-1327 >>>>>>>> goes >>>>>>>>> back to using flexible configurations to prevent users from falling >>>>>>> into >>>>>>>>> that pitfall. >>>>>>>>> >>>>>>>>> Best, Chia-Ping >>>>>>>>> >>>>>>>>> Jun Rao via dev <[email protected]> 於 2026年6月13日週六 上午6:49寫道： >>>>>>>>> >>>>>>>>>> Hi, Jiunn-Yang, >>>>>>>>>> >>>>>>>>>> Thanks for the reply and sorry for the late reply. >>>>>>>>>> >>>>>>>>>> JR1. The design of auto.offset.reset.max.age.ms >>>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$> >>>>>>> still feels weird to >>>>>>>> me. >>>>>>>>>> It >>>>>>>>>> categorizes partitions as new or existing based on the partition >>>>>>>> creation >>>>>>>>>> time. Intuitively, the categorization should be based on the group >>>>>>>> creation >>>>>>>>>> time: all partitions existing when the group is created are existing >>>>>>> and >>>>>>>>>> all partitions created after the group creation are new partitions. >>>>>>>>>> >>>>>>>>>> Jun >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Tue, Jun 9, 2026 at 8:51 AM 黃竣陽 <[email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> Hi all, >>>>>>>>>>> >>>>>>>>>>> Manually bumping this thread. If there is no further >>>>>>>>>>> discussion, I will close the vote. >>>>>>>>>>> >>>>>>>>>>> Best Regards, >>>>>>>>>>> Jiunn-Yang >>>>>>>>>>> >>>>>>>>>>>> 黃竣陽 <[email protected]> 於 2026年6月1日 晚上7:16 寫道： >>>>>>>>>>>> >>>>>>>>>>>> Hello Jian, >>>>>>>>>>>> >>>>>>>>>>>> Thanks for your feedback, >>>>>>>>>>>> >>>>>>>>>>>> Agreed, partition expansion is a common operational task, not an >>>>>>> edge >>>>>>>>>>>> case. I've updated the Motivation section accordingly. >>>>>>>>>>>> >>>>>>>>>>>> Best Regards, >>>>>>>>>>>> Jiunn-Yang >>>>>>>>>>>> >>>>>>>>>>>>> jian fu <[email protected]> 於 2026年6月1日 下午5:49 寫道： >>>>>>>>>>>>> >>>>>>>>>>>>> Hi Jiunn-Yang: >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks for the KIP. I think it would be useful to clarify that >>>>>>> this >>>>>>>>>> is a >>>>>>>>>>>>> common scenario rather than an edge case, which further >>>>>>> demonstrates >>>>>>>>>> the >>>>>>>>>>>>> need for this optimization. For example: >>>>>>>>>>>>> A partition expansion is a common operational task in Kafka: To >>>>>>>>>> balance >>>>>>>>>>>>> resource utilization and cost, topics are typically created with a >>>>>>>>>>> moderate >>>>>>>>>>>>> default partition count. However, as traffic grows over time, it >>>>>>> is >>>>>>>>>>> often >>>>>>>>>>>>> necessary to increase the number of partitions to accommodate the >>>>>>>>>> higher >>>>>>>>>>>>> workload. >>>>>>>>>>>>> >>>>>>>>>>>>> Regards >>>>>>>>>>>>> Jian >>>>>>>>>>>>> >>>>>>>>>>>>> 黃竣陽 <[email protected]> 于2026年5月30日周六 22:31写道： >>>>>>>>>>>>> >>>>>>>>>>>>>> Hello chia, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks for the comments, I have updated the KIP! >>>>>>>>>>>>>> >>>>>>>>>>>>>> Best Regards, >>>>>>>>>>>>>> Jiunn-Yang >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Chia-Ping Tsai <[email protected]> 於 2026年5月30日 晚上8:29 寫道： >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Jiunn-Yang, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Would you mind removing the terms "hot" and "cold" when >>>>>>> describing >>>>>>>>>>>>>>> partitions in the KIP? I understand you are using them to >>>>>>> describe >>>>>>>>>> the >>>>>>>>>>>>>>> "freshness" or the users' need for the records, but applying >>>>>>> these >>>>>>>>>>> terms >>>>>>>>>>>>>> to >>>>>>>>>>>>>>> the partition itself feels a bit unnatural. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> After all, in this scenario, users don't really care whether a >>>>>>>>>>> partition >>>>>>>>>>>>>> is >>>>>>>>>>>>>>> newly expanded or not. Their only expectation is that they won't >>>>>>>>>>> silently >>>>>>>>>>>>>>> lose any live records produced to the topic during their active >>>>>>>>>>>>>> consumption. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Best, Chia-Ping >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 黃竣陽 <[email protected]> 於 2026年5月30日週六 下午12:30寫道： >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hello Jun, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks for the feedback, I have updated the KIP motivation >>>>>>>> section. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Best Regards, >>>>>>>>>>>>>>>> Jiunn-Yang >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Jun Rao via dev <[email protected]> 於 2026年5月30日 凌晨1:12 >>>>>>> 寫道： >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hi, Jiunn-Yang, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks for the reply. I think we need a stronger motivation >>>>>>> for >>>>>>>>>> the >>>>>>>>>>>>>> KIP. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> The KIP says "The core insight is that not all partitions >>>>>>> without >>>>>>>>>> a >>>>>>>>>>>>>>>>> committed offset are the same. A newly expanded partition >>>>>>> (hot) >>>>>>>> is >>>>>>>>>>>>>>>>> fundamentally different from a partition the consumer has >>>>>>> never >>>>>>>>>> seen >>>>>>>>>>>>>>>>> because it predates the group (cold)." Why is the hot >>>>>>> partition >>>>>>>>>>>>>>>>> fundamentally different from the cold? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> The KIP says "The existing by_duration policy is also >>>>>>>> insufficient >>>>>>>>>>>>>>>> because: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> - The calculated seek time (now() - duration) varies across >>>>>>> nodes >>>>>>>>>>> due >>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>> clock skew. To be safe, users must set an overly large >>>>>>> duration, >>>>>>>>>>>>>>>> causing >>>>>>>>>>>>>>>>> unnecessary reprocessing. >>>>>>>>>>>>>>>>> - On network errors, the client recalculates the seek time on >>>>>>>>>> retry, >>>>>>>>>>>>>>>>> shifting the target timestamp forward and risking data loss." >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> However, both of these situations are rare. If these issues >>>>>>>>>> persist, >>>>>>>>>>>>>> more >>>>>>>>>>>>>>>>> severe problems likely exist elsewhere. Rare situations don't >>>>>>>>>> need a >>>>>>>>>>>>>>>> common >>>>>>>>>>>>>>>>> solution. If users care about those rare situations, they can >>>>>>>>>>> implement >>>>>>>>>>>>>>>>> customized logic using >>>>>>>>>>>>>> ConsumerRebalanceListener.onPartitionsAssigned(). >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Jun >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Sun, May 17, 2026 at 6:50 AM 黃竣陽 <[email protected]> >>>>>>> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Hello chia, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Thanks for the feedback, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> If the creation time exists, the returned value should >>>>>>> always >>>>>>>> be >>>>>>>>>>>>>>>> greater >>>>>>>>>>>>>>>>>> than or equal to zero, right? >>>>>>>>>>>>>>>>>> I have explicitly mentioned this in the KIP. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> New Old (MetadataResponse v0–13) positive any >>>>>>>>>>> field >>>>>>>>>>>>>>>>>> absent UnsupportedVersionException >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> The earliest point at which we can detect the version >>>>>>> mismatch >>>>>>>> is >>>>>>>>>>>>>> during >>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>> first metadata fetch after assignment, which occurs inside >>>>>>>>>> poll(). >>>>>>>>>>>>>>>>>> Therefore, the >>>>>>>>>>>>>>>>>> user would encounter an UnsupportedVersionException from >>>>>>> poll(). >>>>>>>>>>> I’ll >>>>>>>>>>>>>>>>>> clarify this in the KIP. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Best Regards, >>>>>>>>>>>>>>>>>> Jiunn-Yang >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <[email protected]> 於 2026年5月17日 下午4:50 >>>>>>> 寫道： >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> hi Jiunn >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> PartitionAgeMs (int64, default -1): The age of this >>>>>>> partition >>>>>>>>>> in >>>>>>>>>>>>>>>>>> milliseconds, computed server-side by the broker as >>>>>>>>>>>>>> broker_current_time >>>>>>>>>>>>>>>> - >>>>>>>>>>>>>>>>>> partition_creation_time. Returns -1 if the broker does not >>>>>>>>>> support >>>>>>>>>>>>>> this >>>>>>>>>>>>>>>>>> feature or the partition creation time is unknown. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> If the creation time exists, the returned value should >>>>>>> always >>>>>>>> be >>>>>>>>>>>>>>>> greater >>>>>>>>>>>>>>>>>> than or equal to zero, right? >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> New Old (MetadataResponse v0–13) positive any >>>>>>>>>>> field >>>>>>>>>>>>>>>>>> absent UnsupportedVersionException >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Will user encounter UnsupportedVersionException when calling >>>>>>>>>>>>>> `poll()`? >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>> Chia-Ping >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> On 2026/05/16 04:30:49 黃竣陽 wrote: >>>>>>>>>>>>>>>>>>>> Hello Jun, chia, >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> I've updated KIP-1327 with a design change based on the >>>>>>>>>>> discussion >>>>>>>>>>>>>>>>>>>> feedback. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> The updated design decouples the new-partition reset >>>>>>> behavior >>>>>>>>>>> from >>>>>>>>>>>>>>>>>>>> the base auto.offset.reset policy: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> - auto.offset.reset.max.age.ms >>>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$> >>>>>>> now applies to all >>>>>>>>>>> auto.offset.reset >>>>>>>>>>>>>>>>>> values >>>>>>>>>>>>>>>>>>>> (latest, earliest, by_duration, none). >>>>>>>>>>>>>>>>>>>> - For new ("hot") partitions, the consumer resets to >>>>>>>>>>>>>>>>>> auto.offset.reset.new >>>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$> >>>>>>> .partitions >>>>>>>>>>>>>>>>>>>> config setting >>>>>>>>>>>>>>>>>>>> - For existing ("cold") partitions, the base >>>>>>> auto.offset.reset >>>>>>>>>>>>>> policy >>>>>>>>>>>>>>>>>> continues >>>>>>>>>>>>>>>>>>>> to apply unchanged. >>>>>>>>>>>>>>>>>>>> - The new-partition reset behavior is represented by a >>>>>>>> separate >>>>>>>>>>>>>>>>>> internal config >>>>>>>>>>>>>>>>>>>> (auto.offset.reset.new >>>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>.partitions, >>>>>>> currently fixed to >>>>>>>>>> earliest). >>>>>>>>>>>>>> This >>>>>>>>>>>>>>>>>> decoupled design makes >>>>>>>>>>>>>>>>>>>> it straightforward to promote the behavior to a public >>>>>>>>>>> user-facing >>>>>>>>>>>>>>>>>> configuration in a future KIP. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Best Regards, >>>>>>>>>>>>>>>>>>>> Jiunn-Yang >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <[email protected]> 於 2026年5月16日 清晨7:46 >>>>>>> 寫道： >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> hi Jun >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> I see what you mean now. The proposal from me is listed >>>>>>>> below: >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> 1) Add auto.offset.reset.new >>>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>.partitions >>>>>>> with a default value >>>>>>>>>> of >>>>>>>>>>>>>>>>>> earliest. It fixes the data loss from both by_duration and >>>>>>>>>> latest, >>>>>>>>>>> and >>>>>>>>>>>>>>>> it >>>>>>>>>>>>>>>>>> does not change the logic of auto.offset.reset=earliest. >>>>>>>>>>>>>>>>>>>>> 2) Mark auto.offset.reset.new >>>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>.partitions >>>>>>> as an internal >>>>>>>>>>>>>>>>>> configuration. auto.offset.reset.new >>>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$> >>>>>>> .partitions=earliest >>>>>>>> already >>>>>>>>>>>>>>>>>> addresses the issue, and we can discuss the use cases of >>>>>>> other >>>>>>>>>>> values >>>>>>>>>>>>>>>> in a >>>>>>>>>>>>>>>>>> separate KIP. >>>>>>>>>>>>>>>>>>>>> 3) Both configs, auto.offset.reset.new >>>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>.partitions >>>>>>> and >>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age.ms >>>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.latest.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfu9JSP4l$>, >>>>>>> will be applied to all for >>>>>>>>>>>>>>>>>> consistency. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> WDYT? >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> On 2026/05/15 20:53:20 Jun Rao via dev wrote: >>>>>>>>>>>>>>>>>>>>>> Hi, Chia-Ping, >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Thanks for the reply. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> 1. In the motivation section, the KIP says "When a Kafka >>>>>>>>>> topic >>>>>>>>>>> is >>>>>>>>>>>>>>>>>> expanded >>>>>>>>>>>>>>>>>>>>>> with new partitions, consumers using the latest auto >>>>>>> offset >>>>>>>>>>> reset >>>>>>>>>>>>>>>>>> policy >>>>>>>>>>>>>>>>>>>>>> will silently miss all records produced to those >>>>>>> partitions >>>>>>>>>>> before >>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>> consumer discovers them.". If a user sets >>>>>>>>>>>>>>>>>>>>>> auto.offset.reset=by_duration=1sec, the same record loss >>>>>>>>>> issue >>>>>>>>>>>>>> could >>>>>>>>>>>>>>>>>> also >>>>>>>>>>>>>>>>>>>>>> happen, right? >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> 2. I was thinking auto.offset.reset.new >>>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>.partitions >>>>>>> will >>>>>>>> take >>>>>>>>>>> the >>>>>>>>>>>>>>>> same >>>>>>>>>>>>>>>>>>>>>> values as auto.offset.reset. So a user could set it >>>>>>>>>>> by_duration if >>>>>>>>>>>>>>>>>> needed. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Jun >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> On Thu, May 14, 2026 at 4:06 PM Chia-Ping Tsai < >>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> hi Jun >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback. I might be missing something >>>>>>>>>>> important >>>>>>>>>>>>>>>> from >>>>>>>>>>>>>>>>>> your >>>>>>>>>>>>>>>>>>>>>>> suggestion, so please bear with me as I try to clarify >>>>>>> with >>>>>>>>>> a >>>>>>>>>>> few >>>>>>>>>>>>>>>>>> questions: >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> 1. Is there a strong use case for extending this logic >>>>>>> to >>>>>>>>>>> other >>>>>>>>>>>>>>>> reset >>>>>>>>>>>>>>>>>>>>>>> policies? Unlike latest, policies like earliest or >>>>>>>>>> by_duration >>>>>>>>>>>>>>>> don't >>>>>>>>>>>>>>>>>> seem >>>>>>>>>>>>>>>>>>>>>>> to suffer from the same silent data loss issue when a >>>>>>>>>>> partition >>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>> expanded. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> 2. What values would we expect users to configure for >>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.new >>>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>.partitions? >>>>>>> If they set it to >>>>>>>>>> earliest >>>>>>>>>>> or >>>>>>>>>>>>>>>>>> latest, >>>>>>>>>>>>>>>>>>>>>>> we might run into the exact same edge cases. For >>>>>>> example, >>>>>>>>>> if a >>>>>>>>>>>>>>>>>> consumer is >>>>>>>>>>>>>>>>>>>>>>> offline for a while and a new partition is created >>>>>>> during >>>>>>>>>> that >>>>>>>>>>>>>>>>>> downtime, >>>>>>>>>>>>>>>>>>>>>>> the user might actually want to skip to latest when >>>>>>>>>> resuming, >>>>>>>>>>>>>>>> rather >>>>>>>>>>>>>>>>>> than >>>>>>>>>>>>>>>>>>>>>>> reading from earliest just because the partition is >>>>>>>>>>> technically >>>>>>>>>>>>>>>>>> "new" to >>>>>>>>>>>>>>>>>>>>>>> the group. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> This is exactly why we opted for introducing a max.age >>>>>>>>>>> threshold. >>>>>>>>>>>>>>>> It >>>>>>>>>>>>>>>>>> gives >>>>>>>>>>>>>>>>>>>>>>> users a time-bound way to define what is genuinely >>>>>>>> "hot/new" >>>>>>>>>>> and >>>>>>>>>>>>>>>>>> what is >>>>>>>>>>>>>>>>>>>>>>> just an old partition they haven't seen yet. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>>>>> Chia-Ping >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> On 2026/05/14 20:48:09 Jun Rao via dev wrote: >>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang, >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> I find auto.offset.reset.latest.max.age a bit weird. It >>>>>>>>>> only >>>>>>>>>>>>>>>>>> applies when >>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset is latest. However, it seems that the >>>>>>>>>>>>>> motivation >>>>>>>>>>>>>>>>>>>>>>> equally >>>>>>>>>>>>>>>>>>>>>>>> applies when auto.offset.reset is set to other values >>>>>>> like >>>>>>>>>>>>>>>>>> by_duration. >>>>>>>>>>>>>>>>>>>>>>> The >>>>>>>>>>>>>>>>>>>>>>>> intention is that we want to have a separate way to >>>>>>>> control >>>>>>>>>>>>>> newly >>>>>>>>>>>>>>>>>> created >>>>>>>>>>>>>>>>>>>>>>>> partitions vs existing partitions when the group >>>>>>> starts. >>>>>>>>>>> Have we >>>>>>>>>>>>>>>>>>>>>>> considered >>>>>>>>>>>>>>>>>>>>>>>> adding a new config like auto.offset.reset.new >>>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$> >>>>>>>> .partitions? >>>>>>>>>>> If >>>>>>>>>>>>>>>> this >>>>>>>>>>>>>>>>>> new >>>>>>>>>>>>>>>>>>>>>>>> config is not set, the offset reset policy defaults to >>>>>>> the >>>>>>>>>>>>>> policy >>>>>>>>>>>>>>>>>> used >>>>>>>>>>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>>>>>>>>>> existing partitions. The user could set it explicitly >>>>>>> to >>>>>>>>>>>>>> customize >>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>> behavior for new partitions. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Jun >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 7, 2026 at 5:07 AM 黃竣陽 <[email protected] >>>>>>>> >>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Hi all, >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> I’d like to manually bump this thread. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Best Regards, >>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> 黃竣陽 <[email protected]> 於 2026年5月1日 晚上10:37 寫道： >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Hello all, >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback. >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> DJ01/DJ02: >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> MetadataResponse bumps from v13 to v14. The >>>>>>>>>>> PartitionMetadata >>>>>>>>>>>>>>>>>> struct >>>>>>>>>>>>>>>>>>>>>>>>> gains a new >>>>>>>>>>>>>>>>>>>>>>>>>> field PartitionAgeMs (int64, default -1), computed >>>>>>>>>>> server-side >>>>>>>>>>>>>>>> by >>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>> broker as >>>>>>>>>>>>>>>>>>>>>>>>>> broker_current_time - partition_creation_time. >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Also add the consumer heartbeat flow. when >>>>>>>>>>> MembershipManager >>>>>>>>>>>>>>>>>> detects >>>>>>>>>>>>>>>>>>>>>>> a >>>>>>>>>>>>>>>>>>>>>>>>> newly assigned >>>>>>>>>>>>>>>>>>>>>>>>>> partition, it explicitly invalidates the metadata for >>>>>>>> the >>>>>>>>>>>>>>>> affected >>>>>>>>>>>>>>>>>>>>>>> topic >>>>>>>>>>>>>>>>>>>>>>>>> and forces a fresh MetadataRequest >>>>>>>>>>>>>>>>>>>>>>>>>> before making the offset reset decision, even if the >>>>>>>>>> topic >>>>>>>>>>> ID >>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>>>> already >>>>>>>>>>>>>>>>>>>>>>>>> in the cache. >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> MB0: >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> The consumer learns the broker's maximum supported >>>>>>>>>>>>>>>>>> MetadataResponse >>>>>>>>>>>>>>>>>>>>>>>>> version via the >>>>>>>>>>>>>>>>>>>>>>>>>> ApiVersions negotiation at connection time. If the >>>>>>>>>>> negotiated >>>>>>>>>>>>>>>>>>>>>>> version is >>>>>>>>>>>>>>>>>>>>>>>>> unsupported, the consumer >>>>>>>>>>>>>>>>>>>>>>>>>> knows the broker does not support PartitionAgeMs at >>>>>>> all >>>>>>>>>> and >>>>>>>>>>>>>> can >>>>>>>>>>>>>>>>>>>>>>> throw an >>>>>>>>>>>>>>>>>>>>>>>>> UnsupportedVersionException >>>>>>>>>>>>>>>>>>>>>>>>>> immediately, rather than silently falling back to >>>>>>> latest >>>>>>>>>>> and >>>>>>>>>>>>>>>>>> risking >>>>>>>>>>>>>>>>>>>>>>>>> data loss without any operator-visible signal. >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> MB1/MB2/MB3: >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> I have addressed these changes in the KIP. >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards, >>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <[email protected]> 於 2026年4月29日 >>>>>>>> 下午4:04 >>>>>>>>>>> 寫道： >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> hi David >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> I agree with the direction of moving the 'age' >>>>>>>>>> resolution >>>>>>>>>>>>>> from >>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>> Heartbeat API to the Metadata API to keep the control >>>>>>>>>> plane >>>>>>>>>>>>>>>> clean. >>>>>>>>>>>>>>>>>> The >>>>>>>>>>>>>>>>>>>>>>> main >>>>>>>>>>>>>>>>>>>>>>>>> trade-off, as we noted before, is introducing >>>>>>>> inter-broker >>>>>>>>>>>>>> clock >>>>>>>>>>>>>>>>>> skew. >>>>>>>>>>>>>>>>>>>>>>> The >>>>>>>>>>>>>>>>>>>>>>>>> Group Coordinator approach provided a single source of >>>>>>>>>> truth >>>>>>>>>>>>>> for >>>>>>>>>>>>>>>>>> time. >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> However, realistically, this time skew should be >>>>>>>>>>> negligible. >>>>>>>>>>>>>>>>>> Given >>>>>>>>>>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>>>>>>>>>> the max.age threshold will likely be configured in >>>>>>>> minutes >>>>>>>>>>> or >>>>>>>>>>>>>>>>>> hours, a >>>>>>>>>>>>>>>>>>>>>>>>> typical NTP skew (in milliseconds) between brokers >>>>>>> won't >>>>>>>>>>> impact >>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>> fallback decision. >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> David Jacot via dev <[email protected]> 於 >>>>>>>>>> 2026年4月29日 >>>>>>>>>>>>>>>> 下午3:29 >>>>>>>>>>>>>>>>>> 寫道： >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all, >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP! >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry, I haven't really followed the previous >>>>>>>>>>> conversation >>>>>>>>>>>>>>>> but I >>>>>>>>>>>>>>>>>>>>>>> took a >>>>>>>>>>>>>>>>>>>>>>>>>>>> quick look at this one. >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ01: I don't clearly understand the flow with the >>>>>>>>>>>>>>>>>>>>>>>>> ConsumerGroupHeartbeat >>>>>>>>>>>>>>>>>>>>>>>>>>>> API after reading the KIP. There is a new boolean; >>>>>>> the >>>>>>>>>>> KIP >>>>>>>>>>>>>>>>>> states >>>>>>>>>>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>>>>>>>>>>>>> partition ages are returned only when this boolean >>>>>>> is >>>>>>>>>>> set. >>>>>>>>>>>>>>>>>>>>>>> Implicitly, >>>>>>>>>>>>>>>>>>>>>>>>> this >>>>>>>>>>>>>>>>>>>>>>>>>>>> means that when the consumer receives a new >>>>>>> partition, >>>>>>>>>> it >>>>>>>>>>>>>> will >>>>>>>>>>>>>>>>>>>>>>> issue a >>>>>>>>>>>>>>>>>>>>>>>>> new >>>>>>>>>>>>>>>>>>>>>>>>>>>> HB request with the boolean set to receive the >>>>>>> ages. >>>>>>>> Is >>>>>>>>>>> my >>>>>>>>>>>>>>>>>>>>>>>>> understanding >>>>>>>>>>>>>>>>>>>>>>>>>>>> correct? We should perhaps clarify the flow and >>>>>>> also >>>>>>>>>>> explain >>>>>>>>>>>>>>>>>> how it >>>>>>>>>>>>>>>>>>>>>>>>> fits >>>>>>>>>>>>>>>>>>>>>>>>>>>> into the existing flow (e.g. list offsets, fetch >>>>>>>>>> offsets, >>>>>>>>>>>>>>>> etc.). >>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ02: It my understanding is correct, I wonder if >>>>>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API is the right place >>>>>>> for >>>>>>>>>>> this >>>>>>>>>>>>>>>> given >>>>>>>>>>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>>>>>>>>>> a new >>>>>>>>>>>>>>>>>>>>>>>>>>>> round trip is done anyway. Alternatively, it could >>>>>>>>>> simply >>>>>>>>>>>>>>>>>> include >>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>> metadata. Generally, we should be rather cautious >>>>>>>> about >>>>>>>>>>> not >>>>>>>>>>>>>>>>>>>>>>> overloading >>>>>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API with unrelated >>>>>>>> concepts. >>>>>>>>>>> The >>>>>>>>>>>>>>>> API >>>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>>>> a >>>>>>>>>>>>>>>>>>>>>>>>>>>> control plane API for assigning or revoking >>>>>>>> partitions. >>>>>>>>>>> The >>>>>>>>>>>>>>>> fact >>>>>>>>>>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>>>>>>>>>> we >>>>>>>>>>>>>>>>>>>>>>>>>>>> don't want to add it to the corresponding Streams >>>>>>> API >>>>>>>>>>> also >>>>>>>>>>>>>>>>>> suggests >>>>>>>>>>>>>>>>>>>>>>>>>>>> something is not quite right. What would we do if >>>>>>> we >>>>>>>>>>> want to >>>>>>>>>>>>>>>>>>>>>>> support >>>>>>>>>>>>>>>>>>>>>>>>>>>> Streams in the future? >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>>>>>>>>>> David >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 29, 2026 at 12:28 AM Muralidhar Basani >>>>>>>> via >>>>>>>>>>> dev >>>>>>>>>>>>>> < >>>>>>>>>>>>>>>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jiunn, >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for this great kip. Good to know about >>>>>>> the >>>>>>>>>>> gap. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-0 - why a new v2 version bump for >>>>>>>>>>> RequestPartitionAges >>>>>>>>>>>>>>>>>> field. >>>>>>>>>>>>>>>>>>>>>>> Can a >>>>>>>>>>>>>>>>>>>>>>>>>>>>> tagged field (for ex: on response, PartitionAges >>>>>>> on >>>>>>>>>>>>>>>>>>>>>>> TopicPartitions) >>>>>>>>>>>>>>>>>>>>>>>>> be >>>>>>>>>>>>>>>>>>>>>>>>>>>>> used here and avoid version bump? >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-1 - For the new config, is there a recommended >>>>>>>>>> value >>>>>>>>>>> or >>>>>>>>>>>>>> a >>>>>>>>>>>>>>>>>>>>>>> ConfigDef >>>>>>>>>>>>>>>>>>>>>>>>>>>>> validator? Probably it should based on the >>>>>>>>>>>>>>>> metadata.max.age.ms >>>>>>> <https://urldefense.com/v3/__http://metadata.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkflKEb5SK$> >>>>>>>>>>>>>>>>>> ? >>>>>>>>>>>>>>>>>>>>>>>>> Sizing >>>>>>>>>>>>>>>>>>>>>>>>>>>>> instructions can be part of javadocs I guess. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-2 - (minor) As there are no changes to Kafka >>>>>>>>>> Streams, >>>>>>>>>>>>>>>> would >>>>>>>>>>>>>>>>>> it >>>>>>>>>>>>>>>>>>>>>>> be >>>>>>>>>>>>>>>>>>>>>>>>> better >>>>>>>>>>>>>>>>>>>>>>>>>>>>> to add this new config >>>>>>>>>> auto.offset.reset.latest.max.age >>>>>>>>>>> to >>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>> StreamsConfig block list >>>>>>>>>>>>>>>>>>>>>>> (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS) >>>>>>>>>>>>>>>>>>>>>>>>> for a >>>>>>>>>>>>>>>>>>>>>>>>>>>>> clear warning, incase users configure it? This is >>>>>>> the >>>>>>>>>>> most >>>>>>>>>>>>>>>>>>>>>>> familiar >>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config and users might easily mistakenly >>>>>>>>>>> configure >>>>>>>>>>>>>>>>>> it. Or >>>>>>>>>>>>>>>>>>>>>>>>> may be >>>>>>>>>>>>>>>>>>>>>>>>>>>>> it's not worth it to add. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-3 - (minor) The phrasing "the consumer falls >>>>>>> back >>>>>>>>>> to >>>>>>>>>>>>>>>>>> earliest" >>>>>>>>>>>>>>>>>>>>>>>>> reads as >>>>>>>>>>>>>>>>>>>>>>>>>>>>> if the config were being changed per-partition >>>>>>> which >>>>>>>>>>> isn't >>>>>>>>>>>>>>>>>>>>>>> supported. >>>>>>>>>>>>>>>>>>>>>>>>> May >>>>>>>>>>>>>>>>>>>>>>>>>>>>> be rephrasing to something like "consumer resolves >>>>>>>> the >>>>>>>>>>>>>>>> initial >>>>>>>>>>>>>>>>>>>>>>>>> position to >>>>>>>>>>>>>>>>>>>>>>>>>>>>> start offset for that partition" as if earliest >>>>>>> was >>>>>>>>>>> applied >>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition only and auto.offset.reset config is >>>>>>>>>>> unchanged. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Murali >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 28, 2026 at 2:48 PM 黃竣陽 < >>>>>>>>>>> [email protected]> >>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi chia, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have updated the KIP to include this change. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <[email protected]> 於 >>>>>>> 2026年4月28日 >>>>>>>>>>> 晚上8:03 >>>>>>>>>>>>>>>> 寫道： >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> chia_0: Should we expose the partition creation >>>>>>>> time >>>>>>>>>>> via >>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>> Admin >>>>>>>>>>>>>>>>>>>>>>>>> API? >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I assume it would be valuable for users to >>>>>>> diagnose >>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>>> troubleshoot >>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> behavior of auto.offset.reset.latest.max.age >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/04/28 10:47:58 黃竣陽 wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a discussion on KIP-1327 >>>>>>>>>>> Prevent >>>>>>>>>>>>>> Hot >>>>>>>>>>>>>>>>>> Data >>>>>>>>>>>>>>>>>>>>>>>>> Loss >>>>>>>>>>>>>>>>>>>>>>>>>>>>> on >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Partition Expansion for Latest Policy >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> < >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$ >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to introduces >>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age, >>>>>>>>>>>>>>>>>>>>>>>>> a >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config that lets the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> latest reset policy distinguish newly expanded >>>>>>>>>> (hot) >>>>>>>>>>>>>>>>>> partitions >>>>>>>>>>>>>>>>>>>>>>>>> from >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> long-existing (cold) ones. Partitions >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> younger than the configured threshold >>>>>>>> automatically >>>>>>>>>>> fall >>>>>>>>>>>>>>>>>> back >>>>>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> earliest, preventing silent data loss >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during topic expansion without forcing a full >>>>>>>>>>> historical >>>>>>>>>>>>>>>>>>>>>>> reprocess. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>> >>> >>> >
