Hello chia, Thanks for the feedback, I have updated the KIP.
Best Regards, Jiunn-Yang > Chia-Ping Tsai <[email protected]> 於 2026年6月17日 凌晨12:47 寫道： > > hi Jiunn-Yang > >> When the config is set on a cluster that has not yet been upgraded... >> classification cannot occur... the consumer falls back to the base >> auto.offset.reset for the affected partitions. No exception is thrown, and >> no operational disruption results. > > Existing group can't take advantage of this excellent new configuration. > Allowing users to modify the group creation time might be overkill. Instead, > we could print a useful warning message to guide users. For example, we can > suggest that they re-create the group with their existing committed offsets > >> Protocol changes > > Would you mind listing those RPC changes in a table format? > >> The full interaction matrix between the base policy and the new-partition >> policy is: > > Please add a filed to describe the target scenario when using these policies > > Best, > Chia-Ping > > > On 2026/06/16 16:14:49 黃竣陽 wrote: >> Hello Jun, chia, >> >> Thanks for the feedback, I have updated the KIP for the new >> approach, PTAL >> >> Best Regards, >> Jiunn-Yang >> >>> Chia-Ping Tsai <[email protected]> 於 2026年6月16日 上午8:23 寫道： >>> >>> hi Jun >>> >>> Yes, your approach is great. I think the combination of latest (for >>> existing partitions) and by_duration (for new partitions) can address 99% >>> of the complaints I have heard regarding this issue. >>> >>> Also, leveraging the group creation time here opens the door to >>> implementing a new policy based on timestamp seek in the future, should the >>> community want to pursue that. >>> >>> Thanks for your patience and constructive feedback. We will update the KIP >>> accordingly. >>> >>> Best, Chia-Ping >>> >>>> Jun Rao via dev <[email protected]> 於 2026年6月16日 清晨5:11 寫道： >>>> >>>> Hi, Chia-Ping, >>>> >>>> Thanks for the reply. >>>> >>>> I agree that it's probably useful to allow a user to configure a different >>>> offset policy for existing partitions vs new partitions. However, using >>>> group creation time to capture that seems more intuitive. Here is another >>>> proposal: remove auto.offset.reset.max.age.ms and categorize new partitions >>>> based on group creation time. Introduce >>>> a new config auto.offset.reset.new.partitions whose values can be earliest, >>>> latest and by_duration, the same as auto.offset.reset. Users can set >>>> `auto.offset.reset.new.partitions` to `earliest` if they want to guarantee >>>> no data loss on new partitions. They can also use by_duration to set an >>>> upper bound on the backlog replayed, which can be different from that of >>>> the existing partitions. This will address your concern about too much >>>> backlog being replayed when the offsets are lost. What do you think? >>>> >>>> Jun >>>> >>>> >>>>> On Mon, Jun 15, 2026 at 10:39 AM Chia-Ping Tsai <[email protected]> >>>>> wrote: >>>>> >>>>> hi Jun >>>>> >>>>> The most important part of this story is how users should expect the data >>>>> they can see when using the latest or by_duration policy with expanded >>>>> partitions. >>>>> >>>>> Yes, the by_duration policy can minimize data loss, but it is >>>>> non-deterministic, which means users will either read too many historical >>>>> records from existing partitions or lose some records from expanded >>>>> partitions. >>>>> >>>>> Also, I agree that auto.offset.reset.max.age.ms >>>>> <https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$> >>>>> is a bit hard to understand, and that is why I preferred having a whole >>>>> new >>>>> policy based entirely on group creation time (KIP-1282) >>>>> >>>>> Best, >>>>> Chia-Ping >>>>> >>>>> Jun Rao via dev <[email protected]> 於 2026年6月16日週二 上午1:08寫道： >>>>> >>>>>> Hi, Chia-Ping and Jiunn-Yang, >>>>>> >>>>>> Thanks for the reply. I am still trying to understand the value of the >>>>>> new >>>>>> configs with the KIP. >>>>>> >>>>>> The motivation of the KIP is that a user doesn't want to miss the data if >>>>>> the backlog is small. The backlog of the existing partition is easy to >>>>>> understand because it relates to retention time. The backlog for the new >>>>>> partition is a bit subtle to understand since it depends on the metadata >>>>>> refresh delay. To set auto.offset.reset.max.age.ms >>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$>, >>>>>> the user needs to >>>>>> understand the metadata refresh delay on the consumer side and use it to >>>>>> set the config. >>>>>> >>>>>> Now, let's consider the alternative: setting the same value for the >>>>>> existing by_duration policy. The KIP lists three issues with this >>>>>> approach. >>>>>> 1. It computes the seek target client-side as now() - duration, which >>>>>> introduces clock skew across consumers and forces operators to choose >>>>>> overly large durations, causing unnecessary reprocessing. >>>>>> 2. The target timestamp is recomputed on each retry, so failed >>>>>> ListOffsetsRequest retries can shift the target forward and potentially >>>>>> miss records produced between attempts. >>>>>> 3. It applies uniformly to all partitions without committed offsets, and >>>>>> cannot distinguish newly expanded partitions from long-existing >>>>>> partitions >>>>>> newly assigned to the group, leading to unnecessary replay. >>>>>> >>>>>> Issues 1 and 2 are uncommon and can be mitigated by adding a bit buffer >>>>>> to >>>>>> the metadata refresh delay. We could also consider improving the >>>>>> implementation. For issue 3, the metadata refresh delay is typically low >>>>>> (in the order of minutes with the classic consumer and tens of seconds >>>>>> with >>>>>> the new consumer). If a user is ok with reading that much backlog for new >>>>>> partitions, it seems they will be ok doing the same for existing >>>>>> partitions. >>>>>> >>>>>> So, instead of introducing a new config, could we just reuse the existing >>>>>> config with better documentation and/or implementation? >>>>>> >>>>>> Jun >>>>>> >>>>>> >>>>>>> On Sat, Jun 13, 2026 at 12:19 AM 黃竣陽 <[email protected]> wrote: >>>>>> >>>>>>> Hello Jun, >>>>>>> >>>>>>> You're right that group creation time is the more intuitive answer at >>>>>>> first glance, >>>>>>> the KIP's own motivation talks about partitions that "predate the group" >>>>>>> vs partitions >>>>>>> "created during group runtime," which directly points to a >>>>>> group-lifecycle >>>>>>> classifier. >>>>>>> I'd like to walk through why we landed on partition age, and the >>>>>>> trade-offs we considered. >>>>>>> >>>>>>> We evaluated three candidate signals: >>>>>>> >>>>>>> 1. `by_duration:5secs` >>>>>>> >>>>>>> This covers the metadata blindness window, but has issues the KIP >>>>>>> currently documents >>>>>>> under "Why not use `by_duration`?": >>>>>>> >>>>>>> - Client-side `now() - duration` introduces clock skew across consumers. >>>>>>> - `ListOffsets` retries shift the target forward, potentially missing >>>>>>> records produced between >>>>>>> attempts. >>>>>>> - It applies uniformly to all partitions without committed offsets, >>>>>>> including pre-existing partitions >>>>>>> newly assigned to the group, causing unnecessary replay. >>>>>>> >>>>>>> 2. Group creation time as classifier >>>>>>> >>>>>>> This works cleanly when the consumer is actively running. Our concern >>>>>>> is the idle / late-rejoin case: >>>>>>> >>>>>>> T=0: Group created. >>>>>>> T=1..T=100: Consumer idle (down, disconnected, etc.). >>>>>>> T=50: Partition added during the idle window. >>>>>>> T=100: Consumer resumes. >>>>>>> >>>>>>> Under group creation time, the new partition is classified as new >>>>>>> (`50 > 0`) and reset to `earliest`, replaying everything from T=50. >>>>>>> But during `[T=1, T=100]`, base partitions also accumulated data that >>>>>>> the consumer accepts as lost — that is precisely the contract of >>>>>>> `auto.offset.reset=latest`. There is no principled reason to treat >>>>>>> the new partition differently; both contain backlog accumulated during >>>>>>> the same idle window. >>>>>>> >>>>>>> This aligns with the "backlog is backlog” principle you raised in >>>>>>> the KIP-1282 thread: a `latest` user has tolerated some backlog on >>>>>>> every other partition during the same idle period; forcing 0-backlog >>>>>>> tolerance only on new partitions would be inconsistent with that >>>>>>> tolerance. >>>>>>> >>>>>>> 3. Partition age vs threshold >>>>>>> >>>>>>> Partition age corresponds to the actual silent data loss window, >>>>>>> the gap between partition creation and the consumer’s metadata >>>>>>> refresh. Within this window, data loss is genuinely silent: the >>>>>>> consumer had no opportunity to know about the partition. Outside this >>>>>>> window, missing data reflects either: >>>>>>> >>>>>>> - (a) the user’s tolerated cost of running with idle consumers, or >>>>>>> - (b) an operational issue to surface via monitoring, not via reset >>>>>> policy. >>>>>>> >>>>>>> We did not choose partition age because it is more elegant than group >>>>>>> creation time — we chose it because its failure mode (requires a >>>>>>> threshold) is >>>>>>> less invasive than the failure mode of group creation time (overrides >>>>>>> user-stated >>>>>>> `latest` intent during idle periods). >>>>>>> >>>>>>> Best Regards, >>>>>>> Jiunn-Yang >>>>>>> >>>>>>>> Chia-Ping Tsai <[email protected]> 於 2026年6月13日 上午11:52 寫道： >>>>>>>> >>>>>>>> Hi Jun, >>>>>>>> >>>>>>>> Relying on both creation times will create an inconsistent scenario. A >>>>>>>> consumer that lost all offsets due to a long sleep will seek to the >>>>>>>> beginning for the partitions created later than the group. >>>>>>>> >>>>>>>> That is why we initially proposed KIP-1282 to fix the inconsistency >>>>>>> using a >>>>>>>> whole new policy. Since KIP-1282 couldn't reach a consensus, KIP-1327 >>>>>>> goes >>>>>>>> back to using flexible configurations to prevent users from falling >>>>>> into >>>>>>>> that pitfall. >>>>>>>> >>>>>>>> Best, Chia-Ping >>>>>>>> >>>>>>>> Jun Rao via dev <[email protected]> 於 2026年6月13日週六 上午6:49寫道： >>>>>>>> >>>>>>>>> Hi, Jiunn-Yang, >>>>>>>>> >>>>>>>>> Thanks for the reply and sorry for the late reply. >>>>>>>>> >>>>>>>>> JR1. The design of auto.offset.reset.max.age.ms >>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$> >>>>>> still feels weird to >>>>>>> me. >>>>>>>>> It >>>>>>>>> categorizes partitions as new or existing based on the partition >>>>>>> creation >>>>>>>>> time. Intuitively, the categorization should be based on the group >>>>>>> creation >>>>>>>>> time: all partitions existing when the group is created are existing >>>>>> and >>>>>>>>> all partitions created after the group creation are new partitions. >>>>>>>>> >>>>>>>>> Jun >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Tue, Jun 9, 2026 at 8:51 AM 黃竣陽 <[email protected]> wrote: >>>>>>>>> >>>>>>>>>> Hi all, >>>>>>>>>> >>>>>>>>>> Manually bumping this thread. If there is no further >>>>>>>>>> discussion, I will close the vote. >>>>>>>>>> >>>>>>>>>> Best Regards, >>>>>>>>>> Jiunn-Yang >>>>>>>>>> >>>>>>>>>>> 黃竣陽 <[email protected]> 於 2026年6月1日 晚上7:16 寫道： >>>>>>>>>>> >>>>>>>>>>> Hello Jian, >>>>>>>>>>> >>>>>>>>>>> Thanks for your feedback, >>>>>>>>>>> >>>>>>>>>>> Agreed, partition expansion is a common operational task, not an >>>>>> edge >>>>>>>>>>> case. I've updated the Motivation section accordingly. >>>>>>>>>>> >>>>>>>>>>> Best Regards, >>>>>>>>>>> Jiunn-Yang >>>>>>>>>>> >>>>>>>>>>>> jian fu <[email protected]> 於 2026年6月1日 下午5:49 寫道： >>>>>>>>>>>> >>>>>>>>>>>> Hi Jiunn-Yang: >>>>>>>>>>>> >>>>>>>>>>>> Thanks for the KIP. I think it would be useful to clarify that >>>>>> this >>>>>>>>> is a >>>>>>>>>>>> common scenario rather than an edge case, which further >>>>>> demonstrates >>>>>>>>> the >>>>>>>>>>>> need for this optimization. For example: >>>>>>>>>>>> A partition expansion is a common operational task in Kafka: To >>>>>>>>> balance >>>>>>>>>>>> resource utilization and cost, topics are typically created with a >>>>>>>>>> moderate >>>>>>>>>>>> default partition count. However, as traffic grows over time, it >>>>>> is >>>>>>>>>> often >>>>>>>>>>>> necessary to increase the number of partitions to accommodate the >>>>>>>>> higher >>>>>>>>>>>> workload. >>>>>>>>>>>> >>>>>>>>>>>> Regards >>>>>>>>>>>> Jian >>>>>>>>>>>> >>>>>>>>>>>> 黃竣陽 <[email protected]> 于2026年5月30日周六 22:31写道： >>>>>>>>>>>> >>>>>>>>>>>>> Hello chia, >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks for the comments, I have updated the KIP! >>>>>>>>>>>>> >>>>>>>>>>>>> Best Regards, >>>>>>>>>>>>> Jiunn-Yang >>>>>>>>>>>>> >>>>>>>>>>>>>> Chia-Ping Tsai <[email protected]> 於 2026年5月30日 晚上8:29 寫道： >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Jiunn-Yang, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Would you mind removing the terms "hot" and "cold" when >>>>>> describing >>>>>>>>>>>>>> partitions in the KIP? I understand you are using them to >>>>>> describe >>>>>>>>> the >>>>>>>>>>>>>> "freshness" or the users' need for the records, but applying >>>>>> these >>>>>>>>>> terms >>>>>>>>>>>>> to >>>>>>>>>>>>>> the partition itself feels a bit unnatural. >>>>>>>>>>>>>> >>>>>>>>>>>>>> After all, in this scenario, users don't really care whether a >>>>>>>>>> partition >>>>>>>>>>>>> is >>>>>>>>>>>>>> newly expanded or not. Their only expectation is that they won't >>>>>>>>>> silently >>>>>>>>>>>>>> lose any live records produced to the topic during their active >>>>>>>>>>>>> consumption. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Best, Chia-Ping >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> 黃竣陽 <[email protected]> 於 2026年5月30日週六 下午12:30寫道： >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hello Jun, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks for the feedback, I have updated the KIP motivation >>>>>>> section. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Best Regards, >>>>>>>>>>>>>>> Jiunn-Yang >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Jun Rao via dev <[email protected]> 於 2026年5月30日 凌晨1:12 >>>>>> 寫道： >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi, Jiunn-Yang, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks for the reply. I think we need a stronger motivation >>>>>> for >>>>>>>>> the >>>>>>>>>>>>> KIP. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> The KIP says "The core insight is that not all partitions >>>>>> without >>>>>>>>> a >>>>>>>>>>>>>>>> committed offset are the same. A newly expanded partition >>>>>> (hot) >>>>>>> is >>>>>>>>>>>>>>>> fundamentally different from a partition the consumer has >>>>>> never >>>>>>>>> seen >>>>>>>>>>>>>>>> because it predates the group (cold)." Why is the hot >>>>>> partition >>>>>>>>>>>>>>>> fundamentally different from the cold? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> The KIP says "The existing by_duration policy is also >>>>>>> insufficient >>>>>>>>>>>>>>> because: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> - The calculated seek time (now() - duration) varies across >>>>>> nodes >>>>>>>>>> due >>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>> clock skew. To be safe, users must set an overly large >>>>>> duration, >>>>>>>>>>>>>>> causing >>>>>>>>>>>>>>>> unnecessary reprocessing. >>>>>>>>>>>>>>>> - On network errors, the client recalculates the seek time on >>>>>>>>> retry, >>>>>>>>>>>>>>>> shifting the target timestamp forward and risking data loss." >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> However, both of these situations are rare. If these issues >>>>>>>>> persist, >>>>>>>>>>>>> more >>>>>>>>>>>>>>>> severe problems likely exist elsewhere. Rare situations don't >>>>>>>>> need a >>>>>>>>>>>>>>> common >>>>>>>>>>>>>>>> solution. If users care about those rare situations, they can >>>>>>>>>> implement >>>>>>>>>>>>>>>> customized logic using >>>>>>>>>>>>> ConsumerRebalanceListener.onPartitionsAssigned(). >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Jun >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Sun, May 17, 2026 at 6:50 AM 黃竣陽 <[email protected]> >>>>>> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hello chia, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks for the feedback, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> If the creation time exists, the returned value should >>>>>> always >>>>>>> be >>>>>>>>>>>>>>> greater >>>>>>>>>>>>>>>>> than or equal to zero, right? >>>>>>>>>>>>>>>>> I have explicitly mentioned this in the KIP. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> New Old (MetadataResponse v0–13) positive any >>>>>>>>>> field >>>>>>>>>>>>>>>>> absent UnsupportedVersionException >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> The earliest point at which we can detect the version >>>>>> mismatch >>>>>>> is >>>>>>>>>>>>> during >>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>> first metadata fetch after assignment, which occurs inside >>>>>>>>> poll(). >>>>>>>>>>>>>>>>> Therefore, the >>>>>>>>>>>>>>>>> user would encounter an UnsupportedVersionException from >>>>>> poll(). >>>>>>>>>> I’ll >>>>>>>>>>>>>>>>> clarify this in the KIP. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Best Regards, >>>>>>>>>>>>>>>>> Jiunn-Yang >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Chia-Ping Tsai <[email protected]> 於 2026年5月17日 下午4:50 >>>>>> 寫道： >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> hi Jiunn >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> PartitionAgeMs (int64, default -1): The age of this >>>>>> partition >>>>>>>>> in >>>>>>>>>>>>>>>>> milliseconds, computed server-side by the broker as >>>>>>>>>>>>> broker_current_time >>>>>>>>>>>>>>> - >>>>>>>>>>>>>>>>> partition_creation_time. Returns -1 if the broker does not >>>>>>>>> support >>>>>>>>>>>>> this >>>>>>>>>>>>>>>>> feature or the partition creation time is unknown. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> If the creation time exists, the returned value should >>>>>> always >>>>>>> be >>>>>>>>>>>>>>> greater >>>>>>>>>>>>>>>>> than or equal to zero, right? >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> New Old (MetadataResponse v0–13) positive any >>>>>>>>>> field >>>>>>>>>>>>>>>>> absent UnsupportedVersionException >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Will user encounter UnsupportedVersionException when calling >>>>>>>>>>>>> `poll()`? >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>> Chia-Ping >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On 2026/05/16 04:30:49 黃竣陽 wrote: >>>>>>>>>>>>>>>>>>> Hello Jun, chia, >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> I've updated KIP-1327 with a design change based on the >>>>>>>>>> discussion >>>>>>>>>>>>>>>>>>> feedback. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> The updated design decouples the new-partition reset >>>>>> behavior >>>>>>>>>> from >>>>>>>>>>>>>>>>>>> the base auto.offset.reset policy: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> - auto.offset.reset.max.age.ms >>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$> >>>>>> now applies to all >>>>>>>>>> auto.offset.reset >>>>>>>>>>>>>>>>> values >>>>>>>>>>>>>>>>>>> (latest, earliest, by_duration, none). >>>>>>>>>>>>>>>>>>> - For new ("hot") partitions, the consumer resets to >>>>>>>>>>>>>>>>> auto.offset.reset.new >>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$> >>>>>> .partitions >>>>>>>>>>>>>>>>>>> config setting >>>>>>>>>>>>>>>>>>> - For existing ("cold") partitions, the base >>>>>> auto.offset.reset >>>>>>>>>>>>> policy >>>>>>>>>>>>>>>>> continues >>>>>>>>>>>>>>>>>>> to apply unchanged. >>>>>>>>>>>>>>>>>>> - The new-partition reset behavior is represented by a >>>>>>> separate >>>>>>>>>>>>>>>>> internal config >>>>>>>>>>>>>>>>>>> (auto.offset.reset.new >>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>.partitions, >>>>>> currently fixed to >>>>>>>>> earliest). >>>>>>>>>>>>> This >>>>>>>>>>>>>>>>> decoupled design makes >>>>>>>>>>>>>>>>>>> it straightforward to promote the behavior to a public >>>>>>>>>> user-facing >>>>>>>>>>>>>>>>> configuration in a future KIP. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Best Regards, >>>>>>>>>>>>>>>>>>> Jiunn-Yang >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <[email protected]> 於 2026年5月16日 清晨7:46 >>>>>> 寫道： >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> hi Jun >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> I see what you mean now. The proposal from me is listed >>>>>>> below: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> 1) Add auto.offset.reset.new >>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>.partitions >>>>>> with a default value >>>>>>>>> of >>>>>>>>>>>>>>>>> earliest. It fixes the data loss from both by_duration and >>>>>>>>> latest, >>>>>>>>>> and >>>>>>>>>>>>>>> it >>>>>>>>>>>>>>>>> does not change the logic of auto.offset.reset=earliest. >>>>>>>>>>>>>>>>>>>> 2) Mark auto.offset.reset.new >>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>.partitions >>>>>> as an internal >>>>>>>>>>>>>>>>> configuration. auto.offset.reset.new >>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$> >>>>>> .partitions=earliest >>>>>>> already >>>>>>>>>>>>>>>>> addresses the issue, and we can discuss the use cases of >>>>>> other >>>>>>>>>> values >>>>>>>>>>>>>>> in a >>>>>>>>>>>>>>>>> separate KIP. >>>>>>>>>>>>>>>>>>>> 3) Both configs, auto.offset.reset.new >>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>.partitions >>>>>> and >>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age.ms >>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.latest.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfu9JSP4l$>, >>>>>> will be applied to all for >>>>>>>>>>>>>>>>> consistency. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> WDYT? >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On 2026/05/15 20:53:20 Jun Rao via dev wrote: >>>>>>>>>>>>>>>>>>>>> Hi, Chia-Ping, >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Thanks for the reply. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> 1. In the motivation section, the KIP says "When a Kafka >>>>>>>>> topic >>>>>>>>>> is >>>>>>>>>>>>>>>>> expanded >>>>>>>>>>>>>>>>>>>>> with new partitions, consumers using the latest auto >>>>>> offset >>>>>>>>>> reset >>>>>>>>>>>>>>>>> policy >>>>>>>>>>>>>>>>>>>>> will silently miss all records produced to those >>>>>> partitions >>>>>>>>>> before >>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>> consumer discovers them.". If a user sets >>>>>>>>>>>>>>>>>>>>> auto.offset.reset=by_duration=1sec, the same record loss >>>>>>>>> issue >>>>>>>>>>>>> could >>>>>>>>>>>>>>>>> also >>>>>>>>>>>>>>>>>>>>> happen, right? >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> 2. I was thinking auto.offset.reset.new >>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>.partitions >>>>>> will >>>>>>> take >>>>>>>>>> the >>>>>>>>>>>>>>> same >>>>>>>>>>>>>>>>>>>>> values as auto.offset.reset. So a user could set it >>>>>>>>>> by_duration if >>>>>>>>>>>>>>>>> needed. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Jun >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> On Thu, May 14, 2026 at 4:06 PM Chia-Ping Tsai < >>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> hi Jun >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback. I might be missing something >>>>>>>>>> important >>>>>>>>>>>>>>> from >>>>>>>>>>>>>>>>> your >>>>>>>>>>>>>>>>>>>>>> suggestion, so please bear with me as I try to clarify >>>>>> with >>>>>>>>> a >>>>>>>>>> few >>>>>>>>>>>>>>>>> questions: >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> 1. Is there a strong use case for extending this logic >>>>>> to >>>>>>>>>> other >>>>>>>>>>>>>>> reset >>>>>>>>>>>>>>>>>>>>>> policies? Unlike latest, policies like earliest or >>>>>>>>> by_duration >>>>>>>>>>>>>>> don't >>>>>>>>>>>>>>>>> seem >>>>>>>>>>>>>>>>>>>>>> to suffer from the same silent data loss issue when a >>>>>>>>>> partition >>>>>>>>>>>>> is >>>>>>>>>>>>>>>>> expanded. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> 2. What values would we expect users to configure for >>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.new >>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>.partitions? >>>>>> If they set it to >>>>>>>>> earliest >>>>>>>>>> or >>>>>>>>>>>>>>>>> latest, >>>>>>>>>>>>>>>>>>>>>> we might run into the exact same edge cases. For >>>>>> example, >>>>>>>>> if a >>>>>>>>>>>>>>>>> consumer is >>>>>>>>>>>>>>>>>>>>>> offline for a while and a new partition is created >>>>>> during >>>>>>>>> that >>>>>>>>>>>>>>>>> downtime, >>>>>>>>>>>>>>>>>>>>>> the user might actually want to skip to latest when >>>>>>>>> resuming, >>>>>>>>>>>>>>> rather >>>>>>>>>>>>>>>>> than >>>>>>>>>>>>>>>>>>>>>> reading from earliest just because the partition is >>>>>>>>>> technically >>>>>>>>>>>>>>>>> "new" to >>>>>>>>>>>>>>>>>>>>>> the group. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> This is exactly why we opted for introducing a max.age >>>>>>>>>> threshold. >>>>>>>>>>>>>>> It >>>>>>>>>>>>>>>>> gives >>>>>>>>>>>>>>>>>>>>>> users a time-bound way to define what is genuinely >>>>>>> "hot/new" >>>>>>>>>> and >>>>>>>>>>>>>>>>> what is >>>>>>>>>>>>>>>>>>>>>> just an old partition they haven't seen yet. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>>>> Chia-Ping >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> On 2026/05/14 20:48:09 Jun Rao via dev wrote: >>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang, >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> I find auto.offset.reset.latest.max.age a bit weird. It >>>>>>>>> only >>>>>>>>>>>>>>>>> applies when >>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset is latest. However, it seems that the >>>>>>>>>>>>> motivation >>>>>>>>>>>>>>>>>>>>>> equally >>>>>>>>>>>>>>>>>>>>>>> applies when auto.offset.reset is set to other values >>>>>> like >>>>>>>>>>>>>>>>> by_duration. >>>>>>>>>>>>>>>>>>>>>> The >>>>>>>>>>>>>>>>>>>>>>> intention is that we want to have a separate way to >>>>>>> control >>>>>>>>>>>>> newly >>>>>>>>>>>>>>>>> created >>>>>>>>>>>>>>>>>>>>>>> partitions vs existing partitions when the group >>>>>> starts. >>>>>>>>>> Have we >>>>>>>>>>>>>>>>>>>>>> considered >>>>>>>>>>>>>>>>>>>>>>> adding a new config like auto.offset.reset.new >>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$> >>>>>>> .partitions? >>>>>>>>>> If >>>>>>>>>>>>>>> this >>>>>>>>>>>>>>>>> new >>>>>>>>>>>>>>>>>>>>>>> config is not set, the offset reset policy defaults to >>>>>> the >>>>>>>>>>>>> policy >>>>>>>>>>>>>>>>> used >>>>>>>>>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>>>>>>>>> existing partitions. The user could set it explicitly >>>>>> to >>>>>>>>>>>>> customize >>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>> behavior for new partitions. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Jun >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> On Thu, May 7, 2026 at 5:07 AM 黃竣陽 <[email protected] >>>>>>> >>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Hi all, >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> I’d like to manually bump this thread. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Best Regards, >>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> 黃竣陽 <[email protected]> 於 2026年5月1日 晚上10:37 寫道： >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Hello all, >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> DJ01/DJ02: >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> MetadataResponse bumps from v13 to v14. The >>>>>>>>>> PartitionMetadata >>>>>>>>>>>>>>>>> struct >>>>>>>>>>>>>>>>>>>>>>>> gains a new >>>>>>>>>>>>>>>>>>>>>>>>> field PartitionAgeMs (int64, default -1), computed >>>>>>>>>> server-side >>>>>>>>>>>>>>> by >>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>> broker as >>>>>>>>>>>>>>>>>>>>>>>>> broker_current_time - partition_creation_time. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Also add the consumer heartbeat flow. when >>>>>>>>>> MembershipManager >>>>>>>>>>>>>>>>> detects >>>>>>>>>>>>>>>>>>>>>> a >>>>>>>>>>>>>>>>>>>>>>>> newly assigned >>>>>>>>>>>>>>>>>>>>>>>>> partition, it explicitly invalidates the metadata for >>>>>>> the >>>>>>>>>>>>>>> affected >>>>>>>>>>>>>>>>>>>>>> topic >>>>>>>>>>>>>>>>>>>>>>>> and forces a fresh MetadataRequest >>>>>>>>>>>>>>>>>>>>>>>>> before making the offset reset decision, even if the >>>>>>>>> topic >>>>>>>>>> ID >>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>>> already >>>>>>>>>>>>>>>>>>>>>>>> in the cache. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> MB0: >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> The consumer learns the broker's maximum supported >>>>>>>>>>>>>>>>> MetadataResponse >>>>>>>>>>>>>>>>>>>>>>>> version via the >>>>>>>>>>>>>>>>>>>>>>>>> ApiVersions negotiation at connection time. If the >>>>>>>>>> negotiated >>>>>>>>>>>>>>>>>>>>>> version is >>>>>>>>>>>>>>>>>>>>>>>> unsupported, the consumer >>>>>>>>>>>>>>>>>>>>>>>>> knows the broker does not support PartitionAgeMs at >>>>>> all >>>>>>>>> and >>>>>>>>>>>>> can >>>>>>>>>>>>>>>>>>>>>> throw an >>>>>>>>>>>>>>>>>>>>>>>> UnsupportedVersionException >>>>>>>>>>>>>>>>>>>>>>>>> immediately, rather than silently falling back to >>>>>> latest >>>>>>>>>> and >>>>>>>>>>>>>>>>> risking >>>>>>>>>>>>>>>>>>>>>>>> data loss without any operator-visible signal. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> MB1/MB2/MB3: >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> I have addressed these changes in the KIP. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Best Regards, >>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <[email protected]> 於 2026年4月29日 >>>>>>> 下午4:04 >>>>>>>>>> 寫道： >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> hi David >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> I agree with the direction of moving the 'age' >>>>>>>>> resolution >>>>>>>>>>>>> from >>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>> Heartbeat API to the Metadata API to keep the control >>>>>>>>> plane >>>>>>>>>>>>>>> clean. >>>>>>>>>>>>>>>>> The >>>>>>>>>>>>>>>>>>>>>> main >>>>>>>>>>>>>>>>>>>>>>>> trade-off, as we noted before, is introducing >>>>>>> inter-broker >>>>>>>>>>>>> clock >>>>>>>>>>>>>>>>> skew. >>>>>>>>>>>>>>>>>>>>>> The >>>>>>>>>>>>>>>>>>>>>>>> Group Coordinator approach provided a single source of >>>>>>>>> truth >>>>>>>>>>>>> for >>>>>>>>>>>>>>>>> time. >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> However, realistically, this time skew should be >>>>>>>>>> negligible. >>>>>>>>>>>>>>>>> Given >>>>>>>>>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>>>>>>>>> the max.age threshold will likely be configured in >>>>>>> minutes >>>>>>>>>> or >>>>>>>>>>>>>>>>> hours, a >>>>>>>>>>>>>>>>>>>>>>>> typical NTP skew (in milliseconds) between brokers >>>>>> won't >>>>>>>>>> impact >>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>> fallback decision. >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> David Jacot via dev <[email protected]> 於 >>>>>>>>> 2026年4月29日 >>>>>>>>>>>>>>> 下午3:29 >>>>>>>>>>>>>>>>> 寫道： >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all, >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP! >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry, I haven't really followed the previous >>>>>>>>>> conversation >>>>>>>>>>>>>>> but I >>>>>>>>>>>>>>>>>>>>>> took a >>>>>>>>>>>>>>>>>>>>>>>>>>> quick look at this one. >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> DJ01: I don't clearly understand the flow with the >>>>>>>>>>>>>>>>>>>>>>>> ConsumerGroupHeartbeat >>>>>>>>>>>>>>>>>>>>>>>>>>> API after reading the KIP. There is a new boolean; >>>>>> the >>>>>>>>>> KIP >>>>>>>>>>>>>>>>> states >>>>>>>>>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>>>>>>>>>>>> partition ages are returned only when this boolean >>>>>> is >>>>>>>>>> set. >>>>>>>>>>>>>>>>>>>>>> Implicitly, >>>>>>>>>>>>>>>>>>>>>>>> this >>>>>>>>>>>>>>>>>>>>>>>>>>> means that when the consumer receives a new >>>>>> partition, >>>>>>>>> it >>>>>>>>>>>>> will >>>>>>>>>>>>>>>>>>>>>> issue a >>>>>>>>>>>>>>>>>>>>>>>> new >>>>>>>>>>>>>>>>>>>>>>>>>>> HB request with the boolean set to receive the >>>>>> ages. >>>>>>> Is >>>>>>>>>> my >>>>>>>>>>>>>>>>>>>>>>>> understanding >>>>>>>>>>>>>>>>>>>>>>>>>>> correct? We should perhaps clarify the flow and >>>>>> also >>>>>>>>>> explain >>>>>>>>>>>>>>>>> how it >>>>>>>>>>>>>>>>>>>>>>>> fits >>>>>>>>>>>>>>>>>>>>>>>>>>> into the existing flow (e.g. list offsets, fetch >>>>>>>>> offsets, >>>>>>>>>>>>>>> etc.). >>>>>>>>>>>>>>>>>>>>>>>>>>> DJ02: It my understanding is correct, I wonder if >>>>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API is the right place >>>>>> for >>>>>>>>>> this >>>>>>>>>>>>>>> given >>>>>>>>>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>>>>>>>>> a new >>>>>>>>>>>>>>>>>>>>>>>>>>> round trip is done anyway. Alternatively, it could >>>>>>>>> simply >>>>>>>>>>>>>>>>> include >>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>> metadata. Generally, we should be rather cautious >>>>>>> about >>>>>>>>>> not >>>>>>>>>>>>>>>>>>>>>> overloading >>>>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API with unrelated >>>>>>> concepts. >>>>>>>>>> The >>>>>>>>>>>>>>> API >>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>>> a >>>>>>>>>>>>>>>>>>>>>>>>>>> control plane API for assigning or revoking >>>>>>> partitions. >>>>>>>>>> The >>>>>>>>>>>>>>> fact >>>>>>>>>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>>>>>>>>> we >>>>>>>>>>>>>>>>>>>>>>>>>>> don't want to add it to the corresponding Streams >>>>>> API >>>>>>>>>> also >>>>>>>>>>>>>>>>> suggests >>>>>>>>>>>>>>>>>>>>>>>>>>> something is not quite right. What would we do if >>>>>> we >>>>>>>>>> want to >>>>>>>>>>>>>>>>>>>>>> support >>>>>>>>>>>>>>>>>>>>>>>>>>> Streams in the future? >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>>>>>>>>> David >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 29, 2026 at 12:28 AM Muralidhar Basani >>>>>>> via >>>>>>>>>> dev >>>>>>>>>>>>> < >>>>>>>>>>>>>>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jiunn, >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for this great kip. Good to know about >>>>>> the >>>>>>>>>> gap. >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-0 - why a new v2 version bump for >>>>>>>>>> RequestPartitionAges >>>>>>>>>>>>>>>>> field. >>>>>>>>>>>>>>>>>>>>>> Can a >>>>>>>>>>>>>>>>>>>>>>>>>>>> tagged field (for ex: on response, PartitionAges >>>>>> on >>>>>>>>>>>>>>>>>>>>>> TopicPartitions) >>>>>>>>>>>>>>>>>>>>>>>> be >>>>>>>>>>>>>>>>>>>>>>>>>>>> used here and avoid version bump? >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-1 - For the new config, is there a recommended >>>>>>>>> value >>>>>>>>>> or >>>>>>>>>>>>> a >>>>>>>>>>>>>>>>>>>>>> ConfigDef >>>>>>>>>>>>>>>>>>>>>>>>>>>> validator? Probably it should based on the >>>>>>>>>>>>>>> metadata.max.age.ms >>>>>> <https://urldefense.com/v3/__http://metadata.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkflKEb5SK$> >>>>>>>>>>>>>>>>> ? >>>>>>>>>>>>>>>>>>>>>>>> Sizing >>>>>>>>>>>>>>>>>>>>>>>>>>>> instructions can be part of javadocs I guess. >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-2 - (minor) As there are no changes to Kafka >>>>>>>>> Streams, >>>>>>>>>>>>>>> would >>>>>>>>>>>>>>>>> it >>>>>>>>>>>>>>>>>>>>>> be >>>>>>>>>>>>>>>>>>>>>>>> better >>>>>>>>>>>>>>>>>>>>>>>>>>>> to add this new config >>>>>>>>> auto.offset.reset.latest.max.age >>>>>>>>>> to >>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>> StreamsConfig block list >>>>>>>>>>>>>>>>>>>>>> (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS) >>>>>>>>>>>>>>>>>>>>>>>> for a >>>>>>>>>>>>>>>>>>>>>>>>>>>> clear warning, incase users configure it? This is >>>>>> the >>>>>>>>>> most >>>>>>>>>>>>>>>>>>>>>> familiar >>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config and users might easily mistakenly >>>>>>>>>> configure >>>>>>>>>>>>>>>>> it. Or >>>>>>>>>>>>>>>>>>>>>>>> may be >>>>>>>>>>>>>>>>>>>>>>>>>>>> it's not worth it to add. >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-3 - (minor) The phrasing "the consumer falls >>>>>> back >>>>>>>>> to >>>>>>>>>>>>>>>>> earliest" >>>>>>>>>>>>>>>>>>>>>>>> reads as >>>>>>>>>>>>>>>>>>>>>>>>>>>> if the config were being changed per-partition >>>>>> which >>>>>>>>>> isn't >>>>>>>>>>>>>>>>>>>>>> supported. >>>>>>>>>>>>>>>>>>>>>>>> May >>>>>>>>>>>>>>>>>>>>>>>>>>>> be rephrasing to something like "consumer resolves >>>>>>> the >>>>>>>>>>>>>>> initial >>>>>>>>>>>>>>>>>>>>>>>> position to >>>>>>>>>>>>>>>>>>>>>>>>>>>> start offset for that partition" as if earliest >>>>>> was >>>>>>>>>> applied >>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>>>>>>>>>>>>> partition only and auto.offset.reset config is >>>>>>>>>> unchanged. >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>>>>>>>>>>> Murali >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 28, 2026 at 2:48 PM 黃竣陽 < >>>>>>>>>> [email protected]> >>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi chia, >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have updated the KIP to include this change. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards, >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <[email protected]> 於 >>>>>> 2026年4月28日 >>>>>>>>>> 晚上8:03 >>>>>>>>>>>>>>> 寫道： >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> chia_0: Should we expose the partition creation >>>>>>> time >>>>>>>>>> via >>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>> Admin >>>>>>>>>>>>>>>>>>>>>>>> API? >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I assume it would be valuable for users to >>>>>> diagnose >>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>> troubleshoot >>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>> behavior of auto.offset.reset.latest.max.age >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/04/28 10:47:58 黃竣陽 wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a discussion on KIP-1327 >>>>>>>>>> Prevent >>>>>>>>>>>>> Hot >>>>>>>>>>>>>>>>> Data >>>>>>>>>>>>>>>>>>>>>>>> Loss >>>>>>>>>>>>>>>>>>>>>>>>>>>> on >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Partition Expansion for Latest Policy >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> < >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>> >>>>>> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$ >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to introduces >>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age, >>>>>>>>>>>>>>>>>>>>>>>> a >>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config that lets the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> latest reset policy distinguish newly expanded >>>>>>>>> (hot) >>>>>>>>>>>>>>>>> partitions >>>>>>>>>>>>>>>>>>>>>>>> from >>>>>>>>>>>>>>>>>>>>>>>>>>>>> long-existing (cold) ones. Partitions >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> younger than the configured threshold >>>>>>> automatically >>>>>>>>>> fall >>>>>>>>>>>>>>>>> back >>>>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>>>>>> earliest, preventing silent data loss >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during topic expansion without forcing a full >>>>>>>>>> historical >>>>>>>>>>>>>>>>>>>>>> reprocess. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >> >> >>
