Hi, Chia-Ping, Thanks for the reply.
1. In the motivation section, the KIP says "When a Kafka topic is expanded with new partitions, consumers using the latest auto offset reset policy will silently miss all records produced to those partitions before the consumer discovers them.". If a user sets auto.offset.reset=by_duration=1sec, the same record loss issue could also happen, right? 2. I was thinking auto.offset.reset.new.partitions will take the same values as auto.offset.reset. So a user could set it by_duration if needed. Jun On Thu, May 14, 2026 at 4:06 PM Chia-Ping Tsai <[email protected]> wrote: > hi Jun > > Thanks for the feedback. I might be missing something important from your > suggestion, so please bear with me as I try to clarify with a few questions: > > 1. Is there a strong use case for extending this logic to other reset > policies? Unlike latest, policies like earliest or by_duration don't seem > to suffer from the same silent data loss issue when a partition is expanded. > > 2. What values would we expect users to configure for > auto.offset.reset.new.partitions? If they set it to earliest or latest, > we might run into the exact same edge cases. For example, if a consumer is > offline for a while and a new partition is created during that downtime, > the user might actually want to skip to latest when resuming, rather than > reading from earliest just because the partition is technically "new" to > the group. > > This is exactly why we opted for introducing a max.age threshold. It gives > users a time-bound way to define what is genuinely "hot/new" and what is > just an old partition they haven't seen yet. > > Best, > Chia-Ping > > On 2026/05/14 20:48:09 Jun Rao via dev wrote: > > Hi, Jiunn-Yang, > > > > Thanks for the KIP. > > > > I find auto.offset.reset.latest.max.age a bit weird. It only applies when > > auto.offset.reset is latest. However, it seems that the motivation > equally > > applies when auto.offset.reset is set to other values like by_duration. > The > > intention is that we want to have a separate way to control newly created > > partitions vs existing partitions when the group starts. Have we > considered > > adding a new config like auto.offset.reset.new.partitions? If this new > > config is not set, the offset reset policy defaults to the policy used > for > > existing partitions. The user could set it explicitly to customize the > > behavior for new partitions. > > > > Jun > > > > On Thu, May 7, 2026 at 5:07 AM 黃竣陽 <[email protected]> wrote: > > > > > Hi all, > > > > > > I’d like to manually bump this thread. > > > > > > Best Regards, > > > Jiunn-Yang > > > > > > > 黃竣陽 <[email protected]> 於 2026年5月1日 晚上10:37 寫道： > > > > > > > > Hello all, > > > > > > > > Thanks for the feedback. > > > > > > > > DJ01/DJ02: > > > > > > > > MetadataResponse bumps from v13 to v14. The PartitionMetadata struct > > > gains a new > > > > field PartitionAgeMs (int64, default -1), computed server-side by the > > > broker as > > > > broker_current_time - partition_creation_time. > > > > > > > > Also add the consumer heartbeat flow. when MembershipManager detects > a > > > newly assigned > > > > partition, it explicitly invalidates the metadata for the affected > topic > > > and forces a fresh MetadataRequest > > > > before making the offset reset decision, even if the topic ID is > already > > > in the cache. > > > > > > > > MB0: > > > > > > > > The consumer learns the broker's maximum supported MetadataResponse > > > version via the > > > > ApiVersions negotiation at connection time. If the negotiated > version is > > > unsupported, the consumer > > > > knows the broker does not support PartitionAgeMs at all and can > throw an > > > UnsupportedVersionException > > > > immediately, rather than silently falling back to latest and risking > > > data loss without any operator-visible signal. > > > > > > > > MB1/MB2/MB3: > > > > > > > > I have addressed these changes in the KIP. > > > > > > > > Best Regards, > > > > Jiunn-Yang > > > > > > > >> Chia-Ping Tsai <[email protected]> 於 2026年4月29日 下午4:04 寫道： > > > >> > > > >> hi David > > > >> > > > >> I agree with the direction of moving the 'age' resolution from the > > > Heartbeat API to the Metadata API to keep the control plane clean. The > main > > > trade-off, as we noted before, is introducing inter-broker clock skew. > The > > > Group Coordinator approach provided a single source of truth for time. > > > >> > > > >> However, realistically, this time skew should be negligible. Given > that > > > the max.age threshold will likely be configured in minutes or hours, a > > > typical NTP skew (in milliseconds) between brokers won't impact the > > > fallback decision. > > > >> > > > >> Best, > > > >> Chia-Ping > > > >> > > > >>> David Jacot via dev <[email protected]> 於 2026年4月29日 下午3:29 寫道： > > > >>> > > > >>> Hi all, > > > >>> > > > >>> Thanks for the KIP! > > > >>> > > > >>> Sorry, I haven't really followed the previous conversation but I > took a > > > >>> quick look at this one. > > > >>> > > > >>> DJ01: I don't clearly understand the flow with the > > > ConsumerGroupHeartbeat > > > >>> API after reading the KIP. There is a new boolean; the KIP states > that > > > >>> partition ages are returned only when this boolean is set. > Implicitly, > > > this > > > >>> means that when the consumer receives a new partition, it will > issue a > > > new > > > >>> HB request with the boolean set to receive the ages. Is my > > > understanding > > > >>> correct? We should perhaps clarify the flow and also explain how it > > > fits > > > >>> into the existing flow (e.g. list offsets, fetch offsets, etc.). > > > >>> DJ02: It my understanding is correct, I wonder if > > > >>> the ConsumerGroupHeartbeat API is the right place for this given > that > > > a new > > > >>> round trip is done anyway. Alternatively, it could simply include > the > > > >>> metadata. Generally, we should be rather cautious about not > overloading > > > >>> the ConsumerGroupHeartbeat API with unrelated concepts. The API is > a > > > >>> control plane API for assigning or revoking partitions. The fact > that > > > we > > > >>> don't want to add it to the corresponding Streams API also suggests > > > >>> something is not quite right. What would we do if we want to > support > > > >>> Streams in the future? > > > >>> > > > >>> Best, > > > >>> David > > > >>> > > > >>>> On Wed, Apr 29, 2026 at 12:28 AM Muralidhar Basani via dev < > > > >>>> [email protected]> wrote: > > > >>>> > > > >>>> Hi Jiunn, > > > >>>> > > > >>>> Thank you for this great kip. Good to know about the gap. > > > >>>> > > > >>>> mb-0 - why a new v2 version bump for RequestPartitionAges field. > Can a > > > >>>> tagged field (for ex: on response, PartitionAges on > TopicPartitions) > > > be > > > >>>> used here and avoid version bump? > > > >>>> > > > >>>> mb-1 - For the new config, is there a recommended value or a > ConfigDef > > > >>>> validator? Probably it should based on the metadata.max.age.ms ? > > > Sizing > > > >>>> instructions can be part of javadocs I guess. > > > >>>> > > > >>>> mb-2 - (minor) As there are no changes to Kafka Streams, would it > be > > > better > > > >>>> to add this new config auto.offset.reset.latest.max.age to the > > > >>>> StreamsConfig block list > (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS) > > > for a > > > >>>> clear warning, incase users configure it? This is the most > familiar > > > >>>> consumer config and users might easily mistakenly configure it. Or > > > may be > > > >>>> it's not worth it to add. > > > >>>> > > > >>>> mb-3 - (minor) The phrasing "the consumer falls back to earliest" > > > reads as > > > >>>> if the config were being changed per-partition which isn't > supported. > > > May > > > >>>> be rephrasing to something like "consumer resolves the initial > > > position to > > > >>>> start offset for that partition" as if earliest was applied to > that > > > >>>> partition only and auto.offset.reset config is unchanged. > > > >>>> > > > >>>> Thanks, > > > >>>> Murali > > > >>>> > > > >>>>> On Tue, Apr 28, 2026 at 2:48 PM 黃竣陽 <[email protected]> wrote: > > > >>>>> > > > >>>>> Hi chia, > > > >>>>> > > > >>>>> I have updated the KIP to include this change. > > > >>>>> > > > >>>>> Best Regards, > > > >>>>> Jiunn-Yang > > > >>>>> > > > >>>>>> Chia-Ping Tsai <[email protected]> 於 2026年4月28日 晚上8:03 寫道： > > > >>>>>> > > > >>>>>> hi Jiunn-Yang > > > >>>>>> > > > >>>>>> chia_0: Should we expose the partition creation time via the > Admin > > > API? > > > >>>>> I assume it would be valuable for users to diagnose and > troubleshoot > > > the > > > >>>>> behavior of auto.offset.reset.latest.max.age > > > >>>>>> > > > >>>>>> Best, > > > >>>>>> Chia-Ping > > > >>>>>> > > > >>>>>> On 2026/04/28 10:47:58 黃竣陽 wrote: > > > >>>>>>> Hello everyone, > > > >>>>>>> > > > >>>>>>> I would like to start a discussion on KIP-1327 Prevent Hot Data > > > Loss > > > >>>> on > > > >>>>> Partition Expansion for Latest Policy > > > >>>>>>> < > > > >>>> > > > > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$ > > > >>>>> > > > >>>>>>> > > > >>>>>>> This proposal aims to introduces > auto.offset.reset.latest.max.age, > > > a > > > >>>>> consumer config that lets the > > > >>>>>>> latest reset policy distinguish newly expanded (hot) partitions > > > from > > > >>>>> long-existing (cold) ones. Partitions > > > >>>>>>> younger than the configured threshold automatically fall back > to > > > >>>>> earliest, preventing silent data loss > > > >>>>>>> during topic expansion without forcing a full historical > reprocess. > > > >>>>>>> > > > >>>>>>> Best regards, > > > >>>>>>> Jiunn-Yang > > > >>>>> > > > >>>>> > > > >>>> > > > > > > > > > > > > >
