Thanks Jiunn for the responses. I don't have any further questions. Regards, Murali
On Thu, May 7, 2026 at 2:07 PM 黃竣陽 <[email protected]> wrote: > Hi all, > > I’d like to manually bump this thread. > > Best Regards, > Jiunn-Yang > > > 黃竣陽 <[email protected]> 於 2026年5月1日 晚上10:37 寫道： > > > > Hello all, > > > > Thanks for the feedback. > > > > DJ01/DJ02: > > > > MetadataResponse bumps from v13 to v14. The PartitionMetadata struct > gains a new > > field PartitionAgeMs (int64, default -1), computed server-side by the > broker as > > broker_current_time - partition_creation_time. > > > > Also add the consumer heartbeat flow. when MembershipManager detects a > newly assigned > > partition, it explicitly invalidates the metadata for the affected topic > and forces a fresh MetadataRequest > > before making the offset reset decision, even if the topic ID is already > in the cache. > > > > MB0: > > > > The consumer learns the broker's maximum supported MetadataResponse > version via the > > ApiVersions negotiation at connection time. If the negotiated version is > unsupported, the consumer > > knows the broker does not support PartitionAgeMs at all and can throw an > UnsupportedVersionException > > immediately, rather than silently falling back to latest and risking > data loss without any operator-visible signal. > > > > MB1/MB2/MB3: > > > > I have addressed these changes in the KIP. > > > > Best Regards, > > Jiunn-Yang > > > >> Chia-Ping Tsai <[email protected]> 於 2026年4月29日 下午4:04 寫道： > >> > >> hi David > >> > >> I agree with the direction of moving the 'age' resolution from the > Heartbeat API to the Metadata API to keep the control plane clean. The main > trade-off, as we noted before, is introducing inter-broker clock skew. The > Group Coordinator approach provided a single source of truth for time. > >> > >> However, realistically, this time skew should be negligible. Given that > the max.age threshold will likely be configured in minutes or hours, a > typical NTP skew (in milliseconds) between brokers won't impact the > fallback decision. > >> > >> Best, > >> Chia-Ping > >> > >>> David Jacot via dev <[email protected]> 於 2026年4月29日 下午3:29 寫道： > >>> > >>> Hi all, > >>> > >>> Thanks for the KIP! > >>> > >>> Sorry, I haven't really followed the previous conversation but I took a > >>> quick look at this one. > >>> > >>> DJ01: I don't clearly understand the flow with the > ConsumerGroupHeartbeat > >>> API after reading the KIP. There is a new boolean; the KIP states that > >>> partition ages are returned only when this boolean is set. Implicitly, > this > >>> means that when the consumer receives a new partition, it will issue a > new > >>> HB request with the boolean set to receive the ages. Is my > understanding > >>> correct? We should perhaps clarify the flow and also explain how it > fits > >>> into the existing flow (e.g. list offsets, fetch offsets, etc.). > >>> DJ02: It my understanding is correct, I wonder if > >>> the ConsumerGroupHeartbeat API is the right place for this given that > a new > >>> round trip is done anyway. Alternatively, it could simply include the > >>> metadata. Generally, we should be rather cautious about not overloading > >>> the ConsumerGroupHeartbeat API with unrelated concepts. The API is a > >>> control plane API for assigning or revoking partitions. The fact that > we > >>> don't want to add it to the corresponding Streams API also suggests > >>> something is not quite right. What would we do if we want to support > >>> Streams in the future? > >>> > >>> Best, > >>> David > >>> > >>>> On Wed, Apr 29, 2026 at 12:28 AM Muralidhar Basani via dev < > >>>> [email protected]> wrote: > >>>> > >>>> Hi Jiunn, > >>>> > >>>> Thank you for this great kip. Good to know about the gap. > >>>> > >>>> mb-0 - why a new v2 version bump for RequestPartitionAges field. Can a > >>>> tagged field (for ex: on response, PartitionAges on TopicPartitions) > be > >>>> used here and avoid version bump? > >>>> > >>>> mb-1 - For the new config, is there a recommended value or a ConfigDef > >>>> validator? Probably it should based on the metadata.max.age.ms ? > Sizing > >>>> instructions can be part of javadocs I guess. > >>>> > >>>> mb-2 - (minor) As there are no changes to Kafka Streams, would it be > better > >>>> to add this new config auto.offset.reset.latest.max.age to the > >>>> StreamsConfig block list (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS) > for a > >>>> clear warning, incase users configure it? This is the most familiar > >>>> consumer config and users might easily mistakenly configure it. Or > may be > >>>> it's not worth it to add. > >>>> > >>>> mb-3 - (minor) The phrasing "the consumer falls back to earliest" > reads as > >>>> if the config were being changed per-partition which isn't supported. > May > >>>> be rephrasing to something like "consumer resolves the initial > position to > >>>> start offset for that partition" as if earliest was applied to that > >>>> partition only and auto.offset.reset config is unchanged. > >>>> > >>>> Thanks, > >>>> Murali > >>>> > >>>>> On Tue, Apr 28, 2026 at 2:48 PM 黃竣陽 <[email protected]> wrote: > >>>>> > >>>>> Hi chia, > >>>>> > >>>>> I have updated the KIP to include this change. > >>>>> > >>>>> Best Regards, > >>>>> Jiunn-Yang > >>>>> > >>>>>> Chia-Ping Tsai <[email protected]> 於 2026年4月28日 晚上8:03 寫道： > >>>>>> > >>>>>> hi Jiunn-Yang > >>>>>> > >>>>>> chia_0: Should we expose the partition creation time via the Admin > API? > >>>>> I assume it would be valuable for users to diagnose and troubleshoot > the > >>>>> behavior of auto.offset.reset.latest.max.age > >>>>>> > >>>>>> Best, > >>>>>> Chia-Ping > >>>>>> > >>>>>> On 2026/04/28 10:47:58 黃竣陽 wrote: > >>>>>>> Hello everyone, > >>>>>>> > >>>>>>> I would like to start a discussion on KIP-1327 Prevent Hot Data > Loss > >>>> on > >>>>> Partition Expansion for Latest Policy > >>>>>>> < > >>>> > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$ > >>>>> > >>>>>>> > >>>>>>> This proposal aims to introduces auto.offset.reset.latest.max.age, > a > >>>>> consumer config that lets the > >>>>>>> latest reset policy distinguish newly expanded (hot) partitions > from > >>>>> long-existing (cold) ones. Partitions > >>>>>>> younger than the configured threshold automatically fall back to > >>>>> earliest, preventing silent data loss > >>>>>>> during topic expansion without forcing a full historical reprocess. > >>>>>>> > >>>>>>> Best regards, > >>>>>>> Jiunn-Yang > >>>>> > >>>>> > >>>> > > > >
