Makes total sense.
On 2026/04/03 12:25:57 Chia-Ping Tsai wrote: > > AS7: I'll have one last try at persuasion here, but I expect I won't > succeed. In KIP-1274, the default group protocol for KafkaConsumer is > changing to "consumer". It is true that "classic" is still supported, but > users will need to explicitly enable it by setting > "group.protocol=classic". I think this means we could change the default > for "auto.offset.reset" to "by_start_time" in AK 5.0. Whether we want to do > it is a different matter. > > I also love this new policy, but changing the default right now is a bit > too eager for this KIP. I would prefer to implement it in 4.4.0 as an > opt-in feature and announce it to users first. If we see good adoption and > positive feedback from the community, we can then open a new KIP to discuss > changing the default in a future release. Let's take it one step at a time > > > Andrew Schofield <[email protected]> 於 2026年4月3日週五 下午7:58寫道： > > > Hi Jiunn-Yang, > > Thanks for your response. > > > > AS1-AS6: lgtm > > > > AS7: I'll have one last try at persuasion here, but I expect I won't > > succeed. In KIP-1274, the default group protocol for KafkaConsumer is > > changing to "consumer". It is true that "classic" is still supported, but > > users will need to explicitly enable it by setting > > "group.protocol=classic". I think this means we could change the default > > for "auto.offset.reset" to "by_start_time" in AK 5.0. Whether we want to do > > it is a different matter. > > > > AS8, AS9: lgtm > > > > AS10: I'm not a fan of the "Epoch-ms" wording, particularly in the Admin > > API javadoc. I would just use "The creation time of the group." or similar. > > > > AS11: I suspect you need to change StreamsGroupHeartbeatResponse too. > > Probably need a Kafka Streams expert to weigh in how > > `auto.offset.reset=by_start_time` would work for the consumer in a streams > > group. > > > > Thanks, > > Andrew > > > > On 2026/04/03 01:18:02 黃竣陽 wrote: > > > Hello all, > > > > > > AS4 > > > > > > I agree that bumping the version for request/response protocols provides > > a > > > meaningful benefit: it allows the consumer to distinguish between "the > > broker > > > does not support this feature" and "the broker supports it, but the > > group has no > > > creation time." This enables more actionable error messages. > > > > > > For __consumer_offsets records (ConsumerGroupMetadataValue and > > > StreamsGroupMetadataValue), I will keep tagged fields. These records > > have no version > > > negotiation. They are written and read internally by the broker. Tagged > > fields allow old brokers > > > to safely ignore unknown tags during log replay, and avoid the need to > > rewrite existing records > > > on upgrade. > > > > > > Best Regards, > > > Jiunn-Yang > > > > > > > Chia-Ping Tsai <[email protected]> 於 2026年4月3日 上午8:01 寫道： > > > > > > > > AS4 > > > > > > > > Bumping the version actually provides a significant benefit. It allows > > the new consumer to offer more actionable error messages to users, because > > it can clearly distinguish between "the broker is too old to support this > > feature" and "the broker supports it, but the group has no creation time." > > > > > > > > Furthermore, by bumping the version, the auto-generated protocol code > > can gracefully handle the serialization (e.g., ignoring the field if the > > client/broker is communicating with an older version). > > > > > > > > On 2026/04/02 15:04:39 黃竣陽 wrote: > > > >> Hi Andrew, > > > >> > > > >> Thanks for the thorough review. > > > >> > > > >> AS1: > > > >> The updated KIP now explicitly addresses these scenarios: > > > >> - Configuring auto.offset.reset=by_start_time with a classic consumer > > group throws > > > >> ConfigException at startup. > > > >> - Classic groups upgraded to modern groups cannot use by_start_time > > immediately, > > > >> because the group creation timestamp was not recorded at the time the > > group was originally > > > >> created. The consumer throws `GroupCreationTimeNotAvailableException` > > in this case. > > > >> Users must delete and recreate the group to obtain a fresh creation > > timestamp. > > > >> > > > >> AS2, AS3, AS5: Updated the KIP accordingly. > > > >> > > > >> AS4: > > > >> > > > >> Tagged fields fit well because GroupCreationTimeMs / CreationTimeMs > > is optional, > > > >> with a default value of -1 ("unknown") that has clear semantics, > > especially for pre-existing > > > >> groups. They are only serialized when non-default, so they add no > > overhead. Older > > > >> brokers simply ignore unknown tags when replaying __consumer_offsets, > > requiring no > > > >> record rewriting during upgrades. > > > >> > > > >> In contrast, a version bump would require expanding validVersions, > > adding version checks > > > >> across read/write paths, and provides no real benefit given the > > field’s optional nature and > > > >> well-defined default. This follows Kafka’s convention: use tagged > > fields for optional additions > > > >> with meaningful defaults, and version bumps for mandatory or semantic > > changes. > > > >> > > > >> AS6: > > > >> > > > >> Thanks for the clarification. I’ve incorporated this into the KIP. > > > >> > > > >> AS7: > > > >> > > > >> I agree that by_start_time better matches user expectations for the > > default offset reset policy. > > > >> However, since it is not supported by classic consumer groups and > > would raise a ConfigException > > > >> at startup, changing the default is only safe after the classic > > protocol is removed from KafkaConsumer. > > > >> Per KIP-1274, classic protocol support will be deprecated in Kafka > > 6.0. After that, the > > > >> default can be changed from latest to by_start_time. > > > >> > > > >> AS8: > > > >> > > > >> InvalidOffsetException is an abstract class that requires subclasses > > to implement partitions(), > > > >> implying the error is tied to specific partitions with invalid > > offsets. > > > >> > > > >> GroupCreationTimeNotAvailableException does not fit this model. The > > root cause is a missing > > > >> group-level timestamp, not an issue with any particular partition. > > Implementing partitions() > > > >> would require returning an empty or artificial set, which violates > > the API’s semantics. > > > >> > > > >> Therefore, I believe extending KafkaException is the better choice. > > > >> > > > >> AS9: > > > >> > > > >> Renamed to GroupCreationTimeNotAvailableException throughout the KIP. > > This better > > > >> describes the state ("not available") rather than the cause. > > > >> > > > >> Happy to hear any further feedback or suggestions. > > > >> > > > >> Best Regards, > > > >> Jiunn-Yang > > > >> > > > >> > > > >>> Andrew Schofield <[email protected]> 於 2026年4月2日 晚上8:34 寫道： > > > >>> > > > >>> Hi Jiunn-Yang, > > > >>> Thanks for this great KIP. Some comments. > > > >>> > > > >>> AS1: A limitation of this KIP is that it will not support classic > > consumer groups. I think this is not a problem and it will be a motivator > > for users to adopt modern consumer groups, but it will be necessary to test > > attempting to use the new policy with classic consumer groups and make sure > > that the ConsumerGroupDescription lacks the creation timestamp and so on. > > There is also the question of classic groups which are upgraded to modern > > consumer groups. > > > >>> AS2: It seems to me that you should also support streams groups, > > both in a StreamsGroupMetadataValue and StreamsGroupDescription. > > > >>> AS3: What you are describing as metadata log changes are actually > > records on the __consumer_offsets topic. > > > >>> AS4: While I understand the need to use tagged fields in the > > records, I do not see why you cannot bump the versions for > > ConsumerGroupHeartbeat and ConsumerGroupDescribe in order to support the > > new fields. > > > >>> AS5: In the Admin API, we already have an optional timestamp in > > TransactionDescription.transactionStartTimeMs which uses the type > > OptionalLong. Any reason not to use the same in ConsumerGroupDescription? > > > >>> AS6: I don't think share group support is needed. "latest" already > > starts at offset 0 for newly added partitions for share groups. > > > >>> AS7: Might you consider making "by_start_time" the default value for > > the auto.offset.reset consumer config in the next major release? I know > > that's a bold move, but I think what you're getting with the new option is > > what people expected "latest" to behave like. > > > >>> AS8: I wonder whether GroupCreationTimeUnknownException should > > extend the abstract InvalidOffsetException. > > > >>> AS9: To follow existing exception names, I think > > UnknownGroupCreationTimeException, GroupCreationTimeNotAvailableException > > (my favourite) or GroupCreationTimeNotFoundException would better match > > what we already have. wdyt? > > > >>> > > > >>> Thanks, > > > >>> Andrew > > > >>> > > > >>> On 2026/04/02 11:34:10 黃竣陽 wrote: > > > >>>> Hi chia, > > > >>>> > > > >>>> chia_13: > > > >>>> `ConsumerGroupDescription` now gains a new method > > `groupCreationTimeMs()`: > > > >>>> > > > >>>> `groupCreationTimeMs()` (`Optional<Long>`): the epoch-ms when this > > consumer group > > > >>>> was first created on the broker. `Optional.empty()` if unknown. > > > >>>> > > > >>>> Please take a look at the updated KIP and let us know if you have > > further questions. > > > >>>> > > > >>>> Best Regards, > > > >>>> Jiunn-Yang > > > >>>> > > > >>>>> Chia-Ping Tsai <[email protected]> 於 2026年4月2日 上午9:58 寫道： > > > >>>>> > > > >>>>> hi Jiunn > > > >>>>> > > > >>>>> chia_13: should we update the public API: ConsumerGroupDescription > > as well? > > > >>>>> > > > >>>>> Best, > > > >>>>> Chia-Ping > > > >>>>> > > > >>>>> On 2026/04/01 11:59:46 黃竣陽 wrote: > > > >>>>>> Hi chia, > > > >>>>>> > > > >>>>>> Thank you for the thoughtful feedback! > > > >>>>>> > > > >>>>>> chia_10: I have added a clarification in the Proposed Changes > > section to address this. In this case, > > > >>>>>> ListOffsetsRequest will resolve to the Log End Offset, but this > > is not a fallback to `latest`. The distinction > > > >>>>>> matters: `latest` is a direct seek to the Log End Offset, whereas > > `by_start_time` always anchors to the > > > >>>>>> group creation timestamp and lets the lookup result follow > > naturally. The consumer will begin consuming > > > >>>>>> any new records produced after this point normally. > > > >>>>>> > > > >>>>>> chia_11: I have introduced a new exception > > `GroupCreationTimeUnknownException` to signal > > > >>>>>> this specific condition. > > > >>>>>> > > > >>>>>> chia_12: I have updated the KIP to expose the group creation > > timestamp via `AdminClient.describeConsumerGroups()` > > > >>>>>> , accessible through > > `ConsumerGroupDescription.groupCreationTimeMs()`. This is backed by a new > > > >>>>>> `GroupCreationTimeMs` field in `ConsumerGroupDescribeResponse. > > > >>>>>> > > > >>>>>> Best Regards, > > > >>>>>> Jiunn-Yang > > > >>>>>> > > > >>>>>>> Chia-Ping Tsai <[email protected]> 於 2026年4月1日 下午2:33 寫道： > > > >>>>>>> > > > >>>>>>> hi Jiunn > > > >>>>>>> > > > >>>>>>> thanks for updating KIP. I have a couple of questions. > > > >>>>>>> > > > >>>>>>> chia_10: What happen if the group creation time is strictly > > greater than the timestamp of the latest record in the partition? Will the > > consumer fall back to the "latest" offset behaviour in this case? It could > > be good to elucidate that in the KIP > > > >>>>>>> > > > >>>>>>> chia_11: Could you specify which exact exception will be thrown > > when the group creation time is unknown > > > >>>>>>> > > > >>>>>>> chia_12: Since the group creation time is now a critical > > semantic anchor, should we also expose it via the `DescribeGroupsResponse`? > > > >>>>>>> > > > >>>>>>> Best, > > > >>>>>>> Chia-Ping > > > >>>>>>> > > > >>>>>>> On 2026/03/05 11:14:31 黃竣陽 wrote: > > > >>>>>>>> Hello everyone, > > > >>>>>>>> > > > >>>>>>>> I would like to start a discussion on KIP-1282: Prevent data > > loss during partition expansion for dynamically added partitions > > > >>>>>>>> <https://cwiki.apache.org/confluence/x/mIY8G> > > > >>>>>>>> > > > >>>>>>>> This proposal aims to introduce a new auto.offset.reset policy > > by_start_time, anchoring the > > > >>>>>>>> offset reset to the consumer's startup timestamp rather than > > partition discovery time, to prevent > > > >>>>>>>> silent data loss during partition expansion. > > > >>>>>>>> > > > >>>>>>>> Best regards, > > > >>>>>>>> Jiunn-Yang > > > >>>>>> > > > >>>>>> > > > >>>> > > > >>>> > > > >> > > > >> > > > > > > > > >
