Hi Andrew,
Thanks for the thorough review.
AS1:
The updated KIP now explicitly addresses these scenarios:
- Configuring auto.offset.reset=by_start_time with a classic consumer group
throws
ConfigException at startup.
- Classic groups upgraded to modern groups cannot use by_start_time
immediately,
because the group creation timestamp was not recorded at the time the group was
originally
created. The consumer throws `GroupCreationTimeNotAvailableException` in this
case.
Users must delete and recreate the group to obtain a fresh creation timestamp.
AS2, AS3, AS5: Updated the KIP accordingly.
AS4:
Tagged fields fit well because GroupCreationTimeMs / CreationTimeMs is
optional,
with a default value of -1 ("unknown") that has clear semantics, especially for
pre-existing
groups. They are only serialized when non-default, so they add no overhead.
Older
brokers simply ignore unknown tags when replaying __consumer_offsets, requiring
no
record rewriting during upgrades.
In contrast, a version bump would require expanding validVersions, adding
version checks
across read/write paths, and provides no real benefit given the field’s
optional nature and
well-defined default. This follows Kafka’s convention: use tagged fields for
optional additions
with meaningful defaults, and version bumps for mandatory or semantic changes.
AS6:
Thanks for the clarification. I’ve incorporated this into the KIP.
AS7:
I agree that by_start_time better matches user expectations for the default
offset reset policy.
However, since it is not supported by classic consumer groups and would raise a
ConfigException
at startup, changing the default is only safe after the classic protocol is
removed from KafkaConsumer.
Per KIP-1274, classic protocol support will be deprecated in Kafka 6.0. After
that, the
default can be changed from latest to by_start_time.
AS8:
InvalidOffsetException is an abstract class that requires subclasses to
implement partitions(),
implying the error is tied to specific partitions with invalid offsets.
GroupCreationTimeNotAvailableException does not fit this model. The root cause
is a missing
group-level timestamp, not an issue with any particular partition. Implementing
partitions()
would require returning an empty or artificial set, which violates the API’s
semantics.
Therefore, I believe extending KafkaException is the better choice.
AS9:
Renamed to GroupCreationTimeNotAvailableException throughout the KIP. This
better
describes the state ("not available") rather than the cause.
Happy to hear any further feedback or suggestions.
Best Regards,
Jiunn-Yang
> Andrew Schofield <[email protected]> 於 2026年4月2日 晚上8:34 寫道：
>
> Hi Jiunn-Yang,
> Thanks for this great KIP. Some comments.
>
> AS1: A limitation of this KIP is that it will not support classic consumer
> groups. I think this is not a problem and it will be a motivator for users to
> adopt modern consumer groups, but it will be necessary to test attempting to
> use the new policy with classic consumer groups and make sure that the
> ConsumerGroupDescription lacks the creation timestamp and so on. There is
> also the question of classic groups which are upgraded to modern consumer
> groups.
> AS2: It seems to me that you should also support streams groups, both in a
> StreamsGroupMetadataValue and StreamsGroupDescription.
> AS3: What you are describing as metadata log changes are actually records on
> the __consumer_offsets topic.
> AS4: While I understand the need to use tagged fields in the records, I do
> not see why you cannot bump the versions for ConsumerGroupHeartbeat and
> ConsumerGroupDescribe in order to support the new fields.
> AS5: In the Admin API, we already have an optional timestamp in
> TransactionDescription.transactionStartTimeMs which uses the type
> OptionalLong. Any reason not to use the same in ConsumerGroupDescription?
> AS6: I don't think share group support is needed. "latest" already starts at
> offset 0 for newly added partitions for share groups.
> AS7: Might you consider making "by_start_time" the default value for the
> auto.offset.reset consumer config in the next major release? I know that's a
> bold move, but I think what you're getting with the new option is what people
> expected "latest" to behave like.
> AS8: I wonder whether GroupCreationTimeUnknownException should extend the
> abstract InvalidOffsetException.
> AS9: To follow existing exception names, I think
> UnknownGroupCreationTimeException, GroupCreationTimeNotAvailableException (my
> favourite) or GroupCreationTimeNotFoundException would better match what we
> already have. wdyt?
>
> Thanks,
> Andrew
>
> On 2026/04/02 11:34:10 黃竣陽 wrote:
>> Hi chia,
>>
>> chia_13:
>> `ConsumerGroupDescription` now gains a new method `groupCreationTimeMs()`:
>>
>> `groupCreationTimeMs()` (`Optional<Long>`): the epoch-ms when this consumer
>> group
>> was first created on the broker. `Optional.empty()` if unknown.
>>
>> Please take a look at the updated KIP and let us know if you have further
>> questions.
>>
>> Best Regards,
>> Jiunn-Yang
>>
>>> Chia-Ping Tsai <[email protected]> 於 2026年4月2日 上午9:58 寫道：
>>>
>>> hi Jiunn
>>>
>>> chia_13: should we update the public API: ConsumerGroupDescription as well?
>>>
>>> Best,
>>> Chia-Ping
>>>
>>> On 2026/04/01 11:59:46 黃竣陽 wrote:
>>>> Hi chia,
>>>>
>>>> Thank you for the thoughtful feedback!
>>>>
>>>> chia_10: I have added a clarification in the Proposed Changes section to
>>>> address this. In this case,
>>>> ListOffsetsRequest will resolve to the Log End Offset, but this is not a
>>>> fallback to `latest`. The distinction
>>>> matters: `latest` is a direct seek to the Log End Offset, whereas
>>>> `by_start_time` always anchors to the
>>>> group creation timestamp and lets the lookup result follow naturally. The
>>>> consumer will begin consuming
>>>> any new records produced after this point normally.
>>>>
>>>> chia_11: I have introduced a new exception
>>>> `GroupCreationTimeUnknownException` to signal
>>>> this specific condition.
>>>>
>>>> chia_12: I have updated the KIP to expose the group creation timestamp via
>>>> `AdminClient.describeConsumerGroups()`
>>>> , accessible through `ConsumerGroupDescription.groupCreationTimeMs()`.
>>>> This is backed by a new
>>>> `GroupCreationTimeMs` field in `ConsumerGroupDescribeResponse.
>>>>
>>>> Best Regards,
>>>> Jiunn-Yang
>>>>
>>>>> Chia-Ping Tsai <[email protected]> 於 2026年4月1日 下午2:33 寫道：
>>>>>
>>>>> hi Jiunn
>>>>>
>>>>> thanks for updating KIP. I have a couple of questions.
>>>>>
>>>>> chia_10: What happen if the group creation time is strictly greater than
>>>>> the timestamp of the latest record in the partition? Will the consumer
>>>>> fall back to the "latest" offset behaviour in this case? It could be good
>>>>> to elucidate that in the KIP
>>>>>
>>>>> chia_11: Could you specify which exact exception will be thrown when the
>>>>> group creation time is unknown
>>>>>
>>>>> chia_12: Since the group creation time is now a critical semantic anchor,
>>>>> should we also expose it via the `DescribeGroupsResponse`?
>>>>>
>>>>> Best,
>>>>> Chia-Ping
>>>>>
>>>>> On 2026/03/05 11:14:31 黃竣陽 wrote:
>>>>>> Hello everyone,
>>>>>>
>>>>>> I would like to start a discussion on KIP-1282: Prevent data loss during
>>>>>> partition expansion for dynamically added partitions
>>>>>> <https://cwiki.apache.org/confluence/x/mIY8G>
>>>>>>
>>>>>> This proposal aims to introduce a new auto.offset.reset policy
>>>>>> by_start_time, anchoring the
>>>>>> offset reset to the consumer's startup timestamp rather than partition
>>>>>> discovery time, to prevent
>>>>>> silent data loss during partition expansion.
>>>>>>
>>>>>> Best regards,
>>>>>> Jiunn-Yang
>>>>
>>>>
>>
>>
