Hi Sean,
Thanks for your thoughtful feedback!
sq1: Thank you for pointing this out — that was indeed a typo. The intended
meaning is to decouple the buffer size from the maximum allowed message size
for writing.
sq2: You are absolutely right. While we could theoretically add validation
during config alteration to enforce that max.message.bytes is not set lower
than the buffer size, this would undoubtedly introduce complexity and require
changes to the valid value definition of max.message.bytes. In practice, the
purpose of this constraint was to indicate that setting an excessively large
buffer size is unnecessary, since records that large won’t actually be
produced. The KIP has been updated accordingly: the <= max.message.bytes
constraint has been removed, and instead an explanatory note has been added to
the documentation.
sq3: The goal is indeed to decouple the buffer size from the maximum allowed
message size. We want to avoid having the buffer cached for occasionally large
messages.
Best,
Lan
At 2025-08-21 16:28:35, "Sean Quah" <sq...@confluent.io.INVALID> wrote:
>Hi Lan, thanks for the KIP!
>
>sq1: Can we make a distinction between the broker-level `message.max.bytes`
>and the topic-level `max.message.bytes`? When `__consumer_offsets` and
>`__share_group_state` are created, their `max.message.bytes` comes from the
>broker-level `message.max.bytes`. The topic-level `max.message.bytes` can
>be changed at any time by an admin afterwards. It is the topic-level config
>that determines the buffer size right now.
>
>sq2: The valid values for the new configs should be `<=` the topic-level
>`max.message.bytes`. I don't think we can stop an admin from reconfiguring
>`max.message.bytes` lower than
>`{group,share}.coordinator.append.max.buffer.size` though.
>
>sq3: Is the motivation to decouple the buffer size from the broker-level
>`message.max.bytes` (intended for applications) or from the topic-level
>`max.message.bytes`? If it's the former, we could consider an alternative
>where we define `offsets.topic.max.message.bytes` and
>`share.coordinator.state.topic.max.message.bytes` instead. We already have
>`{offsets,share.coordinator.state}.topic.{num.partitions,replication.factor,segment.bytes}`
>to override the default topic creation settings. This would address the
>memory issue because the internal Kafka topics will have good defaults.
>
>Thanks,
>Sean
>
>On Thu, Aug 21, 2025 at 4:05 AM Lan Ding <isdin...@163.com> wrote:
>
>> Hi Andrew and Sushant Mahajan,
>>
>> Thank you for the review and suggestions.
>>
>>
>>
>>
>> AS1: You're absolutely right. I've now added the
>> share.coordinator.append.max.buffer.size
>>
>> config with the same default value of 1*1024*1024 as you suggested.
>>
>>
>>
>>
>> Sushant, thank you for highlighting the necessary changes to the
>> CoordinatorRuntime.
>>
>> I've updated the implementation accordingly.
>>
>>
>>
>>
>> Best,
>>
>> Lan
>>
>> At 2025-08-21 03:26:02, "Sushant Mahajan" <sushmah...@gmail.com> wrote:
>> >Hi Lan,
>> >This is a good idea, but as pointed out by Andrew, there are multiple
>> >implementations composing the CoordinatorRuntime and perhaps will increase
>> >in future so this must be addressed.
>> >
>> >Additionally, there will be changes needed in the CoordinatorRuntime
>> >Builder interface itself, for example passing a buffer size config
>> supplier
>> >to fetch the specific config value passed by the coordinator
>> specialization
>> >as we should not refer to config name directly in the runtime.
>> >
>> >Thanks and regards,
>> >Sushant Mahajan
>> >
>> >On Wed, 20 Aug 2025, 20:05 Andrew Schofield, <
>> >andrew_schofield_j...@outlook.com> wrote:
>> >
>> >> Hi Lan,
>> >> Thanks for the KIP. Decoupling these is a good idea.
>> >>
>> >> AS1: This is a change to the coordinator runtime and there
>> >> are two concrete implementations, the group coordinator and
>> >> the share coordinator. I think it would be equally valuable
>> >> to have `share.coordinator.append.max.buffer.size`.
>> >> The same default of 1*1024*1024 seems appropriate to me.
>> >>
>> >> Thanks,
>> >> Andrew
>> >> ________________________________________
>> >> From: Lan Ding <isdin...@163.com>
>> >> Sent: 20 August 2025 14:57
>> >> To: dev@kafka.apache.org <dev@kafka.apache.org>
>> >> Subject: Re: [DISCUSS] KIP-1196: Introduce
>> >> group.coordinator.append.max.bytes config
>> >>
>> >> Hi Chia-Ping,
>> >>
>> >>
>> >>
>> >> Thanks for the feedback. The original configuration name was potentially
>> >> ambiguous;
>> >>
>> >> it has therefore been renamed to
>> >> `group.coordinator.append.max.buffer.size`.
>> >>
>> >>
>> >>
>> >> chia_00:
>> >>
>> >> Update the vaild values of group.coordinator.append.max.buffer.size to
>> be
>> >>
>> >> less than message.max.bytes.
>> >>
>> >>
>> >>
>> >> chia_01:
>> >>
>> >> Yes, I agree. Make this config to be updated dynamically.
>> >>
>> >>
>> >>
>> >> chia_02:
>> >>
>> >> I agree with you. Since the buffer behavior is a soft limit, this change
>> >> should
>> >>
>> >> not impact any existing functionality or require special compatibility
>> >> measures.
>> >>
>> >>
>> >>
>> >> Best,
>> >>
>> >> Lan
>> >>
>> >>
>> >>
>> >>
>> >> At 2025-08-18 00:54:15, "Chia-Ping Tsai" <chia7...@gmail.com> wrote:
>> >> >hi Lan
>> >> >
>> >> >chia_00:
>> >> >
>> >> >The `group.coordinator.append.max.bytes` must be smaller than the max
>> >> >message size; Otherwise, it could generate an invalid record
>> >> >
>> >> >chia_01:
>> >> >
>> >> >Since the max message size is a dynamic configuration, should we also
>> make
>> >> >`group.coordinator.append.max.bytes` dynamic?
>> >> >
>> >> >chia_02:
>> >> >
>> >> >Is compatibility necessary? The max buffer of the new coordinator is
>> >> >currently transparent to users, so I don't think there are any users
>> >> >explicitly setting the max message size to control the buffer used by
>> the
>> >> >new coordinator
>> >> >
>> >> >Best,
>> >> >Chia-Ping
>> >> >
>> >> >
>> >> >Lan Ding <isdin...@163.com> 於 2025年8月18日 週一 上午12:37寫道：
>> >> >
>> >> >> Hello everyone, I'd like to discuss a KIP regarding introducing a new
>> >> >> configuration,
>> >> >> group.coordinator.append.max.bytes Thank you! KIP link:
>> >> >> https://cwiki.apache.org/confluence/x/hA5JFg Best, Lan Ding
>> >>
>>
>>
>>
