Hi Lan, thanks for the KIP!
sq1: Can we make a distinction between the broker-level `message.max.bytes`
and the topic-level `max.message.bytes`? When `__consumer_offsets` and
`__share_group_state` are created, their `max.message.bytes` comes from the
broker-level `message.max.bytes`. The topic-level `max.message.bytes` can
be changed at any time by an admin afterwards. It is the topic-level config
that determines the buffer size right now.
sq2: The valid values for the new configs should be `<=` the topic-level
`max.message.bytes`. I don't think we can stop an admin from reconfiguring
`max.message.bytes` lower than
`{group,share}.coordinator.append.max.buffer.size` though.
sq3: Is the motivation to decouple the buffer size from the broker-level
`message.max.bytes` (intended for applications) or from the topic-level
`max.message.bytes`? If it's the former, we could consider an alternative
where we define `offsets.topic.max.message.bytes` and
`share.coordinator.state.topic.max.message.bytes` instead. We already have
`{offsets,share.coordinator.state}.topic.{num.partitions,replication.factor,segment.bytes}`
to override the default topic creation settings. This would address the
memory issue because the internal Kafka topics will have good defaults.
Thanks,
Sean
On Thu, Aug 21, 2025 at 4:05 AM Lan Ding <isdin...@163.com> wrote:
> Hi Andrew and Sushant Mahajan,
>
> Thank you for the review and suggestions.
>
>
>
>
> AS1: You're absolutely right. I've now added the
> share.coordinator.append.max.buffer.size
>
> config with the same default value of 1*1024*1024 as you suggested.
>
>
>
>
> Sushant, thank you for highlighting the necessary changes to the
> CoordinatorRuntime.
>
> I've updated the implementation accordingly.
>
>
>
>
> Best,
>
> Lan
>
> At 2025-08-21 03:26:02, "Sushant Mahajan" <sushmah...@gmail.com> wrote:
> >Hi Lan,
> >This is a good idea, but as pointed out by Andrew, there are multiple
> >implementations composing the CoordinatorRuntime and perhaps will increase
> >in future so this must be addressed.
> >
> >Additionally, there will be changes needed in the CoordinatorRuntime
> >Builder interface itself, for example passing a buffer size config
> supplier
> >to fetch the specific config value passed by the coordinator
> specialization
> >as we should not refer to config name directly in the runtime.
> >
> >Thanks and regards,
> >Sushant Mahajan
> >
> >On Wed, 20 Aug 2025, 20:05 Andrew Schofield, <
> >andrew_schofield_j...@outlook.com> wrote:
> >
> >> Hi Lan,
> >> Thanks for the KIP. Decoupling these is a good idea.
> >>
> >> AS1: This is a change to the coordinator runtime and there
> >> are two concrete implementations, the group coordinator and
> >> the share coordinator. I think it would be equally valuable
> >> to have `share.coordinator.append.max.buffer.size`.
> >> The same default of 1*1024*1024 seems appropriate to me.
> >>
> >> Thanks,
> >> Andrew
> >> ________________________________________
> >> From: Lan Ding <isdin...@163.com>
> >> Sent: 20 August 2025 14:57
> >> To: dev@kafka.apache.org <dev@kafka.apache.org>
> >> Subject: Re: [DISCUSS] KIP-1196: Introduce
> >> group.coordinator.append.max.bytes config
> >>
> >> Hi Chia-Ping,
> >>
> >>
> >>
> >> Thanks for the feedback. The original configuration name was potentially
> >> ambiguous;
> >>
> >> it has therefore been renamed to
> >> `group.coordinator.append.max.buffer.size`.
> >>
> >>
> >>
> >> chia_00:
> >>
> >> Update the vaild values of group.coordinator.append.max.buffer.size to
> be
> >>
> >> less than message.max.bytes.
> >>
> >>
> >>
> >> chia_01:
> >>
> >> Yes, I agree. Make this config to be updated dynamically.
> >>
> >>
> >>
> >> chia_02:
> >>
> >> I agree with you. Since the buffer behavior is a soft limit, this change
> >> should
> >>
> >> not impact any existing functionality or require special compatibility
> >> measures.
> >>
> >>
> >>
> >> Best,
> >>
> >> Lan
> >>
> >>
> >>
> >>
> >> At 2025-08-18 00:54:15, "Chia-Ping Tsai" <chia7...@gmail.com> wrote:
> >> >hi Lan
> >> >
> >> >chia_00:
> >> >
> >> >The `group.coordinator.append.max.bytes` must be smaller than the max
> >> >message size; Otherwise, it could generate an invalid record
> >> >
> >> >chia_01:
> >> >
> >> >Since the max message size is a dynamic configuration, should we also
> make
> >> >`group.coordinator.append.max.bytes` dynamic?
> >> >
> >> >chia_02:
> >> >
> >> >Is compatibility necessary? The max buffer of the new coordinator is
> >> >currently transparent to users, so I don't think there are any users
> >> >explicitly setting the max message size to control the buffer used by
> the
> >> >new coordinator
> >> >
> >> >Best,
> >> >Chia-Ping
> >> >
> >> >
> >> >Lan Ding <isdin...@163.com> 於 2025年8月18日 週一 上午12:37寫道：
> >> >
> >> >> Hello everyone, I'd like to discuss a KIP regarding introducing a new
> >> >> configuration,
> >> >> group.coordinator.append.max.bytes Thank you! KIP link:
> >> >> https://cwiki.apache.org/confluence/x/hA5JFg Best, Lan Ding
> >>
>
>
>
