Hi Murali,

I want to revisit the motivation. Once KAFKA-21228 is fixed, the timer will
be the safety net for low-throughput partitions. So what is the purpose of
the per-group count?

I think the answer is recovery time. A high-throughput partition can
accumulate many updates within 5 minutes, and the only way to reduce the
replay is the count. However, lowering the broker-level count affects all
other partitions as well. That is why we need the group-level count.

WDYT? If you agree, it would be great to update the motivation accordingly.

Best,
Chia-Ping

On Fri, Nov 13, 2026 05:26 AM, Muralidhar Basani via dev <
[email protected]> wrote:

> Hi Chia,
>
> Thanks for the follow up. I have updated KIP-1349 according to our
> discussion:
>
>  - Switched back to a per-group count config
>  - Raised the broker-level upper bound to 1000
>  - Fixed the inconsistency. group values above the broker limit are now
> clamped at runtime
>
> Hi Sushant, reg metrics, since bytes-since-snapshot no longer fits, I
> replaced it with snapshot-record-rate/total and update-record-rate/total.
>
> Please take a look at the updated KIP and let me know if any feedback.
>
> KIP -
> https://cwiki.apache.org/confluence/spaces/KAFKA/pages/42195
> 8844/KIP-1349+Configurable+snapshot+frequency+for+share+groups
>
> Thanks,
> Murali
>
> On Thu, Nov 12, 2026 at 9:21 AM Chia-Ping Tsai <[email protected]> wrote:
>
> > Hi Murali,
> >
> > Would you mind updating KIP-1349 according to our discussion, i.e.
> > switching back to a per-group count config and raising the broker-level
> > upper bound?
> >
> > Also, the KIP is inconsistent about how a group-level value above the
> > broker-level limit is handled: Public Interfaces says it's rejected at
> > alter time, while Proposed Changes says it's clamped. Since the broker
> > config can change after the group config is set, I think we should clamp
> > the value at runtime.
> >
> > Once the KIP is updated, it may be worth restarting the vote.
> >
> > Best,
> > Chia-Ping
> >
> > On Thu, Oct 08, 2026 02:25 PM, Muralidhar Basani via dev <
> > [email protected]> wrote:
> >
> > > Hi Chia,
> > > Thanks for taking another look. From what I understand, pruning has to
> > wait
> > > for the slowest group on the state partition to snapshot before it can
> > > delete old records.
> > >
> > > For ex: if groups A and B share a state partition:
> > > - A snapshots frequently .. its latest snapshot is at offset 900.
> > > - B has low traffic ... its latest snapshot is still at offset 100.
> > >
> > > So the prune point is min(900, 100) which is 100, so only offsets 0–99
> > can
> > > be deleted. Everything A writes after that stays on disk until B
> > snapshots
> > > again. So B's threshold is controlling the size than A's snapshots,
> > whether
> > > it's in bytes or a count.
> > >
> > > I had moved the KIP to bytes after your earlier suggestion (chia_01),
> > but I
> > > agree the pruning behaviour makes bytes much less useful here than in a
> > > buffering case like KIP-1325, where a byte limit directly bounds
> memory.
> > >
> > > I think we had earlier calculations showed update records are a fairly
> > > constant size, so a count gives nearly the same result.
> > >
> > > I shall revise the KIP back to a count based approach and follow up
> here.
> > >
> > > Thanks,
> > > Murali
> > >
> > > On Tue, Oct 6, 2026 at 8:52 AM Chia-Ping Tsai <[email protected]>
> > wrote:
> > >
> > > > Hi,
> > > >
> > > > After re-reading the KIP, I'm not sure whether bytes is better than
> > > count.
> > > > Eventually the prune must wait for "all" share partitions on the
> state
> > > > partition to be snapshotted. That means users can hardly evaluate the
> > > > retained log size from a per-partition threshold, no matter it is
> count
> > > or
> > > > bytes.
> > > >
> > > > On the other side, the concept of "per-group" is good to me, since it
> > > > allows the low-traffic groups to snapshot more often.
> > > >
> > > > So I'm wondering whether a per-group count config (plus raising the
> > > broker
> > > > upper bound) is enough for this KIP.
> > > >
> > > > Best,
> > > > Chia-Ping
> > > >
> > > > On 2026-05-22 19:30 UTC Muralidhar Basani via dev wrote:
> > > > > Hi all,
> > > > >
> > > > > I would like to start a discussion on KIP-1349, which allows
> > > configurable
> > > > > snapshot frequency of share groups.
> > > > >
> > > > > KIP :
> > > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1349%
> > > 3A+Configurable+snapshot+frequency+for+share+groups
> > > > >
> > > > > Thanks,
> > > > > Murali
> > > > >
> > > >
> > >
> >
>

Reply via email to