Hi Murali, I want to revisit the motivation. Once KAFKA-21228 is fixed, the timer will be the safety net for low-throughput partitions. So what is the purpose of the per-group count?
I think the answer is recovery time. A high-throughput partition can accumulate many updates within 5 minutes, and the only way to reduce the replay is the count. However, lowering the broker-level count affects all other partitions as well. That is why we need the group-level count. WDYT? If you agree, it would be great to update the motivation accordingly. Best, Chia-Ping On Fri, Nov 13, 2026 05:26 AM, Muralidhar Basani via dev < [email protected]> wrote: > Hi Chia, > > Thanks for the follow up. I have updated KIP-1349 according to our > discussion: > > - Switched back to a per-group count config > - Raised the broker-level upper bound to 1000 > - Fixed the inconsistency. group values above the broker limit are now > clamped at runtime > > Hi Sushant, reg metrics, since bytes-since-snapshot no longer fits, I > replaced it with snapshot-record-rate/total and update-record-rate/total. > > Please take a look at the updated KIP and let me know if any feedback. > > KIP - > https://cwiki.apache.org/confluence/spaces/KAFKA/pages/42195 > 8844/KIP-1349+Configurable+snapshot+frequency+for+share+groups > > Thanks, > Murali > > On Thu, Nov 12, 2026 at 9:21 AM Chia-Ping Tsai <[email protected]> wrote: > > > Hi Murali, > > > > Would you mind updating KIP-1349 according to our discussion, i.e. > > switching back to a per-group count config and raising the broker-level > > upper bound? > > > > Also, the KIP is inconsistent about how a group-level value above the > > broker-level limit is handled: Public Interfaces says it's rejected at > > alter time, while Proposed Changes says it's clamped. Since the broker > > config can change after the group config is set, I think we should clamp > > the value at runtime. > > > > Once the KIP is updated, it may be worth restarting the vote. > > > > Best, > > Chia-Ping > > > > On Thu, Oct 08, 2026 02:25 PM, Muralidhar Basani via dev < > > [email protected]> wrote: > > > > > Hi Chia, > > > Thanks for taking another look. From what I understand, pruning has to > > wait > > > for the slowest group on the state partition to snapshot before it can > > > delete old records. > > > > > > For ex: if groups A and B share a state partition: > > > - A snapshots frequently .. its latest snapshot is at offset 900. > > > - B has low traffic ... its latest snapshot is still at offset 100. > > > > > > So the prune point is min(900, 100) which is 100, so only offsets 0–99 > > can > > > be deleted. Everything A writes after that stays on disk until B > > snapshots > > > again. So B's threshold is controlling the size than A's snapshots, > > whether > > > it's in bytes or a count. > > > > > > I had moved the KIP to bytes after your earlier suggestion (chia_01), > > but I > > > agree the pruning behaviour makes bytes much less useful here than in a > > > buffering case like KIP-1325, where a byte limit directly bounds > memory. > > > > > > I think we had earlier calculations showed update records are a fairly > > > constant size, so a count gives nearly the same result. > > > > > > I shall revise the KIP back to a count based approach and follow up > here. > > > > > > Thanks, > > > Murali > > > > > > On Tue, Oct 6, 2026 at 8:52 AM Chia-Ping Tsai <[email protected]> > > wrote: > > > > > > > Hi, > > > > > > > > After re-reading the KIP, I'm not sure whether bytes is better than > > > count. > > > > Eventually the prune must wait for "all" share partitions on the > state > > > > partition to be snapshotted. That means users can hardly evaluate the > > > > retained log size from a per-partition threshold, no matter it is > count > > > or > > > > bytes. > > > > > > > > On the other side, the concept of "per-group" is good to me, since it > > > > allows the low-traffic groups to snapshot more often. > > > > > > > > So I'm wondering whether a per-group count config (plus raising the > > > broker > > > > upper bound) is enough for this KIP. > > > > > > > > Best, > > > > Chia-Ping > > > > > > > > On 2026-05-22 19:30 UTC Muralidhar Basani via dev wrote: > > > > > Hi all, > > > > > > > > > > I would like to start a discussion on KIP-1349, which allows > > > configurable > > > > > snapshot frequency of share groups. > > > > > > > > > > KIP : > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1349% > > > 3A+Configurable+snapshot+frequency+for+share+groups > > > > > > > > > > Thanks, > > > > > Murali > > > > > > > > > > > > > > >
