spetz opened a new pull request, #4063: URL: https://github.com/apache/iggy/pull/4063
Every distinct consumer id stored one offset file and one map entry per partition with no upper bound, so a partition could accumulate durable state until the filesystem or memory gave out and the replica fenced itself. Offset deletes acknowledged locally could also free a primary slot while backups kept every older generation, and follower-served auto-commit cursors could be promoted into committed state across leadership changes. Add a per-partition, per-kind admission limit enforced on the primary before a new key enters consensus or touches disk. Admission counts durable keys plus in-flight reservations, so existing consumers keep updating at the limit while new keys get a typed TooManyConsumerOffsets reply. Replicated partitions now commit every offset store and delete through VSR regardless of the acknowledgement byte, and consumer-group cleanup uses replicated deletes, which keeps file counts identical on all replicas. State transfer exports committed values only, and recovery preserves state above a lowered limit with a warning. Auto-commit hands its reservation to the partition pump through an internal frame instead of a per-poll waiter, phantom cursors are reclaimed on promotion, and the reconciler no longer runs a full pass on every group offset advance. Missing consumer groups return the existing not-found codes. The limit ships as [partition] consumer_offsets_max with a default of 4096, and the new error code is mirrored in the Go, Java, and Node SDKs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
