Hello team, KIP-932 introduces share groups as a new consumption model that provides queue semantics. This directly addresses use cases where:
1. Multiple consumers need to process items efficiently in parallel from a single/multiple topic(s). 2. Messages need explicit acknowledgment/release (to avoid reprocessing or allow retries).Use cases where scaling Flink ML/LLM workload is critical - Shifting Kafka coordination and assignment logic to the broker side would simplify today’s complex Flink source management, making consumption more efficient, scalable, and far less error-prone. Operational Benefits - Higher Throughput: ShareGroupHeartbeat helps in Queue-like workloads, maximum throughput scenarios. Share groups distribute messages at the record level, not partition level, so multiple readers can consume from the same topic with Kafka coordinating message distribution. - Better Availability and Flexible Scaling: consumers assignment logic is simpler in server side and rebalancing frequency is minimised. Let's have discussion over the design and how the checkpointing will work when we use KafkaShareConsumer API from Kafka 4.1 . Regards, Shekhar Prasad Rajak, Blog | Github | Twitter
