Shekharrajak opened a new pull request, #189: URL: https://github.com/apache/flink-connector-kafka/pull/189
Ref https://issues.apache.org/jira/browse/FLINK-38287 This implementation adds Kafka 4.x share group semantics to Flink's Kafka connector while maintaining full backward compatibility with existing code. The code changes are following KIP-932 and FLIP-27 main source architecture and implicit mode acknowledgement. This directly addresses use cases where: 1. Multiple consumers need to process items efficiently in parallel from a single/multiple topic(s). 2. Messages need explicit acknowledgment/release (to avoid reprocessing or allow retries). Use cases where scaling Flink ML/LLM workload is critical - Shifting Kafka coordination and assignment logic to the broker side would simplify today’s complex Flink source management, making consumption more efficient, scalable, and far less error-prone. Operational Benefits - Higher Throughput: ShareGroupHeartbeat helps in Queue-like workloads, maximum throughput scenarios. Share groups distribute messages at the record level, not partition level, so multiple readers can consume from the same topic with Kafka coordinating message distribution. - Better Availability and Flexible Scaling: consumers assignment logic is simpler in server side and rebalancing frequency is minimised. Let's have discussion over the design and how the checkpointing will work when we use KafkaShareConsumer API from Kafka 4.1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
