[
https://issues.apache.org/jira/browse/FLINK-38287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shekhar Prasad Rajak updated FLINK-38287:
-----------------------------------------
Description:
KIP-932 introduces share groups as a new consumption model that provides queue
semantics.
This directly addresses *use cases* where:
1. Multiple consumers need to process items efficiently in parallel from a
single/multiple topic(s).
2. Messages need explicit acknowledgment/release (to avoid reprocessing or
allow retries).
h4. *Operational Benefits*
- {*}Higher Throughput{*}: ShareGroupHeartbeat helps in Queue-like workloads,
maximum throughput scenarios. Share groups distribute messages at the record
level, not partition level, so multiple readers can consume from the same topic
with Kafka coordinating message distribution.
- {*}Better Availability and Flexible Scaling{*}: consumers assignment logic
is simpler in server side and rebalancing frequency is minimised.
- *Reduce Operational overhead at consumer side:* Broker manages partition
state via SharePartitionManager rather than client-side coordination.
Eliminates the need for the complex rebalancing protocol that consumer groups
required and reduced latency for consumer membership changes.
was:
KIP-932 introduces share groups as a new consumption model that provides queue
semantics.
This directly addresses *use cases* where:
1. Multiple consumers need to process items efficiently in parallel from a
single/multiple topic(s).
2. Messages need explicit acknowledgment/release (to avoid reprocessing or
allow retries).
h4. *Operational Benefits*
- {*}Higher Throughput{*}: ShareGroupHeartbeat helps in Queue-like workloads,
maximum throughput scenarios. Share groups distribute messages at the record
level, not partition level, so multiple readers can consume from the same topic
with Kafka coordinating message distribution.
- {*}Better Availability and Flexible Scaling{*}: consumers assignment logic
is simpler in server side and rebalancing frequency is minimised.
> Kafka 4.1.0 Queue Semantics support in Flink Connector Kafka
> ------------------------------------------------------------
>
> Key: FLINK-38287
> URL: https://issues.apache.org/jira/browse/FLINK-38287
> Project: Flink
> Issue Type: New Feature
> Components: API / Core
> Affects Versions: kafka-4.1.0, flink-2.0.0
> Reporter: Shekhar Prasad Rajak
> Priority: Major
> Attachments: Screenshot 2025-09-02 at 10.59.05 AM.png, Screenshot
> 2025-09-02 at 10.59.55 AM.png
>
>
> KIP-932 introduces share groups as a new consumption model that provides
> queue semantics.
> This directly addresses *use cases* where:
> 1. Multiple consumers need to process items efficiently in parallel from a
> single/multiple topic(s).
> 2. Messages need explicit acknowledgment/release (to avoid reprocessing or
> allow retries).
> h4. *Operational Benefits*
> - {*}Higher Throughput{*}: ShareGroupHeartbeat helps in Queue-like
> workloads, maximum throughput scenarios. Share groups distribute messages at
> the record level, not partition level, so multiple readers can consume from
> the same topic with Kafka coordinating message distribution.
> - {*}Better Availability and Flexible Scaling{*}: consumers assignment
> logic is simpler in server side and rebalancing frequency is minimised.
> - *Reduce Operational overhead at consumer side:* Broker manages partition
> state via SharePartitionManager rather than client-side coordination.
> Eliminates the need for the complex rebalancing protocol that consumer groups
> required and reduced latency for consumer membership changes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
