[
https://issues.apache.org/jira/browse/FLINK-38287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18072720#comment-18072720
]
Shekhar Prasad Rajak commented on FLINK-38287:
----------------------------------------------
Key APIs Flink needs to control for checkpoint-based guarantees:
ShareFetch -- fetch ACQUIRED records without auto-acknowledging
ShareAcknowledge with ACCEPT -- only on checkpoint completion
ShareAcknowledge with RELEASE -- on failure recovery
ShareAcknowledge with RENEW -- during checkpoint barrier alignment
AdminClient -- for partition discovery [currently using already ]
The KafkaShareConsumer client class is not suitable because:
No assign() (cannot control partition assignment)
No seek() (cannot rewind to checkpointed offset)
subscribe() involves broker-managed membership (non-deterministic)
IMPLICIT mode auto-ACCEPTs (data loss on crash)
Acquisition lock timeout can RELEASE records during checkpoint (state mismatch)
> Kafka 4.1.0 Queue Semantics support in Flink Connector Kafka
> ------------------------------------------------------------
>
> Key: FLINK-38287
> URL: https://issues.apache.org/jira/browse/FLINK-38287
> Project: Flink
> Issue Type: New Feature
> Components: API / Core
> Affects Versions: kafka-4.1.0
> Reporter: Shekhar Prasad Rajak
> Priority: Major
> Attachments: Screenshot 2025-09-02 at 10.59.05 AM.png, Screenshot
> 2025-09-02 at 10.59.55 AM.png
>
>
> KIP-932 introduces share groups as a new consumption model that provides
> queue semantics.
> This directly addresses *use cases* where:
> 1. Multiple consumers need to process items efficiently in parallel from a
> single/multiple topic(s).
> 2. Messages need explicit acknowledgment/release (to avoid reprocessing or
> allow retries).
> h4. *Operational Benefits*
> - {*}Higher Throughput{*}: ShareGroupHeartbeat helps in Queue-like
> workloads, maximum throughput scenarios. Share groups distribute messages at
> the record level, not partition level, so multiple readers can consume from
> the same topic with Kafka coordinating message distribution.
> - {*}Better Availability and Flexible Scaling{*}: consumers assignment
> logic is simpler in server side and rebalancing frequency is minimised.
> - *Reduce Operational overhead at consumer side:* Broker manages partition
> state via SharePartitionManager rather than client-side coordination.
> Eliminates the need for the complex rebalancing protocol that consumer groups
> required and reduced latency for consumer membership changes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
