[
https://issues.apache.org/jira/browse/SPARK-54321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18047752#comment-18047752
]
Shekhar Prasad Rajak commented on SPARK-54321:
----------------------------------------------
code changes : https://github.com/apache/spark/pull/53626
> Implementing Kafka 4.x Queue Semantics for Spark Streaming - Kafka Source
> -------------------------------------------------------------------------
>
> Key: SPARK-54321
> URL: https://issues.apache.org/jira/browse/SPARK-54321
> Project: Spark
> Issue Type: New Feature
> Components: Connect
> Affects Versions: 4.2.0
> Reporter: Shekhar Prasad Rajak
> Priority: Major
> Labels: pull-request-available
>
> I am initiating a discussion on adding support for Kafka 4.x queue semantics
> within Spark Streaming when using Kafka as a source. Kafka 4 introduces
> queue-like semantics with share groups, enabling multiple consumers to
> process records in parallel from a single partition with per-message
> acknowledgments and retries. This complements traditional Kafka consumer
> groups by offering more flexible and scalable event processing models, ideal
> for high-throughput, real-time data workloads.
> Looking forward to your thoughts and engagement.
> Dev thread with subject: "Discussion: Implementing Kafka 4.x Queue Semantics
> for Spark Streaming - Kafka Source"
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
