Shekhar Prasad Rajak created SPARK-54321:
--------------------------------------------
Summary: Implementing Kafka 4.x Queue Semantics for Spark
Streaming - Kafka Source
Key: SPARK-54321
URL: https://issues.apache.org/jira/browse/SPARK-54321
Project: Spark
Issue Type: New Feature
Components: Connect
Affects Versions: 4.0.0
Reporter: Shekhar Prasad Rajak
I am initiating a discussion on adding support for Kafka 4.x queue semantics
within Spark Streaming when using Kafka as a source. Kafka 4 introduces
queue-like semantics with share groups, enabling multiple consumers to process
records in parallel from a single partition with per-message acknowledgments
and retries. This complements traditional Kafka consumer groups by offering
more flexible and scalable event processing models, ideal for high-throughput,
real-time data workloads.
Looking forward to your thoughts and engagement.
Dev thread with subject: "Discussion: Implementing Kafka 4.x Queue Semantics
for Spark Streaming - Kafka Source"
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
