[
https://issues.apache.org/jira/browse/KAFKA-20934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18105385#comment-18105385
]
Matthias J. Sax commented on KAFKA-20934:
-----------------------------------------
Thanks for filing this ticket. Overall, yes, this could make sense, and sound
like a natural next step. However, as you already pointed out, it won't work
for stateful applications... And I am not sure if it's worth doing for
stateless apps at this point? For stateful application, if QfK would get the
ability to preserve per-key-order, it would become more interesting for KS to
support share groups.
Another aspect is of course, that we are moving towards "streams" groups atm,
so we would need some new hybrid group type to support this?
Overall, this would be a massively big project, so I would only consider it, if
we get very strong demand for it. But it's good to have a ticket wish helps to
capture demand.
> Considering Share Groups for decoupling stateless Kafka Streams processing
> parallelism from source partitions
> -------------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-20934
> URL: https://issues.apache.org/jira/browse/KAFKA-20934
> Project: Kafka
> Issue Type: Improvement
> Reporter: sanghyeok An
> Assignee: sanghyeok An
> Priority: Minor
> Labels: needs-kip, streams
>
> I am creating this Jira ticket for ideation. If Kafka maintainers or the
> community think this direction is worth discussing, I would be happy to write
> a KIP and develop the discussion further.
> Currently, Kafka Streams processing parallelism is closely tied to the number
> of partitions in the source topic. In contrast, Share Groups allow multiple
> consumers to share the same partition, and the number of consumers can exceed
> the number of partitions. Therefore, using Share Groups may provide a way to
> decouple processing parallelism from the number of source partitions.
> Applying this model to existing stateful Kafka Streams topologies does not
> appear to be straightforward. Stateful processing in Kafka Streams is based
> on a model in which a task owns specific input partitions and local state
> stores. In addition, with Share Groups, records from the same partition may
> be processed by different consumers, and partition-level ordering is not
> guaranteed overall.
> However, there may be room to use Share Groups for stateless Kafka Streams
> topologies where record processing is order-independent. In such topologies,
> processing records from the same source partition across multiple Streams
> instances would not introduce conflicts in terms of state ownership,
> potentially allowing the number of processing instances to exceed the number
> of source partitions.
> Although proper performance evaluation would be necessary, this could
> potentially improve throughput for stateless topologies where
> application-side processing is the bottleneck. It could also reduce the need
> to over-partition topics solely to achieve higher processing parallelism,
> which may in turn reduce the operational overhead associated with maintaining
> a large number of partitions. KIP-932 also describes over-partitioning for
> parallel consumption as one of the problems that Share Groups are intended to
> address.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
