[
https://issues.apache.org/jira/browse/KAFKA-20934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18110092#comment-18110092
]
sanghyeok An commented on KAFKA-20934:
--------------------------------------
[~mjsax]
Thanks for checking this! I agree with you.
Given that Share Groups can only support specific nodes in this scenario, the
scope of this work seems too large to justify. As you mentioned, introducing a
new hybrid group type would probably be the biggest challenge.
Another potential challenge that comes to mind is maintaining consistency
across multiple local state stores distributed across different nodes for the
same partition, as well as the changelog on the broker.
> Considering Share Groups for decoupling stateless Kafka Streams processing
> parallelism from source partitions
> -------------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-20934
> URL: https://issues.apache.org/jira/browse/KAFKA-20934
> Project: Kafka
> Issue Type: Improvement
> Components: group-coordinator, streams
> Reporter: sanghyeok An
> Assignee: sanghyeok An
> Priority: Minor
> Labels: needs-kip
>
> I am creating this Jira ticket for ideation. If Kafka maintainers or the
> community think this direction is worth discussing, I would be happy to write
> a KIP and develop the discussion further.
> Currently, Kafka Streams processing parallelism is closely tied to the number
> of partitions in the source topic. In contrast, Share Groups allow multiple
> consumers to share the same partition, and the number of consumers can exceed
> the number of partitions. Therefore, using Share Groups may provide a way to
> decouple processing parallelism from the number of source partitions.
> Applying this model to existing stateful Kafka Streams topologies does not
> appear to be straightforward. Stateful processing in Kafka Streams is based
> on a model in which a task owns specific input partitions and local state
> stores. In addition, with Share Groups, records from the same partition may
> be processed by different consumers, and partition-level ordering is not
> guaranteed overall.
> However, there may be room to use Share Groups for stateless Kafka Streams
> topologies where record processing is order-independent. In such topologies,
> processing records from the same source partition across multiple Streams
> instances would not introduce conflicts in terms of state ownership,
> potentially allowing the number of processing instances to exceed the number
> of source partitions.
> Although proper performance evaluation would be necessary, this could
> potentially improve throughput for stateless topologies where
> application-side processing is the bottleneck. It could also reduce the need
> to over-partition topics solely to achieve higher processing parallelism,
> which may in turn reduce the operational overhead associated with maintaining
> a large number of partitions. KIP-932 also describes over-partitioning for
> parallel consumption as one of the problems that Share Groups are intended to
> address.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
