[
https://issues.apache.org/jira/browse/KAFKA-9281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sophie Blee-Goldman resolved KAFKA-9281.
----------------------------------------
Resolution: Duplicate
> Consider more flexible node grouping for Pattern subscription
> -------------------------------------------------------------
>
> Key: KAFKA-9281
> URL: https://issues.apache.org/jira/browse/KAFKA-9281
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Reporter: Sophie Blee-Goldman
> Priority: Major
>
> Input topics subscribed to using pattern subscription will currently all be
> grouped into the same node group, meaning the number of tasks is determined
> by the maximum partition count of any matching topic. This means less
> overhead per partition and is suitable for some scenarios, but it limits the
> ability to scale out by preventing further parallelization that is possible
> with independent partitions. We should consider making it possible for
> pattern subscription to create a task for every partition summed across all
> matching topics.
> We don't necessarily want to change the default (current) behavior, but we
> could make this more flexible either by autoscaling based on some heuristic,
> or making it customizable by the user. One possibility would be to augment
> the Pattern based source KStream method with an optional parameter that to
> tell Streams how to generate tasks for that pattern, for example
> {code:java}
> public synchronized KStream stream(pattern, numTasks);
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
