[
https://issues.apache.org/jira/browse/KAFKA-9173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16987696#comment-16987696
]
Oleg Muravskiy commented on KAFKA-9173:
---------------------------------------
[~ableegoldman] yes, I subscribe with a pattern that matches 21 topics, each
with 10 partitions.
[~guozhang] Well, I'm citing from the link you just posted above:
{quote}Slightly simplified, the *maximum parallelism* at which your application
may run is bounded by the maximum number of stream tasks, which itself is
determined by maximum number of partitions of the input topic(s) the
application is reading from.
{quote}
So in my case the "maximum number of partitions of the input topic(s)" is 210.
Yet I only get 10 tasks.
I do have just one sub-topology, though:
{{ Sub-topology: 0}}
{{ Source: SOURCE (topics: raw-rrc\d\d)}}
{{ --> PROCESSOR_INGEST}}
{{ Processor: PROCESSOR_INGEST (stores: [BGP-State-Store])}}
{{ --> OUTPUT_ERROR, OUTPUT_MRT}}
{{ <-- SOURCE}}
{{ Sink: OUTPUT_ERROR (topic: error)}}
{{ <-- PROCESSOR_INGEST}}
{{ Sink: OUTPUT_MRT (extractor class:
net.ripe.gii.ris.exabgp.ingestion.IngestExaBgpMessages$$Lambda$29/504582810@27912e3)}}
{{ <-- PROCESSOR_INGEST}}
> StreamsPartitionAssignor assigns partitions to only one worker
> --------------------------------------------------------------
>
> Key: KAFKA-9173
> URL: https://issues.apache.org/jira/browse/KAFKA-9173
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Affects Versions: 2.3.0, 2.2.1
> Reporter: Oleg Muravskiy
> Priority: Major
> Labels: user-experience
> Attachments: StreamsPartitionAssignor.log
>
>
> I'm running a distributed KafkaStreams application on 10 worker nodes,
> subscribed to 21 topics with 10 partitions in each. I'm only using a
> Processor interface, and a persistent state store.
> However, only one worker gets assigned partitions, all other workers get
> nothing. Restarting the application, or cleaning local state stores does not
> help. StreamsPartitionAssignor migrates to other nodes, and eventually picks
> up other node to assign partitions to, but still only one node.
> It's difficult to figure out where to look for the signs of problems, I'm
> attaching the log messages from the StreamsPartitionAssignor. Let me know
> what else I could provide to help resolve this.
> [^StreamsPartitionAssignor.log]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
