Matthias J. Sax created KAFKA-13555:
---------------------------------------
Summary: Consider number if input topic partitions for task
assignment
Key: KAFKA-13555
URL: https://issues.apache.org/jira/browse/KAFKA-13555
Project: Kafka
Issue Type: Improvement
Components: streams
Reporter: Matthias J. Sax
StreamsAssignor tries to distribute tasks evenly across all instances/threads
of a Kafka Streams application. It knows about instances/thread (to give more
capacity to instances with more thread), and it distinguishes between stateless
and stateful tasks. We also try to not move state around but to use a sticky
assignment if possible. However, the assignment does not take the number of
input topic partitions into account.
For example, an upstream tasks could compute two joins, and thus has 3 input
partitions, while a downstream task compute a follow up aggregation with a
single input partitions (from the repartition topic). It could happen that one
thread gets the 3 input partition tasks assigned, while the other thread get
the single input partition tasks assigned resulting to an uneven partition
assignment across both threads.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
