Xu xiaolong created FLINK-22837:
-----------------------------------
Summary: Assign-evenly kafkaTopicPartitions of multiple topics to
flinkKafkaConsumer subtask
Key: FLINK-22837
URL: https://issues.apache.org/jira/browse/FLINK-22837
Project: Flink
Issue Type: Improvement
Components: Connectors / Kafka
Affects Versions: 1.11.0
Reporter: Xu xiaolong
Now ,with flink1.11 kafka connecotr,when we consume multiple kafka topics by
one flinkkafkaconsumer , when we set the consumer parallelism equals with the
total partitions count of multiple topic , make a decision each topic partition
consume by one kafka consumer, so each topic partition count is less than the
subtask count. But,the problem is that currently, some subtask is total free
while someothers workload is very high, this problem is caused by that the
partitionAssigner assign partion of earch topic indepently currently.
Following is one example:
Target topics: topi1, topic2 ,topic3 ,topic4. each has 3 partitions.
In our job we consume the 4 topic by one consumer , our flink standalone
cluster got 9 taskworkers on different nodes. we want balance the workload as
much as possible, so we
set the paralelism of flinkkafkaconsumer to 12. from the UI we notice that the
0-5 subtask is free without partition assigned, the total 12 partiton is
assigned to 6-11 subtask. We learned the source code of
KafkaTopicPartitionAssigner and extend one more partition assign strategy that
can deal with the need we describe up, this stategy can evenly assign partiton
from multiple topic grobally to subtask of consumer.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
