Dong Lin created KAFKA-5335:
-------------------------------
Summary: Controller should batch updatePartitionReassignmentData()
operation
Key: KAFKA-5335
URL: https://issues.apache.org/jira/browse/KAFKA-5335
Project: Kafka
Issue Type: Bug
Reporter: Dong Lin
Assignee: Dong Lin
Currently controller will update partition reassignment data every time a
partition in the reassignment is completed. It means that if user specifies a
huge reassignment znode of size 1 MB to move 10K partitions, controller will
need to write roughly 0.5 MB * 10000 = 5 GB data to zookeeper in order to
complete this reassignment. This is because controller needs to write the
remaining partitions to the znode every time a partition is completely moved.
This is problematic because such a huge reassignment may greatly slow down
Kafka controller. Note that partition reassignment doesn't necessarily cause
data movement between brokers because we may use it only to recorder the
replica list of partitions to evenly distribute preferred leader.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
