Allen Wang created KAFKA-1829:
---------------------------------
Summary: Support adding new partitions on brokers without
reassignment of existing partitions
Key: KAFKA-1829
URL: https://issues.apache.org/jira/browse/KAFKA-1829
Project: Kafka
Issue Type: New Feature
Affects Versions: 0.8.1.1
Reporter: Allen Wang
One solution to deal with increasing incoming traffic to Kafka brokers is to
add new broker instances and new partitions at the same time. This is possible
if disk size is not the bottleneck. In this case, ideally the new partitions
would be assigned to new brokers without any changes to existing partitions.
The reason behind it is that moving existing partitions would cause more
traffic and hence more load.
The way to handle this now is a multi step process:
- add partitions
- generate replica assignment for the new partitions on the new brokers
- execute the assignment
This is complicated. Also, adding partitions and assigning replicas is not
atomic -- when partitions are added, they are already assigned using a default
assignment algorithm.
This feature would greatly help auto scaling Kafka to deal with increasing load.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
