Gokul Ramanan Subramanian created KAFKA-9590:
------------------------------------------------
Summary: Add read-only configuration to limit number of topic
partitions
Key: KAFKA-9590
URL: https://issues.apache.org/jira/browse/KAFKA-9590
Project: Kafka
Issue Type: Improvement
Components: admin
Reporter: Gokul Ramanan Subramanian
Currently, a Kafka cluster does not limit the number of topic partitions
despite the fact that on a given system, having more than a certain number of
topic partitions (without even any messages written / read) renders the cluster
unusable.
The ask here is to have a read-only Kafka configuration named something like
*max.topic.partitions* (an integer, which applies at the broker level and
defaults to INT.MAX) that the administrator the cluster can set to ensure
smooth operation.
Things to keep in mind:
# If a broker already has more than *max.topic.partitions* number of
partitions at the time when the *max.topic.partitions* configuration is first
applied on the broker, the broker should disallow any more topic partitions.
This allows for graceful reassignments and in-place upgrade to a Kafka version
that supports the *max.topic.partitions* configuration.
# Different brokers can technically have different *max.topic.partitions* in
case of a heterogenous cluster, where not all broker nodes operate with the
same system limits (such as CPU, memory etc.)
# When creating a topic partition, the controller has to identify brokers
which will replicate the topic partition. Currently, the controller distributes
the leadership of the topic partitions to ensure uniform load. This algorithm
will need to account for the *max.topic.partitions* configuration.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
