Michael Jaschob created KAFKA-13244:
---------------------------------------
Summary: Control Which Brokers Host Partitions of Newly Created
Topics
Key: KAFKA-13244
URL: https://issues.apache.org/jira/browse/KAFKA-13244
Project: Kafka
Issue Type: New Feature
Components: admin
Reporter: Michael Jaschob
When new topics are created through the admin interface, the Kafka controller
creates the partition assignments for these topics according to the algorithm
defined in {{AdminUtils.assignReplicasToBrokers}}. All (alive) brokers in the
cluster become candidates for hosting partitions.
However, sometimes cluster administrators don't want certain brokers to host
partitions for newly created topics. This can be for a variety of reasons, e.g.:
* a broker may be on a host that is slated for retirement and about to be shut
down
* a broker may be acting as a canary of a new Kafka version and is being
tested with a controlled set of topic-partitions, and should be restricted from
hosting any other topics
* similarly, a broker may in use for administrative operations, such as
dumping log segments, and consequently may be unstable and ideally as
little-used as possible
One solution is for the cluster admins to not give out CREATE permissions to
clients, and instead provide a service/abstraction for creating new topics with
explicit partition assignments. Unfortunately, in a world with Kafka Streams
applications, this is too restrictive: Kafka Streams clients expect to use the
Kafka tooling to create topics when their topology changes, or when they need
to reset their apps.
Since there is currently there is no way to control which brokers are eligible
to be assigned newly created partitions, I am proposing a new configuration
parameter, {{create.topic.broker.filter.policy.class.name}} to allow cluster
administrators to provide a pluggable class to control whether a broker should
be allowed to host new partitions. The class implements a simple interface to
give a binary yes/no verdict on each broker:
{code:java}
boolean isAllowedToHostPartitions(Broker broker);
{code}
This follows a pattern similar to {{create.topic.policy.class.name}} and
{{alter.topic.policy.class.name}}, which also provide cluster-level control
over other aspects of topic creation.
I have a PR ready which I'll submit after creating this ticket. I'm not sure if
this feature idea needs a KIP - technically, it's a change to configuration,
but on the other hand it's a pretty small change that aligns closely to
existing functionality (create/alter topic policies). Please let me know if I
should follow the KIP process.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
