Viktor Somogyi-Vass created KAFKA-18793:
-------------------------------------------
Summary: Virtual Clusters in Kafka
Key: KAFKA-18793
URL: https://issues.apache.org/jira/browse/KAFKA-18793
Project: Kafka
Issue Type: New Feature
Components: core
Reporter: Viktor Somogyi-Vass
Assignee: Viktor Somogyi-Vass
Apache Kafka is widely adopted for building distributed event streaming
platforms. As Kafka clusters grow in size and usage, a common challenge arises:
different teams or workloads often share the same physical Kafka cluster, which
can cause operational and organizational friction.
Currently, Kafka lacks native support for logical separation of tenants (e.g.,
applications, teams) within a shared cluster. This leads to several challenges:
* Operational Complexity: Administrators manually implement workarounds like
naming conventions for topics, quotas, and ACLs.
* Resource Isolation: Shared resources create a risk of noisy neighbor effects,
impacting performance and stability. Therefore it is less effective to host
multiple clusters for multiple teams.
* Management Overhead: Multi-tenant management often involves custom tools or
external systems.
This ticket proposes introducing Virtual Clusters (VCs) to Kafka, a mechanism
to logically separate workloads while sharing the same physical Kafka cluster.
VCs will allow teams to operate independently, improving resource isolation,
security, and management capabilities.
h3. Goals
* Introduce Virtual Clusters as a logical abstraction within a physical Kafka
cluster.
* Enable resource and metadata isolation between Virtual Clusters.
* Simplify multi-tenant management and improve operational efficiency.
* Allow administrators to define and manage VCs without significant overhead.
* Ensure backward compatibility and minimal disruption for existing deployments.
* Provide a transparent view of the cluster for the clients.
h3. Non-Goals
* Introducing hard physical isolation (e.g., dedicated hardware).
* Replacing or deprecating existing multi-tenant mechanisms like quotas and
ACLs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
