[
https://issues.apache.org/jira/browse/KAFKA-18793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Viktor Somogyi-Vass updated KAFKA-18793:
----------------------------------------
Description:
Apache Kafka is widely adopted for building distributed event streaming
platforms. As Kafka clusters grow in size and usage, a common challenge arises:
different teams or workloads often share the same physical Kafka cluster, which
can cause operational and organizational friction.
Currently, Kafka lacks native support for logical separation of tenants (e.g.,
applications, teams) within a shared cluster. This leads to several challenges:
* Operational Complexity: Administrators manually implement workarounds like
naming conventions for topics, quotas, and ACLs.
* Resource Isolation: Shared resources create a risk of noisy neighbor effects,
impacting performance and stability. Therefore it is less effective to host
multiple clusters for multiple teams.
* Management Overhead: Multi-tenant management often involves custom tools or
external systems.
This ticket proposes introducing Virtual Clusters (VCs) to Kafka, a mechanism
to logically separate workloads while sharing the same physical Kafka cluster.
VCs will allow teams to operate independently, improving resource isolation,
security, and management capabilities.
h3. Goals
* Introduce Virtual Clusters as a logical abstraction within a physical Kafka
cluster.
* Enable resource and metadata isolation between Virtual Clusters.
* Simplify multi-tenant management and improve operational efficiency.
* Allow administrators to define and manage VCs without significant overhead.
* Ensure backward compatibility and minimal disruption for existing deployments.
* Provide a transparent view of the cluster for the clients.
h3. Non-Goals
* Introducing hard physical isolation (e.g., dedicated hardware).
* Replacing or deprecating existing multi-tenant mechanisms like quotas and
ACLs.
was:
Apache Kafka is widely adopted for building distributed event streaming
platforms. As Kafka clusters grow in size and usage, a common challenge arises:
different teams or workloads often share the same physical Kafka cluster, which
can cause operational and organizational friction.
Currently, Kafka lacks native support for logical separation of tenants (e.g.,
applications, teams) within a shared cluster. This leads to several challenges:
* Operational Complexity: Administrators manually implement workarounds like
naming conventions for topics, quotas, and ACLs.
* Resource Isolation: Shared resources create a risk of noisy neighbor effects,
impacting performance and stability. Therefore it is less effective to host
multiple clusters for multiple teams.
* Management Overhead: Multi-tenant management often involves custom tools or
external systems.
This ticket proposes introducing Virtual Clusters (VCs) to Kafka, a mechanism
to logically separate workloads while sharing the same physical Kafka cluster.
VCs will allow teams to operate independently, improving resource isolation,
security, and management capabilities.
h3. Goals
* Introduce Virtual Clusters as a logical abstraction within a physical Kafka
cluster.
* Enable resource and metadata isolation between Virtual Clusters.
* Simplify multi-tenant management and improve operational efficiency.
* Allow administrators to define and manage VCs without significant overhead.
* Ensure backward compatibility and minimal disruption for existing deployments.
* Provide a transparent view of the cluster for the clients.
h3. Non-Goals
* Introducing hard physical isolation (e.g., dedicated hardware).
* Replacing or deprecating existing multi-tenant mechanisms like quotas and
ACLs.
> Virtual Clusters in Kafka
> -------------------------
>
> Key: KAFKA-18793
> URL: https://issues.apache.org/jira/browse/KAFKA-18793
> Project: Kafka
> Issue Type: New Feature
> Components: core
> Reporter: Viktor Somogyi-Vass
> Assignee: Viktor Somogyi-Vass
> Priority: Major
>
> Apache Kafka is widely adopted for building distributed event streaming
> platforms. As Kafka clusters grow in size and usage, a common challenge
> arises: different teams or workloads often share the same physical Kafka
> cluster, which can cause operational and organizational friction.
> Currently, Kafka lacks native support for logical separation of tenants
> (e.g., applications, teams) within a shared cluster. This leads to several
> challenges:
> * Operational Complexity: Administrators manually implement workarounds like
> naming conventions for topics, quotas, and ACLs.
> * Resource Isolation: Shared resources create a risk of noisy neighbor
> effects, impacting performance and stability. Therefore it is less effective
> to host multiple clusters for multiple teams.
> * Management Overhead: Multi-tenant management often involves custom tools or
> external systems.
> This ticket proposes introducing Virtual Clusters (VCs) to Kafka, a mechanism
> to logically separate workloads while sharing the same physical Kafka
> cluster. VCs will allow teams to operate independently, improving resource
> isolation, security, and management capabilities.
> h3. Goals
> * Introduce Virtual Clusters as a logical abstraction within a physical Kafka
> cluster.
> * Enable resource and metadata isolation between Virtual Clusters.
> * Simplify multi-tenant management and improve operational efficiency.
> * Allow administrators to define and manage VCs without significant overhead.
> * Ensure backward compatibility and minimal disruption for existing
> deployments.
> * Provide a transparent view of the cluster for the clients.
> h3. Non-Goals
> * Introducing hard physical isolation (e.g., dedicated hardware).
> * Replacing or deprecating existing multi-tenant mechanisms like quotas and
> ACLs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
