Hi David,

Thanks for reading it through in advance. So I think there are 3 options
for resource isolation today, let me go through them for context. The
4th is how virtual clusters help the 3rd option :).
1. We can create completely separate Kafka clusters and host them
separately without any other software running on those hosts. In this
scenario you are completely in control of the hardware (cpu, network, disk
utilization). This however may be the most expensive solution too because
hosting your own servers can be expensive for many users. Also you're
likely left with unused capacity.
2. Then if we think of more economic solutions, we can choose to host Kafka
clusters together in a virtualized environment. There you should be able to
control the hardware resources but from an operation point of view you'll
have a harder time as you'll need to manage more Kafka clusters (need to
set up ACLs, quotas and configuration and keep them in sync if needed).
Also, if there is a need for sharing data between these, it's not easy
because then you'll need to set up replication between those two clusters.
3.To overcome the cluster management overhead, it's better to host one big
Kafka cluster, and this is what larger organizations tend to do. Then a
central team manages the installation (if self-hosted and they're not using
a cloud service) and provides Kafka as a service for other teams. This
often isn't perfect either, because they need to know about every new
client or user to set proper network quotas (besides the catch-all default).
4. With virtual clusters we improve this situation by quotas for virtual
clusters. You would set virtual cluster specific quotas that wouldn't be
possible to exceed for all the clients running together on that VC. Besides
this, virtual cluster admins could be defined who may have better knowledge
about their cluster and their team and set quotas in the clusters
appropriately.

Viktor

On Fri, Feb 14, 2025 at 12:50 AM David Arthur <mum...@gmail.com> wrote:

> Thanks for the KIP, Viktor (et al)!
>
> This certainly looks like an interesting feature. I had one initial
> question upon my first cursory readthrough: how does a virtual cluster help
> with isolation? I see this mentioned as a motivation, but it's not
> immediately obvious to me how it changes resource isolation.
>
> I'll do a more thorough review and come back with more questions :)
>
> -David A
>
>
> On Thu, Feb 13, 2025 at 12:11 PM Viktor Somogyi-Vass
> <viktor.somo...@cloudera.com.invalid> wrote:
>
> > Hi all,
> >
> > I'd like to start a discussion on our new feature proposal: virtual
> > clusters in Kafka. In this KIP we propose the leveling up of
> multi-tenancy
> > to better support organizations who manage their deployments centrally
> for
> > multiple teams.
> > Please take a look if you're interested and I'm happy to take any
> feedback
> > on it.
> >
> > The link to the KIP:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1134%3A+Virtual+Clusters+in+Kafka
> >
> > Thanks,
> > Viktor
> >
>
>
> --
> David Arthur
>

Reply via email to