This is a super cool proposal. Thank you for it! 1) The KIP mentions VCs are allowed ALTER_CONFIG, but doesn't mention what configs will be allowed to get modified and what not. There are a few broker-level configs that shouldn't be modifiable by any single VC imo - https://github.com/apache/kafka/blob/766caaa551a3e7561cf0434e457ac4230435cb54/core/src/main/scala/kafka/server/DynamicBrokerConfig.scala#L86-L93. Kafka also supports changing log levels dynamically, would be weird if one VC could change it for all.
2) re: topic names - the UUID+name combo simply reads weird and is not very user friendly. My understanding is that tenants will see the topic name without the UUID, as the mapping will happen in the broker through the user (principal). If you don't use a principal (i.e no auth), you'd just access the general cluster. On the other hand, I agree with the reasoning of not using KIP-37 style hierarchy. Perhaps we could extend the topic describe command (and metadata responses that carry the info) to contain the tenant info? I just want to make sure we will offer first-class support in managing such clusters 3) metrics. Would be nice to have a few metrics that aggregate usage per VC. I imagine operators will need a way to monitor these things. That being said, I'm not sure how deep into detail you envision going in this single KIP - is the idea here to present it all from a high level, get consensus, and then perhaps split in a few more detailed KIPs? (a-la KIP-500) PS: This is a nit but it's worthwhile to get the terms right. Do we define a "tenant" as a "VC", or is a "tenant" a Principal that is connected to one VC (or more in the future)? Another nit I would suggest for better marketing/interest in the feature is to incorporate the word "multi-tenant" in the proposal. It's industry-standard, very SEO friendly and in general just explains things better. As a simple anecdote, I skipped on reading this proposal at first because I didn't realize it was a multi-tenant solution! In any case, I'm looking forward to seeing this proposal get in! I don't see downsides from having this feature, it's table-stakes for modern data infra in 2025 and as you mentioned - the high overhead cost of Kafka (3 nodes, 3 controllers) is overkill for a majority of use cases. If we want to see greater adoption of Kafka, we need to cater to the smaller fish too (and hope they grow to use more of the platform) As another point - vendors have already offered this in the market for many years (Confluent, RedPanda Serverless) Best, Stan On 2025/03/11 04:10:32 萨尔卡 wrote: > agreed with 3-node-case. and we can do more: both virtual clusters > and hard clusters. > virtual separate like mysql. different users can use different > database. different kafka users use different VC. > how about 10000 kafka instances? but kafka should do HARD isolation for > users to avoid influencing in each others. and keep controll risks UNDER > small-hard-clusters > > > let's just called it VC-Plus with hard isolation > > > 1. little resource use VC but can not ensure influencing > 2. little resource use VC-Plus, no need to do > 3. lots of resource can use VC but worse than VC-Plus > VC-Plus can avoid influencing in each others, keep controll > risks UNDER small-hard-clusters > > > > > 1. little resource, not suggest to use VC > > > > > Administrators would be able to put a cap on the quota of a virtual > > cluster, so they can prevent virtual-clusters from monopolizing > resources. > > This can be thought of as a certain kind of isolation but I'm not sure if > > you meant more than that? I'll add a revised version of the answer to > David > > to the KIP, but the bottom line for why don't we chose the hard kind of > > virtualization where we separate off whole brokers into a cluster of a > > cluster is that it's less efficient on resources too. As I imagine, in a > > hard virtualization case if a 3 node Kafka cluster is split into 2 > virtual > > clusters, we would need to add 3 more brokers if we wanted to satisfy the > > requirement of topics with a replication factor of 3. > > Also with quotas one would be able to control how much produce-, consume- > > and replication-traffic goes through every topic. Adding VC quotas on top > > ensures that resources aren't monopolized. > > > > Best, > > Viktor > > > > On Wed, Mar 5, 2025 at 3:20 AM 萨尔卡 <tigerwe...@qq.com.invalid> wrote: > > > > thanks for your answer. @vk > > in my&nbsp;experience, isolation-in-brokers to avoid influencing each > > other would be a point user cares in-use.&nbsp;
