Hey Daniel and Viktor, Thanks so much for your answers!
1. > The issue we encountered here is how can clients inside a VC can > refer to topics linked in nested VCs? An admin of a VC needs a way to refer > to the resources inside the VC and the child VCs. I would think the only operations referring to resources in multiple VCs at the same time would be in the actual VC operations themselves (Create/Alter/Describe). And since those are new, you can add the necessary context to distinguish between resources. Can you provide an example of a situation where a client in a VC needs to refer to resources in a child VC using the existing APIs? For an example of the way I understood recursive VCs, an AdminClient within a VC would be able to create topics only in its VC, independent of the topics already existing in parent or child VCs. In order for a VC admin to create a topic within a child VC, it would need to first create a user with the appropriate permissions within the child VC. 2a. > I personally think that having some "pointers" > in the name helps a lot in operations. I agree that the tooling should present as much useful information to admins as possible. However, I think we can do this without packing all of the debug information we can into the one free text field we have, and one that necessarily needs to be unique. I imagine that we could opportunistically show context where appropriate, like "topic UUID (topic-a in vc0, topic-b in vc-1)". 2b. > If we don't have topic names, it > means we need to change the protocol (AFAIU, topics without names are not a > thing as of now), and the log directory structure. This is fair. I think having "nameless topics" actually having names that are their UUIDs for backwards compatibility is a reasonable strategy. 3. I don't think I understand your discussion here. Topic directories would certainly require broad protocol changes (which is probably why it hasn't happened yet) and would need to conform to whatever gets implemented in this proposal. It sounds like some of the complication you're mentioning comes from the physical topic names and physical location on disk; that's sorta why I brought up (2). If we transform the log layer to store things by UUID rather than by topic name, we are then free to develop whatever mapping we want between the (virtual cluster, topic directory, topic name) tuple and UUID. Thanks, Greg On Fri, Feb 21, 2025 at 3:32 AM Viktor Somogyi-Vass <viktor.somo...@cloudera.com.invalid> wrote: > Hi Greg, > > Replying to the rest of your points: > > 3. Well, after thinking about it a lot, I think that overall it would > complicate the feature without adding much usefulness. If we assume that a > topic is inside a directory hierarchy, then it wouldn't be transparent > anymore to users. Right now we say that anyone who isn't inside a VC, sees > the physical names of topics. But if we were to introduce directories, then > they would eventually trickle down to the protocols. If cluster admins who > are outside VCs tried to list topics for instance, they would need to be > able to distinguish between mytopic in VC1 and mytopic in VC2. Right now in > our proposal they see the physical names of the topics which are generated > which makes this feature transparent to them. With directories we couldn't > do this because of the non-uniqueness of topic names. > If we were to keep generated topic names and add directory structures that > map VCs on the disk, then we just made an unnecessary complication > too, although you'd be able to list all topics without changing > the protocols because all names are unique. This too however introduces the > problem that in case of a shared topic, we can't place the physical topic > in the right directory. For instance if you created mytopic in VC1 but then > shared it with VC2 and VC3 and removed sharing from VC1, it's weird if on > the disk it's in /kafkalogs/VC1/mytopic-X. It may be a simple "mv" command > to rename the path but there is no good way to choose between VC2 and VC3. > I hope what I'm saying makes sense, but let me know if not :) > > 4. Yes, you're correct in that, I'll fix the KIP. It's rather just a > different way of doing it. > > Best, > Viktor > > On Thu, Feb 20, 2025 at 2:36 PM Dániel Urbán <urb.dani...@gmail.com> > wrote: > > > Hi Greg, > > > > Trying to respond to some of your questions/points: > > > > 1. We started out with the exact model you propose here - if VCs can be > > nested, the root VC can be a special one, and everything else falls into > > place. The issue we encountered here is how can clients inside a VC can > > refer to topics linked in nested VCs? An admin of a VC needs a way to > refer > > to the resources inside the VC and the child VCs. What if the same > physical > > topic is linked in 2 VCs of the same sub-tree? What if the same link name > > is used in different VCs of the sub-tree? This is not an impossible task, > > as we could add VC specifiers when referencing resources such as topics, > > but we had some other priorities as well, namely we didn't want to change > > (most of) the existing protocols. Because of this, we thought that > sticking > > to a single-level VC model is a good tradeoff, as we can ensure full > > backward compatibility, and without touching most of the API requests. As > > you pointed out, we lose a small level of transparency by not allowing > > admins of a VC to create more VCs, but with fine-grained ACL control, we > > can grant admins permissions over multiple VCs while the admins > themselves > > are not bound to any specific VC. This is not as elegant as the tree of > > VCs, but pretty close in terms of functionality. > > > > 2. Skipping topic name generation. I have 2 relevant points here. First > one > > is (assuming the single-level VC model), how can an admin of the physical > > cluster itself manage the cluster and the topics? If we don't name topics > > created through a VC, a cluster admin will have a hard time identifying > > topics and their meanings. I personally think that having some "pointers" > > in the name helps a lot in operations. My second point is about the scope > > of the changes we need to implement VCs. If we don't have topic names, it > > means we need to change the protocol (AFAIU, topics without names are > not a > > thing as of now), and the log directory structure. These are significant > > changes which aren't exactly necessary to have VCs in Kafka - and the KIP > > by itself, with all its impacts on authz, quotas and so on, is already a > > pretty big change. > > > > Thanks, > > Daniel > > > > Greg Harris <greg.har...@aiven.io.invalid> ezt írta (időpont: 2025. > febr. > > 18., K, 20:40): > > > > > Hi All, > > > > > > Thanks for this amazing KIP! I'm glad that Kafka is getting more > > accessible > > > multi-tenancy features, beyond the primitives that are already > available. > > > > > > 1. I think one concept for hierarchical VCs could be to make it so > there > > is > > > a "root VC" corresponding to the physical cluster, and define the > > behavior > > > of child VCs by analogy to the first-tier VCs and this root VC. > > > > > > I think this will be a powerful abstraction if clients are not able to > > > distinguish whether they are in a VC or not, somewhat like a "chroot > > jail" > > > [1]. For example, an organization would be able to apply > > > principle-of-least-privilege to topic aliasing, and use VC credentials > > > instead of distributing root credentials. > > > For organizations wishing to enforce a flat structure, they could > control > > > the creation of VCs with credentials, making hierarchical VCs a matter > of > > > policy and not mechanism [2]. For organizations that require a cluster > > > hierarchy, they won't be forced back to discrete clusters. > > > I could also see flat VCs being implemented first, but designed to be > > > extended to hierarchical VCs later. I think it's too early to reject > > > hierarchical VCs entirely. > > > > > > 2. Could we avoid the name generation logic by not assigning names to > > > topics created in VCs at all? It requires us to make some arbitrary > > > choices, and I'm not sure how those choices will play out. > > > > > > For example: A user creates "topic-a" in some VC0, it is linked to VC1 > > > under the alias "topic-b", and then unlinked from VC0. > > > Is that a supported flow? What do users expect in that circumstance? Of > > > what significance is "topic-a" as part of the underlying topic name? Is > > the > > > topic forever associated with VC0, and can users look that up? > > > Right now, aliasing topics between VCs feels like it should follow the > > > style of file system hard links [3], but it doesn't. > > > > > > 3. There have been hints at a future topic name > > > hierarchy/directory-structure for topics. Do you expect that VCs should > > > replace the need for such a feature, or would a directory-like > structure > > > within VCs still be valuable? > > > > > > 4. For Kafka Connect, the KIP includes this: "This makes it possible to > > > host multiple Connect clusters with the same Kafka cluster." This is > > > incorrect, as multiple Connect clusters can already be connected to a > > > single Kafka cluster by changing the group and topics configurations. > > > MirrorMaker2 uses this feature regularly to isolate flows by > > source/target > > > alias. > > > > > > Thanks, > > > Greg > > > > > > [1] https://manned.org/chroot.2 > > > [2] https://en.wikipedia.org/wiki/Separation_of_mechanism_and_policy > > > [3] https://man7.org/linux/man-pages/man2/link.2.html > > > > > > On Tue, Feb 18, 2025 at 6:49 AM Viktor Somogyi-Vass > > > <viktor.somo...@cloudera.com.invalid> wrote: > > > > > > > Hi Luke, > > > > > > > > 1. ACL after migrating to VC: > > > > Suppose we have an ACL entry to user Bob, after Bob joining VC1, how > > does > > > > the ACL migration work? > > > > Will Bob still have the ACL in cluster-wide level? > > > > > > > > Yes, we didn't plan for this originally, so it would be the > > > administrator's > > > > responsibility to migrate existing ACLs, however thinking about it, > the > > > > migration could be quite simple (essentially we would just need to > use > > > the > > > > same ACLs but move them to under VC1 so they're not global). > Therefore > > if > > > > you think it could be useful, we could perhaps provide tooling for > > this. > > > > > > > > 2. What does topic linking mean? > > > > I don't understand what it means with a topic link to a VC. > > > > If a topic foo links to all VC1, VC2, VC3, does that mean all users > > under > > > > VC 1, 2, 3 can write/read data from it? > > > > It's like a shared topic for all these VCs? > > > > If so, will a consumer group consist of consumer1 in VC1, consumer2 > in > > > > VC2...? > > > > > > > > Yes, topic linking is basically creating virtual topics that exist > > within > > > > the context of a virtual cluster but they point to a real topic. Also > > yes > > > > to your example, we wanted topics to be shareable between virtual > > > clusters, > > > > so VC1, VC2 and VC3 all could have a link to topic "foo" and clients > in > > > > those VCs could read/write them. > > > > For consumer groups, I wouldn't provide this sharing capability. > > Consumer > > > > groups created by users in VCs would be prefixed by the VC name and > > their > > > > offsets would be stored in __consumer-offsets by this name, like > > > > VC1.consumer1. They would be able to consume from topics in other VCs > > via > > > > topic sharing. > > > > > > > > 3. I think a VC inside a VC is not permitted, right? > > > > > > > > Yes, correct. We in fact originally started with a concept of VC > > > > hierarchies which I think is similar in lots of ways, but after > quite a > > > few > > > > brainstorming sessions we discarded this idea as it makes the whole > > > design > > > > very complex vs the current simpler design where we would get the > same > > > > features cheaper. To summarize it we found the following main > problems: > > > > - Topics could be visible just in the VC they're created in or in the > > > > sub-VCs (child VCs?). Both can be arguable and have use-cases, yet > the > > > > implementation really should decide on one, therefore it's too > > > restricting. > > > > If we created a topic sharing feature in this environment, then that > > > would > > > > lead to complicated graphs we don't want as it's hard to work with > > them. > > > > - To refer to topics that in a VC that's inside another VC can be > > > > complicated similarly to a hierarchy. We can choose to have absolute > > > > naming, relative naming or just keep physical naming. It's > complicated > > to > > > > design this. If we allow relative or absolute naming, then we would > > need > > > to > > > > introduce a path separator character in clients, which makes them > > > > incompatible, so users would need to upgrade all their clients too if > > > they > > > > wanted to take advantage of VCs. If we don't allow referencing into > > > > sub-VCs, then they lose some part of their usefulness. They could > still > > > be > > > > used as configuration templates or ACL templates where all resources > > > within > > > > them inherit the same defaults, but I think without topic > shareability > > > > they'd lose their attractiveness. > > > > - Implementing ACLs can be complicated as well. First, whatever we > > > decided > > > > in the previous points, ACLs would have to support that, so its > > > complexity > > > > really depends on the previous points. Then there is an argument > > whether > > > > ACLs are transparent or not. In the current design, they're > > transparent, > > > > which means that being part of a VC wouldn't inherently prevent you > > from > > > > creating a topic if you have the correct permissions on the > > > "kafka-cluster" > > > > resource. But in case of sub-VCs this may or may not be > > > > beneficial, depending on your use-case: do you want sub-VCs to > inherit > > > ACLs > > > > or not. This is again a design decision where we may need to > sacrifice > > a > > > > part of valid use-cases. With the current design this problem doesn't > > > exist > > > > as it's transparent and we don't have VCs. Users would have the > freedom > > > to > > > > write tools using the Admin API that would set the ACLs properly if a > > new > > > > VC is created. > > > > > > > > 4. The view of the VC users. > > > > Supposedly, when a VC user creates a topic "foo", it should be able > to > > > see > > > > the topic "foo" created. > > > > But now it creates a "UUID-foo" topic. Will we do something here to > > help > > > VC > > > > users using virtual-clusters.sh CLI? > > > > > > > > VC users will always see the links or virtual topics instead of the > > real > > > > ones. So the broker will know if a user is assigned to a VC and > > > translates > > > > "UUID.foo" to "foo" and returns that in its response. This naming > > pattern > > > > would be totally transparent for the clients inside a VC. This is > true > > > for > > > > all protocols (produce, consume etc.). For clients outside, we don't > > > really > > > > know better but to return the physical name of the topic which is > > > UUID.foo > > > > . > > > > > > > > 5. Monitoring > > > > How do we monitor the resources for a specific VC? > > > > Do we ever think about it? > > > > > > > > This is a good point, we haven't gotten to it yet. I'll circle back > to > > > you > > > > in a few days. Do you have any ideas in mind? I'd be happy to > consider > > > > them. > > > > > > > > Thanks, > > > > Viktor > > > > > > > > On Tue, Feb 18, 2025 at 2:34 PM Viktor Somogyi-Vass < > > > > viktor.somo...@cloudera.com> wrote: > > > > > > > > > Hi YengTao, > > > > > > > > > > Yes, we left out the naming mechanism but not intentionally. I'll > add > > > it > > > > > after answering your feedback. > > > > > > > > > > - Will this leverage Kafka's existing UUID mechanism, or are > there > > > > plans > > > > > to implement a new naming mechanism? > > > > > - Will there be standardized formats for the length and > structure > > of > > > > > these generated names? > > > > > > > > > > We planned to leverage the UUID naming mechanism for topics that > are > > > > > created by users. Initially the thought was that we could > completely > > > > > replace topic names with UUIDs but unfortunately it can't be done > > > because > > > > > topic names are deeply embedded in Kafka. So we came to the > > conclusion > > > > that > > > > > we should just prefix topic names with UUIDs, like > > > > > "abcd-1234-dcba-3412.mytopic"). This is also advantageous because > > from > > > > the > > > > > debugging perspective, it's easier to identify topics and their > > assumed > > > > > content if their names contain a human-readable part. It also has > to > > > be a > > > > > valid topic name so that non-VC clients can access them if needed > > > > (backward > > > > > compatibility, we don't want to make VC topics incompatible with > > non-VC > > > > > clients). Therefore we could either separate topics by ".", "_" or > > "-". > > > > We > > > > > choose "." but it could be any other. > > > > > Also, topic names that have been created before virtual clusters > (or > > > > > outside virtual clusters but later linked to them), would behave > > > > > normally, so if a user named it "mytopic", then we wouldn't prefix > it > > > > with > > > > > anything. > > > > > > > > > > Thanks, > > > > > Viktor > > > > > > > > > > On Mon, Feb 17, 2025 at 9:22 AM Luke Chen <show...@gmail.com> > wrote: > > > > > > > > > >> Hi Viktor, > > > > >> > > > > >> This is a very interesting KIP! > > > > >> It keeps most of the existing concept and design to create a new > > > virtual > > > > >> cluster concept. Nice! > > > > >> > > > > >> Some questions: > > > > >> 1. ACL after migrating to VC: > > > > >> Suppose we have an ACL entry to user Bob, after Bob joining VC1, > how > > > > does > > > > >> the ACL migration work? > > > > >> Will Bob still have the ACL in cluster-wide level? > > > > >> > > > > >> 2. What does topic linking mean? > > > > >> I don't understand what it means with a topic link to a VC. > > > > >> If a topic foo links to all VC1, VC2, VC3, does that mean all > users > > > > under > > > > >> VC 1, 2, 3 can write/read data from it? > > > > >> It's like a shared topic for all these VCs? > > > > >> If so, will a consumer group consist of consumer1 in VC1, > consumer2 > > in > > > > >> VC2...? > > > > >> > > > > >> 3. I think a VC inside a VC is not permitted, right? > > > > >> > > > > >> 4. The view of the VC users. > > > > >> Supposedly, when a VC user creates a topic "foo", it should be > able > > to > > > > see > > > > >> the topic "foo" created. > > > > >> But now it creates a "UUID-foo" topic. Will we do something here > to > > > help > > > > >> VC > > > > >> users using virtual-clusters.sh CLI? > > > > >> > > > > >> 5. Monitoring > > > > >> How do we monitor the resources for a specific VC? > > > > >> Do we ever think about it? > > > > >> > > > > >> Thanks. > > > > >> Luke > > > > >> > > > > >> On Sat, Feb 15, 2025 at 4:46 PM TengYao Chi <kiting...@gmail.com> > > > > wrote: > > > > >> > > > > >> > Hi Viktor, > > > > >> > > > > > >> > Thank you and all the authors for this great KIP! > > > > >> > > > > > >> > I've taken a quick pass and have some questions about the > > > > >> topic-generated > > > > >> > name mechanism within virtual clusters. > > > > >> > > > > > >> > The KIP mentions that topics created within virtual clusters > will > > > have > > > > >> > generated names to avoid conflicts. However, it doesn't seem to > > > > mention > > > > >> or > > > > >> > define specifications for this name-generation pattern. > > > > >> > > > > > >> > Specifically: > > > > >> > > > > > >> > > > > > >> > - Will this leverage Kafka's existing UUID mechanism, or are > > > there > > > > >> plans > > > > >> > to implement a new naming mechanism? > > > > >> > - Will there be standardized formats for the length and > > structure > > > > of > > > > >> > these generated names? > > > > >> > > > > > >> > Thank you for answering my questions :) > > > > >> > > > > > >> > Best Regards, > > > > >> > TengYao > > > > >> > > > > > >> > > > > > >> > > > > > >> > Viktor Somogyi-Vass <viktor.somo...@cloudera.com.invalid> 於 > > > > 2025年2月14日 > > > > >> 週五 > > > > >> > 下午10:11寫道： > > > > >> > > > > > >> > > Hi David, > > > > >> > > > > > > >> > > Thanks for reading it through in advance. So I think there > are 3 > > > > >> options > > > > >> > > for resource isolation today, let me go through them for > > context. > > > > The > > > > >> > > 4th is how virtual clusters help the 3rd option :). > > > > >> > > 1. We can create completely separate Kafka clusters and host > > them > > > > >> > > separately without any other software running on those hosts. > In > > > > this > > > > >> > > scenario you are completely in control of the hardware (cpu, > > > > network, > > > > >> > disk > > > > >> > > utilization). This however may be the most expensive solution > > too > > > > >> because > > > > >> > > hosting your own servers can be expensive for many users. Also > > > > you're > > > > >> > > likely left with unused capacity. > > > > >> > > 2. Then if we think of more economic solutions, we can choose > to > > > > host > > > > >> > Kafka > > > > >> > > clusters together in a virtualized environment. There you > should > > > be > > > > >> able > > > > >> > to > > > > >> > > control the hardware resources but from an operation point of > > view > > > > >> you'll > > > > >> > > have a harder time as you'll need to manage more Kafka > clusters > > > > (need > > > > >> to > > > > >> > > set up ACLs, quotas and configuration and keep them in sync if > > > > >> needed). > > > > >> > > Also, if there is a need for sharing data between these, it's > > not > > > > easy > > > > >> > > because then you'll need to set up replication between those > two > > > > >> > clusters. > > > > >> > > 3.To overcome the cluster management overhead, it's better to > > host > > > > one > > > > >> > big > > > > >> > > Kafka cluster, and this is what larger organizations tend to > do. > > > > Then > > > > >> a > > > > >> > > central team manages the installation (if self-hosted and > > they're > > > > not > > > > >> > using > > > > >> > > a cloud service) and provides Kafka as a service for other > > teams. > > > > This > > > > >> > > often isn't perfect either, because they need to know about > > every > > > > new > > > > >> > > client or user to set proper network quotas (besides the > > catch-all > > > > >> > > default). > > > > >> > > 4. With virtual clusters we improve this situation by quotas > for > > > > >> virtual > > > > >> > > clusters. You would set virtual cluster specific quotas that > > > > wouldn't > > > > >> be > > > > >> > > possible to exceed for all the clients running together on > that > > > VC. > > > > >> > Besides > > > > >> > > this, virtual cluster admins could be defined who may have > > better > > > > >> > knowledge > > > > >> > > about their cluster and their team and set quotas in the > > clusters > > > > >> > > appropriately. > > > > >> > > > > > > >> > > Viktor > > > > >> > > > > > > >> > > On Fri, Feb 14, 2025 at 12:50 AM David Arthur < > mum...@gmail.com > > > > > > > >> wrote: > > > > >> > > > > > > >> > > > Thanks for the KIP, Viktor (et al)! > > > > >> > > > > > > > >> > > > This certainly looks like an interesting feature. I had one > > > > initial > > > > >> > > > question upon my first cursory readthrough: how does a > virtual > > > > >> cluster > > > > >> > > help > > > > >> > > > with isolation? I see this mentioned as a motivation, but > it's > > > not > > > > >> > > > immediately obvious to me how it changes resource isolation. > > > > >> > > > > > > > >> > > > I'll do a more thorough review and come back with more > > questions > > > > :) > > > > >> > > > > > > > >> > > > -David A > > > > >> > > > > > > > >> > > > > > > > >> > > > On Thu, Feb 13, 2025 at 12:11 PM Viktor Somogyi-Vass > > > > >> > > > <viktor.somo...@cloudera.com.invalid> wrote: > > > > >> > > > > > > > >> > > > > Hi all, > > > > >> > > > > > > > > >> > > > > I'd like to start a discussion on our new feature > proposal: > > > > >> virtual > > > > >> > > > > clusters in Kafka. In this KIP we propose the leveling up > of > > > > >> > > > multi-tenancy > > > > >> > > > > to better support organizations who manage their > deployments > > > > >> > centrally > > > > >> > > > for > > > > >> > > > > multiple teams. > > > > >> > > > > Please take a look if you're interested and I'm happy to > > take > > > > any > > > > >> > > > feedback > > > > >> > > > > on it. > > > > >> > > > > > > > > >> > > > > The link to the KIP: > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1134%3A+Virtual+Clusters+in+Kafka > > > > >> > > > > > > > > >> > > > > Thanks, > > > > >> > > > > Viktor > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > -- > > > > >> > > > David Arthur > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > > > > > > > > > > > >
