Hi Greg, Thank you for your comments!
1. > Can you provide an example of a situation where a client in a VC needs to > refer to resources in a child VC using the existing APIs? I think topic creation/topic listing are both good examples, but I'd argue that a cluster admin might need to be able to perform any kind of operations on the resources, e.g. produce/consume or resetting consumer group offsets are also fair examples. In order for a VC admin to create a topic within a child VC, it would need > to first create a user with the appropriate permissions within the child VC. This is a possible solution, but does sound as a complex user experience: 1. Juggling multiple sets of credentials (1 cred per VC) sounds tedious. Assuming that there is no limit on the number of VCs per level, or the number of levels, this can be a very high number, especially for the admin of the physical cluster/root VC. Having a user to keep track of these credentials and keep switching them sounds like a lot of overhead, and also increases the risk of leaking credentials. If they use an extra system to manage Kafka (e.g. an admin UI) which handles authentication/authorization, they would need to log in and out for the different VCs. 2. Authorization and auditing will be counter-intuitive, as the same cluster admin will use multiple credentials/identities to interact with the same cluster. Following an audit trail or revoking access of a user will need to cover multiple identities instead of just 1. 3. With the many different authentication mechanisms and user security policies, I suspect that there are some use-cases which wouldn't really support it. I think in a sufficiently large org, it can happen (or maybe it is standard) that the Kafka cluster admin has full permissions over the cluster, but has no admin rights in the identity provider backing the authentication. In a situation like this, each time the cluster admin creates a new VC, they would need to request a new set of credentials from the identity provider admin to be able to log into that VC, and start creating resources inside of it. (Maybe this point is closely tied to the previous one.) I think in case we go with nested VCs, we should avoid the above, and introduce new kinds of "references" to resources to allow VC admins to operate across multiple VCs, with the same credentials. (But then we need to touch most of the APIs.) 2a. > I think we can do this without packing all of the debug information we can > into the one free text field we have, and one that necessarily needs to be > unique. I agree, jamming a bunch of different info into a text field smells, adding more fields to the topic metadata would be the clean solution - maybe we can and should do that right at the start, but I think it is still not sufficient: I imagine that we could opportunistically show context where appropriate, > like "topic UUID (topic-a in vc0, topic-b in vc-1)". I think we should discuss separately based on which VC model we are talking about. 2a1. For the flat VC model, showing more context assumes we update tooling and protocols, which means this will not work for "old" clients, both in terms of binary versions and user experience. E.g. kafka-topics --list shows topic names, and nothing else. To show more context, we need to add new options to not break backward compatibility. But then, if the name does not contain any human-readable context (e.g. because it is just the UUID), then the user working with the old client simply lost the context. So my question is what should a user who relied on "kafka-topics --list" pre-VC do post-VC? Do we mandate that they update their client/use different commands? The suggested solution from the KIP will allow them to keep using "kafka-topics --list", even with an older client version, and get the same (or similar value) out of it, which results in a smoother upgrade/migration path. 2a2. Assuming that we go with the nested VC model, and also apply point 1 (client only sees topics of their corresponding VC), it sounds to me like clients will not use topic names anymore, only topic links. (Please correct me if I'm wrong.) Even an admin in the root VC cannot access/refer to topics not linked in the root VC, meaning that we don't need to care about topic names at all, since clients only see "links" instead, corect? The only place where "topic names" matter will be the disk, assuming we do not change the log directory structure. 2b. I think we agree here, for backward compatibility names should be kept in some form. Thanks, Daniel Luke Chen <show...@gmail.com> ezt írta (időpont: 2025. febr. 25., K, 4:25): > Hi Victor, > > > Yes, we didn't plan for this originally, so it would be the > administrator's > responsibility to migrate existing ACLs, however thinking about it, the > migration could be quite simple (essentially we would just need to use the > same ACLs but move them to under VC1 so they're not global). Therefore if > you think it could be useful, we could perhaps provide tooling for this. > > Agree that we provide tooling for this migration. > > > 5. Monitoring > How do we monitor the resources for a specific VC? > Do we ever think about it? > > This is a good point, we haven't gotten to it yet. I'll circle back to you > in a few days. Do you have any ideas in mind? I'd be happy to consider > them. > > In my opinion, we need some basic metadata metrics like > - The VC counts in the cluster > - The topics count in each VC > - The quota status in each VC > - consumer groups under each VC > - ... > > > About the hierarchical VCs > > I originally had the same thought as Greg that it will be helpful for some > scenarios. > But after your explanation, I agree it makes everything more complicated > and it might not be worth investing in it. > I'm open to this and would like to hear the community users' opinions. > > Thanks. > Luke > > On Tue, Feb 25, 2025 at 12:10 AM Viktor Somogyi-Vass > <viktor.somo...@cloudera.com.invalid> wrote: > > > Hi Greg, > > > > 3. > > > > It sounds like some of the complication you're mentioning comes from the > > physical topic names and physical location on disk; that's sorta why I > > brought up (2). > > If we transform the log layer to store things by UUID rather than by > topic > > name, we are then free to develop whatever mapping we want between the > > (virtual cluster, topic directory, topic name) tuple and UUID. > > > > Yes, exactly. The level of complication depends on how we implement it, > but > > all of them would require protocol changes that could limit backward > > compatibility. Also we can store almost every information about topics in > > metadata which is much easier to change and adapt to any requirements. > > Hence, changing the log dir structure is just an extra effort for little > to > > no benefits. > > > > Best, > > Viktor > > > > On Fri, Feb 21, 2025 at 10:20 PM Greg Harris > <greg.har...@aiven.io.invalid > > > > > wrote: > > > > > Hey Daniel and Viktor, > > > > > > Thanks so much for your answers! > > > > > > 1. > > > > The issue we encountered here is how can clients inside a VC can > > > > refer to topics linked in nested VCs? An admin of a VC needs a way to > > > refer > > > > to the resources inside the VC and the child VCs. > > > > > > I would think the only operations referring to resources in multiple > VCs > > at > > > the same time would be in the actual VC operations themselves > > > (Create/Alter/Describe). And since those are new, you can add the > > necessary > > > context to distinguish between resources. > > > Can you provide an example of a situation where a client in a VC needs > to > > > refer to resources in a child VC using the existing APIs? > > > For an example of the way I understood recursive VCs, an AdminClient > > within > > > a VC would be able to create topics only in its VC, independent of the > > > topics already existing in parent or child VCs. In order for a VC admin > > to > > > create a topic within a child VC, it would need to first create a user > > with > > > the appropriate permissions within the child VC. > > > > > > 2a. > > > > I personally think that having some "pointers" > > > > in the name helps a lot in operations. > > > > > > I agree that the tooling should present as much useful information to > > > admins as possible. However, I think we can do this without packing all > > of > > > the debug information we can into the one free text field we have, and > > one > > > that necessarily needs to be unique. > > > I imagine that we could opportunistically show context where > appropriate, > > > like "topic UUID (topic-a in vc0, topic-b in vc-1)". > > > > > > 2b. > > > > If we don't have topic names, it > > > > means we need to change the protocol (AFAIU, topics without names are > > not > > > a > > > > thing as of now), and the log directory structure. > > > > > > This is fair. I think having "nameless topics" actually having names > that > > > are their UUIDs for backwards compatibility is a reasonable strategy. > > > > > > 3. I don't think I understand your discussion here. > > > Topic directories would certainly require broad protocol changes (which > > is > > > probably why it hasn't happened yet) and would need to conform to > > whatever > > > gets implemented in this proposal. > > > It sounds like some of the complication you're mentioning comes from > the > > > physical topic names and physical location on disk; that's sorta why I > > > brought up (2). > > > > > > If we transform the log layer to store things by UUID rather than by > > topic > > > name, we are then free to develop whatever mapping we want between the > > > (virtual cluster, topic directory, topic name) tuple and UUID. > > > > > > Thanks, > > > Greg > > > > > > On Fri, Feb 21, 2025 at 3:32 AM Viktor Somogyi-Vass > > > <viktor.somo...@cloudera.com.invalid> wrote: > > > > > > > Hi Greg, > > > > > > > > Replying to the rest of your points: > > > > > > > > 3. Well, after thinking about it a lot, I think that overall it would > > > > complicate the feature without adding much usefulness. If we assume > > that > > > a > > > > topic is inside a directory hierarchy, then it wouldn't be > transparent > > > > anymore to users. Right now we say that anyone who isn't inside a VC, > > > sees > > > > the physical names of topics. But if we were to introduce > directories, > > > then > > > > they would eventually trickle down to the protocols. If cluster > admins > > > who > > > > are outside VCs tried to list topics for instance, they would need to > > be > > > > able to distinguish between mytopic in VC1 and mytopic in VC2. Right > > now > > > in > > > > our proposal they see the physical names of the topics which are > > > generated > > > > which makes this feature transparent to them. With directories we > > > couldn't > > > > do this because of the non-uniqueness of topic names. > > > > If we were to keep generated topic names and add directory structures > > > that > > > > map VCs on the disk, then we just made an unnecessary complication > > > > too, although you'd be able to list all topics without changing > > > > the protocols because all names are unique. This too however > introduces > > > the > > > > problem that in case of a shared topic, we can't place the physical > > topic > > > > in the right directory. For instance if you created mytopic in VC1 > but > > > then > > > > shared it with VC2 and VC3 and removed sharing from VC1, it's weird > if > > on > > > > the disk it's in /kafkalogs/VC1/mytopic-X. It may be a simple "mv" > > > command > > > > to rename the path but there is no good way to choose between VC2 and > > > VC3. > > > > I hope what I'm saying makes sense, but let me know if not :) > > > > > > > > 4. Yes, you're correct in that, I'll fix the KIP. It's rather just a > > > > different way of doing it. > > > > > > > > Best, > > > > Viktor > > > > > > > > On Thu, Feb 20, 2025 at 2:36 PM Dániel Urbán <urb.dani...@gmail.com> > > > > wrote: > > > > > > > > > Hi Greg, > > > > > > > > > > Trying to respond to some of your questions/points: > > > > > > > > > > 1. We started out with the exact model you propose here - if VCs > can > > be > > > > > nested, the root VC can be a special one, and everything else falls > > > into > > > > > place. The issue we encountered here is how can clients inside a VC > > can > > > > > refer to topics linked in nested VCs? An admin of a VC needs a way > to > > > > refer > > > > > to the resources inside the VC and the child VCs. What if the same > > > > physical > > > > > topic is linked in 2 VCs of the same sub-tree? What if the same > link > > > name > > > > > is used in different VCs of the sub-tree? This is not an impossible > > > task, > > > > > as we could add VC specifiers when referencing resources such as > > > topics, > > > > > but we had some other priorities as well, namely we didn't want to > > > change > > > > > (most of) the existing protocols. Because of this, we thought that > > > > sticking > > > > > to a single-level VC model is a good tradeoff, as we can ensure > full > > > > > backward compatibility, and without touching most of the API > > requests. > > > As > > > > > you pointed out, we lose a small level of transparency by not > > allowing > > > > > admins of a VC to create more VCs, but with fine-grained ACL > control, > > > we > > > > > can grant admins permissions over multiple VCs while the admins > > > > themselves > > > > > are not bound to any specific VC. This is not as elegant as the > tree > > of > > > > > VCs, but pretty close in terms of functionality. > > > > > > > > > > 2. Skipping topic name generation. I have 2 relevant points here. > > First > > > > one > > > > > is (assuming the single-level VC model), how can an admin of the > > > physical > > > > > cluster itself manage the cluster and the topics? If we don't name > > > topics > > > > > created through a VC, a cluster admin will have a hard time > > identifying > > > > > topics and their meanings. I personally think that having some > > > "pointers" > > > > > in the name helps a lot in operations. My second point is about the > > > scope > > > > > of the changes we need to implement VCs. If we don't have topic > > names, > > > it > > > > > means we need to change the protocol (AFAIU, topics without names > are > > > > not a > > > > > thing as of now), and the log directory structure. These are > > > significant > > > > > changes which aren't exactly necessary to have VCs in Kafka - and > the > > > KIP > > > > > by itself, with all its impacts on authz, quotas and so on, is > > already > > > a > > > > > pretty big change. > > > > > > > > > > Thanks, > > > > > Daniel > > > > > > > > > > Greg Harris <greg.har...@aiven.io.invalid> ezt írta (időpont: > 2025. > > > > febr. > > > > > 18., K, 20:40): > > > > > > > > > > > Hi All, > > > > > > > > > > > > Thanks for this amazing KIP! I'm glad that Kafka is getting more > > > > > accessible > > > > > > multi-tenancy features, beyond the primitives that are already > > > > available. > > > > > > > > > > > > 1. I think one concept for hierarchical VCs could be to make it > so > > > > there > > > > > is > > > > > > a "root VC" corresponding to the physical cluster, and define the > > > > > behavior > > > > > > of child VCs by analogy to the first-tier VCs and this root VC. > > > > > > > > > > > > I think this will be a powerful abstraction if clients are not > able > > > to > > > > > > distinguish whether they are in a VC or not, somewhat like a > > "chroot > > > > > jail" > > > > > > [1]. For example, an organization would be able to apply > > > > > > principle-of-least-privilege to topic aliasing, and use VC > > > credentials > > > > > > instead of distributing root credentials. > > > > > > For organizations wishing to enforce a flat structure, they could > > > > control > > > > > > the creation of VCs with credentials, making hierarchical VCs a > > > matter > > > > of > > > > > > policy and not mechanism [2]. For organizations that require a > > > cluster > > > > > > hierarchy, they won't be forced back to discrete clusters. > > > > > > I could also see flat VCs being implemented first, but designed > to > > be > > > > > > extended to hierarchical VCs later. I think it's too early to > > reject > > > > > > hierarchical VCs entirely. > > > > > > > > > > > > 2. Could we avoid the name generation logic by not assigning > names > > to > > > > > > topics created in VCs at all? It requires us to make some > arbitrary > > > > > > choices, and I'm not sure how those choices will play out. > > > > > > > > > > > > For example: A user creates "topic-a" in some VC0, it is linked > to > > > VC1 > > > > > > under the alias "topic-b", and then unlinked from VC0. > > > > > > Is that a supported flow? What do users expect in that > > circumstance? > > > Of > > > > > > what significance is "topic-a" as part of the underlying topic > > name? > > > Is > > > > > the > > > > > > topic forever associated with VC0, and can users look that up? > > > > > > Right now, aliasing topics between VCs feels like it should > follow > > > the > > > > > > style of file system hard links [3], but it doesn't. > > > > > > > > > > > > 3. There have been hints at a future topic name > > > > > > hierarchy/directory-structure for topics. Do you expect that VCs > > > should > > > > > > replace the need for such a feature, or would a directory-like > > > > structure > > > > > > within VCs still be valuable? > > > > > > > > > > > > 4. For Kafka Connect, the KIP includes this: "This makes it > > possible > > > to > > > > > > host multiple Connect clusters with the same Kafka cluster." This > > is > > > > > > incorrect, as multiple Connect clusters can already be connected > > to a > > > > > > single Kafka cluster by changing the group and topics > > configurations. > > > > > > MirrorMaker2 uses this feature regularly to isolate flows by > > > > > source/target > > > > > > alias. > > > > > > > > > > > > Thanks, > > > > > > Greg > > > > > > > > > > > > [1] https://manned.org/chroot.2 > > > > > > [2] > > https://en.wikipedia.org/wiki/Separation_of_mechanism_and_policy > > > > > > [3] https://man7.org/linux/man-pages/man2/link.2.html > > > > > > > > > > > > On Tue, Feb 18, 2025 at 6:49 AM Viktor Somogyi-Vass > > > > > > <viktor.somo...@cloudera.com.invalid> wrote: > > > > > > > > > > > > > Hi Luke, > > > > > > > > > > > > > > 1. ACL after migrating to VC: > > > > > > > Suppose we have an ACL entry to user Bob, after Bob joining > VC1, > > > how > > > > > does > > > > > > > the ACL migration work? > > > > > > > Will Bob still have the ACL in cluster-wide level? > > > > > > > > > > > > > > Yes, we didn't plan for this originally, so it would be the > > > > > > administrator's > > > > > > > responsibility to migrate existing ACLs, however thinking about > > it, > > > > the > > > > > > > migration could be quite simple (essentially we would just need > > to > > > > use > > > > > > the > > > > > > > same ACLs but move them to under VC1 so they're not global). > > > > Therefore > > > > > if > > > > > > > you think it could be useful, we could perhaps provide tooling > > for > > > > > this. > > > > > > > > > > > > > > 2. What does topic linking mean? > > > > > > > I don't understand what it means with a topic link to a VC. > > > > > > > If a topic foo links to all VC1, VC2, VC3, does that mean all > > users > > > > > under > > > > > > > VC 1, 2, 3 can write/read data from it? > > > > > > > It's like a shared topic for all these VCs? > > > > > > > If so, will a consumer group consist of consumer1 in VC1, > > consumer2 > > > > in > > > > > > > VC2...? > > > > > > > > > > > > > > Yes, topic linking is basically creating virtual topics that > > exist > > > > > within > > > > > > > the context of a virtual cluster but they point to a real > topic. > > > Also > > > > > yes > > > > > > > to your example, we wanted topics to be shareable between > virtual > > > > > > clusters, > > > > > > > so VC1, VC2 and VC3 all could have a link to topic "foo" and > > > clients > > > > in > > > > > > > those VCs could read/write them. > > > > > > > For consumer groups, I wouldn't provide this sharing > capability. > > > > > Consumer > > > > > > > groups created by users in VCs would be prefixed by the VC name > > and > > > > > their > > > > > > > offsets would be stored in __consumer-offsets by this name, > like > > > > > > > VC1.consumer1. They would be able to consume from topics in > other > > > VCs > > > > > via > > > > > > > topic sharing. > > > > > > > > > > > > > > 3. I think a VC inside a VC is not permitted, right? > > > > > > > > > > > > > > Yes, correct. We in fact originally started with a concept of > VC > > > > > > > hierarchies which I think is similar in lots of ways, but after > > > > quite a > > > > > > few > > > > > > > brainstorming sessions we discarded this idea as it makes the > > whole > > > > > > design > > > > > > > very complex vs the current simpler design where we would get > the > > > > same > > > > > > > features cheaper. To summarize it we found the following main > > > > problems: > > > > > > > - Topics could be visible just in the VC they're created in or > in > > > the > > > > > > > sub-VCs (child VCs?). Both can be arguable and have use-cases, > > yet > > > > the > > > > > > > implementation really should decide on one, therefore it's too > > > > > > restricting. > > > > > > > If we created a topic sharing feature in this environment, then > > > that > > > > > > would > > > > > > > lead to complicated graphs we don't want as it's hard to work > > with > > > > > them. > > > > > > > - To refer to topics that in a VC that's inside another VC can > be > > > > > > > complicated similarly to a hierarchy. We can choose to have > > > absolute > > > > > > > naming, relative naming or just keep physical naming. It's > > > > complicated > > > > > to > > > > > > > design this. If we allow relative or absolute naming, then we > > would > > > > > need > > > > > > to > > > > > > > introduce a path separator character in clients, which makes > them > > > > > > > incompatible, so users would need to upgrade all their clients > > too > > > if > > > > > > they > > > > > > > wanted to take advantage of VCs. If we don't allow referencing > > into > > > > > > > sub-VCs, then they lose some part of their usefulness. They > could > > > > still > > > > > > be > > > > > > > used as configuration templates or ACL templates where all > > > resources > > > > > > within > > > > > > > them inherit the same defaults, but I think without topic > > > > shareability > > > > > > > they'd lose their attractiveness. > > > > > > > - Implementing ACLs can be complicated as well. First, whatever > > we > > > > > > decided > > > > > > > in the previous points, ACLs would have to support that, so its > > > > > > complexity > > > > > > > really depends on the previous points. Then there is an > argument > > > > > whether > > > > > > > ACLs are transparent or not. In the current design, they're > > > > > transparent, > > > > > > > which means that being part of a VC wouldn't inherently prevent > > you > > > > > from > > > > > > > creating a topic if you have the correct permissions on the > > > > > > "kafka-cluster" > > > > > > > resource. But in case of sub-VCs this may or may not be > > > > > > > beneficial, depending on your use-case: do you want sub-VCs to > > > > inherit > > > > > > ACLs > > > > > > > or not. This is again a design decision where we may need to > > > > sacrifice > > > > > a > > > > > > > part of valid use-cases. With the current design this problem > > > doesn't > > > > > > exist > > > > > > > as it's transparent and we don't have VCs. Users would have the > > > > freedom > > > > > > to > > > > > > > write tools using the Admin API that would set the ACLs > properly > > > if a > > > > > new > > > > > > > VC is created. > > > > > > > > > > > > > > 4. The view of the VC users. > > > > > > > Supposedly, when a VC user creates a topic "foo", it should be > > able > > > > to > > > > > > see > > > > > > > the topic "foo" created. > > > > > > > But now it creates a "UUID-foo" topic. Will we do something > here > > to > > > > > help > > > > > > VC > > > > > > > users using virtual-clusters.sh CLI? > > > > > > > > > > > > > > VC users will always see the links or virtual topics instead of > > the > > > > > real > > > > > > > ones. So the broker will know if a user is assigned to a VC and > > > > > > translates > > > > > > > "UUID.foo" to "foo" and returns that in its response. This > naming > > > > > pattern > > > > > > > would be totally transparent for the clients inside a VC. This > is > > > > true > > > > > > for > > > > > > > all protocols (produce, consume etc.). For clients outside, we > > > don't > > > > > > really > > > > > > > know better but to return the physical name of the topic which > is > > > > > > UUID.foo > > > > > > > . > > > > > > > > > > > > > > 5. Monitoring > > > > > > > How do we monitor the resources for a specific VC? > > > > > > > Do we ever think about it? > > > > > > > > > > > > > > This is a good point, we haven't gotten to it yet. I'll circle > > back > > > > to > > > > > > you > > > > > > > in a few days. Do you have any ideas in mind? I'd be happy to > > > > consider > > > > > > > them. > > > > > > > > > > > > > > Thanks, > > > > > > > Viktor > > > > > > > > > > > > > > On Tue, Feb 18, 2025 at 2:34 PM Viktor Somogyi-Vass < > > > > > > > viktor.somo...@cloudera.com> wrote: > > > > > > > > > > > > > > > Hi YengTao, > > > > > > > > > > > > > > > > Yes, we left out the naming mechanism but not intentionally. > > I'll > > > > add > > > > > > it > > > > > > > > after answering your feedback. > > > > > > > > > > > > > > > > - Will this leverage Kafka's existing UUID mechanism, or > are > > > > there > > > > > > > plans > > > > > > > > to implement a new naming mechanism? > > > > > > > > - Will there be standardized formats for the length and > > > > structure > > > > > of > > > > > > > > these generated names? > > > > > > > > > > > > > > > > We planned to leverage the UUID naming mechanism for topics > > that > > > > are > > > > > > > > created by users. Initially the thought was that we could > > > > completely > > > > > > > > replace topic names with UUIDs but unfortunately it can't be > > done > > > > > > because > > > > > > > > topic names are deeply embedded in Kafka. So we came to the > > > > > conclusion > > > > > > > that > > > > > > > > we should just prefix topic names with UUIDs, like > > > > > > > > "abcd-1234-dcba-3412.mytopic"). This is also advantageous > > because > > > > > from > > > > > > > the > > > > > > > > debugging perspective, it's easier to identify topics and > their > > > > > assumed > > > > > > > > content if their names contain a human-readable part. It also > > has > > > > to > > > > > > be a > > > > > > > > valid topic name so that non-VC clients can access them if > > needed > > > > > > > (backward > > > > > > > > compatibility, we don't want to make VC topics incompatible > > with > > > > > non-VC > > > > > > > > clients). Therefore we could either separate topics by ".", > "_" > > > or > > > > > "-". > > > > > > > We > > > > > > > > choose "." but it could be any other. > > > > > > > > Also, topic names that have been created before virtual > > clusters > > > > (or > > > > > > > > outside virtual clusters but later linked to them), would > > behave > > > > > > > > normally, so if a user named it "mytopic", then we wouldn't > > > prefix > > > > it > > > > > > > with > > > > > > > > anything. > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Viktor > > > > > > > > > > > > > > > > On Mon, Feb 17, 2025 at 9:22 AM Luke Chen <show...@gmail.com > > > > > > wrote: > > > > > > > > > > > > > > > >> Hi Viktor, > > > > > > > >> > > > > > > > >> This is a very interesting KIP! > > > > > > > >> It keeps most of the existing concept and design to create a > > new > > > > > > virtual > > > > > > > >> cluster concept. Nice! > > > > > > > >> > > > > > > > >> Some questions: > > > > > > > >> 1. ACL after migrating to VC: > > > > > > > >> Suppose we have an ACL entry to user Bob, after Bob joining > > VC1, > > > > how > > > > > > > does > > > > > > > >> the ACL migration work? > > > > > > > >> Will Bob still have the ACL in cluster-wide level? > > > > > > > >> > > > > > > > >> 2. What does topic linking mean? > > > > > > > >> I don't understand what it means with a topic link to a VC. > > > > > > > >> If a topic foo links to all VC1, VC2, VC3, does that mean > all > > > > users > > > > > > > under > > > > > > > >> VC 1, 2, 3 can write/read data from it? > > > > > > > >> It's like a shared topic for all these VCs? > > > > > > > >> If so, will a consumer group consist of consumer1 in VC1, > > > > consumer2 > > > > > in > > > > > > > >> VC2...? > > > > > > > >> > > > > > > > >> 3. I think a VC inside a VC is not permitted, right? > > > > > > > >> > > > > > > > >> 4. The view of the VC users. > > > > > > > >> Supposedly, when a VC user creates a topic "foo", it should > be > > > > able > > > > > to > > > > > > > see > > > > > > > >> the topic "foo" created. > > > > > > > >> But now it creates a "UUID-foo" topic. Will we do something > > here > > > > to > > > > > > help > > > > > > > >> VC > > > > > > > >> users using virtual-clusters.sh CLI? > > > > > > > >> > > > > > > > >> 5. Monitoring > > > > > > > >> How do we monitor the resources for a specific VC? > > > > > > > >> Do we ever think about it? > > > > > > > >> > > > > > > > >> Thanks. > > > > > > > >> Luke > > > > > > > >> > > > > > > > >> On Sat, Feb 15, 2025 at 4:46 PM TengYao Chi < > > > kiting...@gmail.com> > > > > > > > wrote: > > > > > > > >> > > > > > > > >> > Hi Viktor, > > > > > > > >> > > > > > > > > >> > Thank you and all the authors for this great KIP! > > > > > > > >> > > > > > > > > >> > I've taken a quick pass and have some questions about the > > > > > > > >> topic-generated > > > > > > > >> > name mechanism within virtual clusters. > > > > > > > >> > > > > > > > > >> > The KIP mentions that topics created within virtual > clusters > > > > will > > > > > > have > > > > > > > >> > generated names to avoid conflicts. However, it doesn't > seem > > > to > > > > > > > mention > > > > > > > >> or > > > > > > > >> > define specifications for this name-generation pattern. > > > > > > > >> > > > > > > > > >> > Specifically: > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > - Will this leverage Kafka's existing UUID mechanism, > or > > > are > > > > > > there > > > > > > > >> plans > > > > > > > >> > to implement a new naming mechanism? > > > > > > > >> > - Will there be standardized formats for the length and > > > > > structure > > > > > > > of > > > > > > > >> > these generated names? > > > > > > > >> > > > > > > > > >> > Thank you for answering my questions :) > > > > > > > >> > > > > > > > > >> > Best Regards, > > > > > > > >> > TengYao > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > Viktor Somogyi-Vass <viktor.somo...@cloudera.com.invalid> > 於 > > > > > > > 2025年2月14日 > > > > > > > >> 週五 > > > > > > > >> > 下午10:11寫道： > > > > > > > >> > > > > > > > > >> > > Hi David, > > > > > > > >> > > > > > > > > > >> > > Thanks for reading it through in advance. So I think > there > > > > are 3 > > > > > > > >> options > > > > > > > >> > > for resource isolation today, let me go through them for > > > > > context. > > > > > > > The > > > > > > > >> > > 4th is how virtual clusters help the 3rd option :). > > > > > > > >> > > 1. We can create completely separate Kafka clusters and > > host > > > > > them > > > > > > > >> > > separately without any other software running on those > > > hosts. > > > > In > > > > > > > this > > > > > > > >> > > scenario you are completely in control of the hardware > > (cpu, > > > > > > > network, > > > > > > > >> > disk > > > > > > > >> > > utilization). This however may be the most expensive > > > solution > > > > > too > > > > > > > >> because > > > > > > > >> > > hosting your own servers can be expensive for many > users. > > > Also > > > > > > > you're > > > > > > > >> > > likely left with unused capacity. > > > > > > > >> > > 2. Then if we think of more economic solutions, we can > > > choose > > > > to > > > > > > > host > > > > > > > >> > Kafka > > > > > > > >> > > clusters together in a virtualized environment. There > you > > > > should > > > > > > be > > > > > > > >> able > > > > > > > >> > to > > > > > > > >> > > control the hardware resources but from an operation > point > > > of > > > > > view > > > > > > > >> you'll > > > > > > > >> > > have a harder time as you'll need to manage more Kafka > > > > clusters > > > > > > > (need > > > > > > > >> to > > > > > > > >> > > set up ACLs, quotas and configuration and keep them in > > sync > > > if > > > > > > > >> needed). > > > > > > > >> > > Also, if there is a need for sharing data between these, > > > it's > > > > > not > > > > > > > easy > > > > > > > >> > > because then you'll need to set up replication between > > those > > > > two > > > > > > > >> > clusters. > > > > > > > >> > > 3.To overcome the cluster management overhead, it's > better > > > to > > > > > host > > > > > > > one > > > > > > > >> > big > > > > > > > >> > > Kafka cluster, and this is what larger organizations > tend > > to > > > > do. > > > > > > > Then > > > > > > > >> a > > > > > > > >> > > central team manages the installation (if self-hosted > and > > > > > they're > > > > > > > not > > > > > > > >> > using > > > > > > > >> > > a cloud service) and provides Kafka as a service for > other > > > > > teams. > > > > > > > This > > > > > > > >> > > often isn't perfect either, because they need to know > > about > > > > > every > > > > > > > new > > > > > > > >> > > client or user to set proper network quotas (besides the > > > > > catch-all > > > > > > > >> > > default). > > > > > > > >> > > 4. With virtual clusters we improve this situation by > > quotas > > > > for > > > > > > > >> virtual > > > > > > > >> > > clusters. You would set virtual cluster specific quotas > > that > > > > > > > wouldn't > > > > > > > >> be > > > > > > > >> > > possible to exceed for all the clients running together > on > > > > that > > > > > > VC. > > > > > > > >> > Besides > > > > > > > >> > > this, virtual cluster admins could be defined who may > have > > > > > better > > > > > > > >> > knowledge > > > > > > > >> > > about their cluster and their team and set quotas in the > > > > > clusters > > > > > > > >> > > appropriately. > > > > > > > >> > > > > > > > > > >> > > Viktor > > > > > > > >> > > > > > > > > > >> > > On Fri, Feb 14, 2025 at 12:50 AM David Arthur < > > > > mum...@gmail.com > > > > > > > > > > > > > >> wrote: > > > > > > > >> > > > > > > > > > >> > > > Thanks for the KIP, Viktor (et al)! > > > > > > > >> > > > > > > > > > > >> > > > This certainly looks like an interesting feature. I > had > > > one > > > > > > > initial > > > > > > > >> > > > question upon my first cursory readthrough: how does a > > > > virtual > > > > > > > >> cluster > > > > > > > >> > > help > > > > > > > >> > > > with isolation? I see this mentioned as a motivation, > > but > > > > it's > > > > > > not > > > > > > > >> > > > immediately obvious to me how it changes resource > > > isolation. > > > > > > > >> > > > > > > > > > > >> > > > I'll do a more thorough review and come back with more > > > > > questions > > > > > > > :) > > > > > > > >> > > > > > > > > > > >> > > > -David A > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > On Thu, Feb 13, 2025 at 12:11 PM Viktor Somogyi-Vass > > > > > > > >> > > > <viktor.somo...@cloudera.com.invalid> wrote: > > > > > > > >> > > > > > > > > > > >> > > > > Hi all, > > > > > > > >> > > > > > > > > > > > >> > > > > I'd like to start a discussion on our new feature > > > > proposal: > > > > > > > >> virtual > > > > > > > >> > > > > clusters in Kafka. In this KIP we propose the > leveling > > > up > > > > of > > > > > > > >> > > > multi-tenancy > > > > > > > >> > > > > to better support organizations who manage their > > > > deployments > > > > > > > >> > centrally > > > > > > > >> > > > for > > > > > > > >> > > > > multiple teams. > > > > > > > >> > > > > Please take a look if you're interested and I'm > happy > > to > > > > > take > > > > > > > any > > > > > > > >> > > > feedback > > > > > > > >> > > > > on it. > > > > > > > >> > > > > > > > > > > > >> > > > > The link to the KIP: > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1134%3A+Virtual+Clusters+in+Kafka > > > > > > > >> > > > > > > > > > > > >> > > > > Thanks, > > > > > > > >> > > > > Viktor > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > -- > > > > > > > >> > > > David Arthur > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
