I'd also like to see clarification regarding the ZK structures. Currently they appear as if user-quotas and client-quotas are equivalent, but this will obviously need to change.
On Sat, Jun 11, 2016 at 1:28 AM, Jun Rao <j...@confluent.io> wrote: > Rajini, > > The new proposal sounds good to me too. My only question is what happens to > those existing quotas on client-id. Do we just treat them as the quota for > that client-id under ANONYMOUS user name? > > Thanks, > > Jun > > On Fri, Jun 10, 2016 at 2:43 PM, Rajini Sivaram < > rajinisiva...@googlemail.com> wrote: > >> Jay, >> >> Thank you for the quick feedback. It shouldn't be too hard since I had a PR >> earlier along these lines anyway. >> >> Jun, are you ok with this approach? If everyone agrees, I will close this >> vote, update the KIP and give some more time for discussions. >> >> >> On Fri, Jun 10, 2016 at 10:27 PM, Jay Kreps <j...@confluent.io> wrote: >> >> > This sounds a lot better to me--hopefully it isn't too much harder! I do >> > think if it is possible to do this directly that will be better for users >> > than having an intermediate step since we always have to work through >> > migrating people who have setup quotas already from the old way to the >> new >> > way. >> > >> > -Jay >> > >> > On Fri, Jun 10, 2016 at 2:12 PM, Rajini Sivaram < >> > rajinisiva...@googlemail.com> wrote: >> > >> > > I do think client-id is a valid and useful grouping for quotas even in >> > > secure clusters - but only if clientA of user1 is treated as a >> different >> > > client-id from clientA of user2. Grouping of clients of a user enables >> > > users to allocate their quota effectively to their clients (eg. >> guarantee >> > > that critical event processing clients are not throttled by auditing >> > > clients). When the KIP was down-sized to support only user-based >> quotas, >> > I >> > > was hoping that we could extend it at a later time to enable >> hierarchical >> > > quotas. But I understand that it can be confusing to switch the >> semantics >> > > of quotas based on modes set in the brokers. So let me try once again >> to >> > > promote the original KIP-55. At the time, I did have a flag to revert >> to >> > > the existing client-id behavior to maintain compatibility. But perhaps >> > that >> > > is not necessary. >> > > >> > > How does this sound? >> > > >> > > - Quotas may be configured for users. Sub-quotas may be configured >> for >> > > client-ids of a user. Quotas may also be configured for client-ids >> of >> > > users >> > > with unlimited quota (Long.MaxValue). >> > > - Users who don't have a quota override are allocated a configurable >> > > default quota. >> > > - Client-ids without a sub-quota override share the remainder of the >> > > user quota if the user has a quota limit. Default quotas may be >> > defined >> > > for >> > > clients of users with unlimited quota. >> > > - For an insecure or single-user secure cluster, the existing >> > client-id >> > > based quota semantics can be achieved by configuring unlimited quota >> > for >> > > the user and sub-quota configuration for client-id that matches the >> > > current >> > > client-id quota configuration. >> > > - For a cluster mixes both secure and insecure access, client-id >> > quotas >> > > can be set for unauthenticated clients (unlimited quota for >> ANONYMOUS, >> > > quotas for client-ids) and user quotas can be set for authenticated >> > > users. >> > > - In a multi-user cluster, it is currently possible to define quotas >> > for >> > > client-ids that span multiple users. This will no longer be >> supported. >> > > >> > > >> > > >> > > >> > > On Fri, Jun 10, 2016 at 6:43 PM, Gwen Shapira <g...@confluent.io> >> wrote: >> > > >> > > > I am not crazy about modes either. An earlier proposal supported both >> > > > client-ids and users at the same time, and it made more sense to me. >> I >> > > > believe it was dropped without proper discussion (Basically, Jun >> > > > mentioned it is complex and Rajini agreed to drop it). We should >> > > > probably rethink the complexity of supporting both vs the limitations >> > > > of "modes". >> > > > >> > > > As you said, we will have secure clients authenticating with users >> and >> > > > insecure clients authenticating with client-ids at the same time. >> > > > >> > > > On Fri, Jun 10, 2016 at 7:19 PM, Jay Kreps <j...@confluent.io> wrote: >> > > > > Hey Rajini, >> > > > > >> > > > > 1. That makes sense to me. Is that reflected in the documentation >> > > > anywhere >> > > > > (I couldn't really find it)? Is there a way to discover that >> > > definition? >> > > > We >> > > > > do way better when we right this stuff down so it has an official >> > > > > definition users and developers can work off of... >> > > > > 2. If client id is a thing that makes sense even when you have >> users, >> > > why >> > > > > would you not want to quota on it? >> > > > > >> > > > > I am not wild about these "modes" where you boot the cluster in >> mode >> > X >> > > > and >> > > > > it behaves in one way and in mode Y and it behaves in another. It >> is >> > > > > complex then for users who expect to be able to set quotas but then >> > > have >> > > > to >> > > > > be able to get access to the filesystem of the kafka nodes to >> > discover >> > > > what >> > > > > mode the cluster is in to know which kind of quota is applicable. >> > > > > >> > > > > I guess there are two ways to think about a feature: one is the >> > > increment >> > > > > from where we are, and the other is the resulting state after that >> > > > > increment is taken. What I am asking is not "is this a low cost >> step >> > > from >> > > > > where we are?" which everyone can agree that it is, but rather >> "does >> > > this >> > > > > make sense as an end state--i.e. if you were starting fresh with >> > > neither >> > > > > users nor client ids nor quotas would you end up with this?". >> > > > > >> > > > > In terms of use cases, I think that we support mixing secure and >> > > insecure >> > > > > access on a single cluster so presumably in that case you would >> want >> > to >> > > > be >> > > > > able to quota insecure users based on client id and secure users >> > based >> > > on >> > > > > user, right? Likewise, as you said, client id is a valid grouping >> > even >> > > in >> > > > > the presence of users, so it might be the case that several apps >> that >> > > are >> > > > > all part of the same system might access Kafka under a single user, >> > but >> > > > you >> > > > > might have different quotas for these different apps. Basically if >> > > client >> > > > > id is a valid grouping even in the presence of users (willing to >> > debate >> > > > > this point, btw!) then you would want to quota on it. >> > > > > >> > > > > -Jay >> > > > > >> > > > > >> > > > > >> > > > > On Fri, Jun 10, 2016 at 4:49 AM, Rajini Sivaram < >> > > > > rajinisiva...@googlemail.com> wrote: >> > > > > >> > > > >> Jay, >> > > > >> >> > > > >> Thank you for the feedback. >> > > > >> >> > > > >> 1. I think it is good to have a single concept of identity, but >> > > multiple >> > > > >> ways of grouping clients for different functions. Client-id is a >> > > logical >> > > > >> grouping of clients with a meaningful name that is used in client >> > > > metrics >> > > > >> and logs. User principal is an authenticated user or a grouping of >> > > > >> unauthenticated users chosen by the broker and is used for ACLs. >> In >> > my >> > > > >> view, in a multi-user system, there is a hierarchy - user owns >> zero >> > or >> > > > more >> > > > >> clients. (principal, client-id) defines a safe group, but the >> > shorter >> > > > >> unsafe client-id is sufficient in client metrics and logs. >> > > > >> >> > > > >> >> > > > >> 2. KIP-55 was initially written to support hierarchical quotas >> > (quotas >> > > > for >> > > > >> user1-clientA, user2-clientA etc), but Jun's view was that the >> > > > complexity >> > > > >> was not justified since there is no clear requirement for this. >> The >> > > > >> cut-down version of the KIP is clearly a lot simpler. But I think >> > your >> > > > >> suggestion is to enable non-hierarchical user quotas and client-id >> > > > quotas >> > > > >> at the same time. Basically treat users and client-ids as distinct >> > > > entities >> > > > >> like topics and allow quotas to be applied to each of these >> > entities. >> > > I >> > > > >> agree that we want to support quotas simultaneously on different >> > > > entities >> > > > >> like topics and users. I am less convinced of client-id and user >> > being >> > > > >> >> > > > >> distinct entities that require separate quotas at the same time. >> And >> > > > >> treating client-id and user as distinct entities makes it harder >> to >> > > > >> implement hierarchical quotas in future. >> > > > >> >> > > > >> >> > > > >> A single user system needs only client-id quotas, and multi-tenant >> > > > system >> > > > >> cannot use client-id quotas since we need to guarantee that one >> > > tenant's >> > > > >> quotas can never be used by another tenant. I suppose a multi-user >> > > > cluster >> > > > >> where users trust each other could apply separate quotas for both >> > > > clients >> > > > >> and users, but I am not sure if there is a usecase that can't be >> > > > satisfied >> > > > >> with just client-id based quotas for this case. Do you have a >> > usecase >> > > in >> > > > >> mind where you want to apply separate quotas for clients and users >> > > > >> simultaneously? >> > > > >> >> > > > >> >> > > > >> >> > > > >> >> > > > >> On Thu, Jun 9, 2016 at 9:40 PM, Jay Kreps <j...@confluent.io> >> wrote: >> > > > >> >> > > > >> > Super sorry to come in late on this one. Rajini, I had two quick >> > > > >> questions >> > > > >> > I think we should be able to answer: >> > > > >> > >> > > > >> > 1. Do client ids make sense in a world which has users? If >> not >> > > > should >> > > > >> we >> > > > >> > unify them the way Hadoop did (without auth the user is a >> kind >> > of >> > > > best >> > > > >> > effort honor system identity). This came up in the discussion >> > > > thread >> > > > >> > but I >> > > > >> > didn't really see a crisp answer. Basically, what is the >> > > > definition of >> > > > >> > "client id" and what is the definition of "user" and how do >> the >> > > > >> concepts >> > > > >> > relate? >> > > > >> > 2. If both client ids and users are sensible distinct notions >> > and >> > > > we >> > > > >> > want to maintain both, why don't we just support quotas on >> > both? >> > > If >> > > > >> they >> > > > >> > both make sense then you would have a reason to set quotas at >> > > both >> > > > >> > levels. >> > > > >> > Why have this "mode" that you set that swaps between only >> being >> > > > able >> > > > >> to >> > > > >> > use >> > > > >> > one and the other? I should be able to set quotas at both >> > levels. >> > > > >> Going >> > > > >> > forward the model we had discussed with quotas was >> potentially >> > > > being >> > > > >> > able >> > > > >> > to set quotas for many things independently (say at the topic >> > > > level), >> > > > >> > and I >> > > > >> > don't think it would make sense to extend this mode approach >> to >> > > > those. >> > > > >> > >> > > > >> > -Jay >> > > > >> > >> > > > >> > On Wed, Jun 8, 2016 at 12:56 PM, Rajini Sivaram < >> > > > >> > rajinisiva...@googlemail.com> wrote: >> > > > >> > >> > > > >> > > I would like to initiate the vote for KIP-55. >> > > > >> > > >> > > > >> > > The KIP details are here: KIP-55: Secure quotas for >> > authenticated >> > > > users >> > > > >> > > < >> > > > >> > > >> > > > >> > >> > > > >> >> > > > >> > > >> > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-55%3A+Secure+Quotas+for+Authenticated+Users >> > > > >> > > > >> > > > >> > > . >> > > > >> > > >> > > > >> > > The JIRA KAFKA-3492 < >> > > > >> https://issues.apache.org/jira/browse/KAFKA-3492 >> > > > >> > > >has >> > > > >> > > a draft PR here: https://github.com/apache/kafka/pull/1256. >> > > > >> > > >> > > > >> > > Thank you... >> > > > >> > > >> > > > >> > > Regards, >> > > > >> > > >> > > > >> > > Rajini >> > > > >> > > >> > > > >> > >> > > > >> >> > > > >> >> > > > >> >> > > > >> -- >> > > > >> Regards, >> > > > >> >> > > > >> Rajini >> > > > >> >> > > > >> > > >> > > >> > > >> > > -- >> > > Regards, >> > > >> > > Rajini >> > > >> > >> >> >> >> -- >> Regards, >> >> Rajini >>
