Re: [VOTE] KIP-55: Secure quotas for authenticated users

Hi all, This vote has passed with 3 binding and one non-binding +1. Many thanks to those who voted. If you have any comments or suggestions, please continue to add them to the discussion thread. On Fri, Jul 15, 2016 at 6:47 AM, Gwen Shapira wrote: > +1 > > Thanks for

Re: [VOTE] KIP-55: Secure quotas for authenticated users

+1 Thanks for working through this, Rajini. On Thu, Jul 7, 2016 at 7:12 AM, Rajini Sivaram wrote: > I would like to initiate voting for KIP-55 ( > https://cwiki.apache.org/confluence/display/KAFKA/KIP-55%3A+Secure+Quotas+for+Authenticated+Users). > Since the KIP

Re: [VOTE] KIP-55: Secure quotas for authenticated users

Rajani, Thanks for the proposal. +1 from me. Jun On Thu, Jul 7, 2016 at 7:12 AM, Rajini Sivaram wrote: > I would like to initiate voting for KIP-55 ( > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-55%3A+Secure+Quotas+for+Authenticated+Users > ). >

Re: [VOTE] KIP-55: Secure quotas for authenticated users

+1 (binding) -Harsha On Thu, Jul 7, 2016 at 2:55 PM Tom Crayford wrote: > +1 (non-binding) > > On Thursday, 7 July 2016, Rajini Sivaram > wrote: > > > I would like to initiate voting for KIP-55 ( > > > > >

Re: [VOTE] KIP-55: Secure quotas for authenticated users

+1 (non-binding) On Thursday, 7 July 2016, Rajini Sivaram wrote: > I would like to initiate voting for KIP-55 ( > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-55%3A+Secure+Quotas+for+Authenticated+Users > ). > Since the KIP has changed quite a lot since

[VOTE] KIP-55: Secure quotas for authenticated users

I would like to initiate voting for KIP-55 ( https://cwiki.apache.org/confluence/display/KAFKA/KIP-55%3A+Secure+Quotas+for+Authenticated+Users). Since the KIP has changed quite a lot since the last vote, we will discard the previous vote and start this new voting thread. KIP-55 extends the

Re: [VOTE] KIP-55: Secure quotas for authenticated users

Thank you, Jun. Hi all, Please let me know if you have any comments or suggestions on the updated KIP. If there are no objections, I will initiate voting next week. Thank you... On Thu, Jun 30, 2016 at 10:37 PM, Jun Rao wrote: > Rajini, > > The latest wiki looks good to

Re: [VOTE] KIP-55: Secure quotas for authenticated users

Rajini, The latest wiki looks good to me. Perhaps you want to ask other people to also take a look and then we can start the voting. Thanks, Jun On Tue, Jun 28, 2016 at 6:27 AM, Rajini Sivaram < rajinisiva...@googlemail.com> wrote: > Jun, > > Thank you for the review. I have changed all

Re: [VOTE] KIP-55: Secure quotas for authenticated users

Jun, Thank you for the review. I have changed all default property configs to be stored with the node name . So the defaults are /config/clients/ for default client-id quota, /config/users/ for default user quota and /config/users/ for default quota. Hope that makes sense. On

Re: [VOTE] KIP-55: Secure quotas for authenticated users

Rajini, Thanks for the update. Looks good to me. My only comment is that instead of /config/users//clients, would it be better to represent it as /config/users//clients/ so that it's more consistent? Jun On Thu, Jun 23, 2016 at 2:16 PM, Rajini Sivaram < rajinisiva...@googlemail.com> wrote: >

Re: [VOTE] KIP-55: Secure quotas for authenticated users

Jun, Yes, I agree that it makes sense to retain the existing semantics for client-id quotas for compatibility. Especially if we can provide the option to enable secure client-id quotas for multi-user clusters as well. I have updated the KIP - each of these levels can have defaults as well as

Re: [VOTE] KIP-55: Secure quotas for authenticated users

Hi, Rajini, For the following statements, would it be better to allocate the quota to all connections whose client-id is clientX? This way, existing client-id quotas are fully compatible in the new release whether the cluster is in a single user or multi-user environment. 4. If client-id quota

Re: [VOTE] KIP-55: Secure quotas for authenticated users

Ismael, Jun, Thank you both for the feedback. Have updated the KIP to add dynamic default quotas for client-id with deprecation of existing static default properties. On Wed, Jun 22, 2016 at 12:50 AM, Jun Rao wrote: > Yes, for consistency, perhaps we can allow client-id

Re: [VOTE] KIP-55: Secure quotas for authenticated users

Yes, for consistency, perhaps we can allow client-id quota to be configured dynamically too and mark the static config in the broker as deprecated. If both are set, the dynamic one wins. Thanks, Jun On Tue, Jun 21, 2016 at 3:56 AM, Ismael Juma wrote: > On Tue, Jun 21, 2016

Re: [VOTE] KIP-55: Secure quotas for authenticated users

On Tue, Jun 21, 2016 at 12:50 PM, Rajini Sivaram < rajinisiva...@googlemail.com> wrote: > It is actually quite tempting to do the same for client-id quotas as well, > but I suppose we can't break existing users who have configured defaults in > server.properties and providing two ways of setting

Re: [VOTE] KIP-55: Secure quotas for authenticated users

Jun, I think that is a good idea. It keeps all user quotas in one place with an intuitive hierarchy, avoids properties that are applied based on various conditions and also enables dynamic updates. Dynamic updates obviously need to get applied to all clients using the default, but I think that

Re: [VOTE] KIP-55: Secure quotas for authenticated users

Rajini, Another thing that's probably worth thinking through is whether it's better to make the default user quota dynamic as well. So, instead of adding quota.user.producer.default and quota.user.consumer.default in the broker config, another way is to set them using sth like the following.

Re: [VOTE] KIP-55: Secure quotas for authenticated users

Gwen/Jay, Have you had a chance to look at the updated KIP? It will be good to get your feedback as well before restarting vote on the updated KIP. If there are no objections, I will start the vote tomorrow. On Fri, Jun 17, 2016 at 6:59 PM, Rajini Sivaram < rajinisiva...@googlemail.com> wrote:

Re: [VOTE] KIP-55: Secure quotas for authenticated users

Thank you, Jun. I have removed user_principal from the KIP. On Fri, Jun 17, 2016 at 6:00 PM, Jun Rao wrote: > Rajini, > > 10. Yes, then we can probably leave out the user_principal field and keep > the version to be 1. > > Other than that, the KIP looks good to me. > >

Re: [VOTE] KIP-55: Secure quotas for authenticated users

Rajini, 10. Yes, then we can probably leave out the user_principal field and keep the version to be 1. Other than that, the KIP looks good to me. Thanks, Jun On Fri, Jun 17, 2016 at 3:29 AM, Rajini Sivaram < rajinisiva...@googlemail.com> wrote: > Jun, > > 10. Since entity_type "users" is

Re: [VOTE] KIP-55: Secure quotas for authenticated users

Jun, 10. Since entity_type "users" is new, shouldn't the JSON for these entities have version 1? I have moved "user_principal" out of the config in the samples and added to the entries as well. But actually, do we need to store the non-encoded principal at all? The node name is

Re: [VOTE] KIP-55: Secure quotas for authenticated users

Rajini, Thanks for the update. A few more questions/comments. 10. For the quota value stored in ZK, since we are adding an optional user_principal field in the json, we should bump the version from 1 to 2. Also, user_principal is not really part of the config values. So, perhaps we should

Re: [VOTE] KIP-55: Secure quotas for authenticated users

Jun, Actually, with quotas stored in different nodes in ZK, it is better to store remainder quota rather than total quota under /users/ so that quota calculations are not dependent on the order of notifications. I have updated the KIP to reflect that. So the quotas in ZK now always reflect the

Re: [VOTE] KIP-55: Secure quotas for authenticated users

Jun, Thank you for the review. I have updated the KIP: 1. Added an overview section. Slightly reworded since it is better to treat user and client-id as different levels rather than the same level. 2. Yes, it is neater to store quota for each entity in a different path in Zookeeper.

Re: [VOTE] KIP-55: Secure quotas for authenticated users

Hi, Rajini, Thanks for the updated wiki. Overall, I like the new approach. It covers the common use cases now, is extensible, and is backward compatible. A few comments below. 1. It would be useful to describe a bit at the high level, how the new approach works. I think this can be summarized as

Re: [VOTE] KIP-55: Secure quotas for authenticated users

Harsha, The sample configuration under https://cwiki.apache.org/confluence/display/KAFKA/KIP-55%3A+Secure+Quotas+for+Authenticated+Users#KIP-55:SecureQuotasforAuthenticatedUsers-QuotaConfiguration shows the Zookeeper data for different scenarios. - *user1* (/users/user1 in ZK) has only

Re: [VOTE] KIP-55: Secure quotas for authenticated users

Rajini, How does sub-quotas works in case of authenticated users. Where are we maintaining the relation between users and their client Ids. Can you add an example of zk data under /users. Thanks, Harsha On Mon, Jun 13, 2016, at 05:01 AM, Rajini Sivaram wrote: > I

Re: [VOTE] KIP-55: Secure quotas for authenticated users

I have updated KIP-55 to reflect the changes from the discussions in the voting thread ( https://www.mail-archive.com/dev@kafka.apache.org/msg51610.html). Jun/Gwen, Existing client-id quotas will be used as default client-id quotas for users when no user quotas are configured - i.e., default

Re: [VOTE] KIP-55: Secure quotas for authenticated users

Thank you all for the feedback. Closing this voting thread to continue discussions on the updated KIP in the discuss thread. On Sat, Jun 11, 2016 at 7:45 AM, Gwen Shapira wrote: > I'd also like to see clarification regarding the ZK structures. > Currently they appear as if

Re: [VOTE] KIP-55: Secure quotas for authenticated users

I'd also like to see clarification regarding the ZK structures. Currently they appear as if user-quotas and client-quotas are equivalent, but this will obviously need to change. On Sat, Jun 11, 2016 at 1:28 AM, Jun Rao wrote: > Rajini, > > The new proposal sounds good to me

Re: [VOTE] KIP-55: Secure quotas for authenticated users

Rajini, The new proposal sounds good to me too. My only question is what happens to those existing quotas on client-id. Do we just treat them as the quota for that client-id under ANONYMOUS user name? Thanks, Jun On Fri, Jun 10, 2016 at 2:43 PM, Rajini Sivaram < rajinisiva...@googlemail.com>

Re: [VOTE] KIP-55: Secure quotas for authenticated users

Jay, Thank you for the quick feedback. It shouldn't be too hard since I had a PR earlier along these lines anyway. Jun, are you ok with this approach? If everyone agrees, I will close this vote, update the KIP and give some more time for discussions. On Fri, Jun 10, 2016 at 10:27 PM, Jay Kreps

Re: [VOTE] KIP-55: Secure quotas for authenticated users

This sounds a lot better to me--hopefully it isn't too much harder! I do think if it is possible to do this directly that will be better for users than having an intermediate step since we always have to work through migrating people who have setup quotas already from the old way to the new way.

Re: [VOTE] KIP-55: Secure quotas for authenticated users

I do think client-id is a valid and useful grouping for quotas even in secure clusters - but only if clientA of user1 is treated as a different client-id from clientA of user2. Grouping of clients of a user enables users to allocate their quota effectively to their clients (eg. guarantee that

Re: [VOTE] KIP-55: Secure quotas for authenticated users

I am not crazy about modes either. An earlier proposal supported both client-ids and users at the same time, and it made more sense to me. I believe it was dropped without proper discussion (Basically, Jun mentioned it is complex and Rajini agreed to drop it). We should probably rethink the

Re: [VOTE] KIP-55: Secure quotas for authenticated users

Hey Rajini, 1. That makes sense to me. Is that reflected in the documentation anywhere (I couldn't really find it)? Is there a way to discover that definition? We do way better when we right this stuff down so it has an official definition users and developers can work off of... 2. If client id

Re: [VOTE] KIP-55: Secure quotas for authenticated users

Jay, Thank you for the feedback. 1. I think it is good to have a single concept of identity, but multiple ways of grouping clients for different functions. Client-id is a logical grouping of clients with a meaningful name that is used in client metrics and logs. User principal is an

Re: [VOTE] KIP-55: Secure quotas for authenticated users

Super sorry to come in late on this one. Rajini, I had two quick questions I think we should be able to answer: 1. Do client ids make sense in a world which has users? If not should we unify them the way Hadoop did (without auth the user is a kind of best effort honor system identity).

Re: [VOTE] KIP-55: Secure quotas for authenticated users

+1 (non-binding) On Wed, Jun 8, 2016 at 8:56 PM, Rajini Sivaram wrote: > I would like to initiate the vote for KIP-55. > > The KIP details are here: KIP-55: Secure quotas for authenticated users > < >

Re: [VOTE] KIP-55: Secure quotas for authenticated users

Jun, Oops, sorry, I hadn't realized that the last note was on the discuss thread. Thank you for pointing it out. I have sent another note for voting. On Wed, Jun 8, 2016 at 4:30 PM, Jun Rao wrote: > Rajini, > > Perhaps it will be clearer if you start the voting in a new

[VOTE] KIP-55: Secure quotas for authenticated users

I would like to initiate the vote for KIP-55. The KIP details are here: KIP-55: Secure quotas for authenticated users . The JIRA KAFKA-3492 has

Re: [VOTE] KIP-55: Secure quotas for authenticated users

Rajini, Perhaps it will be clearer if you start the voting in a new thread (with VOTE in the subject). Thanks, Jun On Tue, Jun 7, 2016 at 1:55 PM, Rajini Sivaram wrote: > I would like to initiate the vote for KIP-55. > > The KIP details are here: KIP-55: Secure

[VOTE] KIP-55: Secure quotas for authenticated users

I would like to initiate the vote for KIP-55. The KIP details are here: KIP-55: Secure quotas for authenticated users . The JIRA KAFKA-3492 has