Re: [VOTE] KIP-55: Secure quotas for authenticated users

[Rajini Sivaram]

Hi all,

This vote has passed with 3 binding and one non-binding +1. Many thanks to
those who voted.

If you have any comments or suggestions, please continue to add them to the
discussion thread.


On Fri, Jul 15, 2016 at 6:47 AM, Gwen Shapira  wrote:

> +1
>
> Thanks for working through this, Rajini.
>
> On Thu, Jul 7, 2016 at 7:12 AM, Rajini Sivaram
>  wrote:
> > I would like to initiate voting for KIP-55 (
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-55%3A+Secure+Quotas+for+Authenticated+Users
> ).
> > Since the KIP has changed quite a lot since the last vote, we will
> discard
> > the previous vote and start this new voting thread.
> >
> > KIP-55 extends the existing client-id quota implementation to enable
> secure
> > quotas for multi-user environments. The KIP proposes a flexible, unified
> > design that supports quotas at ,  or 
> > levels. It retains compatibility with the existing  quotas
> when
> > new user level quotas are not configured.
> >
> > Thank you...
> >
> >
> > Regards,
> >
> > Rajini
>



-- 
Regards,

Rajini

Re: [VOTE] KIP-55: Secure quotas for authenticated users

[Gwen Shapira]

+1

Thanks for working through this, Rajini.

On Thu, Jul 7, 2016 at 7:12 AM, Rajini Sivaram
 wrote:
> I would like to initiate voting for KIP-55 (
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-55%3A+Secure+Quotas+for+Authenticated+Users).
> Since the KIP has changed quite a lot since the last vote, we will discard
> the previous vote and start this new voting thread.
>
> KIP-55 extends the existing client-id quota implementation to enable secure
> quotas for multi-user environments. The KIP proposes a flexible, unified
> design that supports quotas at ,  or 
> levels. It retains compatibility with the existing  quotas when
> new user level quotas are not configured.
>
> Thank you...
>
>
> Regards,
>
> Rajini

Re: [VOTE] KIP-55: Secure quotas for authenticated users

[Jun Rao]

Rajani,

Thanks for the proposal. +1 from me.

Jun

On Thu, Jul 7, 2016 at 7:12 AM, Rajini Sivaram  wrote:

> I would like to initiate voting for KIP-55 (
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-55%3A+Secure+Quotas+for+Authenticated+Users
> ).
> Since the KIP has changed quite a lot since the last vote, we will discard
> the previous vote and start this new voting thread.
>
> KIP-55 extends the existing client-id quota implementation to enable secure
> quotas for multi-user environments. The KIP proposes a flexible, unified
> design that supports quotas at ,  or 
> levels. It retains compatibility with the existing  quotas when
> new user level quotas are not configured.
>
> Thank you...
>
>
> Regards,
>
> Rajini
>

Re: [VOTE] KIP-55: Secure quotas for authenticated users

[Harsha Chintalapani]

+1 (binding)
-Harsha
On Thu, Jul 7, 2016 at 2:55 PM Tom Crayford  wrote:

> +1 (non-binding)
>
> On Thursday, 7 July 2016, Rajini Sivaram 
> wrote:
>
> > I would like to initiate voting for KIP-55 (
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-55%3A+Secure+Quotas+for+Authenticated+Users
> > ).
> > Since the KIP has changed quite a lot since the last vote, we will
> discard
> > the previous vote and start this new voting thread.
> >
> > KIP-55 extends the existing client-id quota implementation to enable
> secure
> > quotas for multi-user environments. The KIP proposes a flexible, unified
> > design that supports quotas at ,  or 
> > levels. It retains compatibility with the existing  quotas
> when
> > new user level quotas are not configured.
> >
> > Thank you...
> >
> >
> > Regards,
> >
> > Rajini
> >
>

Re: [VOTE] KIP-55: Secure quotas for authenticated users

[Tom Crayford]

+1 (non-binding)

On Thursday, 7 July 2016, Rajini Sivaram 
wrote:

> I would like to initiate voting for KIP-55 (
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-55%3A+Secure+Quotas+for+Authenticated+Users
> ).
> Since the KIP has changed quite a lot since the last vote, we will discard
> the previous vote and start this new voting thread.
>
> KIP-55 extends the existing client-id quota implementation to enable secure
> quotas for multi-user environments. The KIP proposes a flexible, unified
> design that supports quotas at ,  or 
> levels. It retains compatibility with the existing  quotas when
> new user level quotas are not configured.
>
> Thank you...
>
>
> Regards,
>
> Rajini
>

[VOTE] KIP-55: Secure quotas for authenticated users

[Rajini Sivaram]

I would like to initiate voting for KIP-55 (
https://cwiki.apache.org/confluence/display/KAFKA/KIP-55%3A+Secure+Quotas+for+Authenticated+Users).
Since the KIP has changed quite a lot since the last vote, we will discard
the previous vote and start this new voting thread.

KIP-55 extends the existing client-id quota implementation to enable secure
quotas for multi-user environments. The KIP proposes a flexible, unified
design that supports quotas at ,  or 
levels. It retains compatibility with the existing  quotas when
new user level quotas are not configured.

Thank you...


Regards,

Rajini

Re: [VOTE] KIP-55: Secure quotas for authenticated users

[Rajini Sivaram]

Thank you, Jun.

Hi all,

Please let me know if you have any comments or suggestions on the updated
KIP. If there are no objections, I will initiate voting next week.

Thank you...


On Thu, Jun 30, 2016 at 10:37 PM, Jun Rao  wrote:

> Rajini,
>
> The latest wiki looks good to me. Perhaps you want to ask other people to
> also take a look and then we can start the voting.
>
> Thanks,
>
> Jun
>
> On Tue, Jun 28, 2016 at 6:27 AM, Rajini Sivaram <
> rajinisiva...@googlemail.com> wrote:
>
> > Jun,
> >
> > Thank you for the review. I have changed all default property configs to
> be
> > stored with the node name . So the defaults are
> > /config/clients/ for default client-id quota,
> > /config/users/ for default user quota and
> > /config/users/ for default 
> > quota. Hope that makes sense.
> >
> > On Mon, Jun 27, 2016 at 10:25 PM, Jun Rao  wrote:
> >
> > > Rajini,
> > >
> > > Thanks for the update. Looks good to me. My only comment is that
> > > instead of /config/users//clients,
> > > would it be better to represent it as
> > > /config/users//clients/
> > > so that it's more consistent?
> > >
> > > Jun
> > >
> > >
> > > On Thu, Jun 23, 2016 at 2:16 PM, Rajini Sivaram <
> > > rajinisiva...@googlemail.com> wrote:
> > >
> > > > Jun,
> > > >
> > > > Yes, I agree that it makes sense to retain the existing semantics for
> > > > client-id quotas for compatibility. Especially if we can provide the
> > > option
> > > > to enable secure client-id quotas for multi-user clusters as well.
> > > >
> > > > I have updated the KIP - each of these levels can have defaults as
> well
> > > as
> > > > specific entries:
> > > >
> > > >- /config/clients : Insecure  quotas with the same
> > > semantics
> > > >as now
> > > >- /config/users: User quotas
> > > >- /config/users/userA/clients:  quotas for userA
> > > >- /config/users//clients: Default 
> quotas
> > > >
> > > > Now it is fully flexible as well as compatible with the current
> > > > implementation. I used /config/users//clients rather than
> > > > /config/users/clients since "clients" is a valid (unlikely, but still
> > > > possible) user principal. I used , but it could be anything
> > that
> > > > is a valid Zookeeper node name, but not a valid URL-encoded name.
> > > >
> > > > Thank you,
> > > >
> > > > Rajini
> > > >
> > > > On Thu, Jun 23, 2016 at 3:43 PM, Jun Rao  wrote:
> > > >
> > > > > Hi, Rajini,
> > > > >
> > > > > For the following statements, would it be better to allocate the
> > quota
> > > to
> > > > > all connections whose client-id is clientX? This way, existing
> > > client-id
> > > > > quotas are fully compatible in the new release whether the cluster
> is
> > > in
> > > > a
> > > > > single user or multi-user environment.
> > > > >
> > > > > 4. If client-id quota override is defined for clientX in
> > > > > /config/clients/clientX, this quota is allocated for the sole use
> of
> > > > >  > > > > clientX>
> > > > > 5. If dynamic client-id default is configured in /config/clients,
> > this
> > > > > default quota is allocated for the sole use of 
> > > > > 6. If quota.producer.default is configured for the broker in
> > > > > server.properties, this default quota is allocated for the sole use
> > of
> > > > >  > > > > clientX>
> > > > >
> > > > > We can potentially add a default quota for both user and client at
> > path
> > > > > /config/users/clients?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > > On Wed, Jun 22, 2016 at 3:01 AM, Rajini Sivaram <
> > > > > rajinisiva...@googlemail.com> wrote:
> > > > >
> > > > > > Ismael, Jun,
> > > > > >
> > > > > > Thank you both for the feedback. Have updated the KIP to add
> > dynamic
> > > > > > default quotas for client-id with deprecation of existing static
> > > > default
> > > > > > properties.
> > > > > >
> > > > > >
> > > > > > On Wed, Jun 22, 2016 at 12:50 AM, Jun Rao 
> > wrote:
> > > > > >
> > > > > > > Yes, for consistency, perhaps we can allow client-id quota to
> be
> > > > > > configured
> > > > > > > dynamically too and mark the static config in the broker as
> > > > deprecated.
> > > > > > If
> > > > > > > both are set, the dynamic one wins.
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > > On Tue, Jun 21, 2016 at 3:56 AM, Ismael Juma <
> ism...@juma.me.uk>
> > > > > wrote:
> > > > > > >
> > > > > > > > On Tue, Jun 21, 2016 at 12:50 PM, Rajini Sivaram <
> > > > > > > > rajinisiva...@googlemail.com> wrote:
> > > > > > > >
> > > > > > > > > It is actually quite tempting to do the same for client-id
> > > quotas
> > > > > as
> > > > > > > > well,
> > > > > > > > > but I suppose we can't break existing users who have
> > configured
> > > > > > > defaults
> > > > > > > > in
> > > > > > > > > server.properties and providing two ways of setting
> client-id

Re: [VOTE] KIP-55: Secure quotas for authenticated users

[Jun Rao]

Rajini,

The latest wiki looks good to me. Perhaps you want to ask other people to
also take a look and then we can start the voting.

Thanks,

Jun

On Tue, Jun 28, 2016 at 6:27 AM, Rajini Sivaram <
rajinisiva...@googlemail.com> wrote:

> Jun,
>
> Thank you for the review. I have changed all default property configs to be
> stored with the node name . So the defaults are
> /config/clients/ for default client-id quota,
> /config/users/ for default user quota and
> /config/users/ for default 
> quota. Hope that makes sense.
>
> On Mon, Jun 27, 2016 at 10:25 PM, Jun Rao  wrote:
>
> > Rajini,
> >
> > Thanks for the update. Looks good to me. My only comment is that
> > instead of /config/users//clients,
> > would it be better to represent it as
> > /config/users//clients/
> > so that it's more consistent?
> >
> > Jun
> >
> >
> > On Thu, Jun 23, 2016 at 2:16 PM, Rajini Sivaram <
> > rajinisiva...@googlemail.com> wrote:
> >
> > > Jun,
> > >
> > > Yes, I agree that it makes sense to retain the existing semantics for
> > > client-id quotas for compatibility. Especially if we can provide the
> > option
> > > to enable secure client-id quotas for multi-user clusters as well.
> > >
> > > I have updated the KIP - each of these levels can have defaults as well
> > as
> > > specific entries:
> > >
> > >- /config/clients : Insecure  quotas with the same
> > semantics
> > >as now
> > >- /config/users: User quotas
> > >- /config/users/userA/clients:  quotas for userA
> > >- /config/users//clients: Default  quotas
> > >
> > > Now it is fully flexible as well as compatible with the current
> > > implementation. I used /config/users//clients rather than
> > > /config/users/clients since "clients" is a valid (unlikely, but still
> > > possible) user principal. I used , but it could be anything
> that
> > > is a valid Zookeeper node name, but not a valid URL-encoded name.
> > >
> > > Thank you,
> > >
> > > Rajini
> > >
> > > On Thu, Jun 23, 2016 at 3:43 PM, Jun Rao  wrote:
> > >
> > > > Hi, Rajini,
> > > >
> > > > For the following statements, would it be better to allocate the
> quota
> > to
> > > > all connections whose client-id is clientX? This way, existing
> > client-id
> > > > quotas are fully compatible in the new release whether the cluster is
> > in
> > > a
> > > > single user or multi-user environment.
> > > >
> > > > 4. If client-id quota override is defined for clientX in
> > > > /config/clients/clientX, this quota is allocated for the sole use of
> > > >  > > > clientX>
> > > > 5. If dynamic client-id default is configured in /config/clients,
> this
> > > > default quota is allocated for the sole use of 
> > > > 6. If quota.producer.default is configured for the broker in
> > > > server.properties, this default quota is allocated for the sole use
> of
> > > >  > > > clientX>
> > > >
> > > > We can potentially add a default quota for both user and client at
> path
> > > > /config/users/clients?
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Wed, Jun 22, 2016 at 3:01 AM, Rajini Sivaram <
> > > > rajinisiva...@googlemail.com> wrote:
> > > >
> > > > > Ismael, Jun,
> > > > >
> > > > > Thank you both for the feedback. Have updated the KIP to add
> dynamic
> > > > > default quotas for client-id with deprecation of existing static
> > > default
> > > > > properties.
> > > > >
> > > > >
> > > > > On Wed, Jun 22, 2016 at 12:50 AM, Jun Rao 
> wrote:
> > > > >
> > > > > > Yes, for consistency, perhaps we can allow client-id quota to be
> > > > > configured
> > > > > > dynamically too and mark the static config in the broker as
> > > deprecated.
> > > > > If
> > > > > > both are set, the dynamic one wins.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > > On Tue, Jun 21, 2016 at 3:56 AM, Ismael Juma 
> > > > wrote:
> > > > > >
> > > > > > > On Tue, Jun 21, 2016 at 12:50 PM, Rajini Sivaram <
> > > > > > > rajinisiva...@googlemail.com> wrote:
> > > > > > >
> > > > > > > > It is actually quite tempting to do the same for client-id
> > quotas
> > > > as
> > > > > > > well,
> > > > > > > > but I suppose we can't break existing users who have
> configured
> > > > > > defaults
> > > > > > > in
> > > > > > > > server.properties and providing two ways of setting client-id
> > > > > defaults
> > > > > > > > would be just too confusing.
> > > > > > > >
> > > > > > >
> > > > > > > Using two different approaches for client-id versus user quota
> > > > defaults
> > > > > > is
> > > > > > > also not great. We could deprecate the server.properties
> default
> > > > > configs
> > > > > > > for client-id quotas and remove them in the future. In the
> > > meantime,
> > > > we
> > > > > > > would have to live with 2 level defaults.
> > > > > > >
> > > > > > > Jun, what are your thoughts on this?
> > > > > > >
> > > > > > 

Re: [VOTE] KIP-55: Secure quotas for authenticated users

[Rajini Sivaram]

Jun,

Thank you for the review. I have changed all default property configs to be
stored with the node name . So the defaults are
/config/clients/ for default client-id quota,
/config/users/ for default user quota and
/config/users/ for default 
quota. Hope that makes sense.

On Mon, Jun 27, 2016 at 10:25 PM, Jun Rao  wrote:

> Rajini,
>
> Thanks for the update. Looks good to me. My only comment is that
> instead of /config/users//clients,
> would it be better to represent it as
> /config/users//clients/
> so that it's more consistent?
>
> Jun
>
>
> On Thu, Jun 23, 2016 at 2:16 PM, Rajini Sivaram <
> rajinisiva...@googlemail.com> wrote:
>
> > Jun,
> >
> > Yes, I agree that it makes sense to retain the existing semantics for
> > client-id quotas for compatibility. Especially if we can provide the
> option
> > to enable secure client-id quotas for multi-user clusters as well.
> >
> > I have updated the KIP - each of these levels can have defaults as well
> as
> > specific entries:
> >
> >- /config/clients : Insecure  quotas with the same
> semantics
> >as now
> >- /config/users: User quotas
> >- /config/users/userA/clients:  quotas for userA
> >- /config/users//clients: Default  quotas
> >
> > Now it is fully flexible as well as compatible with the current
> > implementation. I used /config/users//clients rather than
> > /config/users/clients since "clients" is a valid (unlikely, but still
> > possible) user principal. I used , but it could be anything that
> > is a valid Zookeeper node name, but not a valid URL-encoded name.
> >
> > Thank you,
> >
> > Rajini
> >
> > On Thu, Jun 23, 2016 at 3:43 PM, Jun Rao  wrote:
> >
> > > Hi, Rajini,
> > >
> > > For the following statements, would it be better to allocate the quota
> to
> > > all connections whose client-id is clientX? This way, existing
> client-id
> > > quotas are fully compatible in the new release whether the cluster is
> in
> > a
> > > single user or multi-user environment.
> > >
> > > 4. If client-id quota override is defined for clientX in
> > > /config/clients/clientX, this quota is allocated for the sole use of
> > >  > > clientX>
> > > 5. If dynamic client-id default is configured in /config/clients, this
> > > default quota is allocated for the sole use of 
> > > 6. If quota.producer.default is configured for the broker in
> > > server.properties, this default quota is allocated for the sole use of
> > >  > > clientX>
> > >
> > > We can potentially add a default quota for both user and client at path
> > > /config/users/clients?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Wed, Jun 22, 2016 at 3:01 AM, Rajini Sivaram <
> > > rajinisiva...@googlemail.com> wrote:
> > >
> > > > Ismael, Jun,
> > > >
> > > > Thank you both for the feedback. Have updated the KIP to add dynamic
> > > > default quotas for client-id with deprecation of existing static
> > default
> > > > properties.
> > > >
> > > >
> > > > On Wed, Jun 22, 2016 at 12:50 AM, Jun Rao  wrote:
> > > >
> > > > > Yes, for consistency, perhaps we can allow client-id quota to be
> > > > configured
> > > > > dynamically too and mark the static config in the broker as
> > deprecated.
> > > > If
> > > > > both are set, the dynamic one wins.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > > On Tue, Jun 21, 2016 at 3:56 AM, Ismael Juma 
> > > wrote:
> > > > >
> > > > > > On Tue, Jun 21, 2016 at 12:50 PM, Rajini Sivaram <
> > > > > > rajinisiva...@googlemail.com> wrote:
> > > > > >
> > > > > > > It is actually quite tempting to do the same for client-id
> quotas
> > > as
> > > > > > well,
> > > > > > > but I suppose we can't break existing users who have configured
> > > > > defaults
> > > > > > in
> > > > > > > server.properties and providing two ways of setting client-id
> > > > defaults
> > > > > > > would be just too confusing.
> > > > > > >
> > > > > >
> > > > > > Using two different approaches for client-id versus user quota
> > > defaults
> > > > > is
> > > > > > also not great. We could deprecate the server.properties default
> > > > configs
> > > > > > for client-id quotas and remove them in the future. In the
> > meantime,
> > > we
> > > > > > would have to live with 2 level defaults.
> > > > > >
> > > > > > Jun, what are your thoughts on this?
> > > > > >
> > > > > > Ismael
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Regards,
> > > >
> > > > Rajini
> > > >
> > >
> >
> >
> >
> > --
> > Regards,
> >
> > Rajini
> >
>



-- 
Regards,

Rajini

Re: [VOTE] KIP-55: Secure quotas for authenticated users

[Jun Rao]

Rajini,

Thanks for the update. Looks good to me. My only comment is that
instead of /config/users//clients,
would it be better to represent it as /config/users//clients/
so that it's more consistent?

Jun


On Thu, Jun 23, 2016 at 2:16 PM, Rajini Sivaram <
rajinisiva...@googlemail.com> wrote:

> Jun,
>
> Yes, I agree that it makes sense to retain the existing semantics for
> client-id quotas for compatibility. Especially if we can provide the option
> to enable secure client-id quotas for multi-user clusters as well.
>
> I have updated the KIP - each of these levels can have defaults as well as
> specific entries:
>
>- /config/clients : Insecure  quotas with the same semantics
>as now
>- /config/users: User quotas
>- /config/users/userA/clients:  quotas for userA
>- /config/users//clients: Default  quotas
>
> Now it is fully flexible as well as compatible with the current
> implementation. I used /config/users//clients rather than
> /config/users/clients since "clients" is a valid (unlikely, but still
> possible) user principal. I used , but it could be anything that
> is a valid Zookeeper node name, but not a valid URL-encoded name.
>
> Thank you,
>
> Rajini
>
> On Thu, Jun 23, 2016 at 3:43 PM, Jun Rao  wrote:
>
> > Hi, Rajini,
> >
> > For the following statements, would it be better to allocate the quota to
> > all connections whose client-id is clientX? This way, existing client-id
> > quotas are fully compatible in the new release whether the cluster is in
> a
> > single user or multi-user environment.
> >
> > 4. If client-id quota override is defined for clientX in
> > /config/clients/clientX, this quota is allocated for the sole use of
> >  > clientX>
> > 5. If dynamic client-id default is configured in /config/clients, this
> > default quota is allocated for the sole use of 
> > 6. If quota.producer.default is configured for the broker in
> > server.properties, this default quota is allocated for the sole use of
> >  > clientX>
> >
> > We can potentially add a default quota for both user and client at path
> > /config/users/clients?
> >
> > Thanks,
> >
> > Jun
> >
> > On Wed, Jun 22, 2016 at 3:01 AM, Rajini Sivaram <
> > rajinisiva...@googlemail.com> wrote:
> >
> > > Ismael, Jun,
> > >
> > > Thank you both for the feedback. Have updated the KIP to add dynamic
> > > default quotas for client-id with deprecation of existing static
> default
> > > properties.
> > >
> > >
> > > On Wed, Jun 22, 2016 at 12:50 AM, Jun Rao  wrote:
> > >
> > > > Yes, for consistency, perhaps we can allow client-id quota to be
> > > configured
> > > > dynamically too and mark the static config in the broker as
> deprecated.
> > > If
> > > > both are set, the dynamic one wins.
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Tue, Jun 21, 2016 at 3:56 AM, Ismael Juma 
> > wrote:
> > > >
> > > > > On Tue, Jun 21, 2016 at 12:50 PM, Rajini Sivaram <
> > > > > rajinisiva...@googlemail.com> wrote:
> > > > >
> > > > > > It is actually quite tempting to do the same for client-id quotas
> > as
> > > > > well,
> > > > > > but I suppose we can't break existing users who have configured
> > > > defaults
> > > > > in
> > > > > > server.properties and providing two ways of setting client-id
> > > defaults
> > > > > > would be just too confusing.
> > > > > >
> > > > >
> > > > > Using two different approaches for client-id versus user quota
> > defaults
> > > > is
> > > > > also not great. We could deprecate the server.properties default
> > > configs
> > > > > for client-id quotas and remove them in the future. In the
> meantime,
> > we
> > > > > would have to live with 2 level defaults.
> > > > >
> > > > > Jun, what are your thoughts on this?
> > > > >
> > > > > Ismael
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Regards,
> > >
> > > Rajini
> > >
> >
>
>
>
> --
> Regards,
>
> Rajini
>

Re: [VOTE] KIP-55: Secure quotas for authenticated users

[Rajini Sivaram]

Jun,

Yes, I agree that it makes sense to retain the existing semantics for
client-id quotas for compatibility. Especially if we can provide the option
to enable secure client-id quotas for multi-user clusters as well.

I have updated the KIP - each of these levels can have defaults as well as
specific entries:

   - /config/clients : Insecure  quotas with the same semantics
   as now
   - /config/users: User quotas
   - /config/users/userA/clients:  quotas for userA
   - /config/users//clients: Default  quotas

Now it is fully flexible as well as compatible with the current
implementation. I used /config/users//clients rather than
/config/users/clients since "clients" is a valid (unlikely, but still
possible) user principal. I used , but it could be anything that
is a valid Zookeeper node name, but not a valid URL-encoded name.

Thank you,

Rajini

On Thu, Jun 23, 2016 at 3:43 PM, Jun Rao  wrote:

> Hi, Rajini,
>
> For the following statements, would it be better to allocate the quota to
> all connections whose client-id is clientX? This way, existing client-id
> quotas are fully compatible in the new release whether the cluster is in a
> single user or multi-user environment.
>
> 4. If client-id quota override is defined for clientX in
> /config/clients/clientX, this quota is allocated for the sole use of
>  clientX>
> 5. If dynamic client-id default is configured in /config/clients, this
> default quota is allocated for the sole use of 
> 6. If quota.producer.default is configured for the broker in
> server.properties, this default quota is allocated for the sole use of
>  clientX>
>
> We can potentially add a default quota for both user and client at path
> /config/users/clients?
>
> Thanks,
>
> Jun
>
> On Wed, Jun 22, 2016 at 3:01 AM, Rajini Sivaram <
> rajinisiva...@googlemail.com> wrote:
>
> > Ismael, Jun,
> >
> > Thank you both for the feedback. Have updated the KIP to add dynamic
> > default quotas for client-id with deprecation of existing static default
> > properties.
> >
> >
> > On Wed, Jun 22, 2016 at 12:50 AM, Jun Rao  wrote:
> >
> > > Yes, for consistency, perhaps we can allow client-id quota to be
> > configured
> > > dynamically too and mark the static config in the broker as deprecated.
> > If
> > > both are set, the dynamic one wins.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Tue, Jun 21, 2016 at 3:56 AM, Ismael Juma 
> wrote:
> > >
> > > > On Tue, Jun 21, 2016 at 12:50 PM, Rajini Sivaram <
> > > > rajinisiva...@googlemail.com> wrote:
> > > >
> > > > > It is actually quite tempting to do the same for client-id quotas
> as
> > > > well,
> > > > > but I suppose we can't break existing users who have configured
> > > defaults
> > > > in
> > > > > server.properties and providing two ways of setting client-id
> > defaults
> > > > > would be just too confusing.
> > > > >
> > > >
> > > > Using two different approaches for client-id versus user quota
> defaults
> > > is
> > > > also not great. We could deprecate the server.properties default
> > configs
> > > > for client-id quotas and remove them in the future. In the meantime,
> we
> > > > would have to live with 2 level defaults.
> > > >
> > > > Jun, what are your thoughts on this?
> > > >
> > > > Ismael
> > > >
> > >
> >
> >
> >
> > --
> > Regards,
> >
> > Rajini
> >
>



-- 
Regards,

Rajini

Re: [VOTE] KIP-55: Secure quotas for authenticated users

[Jun Rao]

Hi, Rajini,

For the following statements, would it be better to allocate the quota to
all connections whose client-id is clientX? This way, existing client-id
quotas are fully compatible in the new release whether the cluster is in a
single user or multi-user environment.

4. If client-id quota override is defined for clientX in
/config/clients/clientX, this quota is allocated for the sole use of 
5. If dynamic client-id default is configured in /config/clients, this
default quota is allocated for the sole use of 
6. If quota.producer.default is configured for the broker in
server.properties, this default quota is allocated for the sole use of 

We can potentially add a default quota for both user and client at path
/config/users/clients?

Thanks,

Jun

On Wed, Jun 22, 2016 at 3:01 AM, Rajini Sivaram <
rajinisiva...@googlemail.com> wrote:

> Ismael, Jun,
>
> Thank you both for the feedback. Have updated the KIP to add dynamic
> default quotas for client-id with deprecation of existing static default
> properties.
>
>
> On Wed, Jun 22, 2016 at 12:50 AM, Jun Rao  wrote:
>
> > Yes, for consistency, perhaps we can allow client-id quota to be
> configured
> > dynamically too and mark the static config in the broker as deprecated.
> If
> > both are set, the dynamic one wins.
> >
> > Thanks,
> >
> > Jun
> >
> > On Tue, Jun 21, 2016 at 3:56 AM, Ismael Juma  wrote:
> >
> > > On Tue, Jun 21, 2016 at 12:50 PM, Rajini Sivaram <
> > > rajinisiva...@googlemail.com> wrote:
> > >
> > > > It is actually quite tempting to do the same for client-id quotas as
> > > well,
> > > > but I suppose we can't break existing users who have configured
> > defaults
> > > in
> > > > server.properties and providing two ways of setting client-id
> defaults
> > > > would be just too confusing.
> > > >
> > >
> > > Using two different approaches for client-id versus user quota defaults
> > is
> > > also not great. We could deprecate the server.properties default
> configs
> > > for client-id quotas and remove them in the future. In the meantime, we
> > > would have to live with 2 level defaults.
> > >
> > > Jun, what are your thoughts on this?
> > >
> > > Ismael
> > >
> >
>
>
>
> --
> Regards,
>
> Rajini
>

Re: [VOTE] KIP-55: Secure quotas for authenticated users

[Rajini Sivaram]

Ismael, Jun,

Thank you both for the feedback. Have updated the KIP to add dynamic
default quotas for client-id with deprecation of existing static default
properties.


On Wed, Jun 22, 2016 at 12:50 AM, Jun Rao  wrote:

> Yes, for consistency, perhaps we can allow client-id quota to be configured
> dynamically too and mark the static config in the broker as deprecated. If
> both are set, the dynamic one wins.
>
> Thanks,
>
> Jun
>
> On Tue, Jun 21, 2016 at 3:56 AM, Ismael Juma  wrote:
>
> > On Tue, Jun 21, 2016 at 12:50 PM, Rajini Sivaram <
> > rajinisiva...@googlemail.com> wrote:
> >
> > > It is actually quite tempting to do the same for client-id quotas as
> > well,
> > > but I suppose we can't break existing users who have configured
> defaults
> > in
> > > server.properties and providing two ways of setting client-id defaults
> > > would be just too confusing.
> > >
> >
> > Using two different approaches for client-id versus user quota defaults
> is
> > also not great. We could deprecate the server.properties default configs
> > for client-id quotas and remove them in the future. In the meantime, we
> > would have to live with 2 level defaults.
> >
> > Jun, what are your thoughts on this?
> >
> > Ismael
> >
>



-- 
Regards,

Rajini

Re: [VOTE] KIP-55: Secure quotas for authenticated users

[Jun Rao]

Yes, for consistency, perhaps we can allow client-id quota to be configured
dynamically too and mark the static config in the broker as deprecated. If
both are set, the dynamic one wins.

Thanks,

Jun

On Tue, Jun 21, 2016 at 3:56 AM, Ismael Juma  wrote:

> On Tue, Jun 21, 2016 at 12:50 PM, Rajini Sivaram <
> rajinisiva...@googlemail.com> wrote:
>
> > It is actually quite tempting to do the same for client-id quotas as
> well,
> > but I suppose we can't break existing users who have configured defaults
> in
> > server.properties and providing two ways of setting client-id defaults
> > would be just too confusing.
> >
>
> Using two different approaches for client-id versus user quota defaults is
> also not great. We could deprecate the server.properties default configs
> for client-id quotas and remove them in the future. In the meantime, we
> would have to live with 2 level defaults.
>
> Jun, what are your thoughts on this?
>
> Ismael
>

Re: [VOTE] KIP-55: Secure quotas for authenticated users

[Ismael Juma]

On Tue, Jun 21, 2016 at 12:50 PM, Rajini Sivaram <
rajinisiva...@googlemail.com> wrote:

> It is actually quite tempting to do the same for client-id quotas as well,
> but I suppose we can't break existing users who have configured defaults in
> server.properties and providing two ways of setting client-id defaults
> would be just too confusing.
>

Using two different approaches for client-id versus user quota defaults is
also not great. We could deprecate the server.properties default configs
for client-id quotas and remove them in the future. In the meantime, we
would have to live with 2 level defaults.

Jun, what are your thoughts on this?

Ismael

Re: [VOTE] KIP-55: Secure quotas for authenticated users

[Rajini Sivaram]

Jun,

I think that is a good idea. It keeps all user quotas in one place with an
intuitive hierarchy, avoids properties that are applied based on various
conditions and also enables dynamic updates. Dynamic updates obviously need
to get applied to all clients using the default, but I think that should be
ok.

It is actually quite tempting to do the same for client-id quotas as well,
but I suppose we can't break existing users who have configured defaults in
server.properties and providing two ways of setting client-id defaults
would be just too confusing.

I have updated the KIP to remove the default user quota properties and set
default user quotas in /config/users.

Thank you,

Rajini

On Tue, Jun 21, 2016 at 6:04 AM, Jun Rao  wrote:

> Rajini,
>
> Another thing that's probably worth thinking through is whether it's better
> to make the default user quota dynamic as well. So, instead of adding
> quota.user.producer.default
> and quota.user.consumer.default in the broker config, another way is to set
> them using sth like the following.
>
> bin/kafka-configs  --zookeeper localhost:2181 --alter --add-config
> 'producer_byte_rate=10,consumer_byte_rate=20' --entity-name=* --entity-type
> users
>
> We may add other types of quotas in the future and we probably don't want
> to keep adding static configs.
>
> Thanks,
>
> Jun
>
>
>
> On Mon, Jun 20, 2016 at 2:32 AM, Rajini Sivaram <
> rajinisiva...@googlemail.com> wrote:
>
> > Gwen/Jay,
> >
> > Have you had a chance to look at the updated KIP? It will be good to get
> > your feedback as well before restarting vote on the updated KIP.
> >
> > If there are no objections, I will start the vote tomorrow.
> >
> >
> > On Fri, Jun 17, 2016 at 6:59 PM, Rajini Sivaram <
> > rajinisiva...@googlemail.com> wrote:
> >
> > > Thank you, Jun. I have removed user_principal from the KIP.
> > >
> > > On Fri, Jun 17, 2016 at 6:00 PM, Jun Rao  wrote:
> > >
> > >> Rajini,
> > >>
> > >> 10. Yes, then we can probably leave out the user_principal field and
> > keep
> > >> the version to be 1.
> > >>
> > >> Other than that, the KIP looks good to me.
> > >>
> > >> Thanks,
> > >>
> > >> Jun
> > >>
> > >> On Fri, Jun 17, 2016 at 3:29 AM, Rajini Sivaram <
> > >> rajinisiva...@googlemail.com> wrote:
> > >>
> > >> > Jun,
> > >> >
> > >> > 10. Since entity_type "users" is new, shouldn't the JSON for these
> > >> entities
> > >> > have version 1? I have moved "user_principal" out of the config in
> the
> > >> > samples and added to the  entries as well. But
> actually,
> > >> do
> > >> > we need to store the non-encoded principal at all? The node name is
> > >> > URL-encoded user principal, so it is fairly readable if you are
> > looking
> > >> in
> > >> > ZK and *kafka_configs.sh* will show the non-encoded principal by
> > >> decoding
> > >> > the name from the path (since it needs to do encoding anyway because
> > the
> > >> > names specified on the command line will be non-encoded principals,
> it
> > >> can
> > >> > do decoding too). Perhaps that is sufficient?
> > >> >
> > >> > 11. I liked the second approach since it looks neat and
> future-proof.
> > >> Have
> > >> > updated the KIP.
> > >> >
> > >> > 12. Yes, that is correct.
> > >> >
> > >> > Many thanks,
> > >> >
> > >> > Rajini
> > >> >
> > >> >
> > >> > On Thu, Jun 16, 2016 at 11:36 PM, Jun Rao  wrote:
> > >> >
> > >> > > Rajini,
> > >> > >
> > >> > > Thanks for the update. A few more questions/comments.
> > >> > >
> > >> > > 10. For the quota value stored in ZK, since we are adding an
> > optional
> > >> > > user_principal field in the json, we should bump the version from
> 1
> > >> to 2.
> > >> > > Also, user_principal is not really part of the config values. So,
> > >> perhaps
> > >> > > we should represent it as the following?
> > >> > > {
> > >> > > "version":2,
> > >> > > "config": {
> > >> > > "producer_byte_rate":"1024",
> > >> > > "consumer_byte_rate":"2048"
> > >> > > },
> > >> > > "user_principal" : "user1"
> > >> > > }
> > >> > >
> > >> > >  Also, we should store user_principal in the following json too,
> > >> right?
> > >> > > // Zookeeper persistence path
> /users//clients/clientA
> > >> > > {
> > >> > > "version":1,
> > >> > > "config": {
> > >> > > "producer_byte_rate":"10",
> > >> > > "consumer_byte_rate":"30"
> > >> > > }
> > >> > > }
> > >> > >
> > >> > > 11. For the change notification path, would it be better to change
> > it
> > >> to
> > >> > > something like the following and bump up version to 2?
> > >> > > // Change notification for quota of 
> > >> > > {
> > >> > > "version":2,
> > >> > > [
> > >> > >   { "entity_type": "users",
> > >> > > "entity_name": "user2"
> > >> > >   },
> > >> > >   { "entity_type": "clients",
> > >> > > "entity_name": "clientA"
> > >> > >   }
> > >> > > ]
> > >> > >  }
> > >> > >
> > >> > > 

Re: [VOTE] KIP-55: Secure quotas for authenticated users

[Jun Rao]

Rajini,

Another thing that's probably worth thinking through is whether it's better
to make the default user quota dynamic as well. So, instead of adding
quota.user.producer.default
and quota.user.consumer.default in the broker config, another way is to set
them using sth like the following.

bin/kafka-configs  --zookeeper localhost:2181 --alter --add-config
'producer_byte_rate=10,consumer_byte_rate=20' --entity-name=* --entity-type
users

We may add other types of quotas in the future and we probably don't want
to keep adding static configs.

Thanks,

Jun



On Mon, Jun 20, 2016 at 2:32 AM, Rajini Sivaram <
rajinisiva...@googlemail.com> wrote:

> Gwen/Jay,
>
> Have you had a chance to look at the updated KIP? It will be good to get
> your feedback as well before restarting vote on the updated KIP.
>
> If there are no objections, I will start the vote tomorrow.
>
>
> On Fri, Jun 17, 2016 at 6:59 PM, Rajini Sivaram <
> rajinisiva...@googlemail.com> wrote:
>
> > Thank you, Jun. I have removed user_principal from the KIP.
> >
> > On Fri, Jun 17, 2016 at 6:00 PM, Jun Rao  wrote:
> >
> >> Rajini,
> >>
> >> 10. Yes, then we can probably leave out the user_principal field and
> keep
> >> the version to be 1.
> >>
> >> Other than that, the KIP looks good to me.
> >>
> >> Thanks,
> >>
> >> Jun
> >>
> >> On Fri, Jun 17, 2016 at 3:29 AM, Rajini Sivaram <
> >> rajinisiva...@googlemail.com> wrote:
> >>
> >> > Jun,
> >> >
> >> > 10. Since entity_type "users" is new, shouldn't the JSON for these
> >> entities
> >> > have version 1? I have moved "user_principal" out of the config in the
> >> > samples and added to the  entries as well. But actually,
> >> do
> >> > we need to store the non-encoded principal at all? The node name is
> >> > URL-encoded user principal, so it is fairly readable if you are
> looking
> >> in
> >> > ZK and *kafka_configs.sh* will show the non-encoded principal by
> >> decoding
> >> > the name from the path (since it needs to do encoding anyway because
> the
> >> > names specified on the command line will be non-encoded principals, it
> >> can
> >> > do decoding too). Perhaps that is sufficient?
> >> >
> >> > 11. I liked the second approach since it looks neat and future-proof.
> >> Have
> >> > updated the KIP.
> >> >
> >> > 12. Yes, that is correct.
> >> >
> >> > Many thanks,
> >> >
> >> > Rajini
> >> >
> >> >
> >> > On Thu, Jun 16, 2016 at 11:36 PM, Jun Rao  wrote:
> >> >
> >> > > Rajini,
> >> > >
> >> > > Thanks for the update. A few more questions/comments.
> >> > >
> >> > > 10. For the quota value stored in ZK, since we are adding an
> optional
> >> > > user_principal field in the json, we should bump the version from 1
> >> to 2.
> >> > > Also, user_principal is not really part of the config values. So,
> >> perhaps
> >> > > we should represent it as the following?
> >> > > {
> >> > > "version":2,
> >> > > "config": {
> >> > > "producer_byte_rate":"1024",
> >> > > "consumer_byte_rate":"2048"
> >> > > },
> >> > > "user_principal" : "user1"
> >> > > }
> >> > >
> >> > >  Also, we should store user_principal in the following json too,
> >> right?
> >> > > // Zookeeper persistence path /users//clients/clientA
> >> > > {
> >> > > "version":1,
> >> > > "config": {
> >> > > "producer_byte_rate":"10",
> >> > > "consumer_byte_rate":"30"
> >> > > }
> >> > > }
> >> > >
> >> > > 11. For the change notification path, would it be better to change
> it
> >> to
> >> > > something like the following and bump up version to 2?
> >> > > // Change notification for quota of 
> >> > > {
> >> > > "version":2,
> >> > > [
> >> > >   { "entity_type": "users",
> >> > > "entity_name": "user2"
> >> > >   },
> >> > >   { "entity_type": "clients",
> >> > > "entity_name": "clientA"
> >> > >   }
> >> > > ]
> >> > >  }
> >> > >
> >> > > Alternatively, we could change it to
> >> > > // Change notification for quota of 
> >> > > {
> >> > > "version":2,
> >> > > "entity_path": "users/user2"
> >> > > }
> >> > >
> >> > > {
> >> > > "version":2,
> >> > > "entity_path": "users/user2/clients/clientA"
> >> > > }
> >> > >
> >> > > 12. Just to clarify on the meaning of remainder quota. If you have
> >> quotas
> >> > > like the following,
> >> > >   = 5
> >> > >   = 10
> >> > >   = 12
> >> > > it means that all connections with user1 whose client-id is neither
> >> > client1
> >> > > nor client2 will be sharing a quota of 12, right? In other words,
> the
> >> > quota
> >> > > of  doesn't include the quota for  and
>  >> > > client2>.
> >> > >
> >> > > Thanks,
> >> > >
> >> > > Jun
> >> > >
> >> > >
> >> > > On Thu, Jun 16, 2016 at 5:03 AM, Rajini Sivaram <
> >> > > rajinisiva...@googlemail.com> wrote:
> >> > >
> >> > > > Jun,
> >> > > >
> >> > > > Actually, with quotas stored in 

Re: [VOTE] KIP-55: Secure quotas for authenticated users

[Rajini Sivaram]

Gwen/Jay,

Have you had a chance to look at the updated KIP? It will be good to get
your feedback as well before restarting vote on the updated KIP.

If there are no objections, I will start the vote tomorrow.


On Fri, Jun 17, 2016 at 6:59 PM, Rajini Sivaram <
rajinisiva...@googlemail.com> wrote:

> Thank you, Jun. I have removed user_principal from the KIP.
>
> On Fri, Jun 17, 2016 at 6:00 PM, Jun Rao  wrote:
>
>> Rajini,
>>
>> 10. Yes, then we can probably leave out the user_principal field and keep
>> the version to be 1.
>>
>> Other than that, the KIP looks good to me.
>>
>> Thanks,
>>
>> Jun
>>
>> On Fri, Jun 17, 2016 at 3:29 AM, Rajini Sivaram <
>> rajinisiva...@googlemail.com> wrote:
>>
>> > Jun,
>> >
>> > 10. Since entity_type "users" is new, shouldn't the JSON for these
>> entities
>> > have version 1? I have moved "user_principal" out of the config in the
>> > samples and added to the  entries as well. But actually,
>> do
>> > we need to store the non-encoded principal at all? The node name is
>> > URL-encoded user principal, so it is fairly readable if you are looking
>> in
>> > ZK and *kafka_configs.sh* will show the non-encoded principal by
>> decoding
>> > the name from the path (since it needs to do encoding anyway because the
>> > names specified on the command line will be non-encoded principals, it
>> can
>> > do decoding too). Perhaps that is sufficient?
>> >
>> > 11. I liked the second approach since it looks neat and future-proof.
>> Have
>> > updated the KIP.
>> >
>> > 12. Yes, that is correct.
>> >
>> > Many thanks,
>> >
>> > Rajini
>> >
>> >
>> > On Thu, Jun 16, 2016 at 11:36 PM, Jun Rao  wrote:
>> >
>> > > Rajini,
>> > >
>> > > Thanks for the update. A few more questions/comments.
>> > >
>> > > 10. For the quota value stored in ZK, since we are adding an optional
>> > > user_principal field in the json, we should bump the version from 1
>> to 2.
>> > > Also, user_principal is not really part of the config values. So,
>> perhaps
>> > > we should represent it as the following?
>> > > {
>> > > "version":2,
>> > > "config": {
>> > > "producer_byte_rate":"1024",
>> > > "consumer_byte_rate":"2048"
>> > > },
>> > > "user_principal" : "user1"
>> > > }
>> > >
>> > >  Also, we should store user_principal in the following json too,
>> right?
>> > > // Zookeeper persistence path /users//clients/clientA
>> > > {
>> > > "version":1,
>> > > "config": {
>> > > "producer_byte_rate":"10",
>> > > "consumer_byte_rate":"30"
>> > > }
>> > > }
>> > >
>> > > 11. For the change notification path, would it be better to change it
>> to
>> > > something like the following and bump up version to 2?
>> > > // Change notification for quota of 
>> > > {
>> > > "version":2,
>> > > [
>> > >   { "entity_type": "users",
>> > > "entity_name": "user2"
>> > >   },
>> > >   { "entity_type": "clients",
>> > > "entity_name": "clientA"
>> > >   }
>> > > ]
>> > >  }
>> > >
>> > > Alternatively, we could change it to
>> > > // Change notification for quota of 
>> > > {
>> > > "version":2,
>> > > "entity_path": "users/user2"
>> > > }
>> > >
>> > > {
>> > > "version":2,
>> > > "entity_path": "users/user2/clients/clientA"
>> > > }
>> > >
>> > > 12. Just to clarify on the meaning of remainder quota. If you have
>> quotas
>> > > like the following,
>> > >   = 5
>> > >   = 10
>> > >   = 12
>> > > it means that all connections with user1 whose client-id is neither
>> > client1
>> > > nor client2 will be sharing a quota of 12, right? In other words, the
>> > quota
>> > > of  doesn't include the quota for  and > > > client2>.
>> > >
>> > > Thanks,
>> > >
>> > > Jun
>> > >
>> > >
>> > > On Thu, Jun 16, 2016 at 5:03 AM, Rajini Sivaram <
>> > > rajinisiva...@googlemail.com> wrote:
>> > >
>> > > > Jun,
>> > > >
>> > > > Actually, with quotas stored in different nodes in ZK, it is better
>> to
>> > > > store remainder quota rather than total quota under /users/ so
>> > that
>> > > > quota calculations are not dependent on the order of notifications.
>> I
>> > > have
>> > > > updated the KIP to reflect that. So the quotas in ZK now always
>> reflect
>> > > the
>> > > > quota applied to a group of client connections and use the same
>> format
>> > as
>> > > > client-id quotas. But it is not hierarchical, making the
>> configuration
>> > > > simpler.
>> > > >
>> > > > On Thu, Jun 16, 2016 at 11:49 AM, Rajini Sivaram <
>> > > > rajinisiva...@googlemail.com> wrote:
>> > > >
>> > > > > Jun,
>> > > > >
>> > > > > Thank you for the review. I have updated the KIP:
>> > > > >
>> > > > >
>> > > > >1. Added an overview section. Slightly reworded since it is
>> better
>> > > to
>> > > > >treat user and client-id as different levels rather than the
>> same
>> > > > level.
>> > > > >

Re: [VOTE] KIP-55: Secure quotas for authenticated users

[Rajini Sivaram]

Thank you, Jun. I have removed user_principal from the KIP.

On Fri, Jun 17, 2016 at 6:00 PM, Jun Rao  wrote:

> Rajini,
>
> 10. Yes, then we can probably leave out the user_principal field and keep
> the version to be 1.
>
> Other than that, the KIP looks good to me.
>
> Thanks,
>
> Jun
>
> On Fri, Jun 17, 2016 at 3:29 AM, Rajini Sivaram <
> rajinisiva...@googlemail.com> wrote:
>
> > Jun,
> >
> > 10. Since entity_type "users" is new, shouldn't the JSON for these
> entities
> > have version 1? I have moved "user_principal" out of the config in the
> > samples and added to the  entries as well. But actually, do
> > we need to store the non-encoded principal at all? The node name is
> > URL-encoded user principal, so it is fairly readable if you are looking
> in
> > ZK and *kafka_configs.sh* will show the non-encoded principal by decoding
> > the name from the path (since it needs to do encoding anyway because the
> > names specified on the command line will be non-encoded principals, it
> can
> > do decoding too). Perhaps that is sufficient?
> >
> > 11. I liked the second approach since it looks neat and future-proof.
> Have
> > updated the KIP.
> >
> > 12. Yes, that is correct.
> >
> > Many thanks,
> >
> > Rajini
> >
> >
> > On Thu, Jun 16, 2016 at 11:36 PM, Jun Rao  wrote:
> >
> > > Rajini,
> > >
> > > Thanks for the update. A few more questions/comments.
> > >
> > > 10. For the quota value stored in ZK, since we are adding an optional
> > > user_principal field in the json, we should bump the version from 1 to
> 2.
> > > Also, user_principal is not really part of the config values. So,
> perhaps
> > > we should represent it as the following?
> > > {
> > > "version":2,
> > > "config": {
> > > "producer_byte_rate":"1024",
> > > "consumer_byte_rate":"2048"
> > > },
> > > "user_principal" : "user1"
> > > }
> > >
> > >  Also, we should store user_principal in the following json too, right?
> > > // Zookeeper persistence path /users//clients/clientA
> > > {
> > > "version":1,
> > > "config": {
> > > "producer_byte_rate":"10",
> > > "consumer_byte_rate":"30"
> > > }
> > > }
> > >
> > > 11. For the change notification path, would it be better to change it
> to
> > > something like the following and bump up version to 2?
> > > // Change notification for quota of 
> > > {
> > > "version":2,
> > > [
> > >   { "entity_type": "users",
> > > "entity_name": "user2"
> > >   },
> > >   { "entity_type": "clients",
> > > "entity_name": "clientA"
> > >   }
> > > ]
> > >  }
> > >
> > > Alternatively, we could change it to
> > > // Change notification for quota of 
> > > {
> > > "version":2,
> > > "entity_path": "users/user2"
> > > }
> > >
> > > {
> > > "version":2,
> > > "entity_path": "users/user2/clients/clientA"
> > > }
> > >
> > > 12. Just to clarify on the meaning of remainder quota. If you have
> quotas
> > > like the following,
> > >   = 5
> > >   = 10
> > >   = 12
> > > it means that all connections with user1 whose client-id is neither
> > client1
> > > nor client2 will be sharing a quota of 12, right? In other words, the
> > quota
> > > of  doesn't include the quota for  and  > > client2>.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Thu, Jun 16, 2016 at 5:03 AM, Rajini Sivaram <
> > > rajinisiva...@googlemail.com> wrote:
> > >
> > > > Jun,
> > > >
> > > > Actually, with quotas stored in different nodes in ZK, it is better
> to
> > > > store remainder quota rather than total quota under /users/ so
> > that
> > > > quota calculations are not dependent on the order of notifications. I
> > > have
> > > > updated the KIP to reflect that. So the quotas in ZK now always
> reflect
> > > the
> > > > quota applied to a group of client connections and use the same
> format
> > as
> > > > client-id quotas. But it is not hierarchical, making the
> configuration
> > > > simpler.
> > > >
> > > > On Thu, Jun 16, 2016 at 11:49 AM, Rajini Sivaram <
> > > > rajinisiva...@googlemail.com> wrote:
> > > >
> > > > > Jun,
> > > > >
> > > > > Thank you for the review. I have updated the KIP:
> > > > >
> > > > >
> > > > >1. Added an overview section. Slightly reworded since it is
> better
> > > to
> > > > >treat user and client-id as different levels rather than the
> same
> > > > level.
> > > > >2. Yes, it is neater to store quota for each entity in a
> different
> > > > >path in Zookeeper. I put clients under users rather than the
> other
> > > way
> > > > >round since that reflects the hierarchy and also keeps a user's
> > > quotas
> > > > >together under a single sub-tree. I had initially used a single
> > node
> > > > to
> > > > >keep quotas and sub-quotas of a user together so that updates
> are
> > > > atomic
> > > > >since changes 

Re: [VOTE] KIP-55: Secure quotas for authenticated users

[Jun Rao]

Rajini,

10. Yes, then we can probably leave out the user_principal field and keep
the version to be 1.

Other than that, the KIP looks good to me.

Thanks,

Jun

On Fri, Jun 17, 2016 at 3:29 AM, Rajini Sivaram <
rajinisiva...@googlemail.com> wrote:

> Jun,
>
> 10. Since entity_type "users" is new, shouldn't the JSON for these entities
> have version 1? I have moved "user_principal" out of the config in the
> samples and added to the  entries as well. But actually, do
> we need to store the non-encoded principal at all? The node name is
> URL-encoded user principal, so it is fairly readable if you are looking in
> ZK and *kafka_configs.sh* will show the non-encoded principal by decoding
> the name from the path (since it needs to do encoding anyway because the
> names specified on the command line will be non-encoded principals, it can
> do decoding too). Perhaps that is sufficient?
>
> 11. I liked the second approach since it looks neat and future-proof. Have
> updated the KIP.
>
> 12. Yes, that is correct.
>
> Many thanks,
>
> Rajini
>
>
> On Thu, Jun 16, 2016 at 11:36 PM, Jun Rao  wrote:
>
> > Rajini,
> >
> > Thanks for the update. A few more questions/comments.
> >
> > 10. For the quota value stored in ZK, since we are adding an optional
> > user_principal field in the json, we should bump the version from 1 to 2.
> > Also, user_principal is not really part of the config values. So, perhaps
> > we should represent it as the following?
> > {
> > "version":2,
> > "config": {
> > "producer_byte_rate":"1024",
> > "consumer_byte_rate":"2048"
> > },
> > "user_principal" : "user1"
> > }
> >
> >  Also, we should store user_principal in the following json too, right?
> > // Zookeeper persistence path /users//clients/clientA
> > {
> > "version":1,
> > "config": {
> > "producer_byte_rate":"10",
> > "consumer_byte_rate":"30"
> > }
> > }
> >
> > 11. For the change notification path, would it be better to change it to
> > something like the following and bump up version to 2?
> > // Change notification for quota of 
> > {
> > "version":2,
> > [
> >   { "entity_type": "users",
> > "entity_name": "user2"
> >   },
> >   { "entity_type": "clients",
> > "entity_name": "clientA"
> >   }
> > ]
> >  }
> >
> > Alternatively, we could change it to
> > // Change notification for quota of 
> > {
> > "version":2,
> > "entity_path": "users/user2"
> > }
> >
> > {
> > "version":2,
> > "entity_path": "users/user2/clients/clientA"
> > }
> >
> > 12. Just to clarify on the meaning of remainder quota. If you have quotas
> > like the following,
> >   = 5
> >   = 10
> >   = 12
> > it means that all connections with user1 whose client-id is neither
> client1
> > nor client2 will be sharing a quota of 12, right? In other words, the
> quota
> > of  doesn't include the quota for  and  > client2>.
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Thu, Jun 16, 2016 at 5:03 AM, Rajini Sivaram <
> > rajinisiva...@googlemail.com> wrote:
> >
> > > Jun,
> > >
> > > Actually, with quotas stored in different nodes in ZK, it is better to
> > > store remainder quota rather than total quota under /users/ so
> that
> > > quota calculations are not dependent on the order of notifications. I
> > have
> > > updated the KIP to reflect that. So the quotas in ZK now always reflect
> > the
> > > quota applied to a group of client connections and use the same format
> as
> > > client-id quotas. But it is not hierarchical, making the configuration
> > > simpler.
> > >
> > > On Thu, Jun 16, 2016 at 11:49 AM, Rajini Sivaram <
> > > rajinisiva...@googlemail.com> wrote:
> > >
> > > > Jun,
> > > >
> > > > Thank you for the review. I have updated the KIP:
> > > >
> > > >
> > > >1. Added an overview section. Slightly reworded since it is better
> > to
> > > >treat user and client-id as different levels rather than the same
> > > level.
> > > >2. Yes, it is neater to store quota for each entity in a different
> > > >path in Zookeeper. I put clients under users rather than the other
> > way
> > > >round since that reflects the hierarchy and also keeps a user's
> > quotas
> > > >together under a single sub-tree. I had initially used a single
> node
> > > to
> > > >keep quotas and sub-quotas of a user together so that updates are
> > > atomic
> > > >since changes to sub-quotas also affect remainder quotas for other
> > > clients.
> > > >But I imagine, updates to configs are rare and it is not a big
> > issue.
> > > >3. I haven't modified the JSON for configuration change
> > notifications.
> > > >The entity_name can now be a subpath that has both user and
> client.
> > > Have
> > > >added an example to the KIP. The downside of keeping clients under
> > > users in
> > > >ZK in 2) is that the 

Re: [VOTE] KIP-55: Secure quotas for authenticated users

[Rajini Sivaram]

Jun,

10. Since entity_type "users" is new, shouldn't the JSON for these entities
have version 1? I have moved "user_principal" out of the config in the
samples and added to the  entries as well. But actually, do
we need to store the non-encoded principal at all? The node name is
URL-encoded user principal, so it is fairly readable if you are looking in
ZK and *kafka_configs.sh* will show the non-encoded principal by decoding
the name from the path (since it needs to do encoding anyway because the
names specified on the command line will be non-encoded principals, it can
do decoding too). Perhaps that is sufficient?

11. I liked the second approach since it looks neat and future-proof. Have
updated the KIP.

12. Yes, that is correct.

Many thanks,

Rajini


On Thu, Jun 16, 2016 at 11:36 PM, Jun Rao  wrote:

> Rajini,
>
> Thanks for the update. A few more questions/comments.
>
> 10. For the quota value stored in ZK, since we are adding an optional
> user_principal field in the json, we should bump the version from 1 to 2.
> Also, user_principal is not really part of the config values. So, perhaps
> we should represent it as the following?
> {
> "version":2,
> "config": {
> "producer_byte_rate":"1024",
> "consumer_byte_rate":"2048"
> },
> "user_principal" : "user1"
> }
>
>  Also, we should store user_principal in the following json too, right?
> // Zookeeper persistence path /users//clients/clientA
> {
> "version":1,
> "config": {
> "producer_byte_rate":"10",
> "consumer_byte_rate":"30"
> }
> }
>
> 11. For the change notification path, would it be better to change it to
> something like the following and bump up version to 2?
> // Change notification for quota of 
> {
> "version":2,
> [
>   { "entity_type": "users",
> "entity_name": "user2"
>   },
>   { "entity_type": "clients",
> "entity_name": "clientA"
>   }
> ]
>  }
>
> Alternatively, we could change it to
> // Change notification for quota of 
> {
> "version":2,
> "entity_path": "users/user2"
> }
>
> {
> "version":2,
> "entity_path": "users/user2/clients/clientA"
> }
>
> 12. Just to clarify on the meaning of remainder quota. If you have quotas
> like the following,
>   = 5
>   = 10
>   = 12
> it means that all connections with user1 whose client-id is neither client1
> nor client2 will be sharing a quota of 12, right? In other words, the quota
> of  doesn't include the quota for  and  client2>.
>
> Thanks,
>
> Jun
>
>
> On Thu, Jun 16, 2016 at 5:03 AM, Rajini Sivaram <
> rajinisiva...@googlemail.com> wrote:
>
> > Jun,
> >
> > Actually, with quotas stored in different nodes in ZK, it is better to
> > store remainder quota rather than total quota under /users/ so that
> > quota calculations are not dependent on the order of notifications. I
> have
> > updated the KIP to reflect that. So the quotas in ZK now always reflect
> the
> > quota applied to a group of client connections and use the same format as
> > client-id quotas. But it is not hierarchical, making the configuration
> > simpler.
> >
> > On Thu, Jun 16, 2016 at 11:49 AM, Rajini Sivaram <
> > rajinisiva...@googlemail.com> wrote:
> >
> > > Jun,
> > >
> > > Thank you for the review. I have updated the KIP:
> > >
> > >
> > >1. Added an overview section. Slightly reworded since it is better
> to
> > >treat user and client-id as different levels rather than the same
> > level.
> > >2. Yes, it is neater to store quota for each entity in a different
> > >path in Zookeeper. I put clients under users rather than the other
> way
> > >round since that reflects the hierarchy and also keeps a user's
> quotas
> > >together under a single sub-tree. I had initially used a single node
> > to
> > >keep quotas and sub-quotas of a user together so that updates are
> > atomic
> > >since changes to sub-quotas also affect remainder quotas for other
> > clients.
> > >But I imagine, updates to configs are rare and it is not a big
> issue.
> > >3. I haven't modified the JSON for configuration change
> notifications.
> > >The entity_name can now be a subpath that has both user and client.
> > Have
> > >added an example to the KIP. The downside of keeping clients under
> > users in
> > >ZK in 2) is that the change notification for sub-quota has
> entity_type
> > >"users". I could extend the JSON to include client separately, but
> > since
> > >changes to a client sub-quota does impact other clients of the user
> > as well
> > >due to change in remainder quota, it may be ok as it is. Do let me
> > know if
> > >it looks confusing in the example.
> > >4. Agree, updated.
> > >
> > >
> > > On Wed, Jun 15, 2016 at 10:27 PM, Jun Rao  wrote:
> > >
> > >> Hi, Rajini,
> > >>
> > >> Thanks for the updated 

Re: [VOTE] KIP-55: Secure quotas for authenticated users

[Jun Rao]

Rajini,

Thanks for the update. A few more questions/comments.

10. For the quota value stored in ZK, since we are adding an optional
user_principal field in the json, we should bump the version from 1 to 2.
Also, user_principal is not really part of the config values. So, perhaps
we should represent it as the following?
{
"version":2,
"config": {
"producer_byte_rate":"1024",
"consumer_byte_rate":"2048"
},
"user_principal" : "user1"
}

 Also, we should store user_principal in the following json too, right?
// Zookeeper persistence path /users//clients/clientA
{
"version":1,
"config": {
"producer_byte_rate":"10",
"consumer_byte_rate":"30"
}
}

11. For the change notification path, would it be better to change it to
something like the following and bump up version to 2?
// Change notification for quota of 
{
"version":2,
[
  { "entity_type": "users",
"entity_name": "user2"
  },
  { "entity_type": "clients",
"entity_name": "clientA"
  }
]
 }

Alternatively, we could change it to
// Change notification for quota of 
{
"version":2,
"entity_path": "users/user2"
}

{
"version":2,
"entity_path": "users/user2/clients/clientA"
}

12. Just to clarify on the meaning of remainder quota. If you have quotas
like the following,
  = 5
  = 10
  = 12
it means that all connections with user1 whose client-id is neither client1
nor client2 will be sharing a quota of 12, right? In other words, the quota
of  doesn't include the quota for  and .

Thanks,

Jun


On Thu, Jun 16, 2016 at 5:03 AM, Rajini Sivaram <
rajinisiva...@googlemail.com> wrote:

> Jun,
>
> Actually, with quotas stored in different nodes in ZK, it is better to
> store remainder quota rather than total quota under /users/ so that
> quota calculations are not dependent on the order of notifications. I have
> updated the KIP to reflect that. So the quotas in ZK now always reflect the
> quota applied to a group of client connections and use the same format as
> client-id quotas. But it is not hierarchical, making the configuration
> simpler.
>
> On Thu, Jun 16, 2016 at 11:49 AM, Rajini Sivaram <
> rajinisiva...@googlemail.com> wrote:
>
> > Jun,
> >
> > Thank you for the review. I have updated the KIP:
> >
> >
> >1. Added an overview section. Slightly reworded since it is better to
> >treat user and client-id as different levels rather than the same
> level.
> >2. Yes, it is neater to store quota for each entity in a different
> >path in Zookeeper. I put clients under users rather than the other way
> >round since that reflects the hierarchy and also keeps a user's quotas
> >together under a single sub-tree. I had initially used a single node
> to
> >keep quotas and sub-quotas of a user together so that updates are
> atomic
> >since changes to sub-quotas also affect remainder quotas for other
> clients.
> >But I imagine, updates to configs are rare and it is not a big issue.
> >3. I haven't modified the JSON for configuration change notifications.
> >The entity_name can now be a subpath that has both user and client.
> Have
> >added an example to the KIP. The downside of keeping clients under
> users in
> >ZK in 2) is that the change notification for sub-quota has entity_type
> >"users". I could extend the JSON to include client separately, but
> since
> >changes to a client sub-quota does impact other clients of the user
> as well
> >due to change in remainder quota, it may be ok as it is. Do let me
> know if
> >it looks confusing in the example.
> >4. Agree, updated.
> >
> >
> > On Wed, Jun 15, 2016 at 10:27 PM, Jun Rao  wrote:
> >
> >> Hi, Rajini,
> >>
> >> Thanks for the updated wiki. Overall, I like the new approach. It covers
> >> the common use cases now, is extensible, and is backward compatible. A
> few
> >> comments below.
> >>
> >> 1. It would be useful to describe a bit at the high level, how the new
> >> approach works. I think this can be summarized as follows. Quotas can be
> >> set at user, client-id or  levels. For a given client
> >> connection, the most specific quota matching the connection will be
> >> applied. For example, if both a  and a  quota
> match
> >> a connection, the  quota will be used. If more than 1
> >> quota at the same level (e.g., a quota on a user and another quota on a
> >> client-id) match the connection, the user level quota will be used,
> i.e.,
> >> user quota takes precedence over client-id quota.
> >>
> >> 2. For the ZK data structure, would it be better to store  >> client-id>
> >> quota as the following. Then the format of the value in each path is the
> >> same. The wiki also mentions that we want to include the original user
> >> name
> >> in the ZK value. 

Re: [VOTE] KIP-55: Secure quotas for authenticated users

[Rajini Sivaram]

Jun,

Actually, with quotas stored in different nodes in ZK, it is better to
store remainder quota rather than total quota under /users/ so that
quota calculations are not dependent on the order of notifications. I have
updated the KIP to reflect that. So the quotas in ZK now always reflect the
quota applied to a group of client connections and use the same format as
client-id quotas. But it is not hierarchical, making the configuration
simpler.

On Thu, Jun 16, 2016 at 11:49 AM, Rajini Sivaram <
rajinisiva...@googlemail.com> wrote:

> Jun,
>
> Thank you for the review. I have updated the KIP:
>
>
>1. Added an overview section. Slightly reworded since it is better to
>treat user and client-id as different levels rather than the same level.
>2. Yes, it is neater to store quota for each entity in a different
>path in Zookeeper. I put clients under users rather than the other way
>round since that reflects the hierarchy and also keeps a user's quotas
>together under a single sub-tree. I had initially used a single node to
>keep quotas and sub-quotas of a user together so that updates are atomic
>since changes to sub-quotas also affect remainder quotas for other clients.
>But I imagine, updates to configs are rare and it is not a big issue.
>3. I haven't modified the JSON for configuration change notifications.
>The entity_name can now be a subpath that has both user and client. Have
>added an example to the KIP. The downside of keeping clients under users in
>ZK in 2) is that the change notification for sub-quota has entity_type
>"users". I could extend the JSON to include client separately, but since
>changes to a client sub-quota does impact other clients of the user as well
>due to change in remainder quota, it may be ok as it is. Do let me know if
>it looks confusing in the example.
>4. Agree, updated.
>
>
> On Wed, Jun 15, 2016 at 10:27 PM, Jun Rao  wrote:
>
>> Hi, Rajini,
>>
>> Thanks for the updated wiki. Overall, I like the new approach. It covers
>> the common use cases now, is extensible, and is backward compatible. A few
>> comments below.
>>
>> 1. It would be useful to describe a bit at the high level, how the new
>> approach works. I think this can be summarized as follows. Quotas can be
>> set at user, client-id or  levels. For a given client
>> connection, the most specific quota matching the connection will be
>> applied. For example, if both a  and a  quota match
>> a connection, the  quota will be used. If more than 1
>> quota at the same level (e.g., a quota on a user and another quota on a
>> client-id) match the connection, the user level quota will be used, i.e.,
>> user quota takes precedence over client-id quota.
>>
>> 2. For the ZK data structure, would it be better to store > client-id>
>> quota as the following. Then the format of the value in each path is the
>> same. The wiki also mentions that we want to include the original user
>> name
>> in the ZK value. Could you describe that in an example?
>> // Zookeeper persistence path /clients/clientA/users/
>> {
>> "version":1,
>> "config": {
>> "producer_byte_rate":"10",
>> "consumer_byte_rate":"20"
>> }
>> }
>>
>> 3. Could you document the format change of the ZK value in
>> /config/changes/config_change_xxx, if any?
>>
>> 4. For the config command, could we specify the sub-quota like the
>> following, instead of in the config value? This seems more intuitive.
>>
>> bin/kafka-configs  --zookeeper localhost:2181 --alter --add-config
>> 'producer_byte_rate=1024,consumer_byte_rate=2048' --entity-name
>> clientA --entity-type clients --entity-name user2 --entity-type users
>>
>> Thanks,
>>
>> Jun
>>
>> On Wed, Jun 15, 2016 at 10:35 AM, Rajini Sivaram <
>> rajinisiva...@googlemail.com> wrote:
>>
>> > Harsha,
>> >
>> > The sample configuration under
>> >
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-55%3A+Secure+Quotas+for+Authenticated+Users#KIP-55:SecureQuotasforAuthenticatedUsers-QuotaConfiguration
>> > shows
>> > the Zookeeper data for different scenarios.
>> >
>> >- *user1* (/users/user1 in ZK) has only user-level quotas
>> >- *user2* (/users/user2 in ZK) defines user-level quotas and
>> sub-quotas
>> >for clients *clientA* and *clientB*. Other client-ids of *user2*
>> share
>> >the remaining quota of *user2*.
>> >
>> >
>> > On Wed, Jun 15, 2016 at 5:30 PM, Harsha  wrote:
>> >
>> > > Rajini,
>> > >   How does sub-quotas works in case of authenticated users.
>> > >   Where are we maintaining the relation between users and
>> their
>> > >   client Ids. Can you add an example of zk data under /users.
>> > > Thanks,
>> > > Harsha
>> > >
>> > > On Mon, Jun 13, 2016, at 05:01 AM, Rajini Sivaram wrote:
>> > > > I have updated KIP-55 to reflect the changes from the discussions in
>> > the
>> > > 

Re: [VOTE] KIP-55: Secure quotas for authenticated users

[Rajini Sivaram]

Jun,

Thank you for the review. I have updated the KIP:


   1. Added an overview section. Slightly reworded since it is better to
   treat user and client-id as different levels rather than the same level.
   2. Yes, it is neater to store quota for each entity in a different path
   in Zookeeper. I put clients under users rather than the other way round
   since that reflects the hierarchy and also keeps a user's quotas together
   under a single sub-tree. I had initially used a single node to keep quotas
   and sub-quotas of a user together so that updates are atomic since changes
   to sub-quotas also affect remainder quotas for other clients. But I
   imagine, updates to configs are rare and it is not a big issue.
   3. I haven't modified the JSON for configuration change notifications.
   The entity_name can now be a subpath that has both user and client. Have
   added an example to the KIP. The downside of keeping clients under users in
   ZK in 2) is that the change notification for sub-quota has entity_type
   "users". I could extend the JSON to include client separately, but since
   changes to a client sub-quota does impact other clients of the user as well
   due to change in remainder quota, it may be ok as it is. Do let me know if
   it looks confusing in the example.
   4. Agree, updated.


On Wed, Jun 15, 2016 at 10:27 PM, Jun Rao  wrote:

> Hi, Rajini,
>
> Thanks for the updated wiki. Overall, I like the new approach. It covers
> the common use cases now, is extensible, and is backward compatible. A few
> comments below.
>
> 1. It would be useful to describe a bit at the high level, how the new
> approach works. I think this can be summarized as follows. Quotas can be
> set at user, client-id or  levels. For a given client
> connection, the most specific quota matching the connection will be
> applied. For example, if both a  and a  quota match
> a connection, the  quota will be used. If more than 1
> quota at the same level (e.g., a quota on a user and another quota on a
> client-id) match the connection, the user level quota will be used, i.e.,
> user quota takes precedence over client-id quota.
>
> 2. For the ZK data structure, would it be better to store 
> quota as the following. Then the format of the value in each path is the
> same. The wiki also mentions that we want to include the original user name
> in the ZK value. Could you describe that in an example?
> // Zookeeper persistence path /clients/clientA/users/
> {
> "version":1,
> "config": {
> "producer_byte_rate":"10",
> "consumer_byte_rate":"20"
> }
> }
>
> 3. Could you document the format change of the ZK value in
> /config/changes/config_change_xxx, if any?
>
> 4. For the config command, could we specify the sub-quota like the
> following, instead of in the config value? This seems more intuitive.
>
> bin/kafka-configs  --zookeeper localhost:2181 --alter --add-config
> 'producer_byte_rate=1024,consumer_byte_rate=2048' --entity-name
> clientA --entity-type clients --entity-name user2 --entity-type users
>
> Thanks,
>
> Jun
>
> On Wed, Jun 15, 2016 at 10:35 AM, Rajini Sivaram <
> rajinisiva...@googlemail.com> wrote:
>
> > Harsha,
> >
> > The sample configuration under
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-55%3A+Secure+Quotas+for+Authenticated+Users#KIP-55:SecureQuotasforAuthenticatedUsers-QuotaConfiguration
> > shows
> > the Zookeeper data for different scenarios.
> >
> >- *user1* (/users/user1 in ZK) has only user-level quotas
> >- *user2* (/users/user2 in ZK) defines user-level quotas and
> sub-quotas
> >for clients *clientA* and *clientB*. Other client-ids of *user2* share
> >the remaining quota of *user2*.
> >
> >
> > On Wed, Jun 15, 2016 at 5:30 PM, Harsha  wrote:
> >
> > > Rajini,
> > >   How does sub-quotas works in case of authenticated users.
> > >   Where are we maintaining the relation between users and their
> > >   client Ids. Can you add an example of zk data under /users.
> > > Thanks,
> > > Harsha
> > >
> > > On Mon, Jun 13, 2016, at 05:01 AM, Rajini Sivaram wrote:
> > > > I have updated KIP-55 to reflect the changes from the discussions in
> > the
> > > > voting thread (
> > > > https://www.mail-archive.com/dev@kafka.apache.org/msg51610.html).
> > > >
> > > > Jun/Gwen,
> > > >
> > > > Existing client-id quotas will be used as default client-id quotas
> for
> > > > users when no user quotas are configured - i.e., default user quota
> is
> > > > unlimited and no user-specific quota override is specified. This
> > enables
> > > > user rate limits to be configured for ANONYMOUS if required in a
> > cluster
> > > > that has both PLAINTEXT and SSL/SASL. By default, without any user
> rate
> > > > limits set, rate limits for client-ids will apply, retaining the
> > current
> > > > client-id quota configuration for 

Re: [VOTE] KIP-55: Secure quotas for authenticated users

[Jun Rao]

Hi, Rajini,

Thanks for the updated wiki. Overall, I like the new approach. It covers
the common use cases now, is extensible, and is backward compatible. A few
comments below.

1. It would be useful to describe a bit at the high level, how the new
approach works. I think this can be summarized as follows. Quotas can be
set at user, client-id or  levels. For a given client
connection, the most specific quota matching the connection will be
applied. For example, if both a  and a  quota match
a connection, the  quota will be used. If more than 1
quota at the same level (e.g., a quota on a user and another quota on a
client-id) match the connection, the user level quota will be used, i.e.,
user quota takes precedence over client-id quota.

2. For the ZK data structure, would it be better to store 
quota as the following. Then the format of the value in each path is the
same. The wiki also mentions that we want to include the original user name
in the ZK value. Could you describe that in an example?
// Zookeeper persistence path /clients/clientA/users/
{
"version":1,
"config": {
"producer_byte_rate":"10",
"consumer_byte_rate":"20"
}
}

3. Could you document the format change of the ZK value in
/config/changes/config_change_xxx, if any?

4. For the config command, could we specify the sub-quota like the
following, instead of in the config value? This seems more intuitive.

bin/kafka-configs  --zookeeper localhost:2181 --alter --add-config
'producer_byte_rate=1024,consumer_byte_rate=2048' --entity-name
clientA --entity-type clients --entity-name user2 --entity-type users

Thanks,

Jun

On Wed, Jun 15, 2016 at 10:35 AM, Rajini Sivaram <
rajinisiva...@googlemail.com> wrote:

> Harsha,
>
> The sample configuration under
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-55%3A+Secure+Quotas+for+Authenticated+Users#KIP-55:SecureQuotasforAuthenticatedUsers-QuotaConfiguration
> shows
> the Zookeeper data for different scenarios.
>
>- *user1* (/users/user1 in ZK) has only user-level quotas
>- *user2* (/users/user2 in ZK) defines user-level quotas and sub-quotas
>for clients *clientA* and *clientB*. Other client-ids of *user2* share
>the remaining quota of *user2*.
>
>
> On Wed, Jun 15, 2016 at 5:30 PM, Harsha  wrote:
>
> > Rajini,
> >   How does sub-quotas works in case of authenticated users.
> >   Where are we maintaining the relation between users and their
> >   client Ids. Can you add an example of zk data under /users.
> > Thanks,
> > Harsha
> >
> > On Mon, Jun 13, 2016, at 05:01 AM, Rajini Sivaram wrote:
> > > I have updated KIP-55 to reflect the changes from the discussions in
> the
> > > voting thread (
> > > https://www.mail-archive.com/dev@kafka.apache.org/msg51610.html).
> > >
> > > Jun/Gwen,
> > >
> > > Existing client-id quotas will be used as default client-id quotas for
> > > users when no user quotas are configured - i.e., default user quota is
> > > unlimited and no user-specific quota override is specified. This
> enables
> > > user rate limits to be configured for ANONYMOUS if required in a
> cluster
> > > that has both PLAINTEXT and SSL/SASL. By default, without any user rate
> > > limits set, rate limits for client-ids will apply, retaining the
> current
> > > client-id quota configuration for single-user clusters.
> > >
> > > Zookeeper will have two paths /clients with client-id quotas that apply
> > > only when user quota is unlimited similar to now. And /users which
> > > persists
> > > user quotas for any user including ANONYMOUS.
> > >
> > > Comments and feedback are appreciated.
> > >
> > > Regards,
> > >
> > > Rajini
> > >
> > >
> > > On Wed, Jun 8, 2016 at 9:00 PM, Rajini Sivaram
> > >  > > > wrote:
> > >
> > > > Jun,
> > > >
> > > > Oops, sorry, I hadn't realized that the last note was on the discuss
> > > > thread. Thank you for pointing it out. I have sent another note for
> > voting.
> > > >
> > > >
> > > > On Wed, Jun 8, 2016 at 4:30 PM, Jun Rao  wrote:
> > > >
> > > >> Rajini,
> > > >>
> > > >> Perhaps it will be clearer if you start the voting in a new thread
> > (with
> > > >> VOTE in the subject).
> > > >>
> > > >> Thanks,
> > > >>
> > > >> Jun
> > > >>
> > > >> On Tue, Jun 7, 2016 at 1:55 PM, Rajini Sivaram <
> > > >> rajinisiva...@googlemail.com
> > > >> > wrote:
> > > >>
> > > >> > I would like to initiate the vote for KIP-55.
> > > >> >
> > > >> > The KIP details are here: KIP-55: Secure quotas for authenticated
> > users
> > > >> > <
> > > >> >
> > > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-55%3A+Secure+Quotas+for+Authenticated+Users
> > > >> > >
> > > >> > .
> > > >> >
> > > >> > The JIRA  KAFKA-3492  <
> > https://issues.apache.org/jira/browse/KAFKA-3492
> > > >> > >has
> > > >> > a draft PR here: https://github.com/apache/kafka/pull/1256.
> > > 

Re: [VOTE] KIP-55: Secure quotas for authenticated users

[Rajini Sivaram]

Harsha,

The sample configuration under
https://cwiki.apache.org/confluence/display/KAFKA/KIP-55%3A+Secure+Quotas+for+Authenticated+Users#KIP-55:SecureQuotasforAuthenticatedUsers-QuotaConfiguration
shows
the Zookeeper data for different scenarios.

   - *user1* (/users/user1 in ZK) has only user-level quotas
   - *user2* (/users/user2 in ZK) defines user-level quotas and sub-quotas
   for clients *clientA* and *clientB*. Other client-ids of *user2* share
   the remaining quota of *user2*.


On Wed, Jun 15, 2016 at 5:30 PM, Harsha  wrote:

> Rajini,
>   How does sub-quotas works in case of authenticated users.
>   Where are we maintaining the relation between users and their
>   client Ids. Can you add an example of zk data under /users.
> Thanks,
> Harsha
>
> On Mon, Jun 13, 2016, at 05:01 AM, Rajini Sivaram wrote:
> > I have updated KIP-55 to reflect the changes from the discussions in the
> > voting thread (
> > https://www.mail-archive.com/dev@kafka.apache.org/msg51610.html).
> >
> > Jun/Gwen,
> >
> > Existing client-id quotas will be used as default client-id quotas for
> > users when no user quotas are configured - i.e., default user quota is
> > unlimited and no user-specific quota override is specified. This enables
> > user rate limits to be configured for ANONYMOUS if required in a cluster
> > that has both PLAINTEXT and SSL/SASL. By default, without any user rate
> > limits set, rate limits for client-ids will apply, retaining the current
> > client-id quota configuration for single-user clusters.
> >
> > Zookeeper will have two paths /clients with client-id quotas that apply
> > only when user quota is unlimited similar to now. And /users which
> > persists
> > user quotas for any user including ANONYMOUS.
> >
> > Comments and feedback are appreciated.
> >
> > Regards,
> >
> > Rajini
> >
> >
> > On Wed, Jun 8, 2016 at 9:00 PM, Rajini Sivaram
> >  > > wrote:
> >
> > > Jun,
> > >
> > > Oops, sorry, I hadn't realized that the last note was on the discuss
> > > thread. Thank you for pointing it out. I have sent another note for
> voting.
> > >
> > >
> > > On Wed, Jun 8, 2016 at 4:30 PM, Jun Rao  wrote:
> > >
> > >> Rajini,
> > >>
> > >> Perhaps it will be clearer if you start the voting in a new thread
> (with
> > >> VOTE in the subject).
> > >>
> > >> Thanks,
> > >>
> > >> Jun
> > >>
> > >> On Tue, Jun 7, 2016 at 1:55 PM, Rajini Sivaram <
> > >> rajinisiva...@googlemail.com
> > >> > wrote:
> > >>
> > >> > I would like to initiate the vote for KIP-55.
> > >> >
> > >> > The KIP details are here: KIP-55: Secure quotas for authenticated
> users
> > >> > <
> > >> >
> > >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-55%3A+Secure+Quotas+for+Authenticated+Users
> > >> > >
> > >> > .
> > >> >
> > >> > The JIRA  KAFKA-3492  <
> https://issues.apache.org/jira/browse/KAFKA-3492
> > >> > >has
> > >> > a draft PR here: https://github.com/apache/kafka/pull/1256.
> > >> >
> > >> > Thank you...
> > >> >
> > >> >
> > >> > Regards,
> > >> >
> > >> > Rajini
> > >> >
> > >>
> > >
> > >
> > >
> > > --
> > > Regards,
> > >
> > > Rajini
> > >
> >
> >
> >
> > --
> > Regards,
> >
> > Rajini
>



-- 
Regards,

Rajini

Re: [VOTE] KIP-55: Secure quotas for authenticated users

[Harsha]

Rajini,
  How does sub-quotas works in case of authenticated users.
  Where are we maintaining the relation between users and their
  client Ids. Can you add an example of zk data under /users.
Thanks,
Harsha

On Mon, Jun 13, 2016, at 05:01 AM, Rajini Sivaram wrote:
> I have updated KIP-55 to reflect the changes from the discussions in the
> voting thread (
> https://www.mail-archive.com/dev@kafka.apache.org/msg51610.html).
> 
> Jun/Gwen,
> 
> Existing client-id quotas will be used as default client-id quotas for
> users when no user quotas are configured - i.e., default user quota is
> unlimited and no user-specific quota override is specified. This enables
> user rate limits to be configured for ANONYMOUS if required in a cluster
> that has both PLAINTEXT and SSL/SASL. By default, without any user rate
> limits set, rate limits for client-ids will apply, retaining the current
> client-id quota configuration for single-user clusters.
> 
> Zookeeper will have two paths /clients with client-id quotas that apply
> only when user quota is unlimited similar to now. And /users which
> persists
> user quotas for any user including ANONYMOUS.
> 
> Comments and feedback are appreciated.
> 
> Regards,
> 
> Rajini
> 
> 
> On Wed, Jun 8, 2016 at 9:00 PM, Rajini Sivaram
>  > wrote:
> 
> > Jun,
> >
> > Oops, sorry, I hadn't realized that the last note was on the discuss
> > thread. Thank you for pointing it out. I have sent another note for voting.
> >
> >
> > On Wed, Jun 8, 2016 at 4:30 PM, Jun Rao  wrote:
> >
> >> Rajini,
> >>
> >> Perhaps it will be clearer if you start the voting in a new thread (with
> >> VOTE in the subject).
> >>
> >> Thanks,
> >>
> >> Jun
> >>
> >> On Tue, Jun 7, 2016 at 1:55 PM, Rajini Sivaram <
> >> rajinisiva...@googlemail.com
> >> > wrote:
> >>
> >> > I would like to initiate the vote for KIP-55.
> >> >
> >> > The KIP details are here: KIP-55: Secure quotas for authenticated users
> >> > <
> >> >
> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-55%3A+Secure+Quotas+for+Authenticated+Users
> >> > >
> >> > .
> >> >
> >> > The JIRA  KAFKA-3492   >> > >has
> >> > a draft PR here: https://github.com/apache/kafka/pull/1256.
> >> >
> >> > Thank you...
> >> >
> >> >
> >> > Regards,
> >> >
> >> > Rajini
> >> >
> >>
> >
> >
> >
> > --
> > Regards,
> >
> > Rajini
> >
> 
> 
> 
> -- 
> Regards,
> 
> Rajini

Re: [VOTE] KIP-55: Secure quotas for authenticated users

[Rajini Sivaram]

I have updated KIP-55 to reflect the changes from the discussions in the
voting thread (
https://www.mail-archive.com/dev@kafka.apache.org/msg51610.html).

Jun/Gwen,

Existing client-id quotas will be used as default client-id quotas for
users when no user quotas are configured - i.e., default user quota is
unlimited and no user-specific quota override is specified. This enables
user rate limits to be configured for ANONYMOUS if required in a cluster
that has both PLAINTEXT and SSL/SASL. By default, without any user rate
limits set, rate limits for client-ids will apply, retaining the current
client-id quota configuration for single-user clusters.

Zookeeper will have two paths /clients with client-id quotas that apply
only when user quota is unlimited similar to now. And /users which persists
user quotas for any user including ANONYMOUS.

Comments and feedback are appreciated.

Regards,

Rajini


On Wed, Jun 8, 2016 at 9:00 PM, Rajini Sivaram  wrote:

> Jun,
>
> Oops, sorry, I hadn't realized that the last note was on the discuss
> thread. Thank you for pointing it out. I have sent another note for voting.
>
>
> On Wed, Jun 8, 2016 at 4:30 PM, Jun Rao  wrote:
>
>> Rajini,
>>
>> Perhaps it will be clearer if you start the voting in a new thread (with
>> VOTE in the subject).
>>
>> Thanks,
>>
>> Jun
>>
>> On Tue, Jun 7, 2016 at 1:55 PM, Rajini Sivaram <
>> rajinisiva...@googlemail.com
>> > wrote:
>>
>> > I would like to initiate the vote for KIP-55.
>> >
>> > The KIP details are here: KIP-55: Secure quotas for authenticated users
>> > <
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-55%3A+Secure+Quotas+for+Authenticated+Users
>> > >
>> > .
>> >
>> > The JIRA  KAFKA-3492  > > >has
>> > a draft PR here: https://github.com/apache/kafka/pull/1256.
>> >
>> > Thank you...
>> >
>> >
>> > Regards,
>> >
>> > Rajini
>> >
>>
>
>
>
> --
> Regards,
>
> Rajini
>



-- 
Regards,

Rajini

Re: [VOTE] KIP-55: Secure quotas for authenticated users

[Rajini Sivaram]

Thank you all for the feedback. Closing this voting thread to continue
discussions on the updated KIP in the discuss thread.

On Sat, Jun 11, 2016 at 7:45 AM, Gwen Shapira  wrote:

> I'd also like to see clarification regarding the ZK structures.
> Currently they appear as if user-quotas and client-quotas are
> equivalent, but this will obviously need to change.
>
>
> On Sat, Jun 11, 2016 at 1:28 AM, Jun Rao  wrote:
> > Rajini,
> >
> > The new proposal sounds good to me too. My only question is what happens
> to
> > those existing quotas on client-id. Do we just treat them as the quota
> for
> > that client-id under ANONYMOUS user name?
> >
> > Thanks,
> >
> > Jun
> >
> > On Fri, Jun 10, 2016 at 2:43 PM, Rajini Sivaram <
> > rajinisiva...@googlemail.com> wrote:
> >
> >> Jay,
> >>
> >> Thank you for the quick feedback. It shouldn't be too hard since I had
> a PR
> >> earlier along these lines anyway.
> >>
> >> Jun, are you ok with this approach? If everyone agrees, I will close
> this
> >> vote, update the KIP and give some more time for discussions.
> >>
> >>
> >> On Fri, Jun 10, 2016 at 10:27 PM, Jay Kreps  wrote:
> >>
> >> > This sounds a lot better to me--hopefully it isn't too much harder! I
> do
> >> > think if it is possible to do this directly that will be better for
> users
> >> > than having an intermediate step since we always have to work through
> >> > migrating people who have setup quotas already from the old way to the
> >> new
> >> > way.
> >> >
> >> > -Jay
> >> >
> >> > On Fri, Jun 10, 2016 at 2:12 PM, Rajini Sivaram <
> >> > rajinisiva...@googlemail.com> wrote:
> >> >
> >> > > I do think client-id is a valid and useful grouping for quotas even
> in
> >> > > secure clusters - but only if clientA of user1 is treated as a
> >> different
> >> > > client-id from clientA of user2. Grouping of clients of a user
> enables
> >> > > users to allocate their quota effectively to their clients (eg.
> >> guarantee
> >> > > that critical event processing clients are not throttled by auditing
> >> > > clients). When the KIP was down-sized to support only user-based
> >> quotas,
> >> > I
> >> > > was hoping that we could extend it at a later time to enable
> >> hierarchical
> >> > > quotas. But I understand that it can be confusing to switch the
> >> semantics
> >> > > of quotas based on modes set in the brokers. So let me try once
> again
> >> to
> >> > > promote the original KIP-55. At the time, I did have a flag to
> revert
> >> to
> >> > > the existing client-id behavior to maintain compatibility. But
> perhaps
> >> > that
> >> > > is not necessary.
> >> > >
> >> > > How does this sound?
> >> > >
> >> > >- Quotas may be configured for users. Sub-quotas may be
> configured
> >> for
> >> > >client-ids of a user. Quotas may also be configured for
> client-ids
> >> of
> >> > > users
> >> > >with unlimited quota (Long.MaxValue).
> >> > >- Users who don't have a quota override are allocated a
> configurable
> >> > >default quota.
> >> > >- Client-ids without a sub-quota override share the remainder of
> the
> >> > >user quota if the user has a quota limit. Default quotas may be
> >> > defined
> >> > > for
> >> > >clients of users with unlimited quota.
> >> > >- For an insecure or single-user secure cluster, the existing
> >> > client-id
> >> > >based quota semantics can be achieved by configuring unlimited
> quota
> >> > for
> >> > >the user and sub-quota configuration for client-id that matches
> the
> >> > > current
> >> > >client-id quota configuration.
> >> > >- For a cluster mixes both secure and insecure access, client-id
> >> > quotas
> >> > >can be set for unauthenticated clients (unlimited quota for
> >> ANONYMOUS,
> >> > >quotas for client-ids) and user quotas can be set for
> authenticated
> >> > > users.
> >> > >- In a multi-user cluster, it is currently possible to define
> quotas
> >> > for
> >> > >client-ids that span multiple users. This will no longer be
> >> supported.
> >> > >
> >> > >
> >> > >
> >> > >
> >> > > On Fri, Jun 10, 2016 at 6:43 PM, Gwen Shapira 
> >> wrote:
> >> > >
> >> > > > I am not crazy about modes either. An earlier proposal supported
> both
> >> > > > client-ids and users at the same time, and it made more sense to
> me.
> >> I
> >> > > > believe it was dropped without proper discussion (Basically, Jun
> >> > > > mentioned it is complex and Rajini agreed to drop it). We should
> >> > > > probably rethink the complexity of supporting both vs the
> limitations
> >> > > > of "modes".
> >> > > >
> >> > > > As you said, we will have secure clients authenticating with users
> >> and
> >> > > > insecure clients authenticating with client-ids at the same time.
> >> > > >
> >> > > > On Fri, Jun 10, 2016 at 7:19 PM, Jay Kreps 
> wrote:
> >> > > > > Hey Rajini,
> >> > > > >
> >> > > > > 1. That makes sense to me. Is 

Re: [VOTE] KIP-55: Secure quotas for authenticated users

[Gwen Shapira]

I'd also like to see clarification regarding the ZK structures.
Currently they appear as if user-quotas and client-quotas are
equivalent, but this will obviously need to change.


On Sat, Jun 11, 2016 at 1:28 AM, Jun Rao  wrote:
> Rajini,
>
> The new proposal sounds good to me too. My only question is what happens to
> those existing quotas on client-id. Do we just treat them as the quota for
> that client-id under ANONYMOUS user name?
>
> Thanks,
>
> Jun
>
> On Fri, Jun 10, 2016 at 2:43 PM, Rajini Sivaram <
> rajinisiva...@googlemail.com> wrote:
>
>> Jay,
>>
>> Thank you for the quick feedback. It shouldn't be too hard since I had a PR
>> earlier along these lines anyway.
>>
>> Jun, are you ok with this approach? If everyone agrees, I will close this
>> vote, update the KIP and give some more time for discussions.
>>
>>
>> On Fri, Jun 10, 2016 at 10:27 PM, Jay Kreps  wrote:
>>
>> > This sounds a lot better to me--hopefully it isn't too much harder! I do
>> > think if it is possible to do this directly that will be better for users
>> > than having an intermediate step since we always have to work through
>> > migrating people who have setup quotas already from the old way to the
>> new
>> > way.
>> >
>> > -Jay
>> >
>> > On Fri, Jun 10, 2016 at 2:12 PM, Rajini Sivaram <
>> > rajinisiva...@googlemail.com> wrote:
>> >
>> > > I do think client-id is a valid and useful grouping for quotas even in
>> > > secure clusters - but only if clientA of user1 is treated as a
>> different
>> > > client-id from clientA of user2. Grouping of clients of a user enables
>> > > users to allocate their quota effectively to their clients (eg.
>> guarantee
>> > > that critical event processing clients are not throttled by auditing
>> > > clients). When the KIP was down-sized to support only user-based
>> quotas,
>> > I
>> > > was hoping that we could extend it at a later time to enable
>> hierarchical
>> > > quotas. But I understand that it can be confusing to switch the
>> semantics
>> > > of quotas based on modes set in the brokers. So let me try once again
>> to
>> > > promote the original KIP-55. At the time, I did have a flag to revert
>> to
>> > > the existing client-id behavior to maintain compatibility. But perhaps
>> > that
>> > > is not necessary.
>> > >
>> > > How does this sound?
>> > >
>> > >- Quotas may be configured for users. Sub-quotas may be configured
>> for
>> > >client-ids of a user. Quotas may also be configured for client-ids
>> of
>> > > users
>> > >with unlimited quota (Long.MaxValue).
>> > >- Users who don't have a quota override are allocated a configurable
>> > >default quota.
>> > >- Client-ids without a sub-quota override share the remainder of the
>> > >user quota if the user has a quota limit. Default quotas may be
>> > defined
>> > > for
>> > >clients of users with unlimited quota.
>> > >- For an insecure or single-user secure cluster, the existing
>> > client-id
>> > >based quota semantics can be achieved by configuring unlimited quota
>> > for
>> > >the user and sub-quota configuration for client-id that matches the
>> > > current
>> > >client-id quota configuration.
>> > >- For a cluster mixes both secure and insecure access, client-id
>> > quotas
>> > >can be set for unauthenticated clients (unlimited quota for
>> ANONYMOUS,
>> > >quotas for client-ids) and user quotas can be set for authenticated
>> > > users.
>> > >- In a multi-user cluster, it is currently possible to define quotas
>> > for
>> > >client-ids that span multiple users. This will no longer be
>> supported.
>> > >
>> > >
>> > >
>> > >
>> > > On Fri, Jun 10, 2016 at 6:43 PM, Gwen Shapira 
>> wrote:
>> > >
>> > > > I am not crazy about modes either. An earlier proposal supported both
>> > > > client-ids and users at the same time, and it made more sense to me.
>> I
>> > > > believe it was dropped without proper discussion (Basically, Jun
>> > > > mentioned it is complex and Rajini agreed to drop it). We should
>> > > > probably rethink the complexity of supporting both vs the limitations
>> > > > of "modes".
>> > > >
>> > > > As you said, we will have secure clients authenticating with users
>> and
>> > > > insecure clients authenticating with client-ids at the same time.
>> > > >
>> > > > On Fri, Jun 10, 2016 at 7:19 PM, Jay Kreps  wrote:
>> > > > > Hey Rajini,
>> > > > >
>> > > > > 1. That makes sense to me. Is that reflected in the documentation
>> > > > anywhere
>> > > > > (I couldn't really find it)? Is there a way to discover that
>> > > definition?
>> > > > We
>> > > > > do way better when we right this stuff down so it has an official
>> > > > > definition users and developers can work off of...
>> > > > > 2. If client id is a thing that makes sense even when you have
>> users,
>> > > why
>> > > > > would you not want to quota on it?
>> > > > >
>> > > > > I am not wild about these 

Re: [VOTE] KIP-55: Secure quotas for authenticated users

[Jun Rao]

Rajini,

The new proposal sounds good to me too. My only question is what happens to
those existing quotas on client-id. Do we just treat them as the quota for
that client-id under ANONYMOUS user name?

Thanks,

Jun

On Fri, Jun 10, 2016 at 2:43 PM, Rajini Sivaram <
rajinisiva...@googlemail.com> wrote:

> Jay,
>
> Thank you for the quick feedback. It shouldn't be too hard since I had a PR
> earlier along these lines anyway.
>
> Jun, are you ok with this approach? If everyone agrees, I will close this
> vote, update the KIP and give some more time for discussions.
>
>
> On Fri, Jun 10, 2016 at 10:27 PM, Jay Kreps  wrote:
>
> > This sounds a lot better to me--hopefully it isn't too much harder! I do
> > think if it is possible to do this directly that will be better for users
> > than having an intermediate step since we always have to work through
> > migrating people who have setup quotas already from the old way to the
> new
> > way.
> >
> > -Jay
> >
> > On Fri, Jun 10, 2016 at 2:12 PM, Rajini Sivaram <
> > rajinisiva...@googlemail.com> wrote:
> >
> > > I do think client-id is a valid and useful grouping for quotas even in
> > > secure clusters - but only if clientA of user1 is treated as a
> different
> > > client-id from clientA of user2. Grouping of clients of a user enables
> > > users to allocate their quota effectively to their clients (eg.
> guarantee
> > > that critical event processing clients are not throttled by auditing
> > > clients). When the KIP was down-sized to support only user-based
> quotas,
> > I
> > > was hoping that we could extend it at a later time to enable
> hierarchical
> > > quotas. But I understand that it can be confusing to switch the
> semantics
> > > of quotas based on modes set in the brokers. So let me try once again
> to
> > > promote the original KIP-55. At the time, I did have a flag to revert
> to
> > > the existing client-id behavior to maintain compatibility. But perhaps
> > that
> > > is not necessary.
> > >
> > > How does this sound?
> > >
> > >- Quotas may be configured for users. Sub-quotas may be configured
> for
> > >client-ids of a user. Quotas may also be configured for client-ids
> of
> > > users
> > >with unlimited quota (Long.MaxValue).
> > >- Users who don't have a quota override are allocated a configurable
> > >default quota.
> > >- Client-ids without a sub-quota override share the remainder of the
> > >user quota if the user has a quota limit. Default quotas may be
> > defined
> > > for
> > >clients of users with unlimited quota.
> > >- For an insecure or single-user secure cluster, the existing
> > client-id
> > >based quota semantics can be achieved by configuring unlimited quota
> > for
> > >the user and sub-quota configuration for client-id that matches the
> > > current
> > >client-id quota configuration.
> > >- For a cluster mixes both secure and insecure access, client-id
> > quotas
> > >can be set for unauthenticated clients (unlimited quota for
> ANONYMOUS,
> > >quotas for client-ids) and user quotas can be set for authenticated
> > > users.
> > >- In a multi-user cluster, it is currently possible to define quotas
> > for
> > >client-ids that span multiple users. This will no longer be
> supported.
> > >
> > >
> > >
> > >
> > > On Fri, Jun 10, 2016 at 6:43 PM, Gwen Shapira 
> wrote:
> > >
> > > > I am not crazy about modes either. An earlier proposal supported both
> > > > client-ids and users at the same time, and it made more sense to me.
> I
> > > > believe it was dropped without proper discussion (Basically, Jun
> > > > mentioned it is complex and Rajini agreed to drop it). We should
> > > > probably rethink the complexity of supporting both vs the limitations
> > > > of "modes".
> > > >
> > > > As you said, we will have secure clients authenticating with users
> and
> > > > insecure clients authenticating with client-ids at the same time.
> > > >
> > > > On Fri, Jun 10, 2016 at 7:19 PM, Jay Kreps  wrote:
> > > > > Hey Rajini,
> > > > >
> > > > > 1. That makes sense to me. Is that reflected in the documentation
> > > > anywhere
> > > > > (I couldn't really find it)? Is there a way to discover that
> > > definition?
> > > > We
> > > > > do way better when we right this stuff down so it has an official
> > > > > definition users and developers can work off of...
> > > > > 2. If client id is a thing that makes sense even when you have
> users,
> > > why
> > > > > would you not want to quota on it?
> > > > >
> > > > > I am not wild about these "modes" where you boot the cluster in
> mode
> > X
> > > > and
> > > > > it behaves in one way and in mode Y and it behaves in another. It
> is
> > > > > complex then for users who expect to be able to set quotas but then
> > > have
> > > > to
> > > > > be able to get access to the filesystem of the kafka nodes to
> > discover
> > > > what
> > > > > mode the cluster is in to 

Re: [VOTE] KIP-55: Secure quotas for authenticated users

[Rajini Sivaram]

Jay,

Thank you for the quick feedback. It shouldn't be too hard since I had a PR
earlier along these lines anyway.

Jun, are you ok with this approach? If everyone agrees, I will close this
vote, update the KIP and give some more time for discussions.


On Fri, Jun 10, 2016 at 10:27 PM, Jay Kreps  wrote:

> This sounds a lot better to me--hopefully it isn't too much harder! I do
> think if it is possible to do this directly that will be better for users
> than having an intermediate step since we always have to work through
> migrating people who have setup quotas already from the old way to the new
> way.
>
> -Jay
>
> On Fri, Jun 10, 2016 at 2:12 PM, Rajini Sivaram <
> rajinisiva...@googlemail.com> wrote:
>
> > I do think client-id is a valid and useful grouping for quotas even in
> > secure clusters - but only if clientA of user1 is treated as a different
> > client-id from clientA of user2. Grouping of clients of a user enables
> > users to allocate their quota effectively to their clients (eg. guarantee
> > that critical event processing clients are not throttled by auditing
> > clients). When the KIP was down-sized to support only user-based quotas,
> I
> > was hoping that we could extend it at a later time to enable hierarchical
> > quotas. But I understand that it can be confusing to switch the semantics
> > of quotas based on modes set in the brokers. So let me try once again to
> > promote the original KIP-55. At the time, I did have a flag to revert to
> > the existing client-id behavior to maintain compatibility. But perhaps
> that
> > is not necessary.
> >
> > How does this sound?
> >
> >- Quotas may be configured for users. Sub-quotas may be configured for
> >client-ids of a user. Quotas may also be configured for client-ids of
> > users
> >with unlimited quota (Long.MaxValue).
> >- Users who don't have a quota override are allocated a configurable
> >default quota.
> >- Client-ids without a sub-quota override share the remainder of the
> >user quota if the user has a quota limit. Default quotas may be
> defined
> > for
> >clients of users with unlimited quota.
> >- For an insecure or single-user secure cluster, the existing
> client-id
> >based quota semantics can be achieved by configuring unlimited quota
> for
> >the user and sub-quota configuration for client-id that matches the
> > current
> >client-id quota configuration.
> >- For a cluster mixes both secure and insecure access, client-id
> quotas
> >can be set for unauthenticated clients (unlimited quota for ANONYMOUS,
> >quotas for client-ids) and user quotas can be set for authenticated
> > users.
> >- In a multi-user cluster, it is currently possible to define quotas
> for
> >client-ids that span multiple users. This will no longer be supported.
> >
> >
> >
> >
> > On Fri, Jun 10, 2016 at 6:43 PM, Gwen Shapira  wrote:
> >
> > > I am not crazy about modes either. An earlier proposal supported both
> > > client-ids and users at the same time, and it made more sense to me. I
> > > believe it was dropped without proper discussion (Basically, Jun
> > > mentioned it is complex and Rajini agreed to drop it). We should
> > > probably rethink the complexity of supporting both vs the limitations
> > > of "modes".
> > >
> > > As you said, we will have secure clients authenticating with users and
> > > insecure clients authenticating with client-ids at the same time.
> > >
> > > On Fri, Jun 10, 2016 at 7:19 PM, Jay Kreps  wrote:
> > > > Hey Rajini,
> > > >
> > > > 1. That makes sense to me. Is that reflected in the documentation
> > > anywhere
> > > > (I couldn't really find it)? Is there a way to discover that
> > definition?
> > > We
> > > > do way better when we right this stuff down so it has an official
> > > > definition users and developers can work off of...
> > > > 2. If client id is a thing that makes sense even when you have users,
> > why
> > > > would you not want to quota on it?
> > > >
> > > > I am not wild about these "modes" where you boot the cluster in mode
> X
> > > and
> > > > it behaves in one way and in mode Y and it behaves in another. It is
> > > > complex then for users who expect to be able to set quotas but then
> > have
> > > to
> > > > be able to get access to the filesystem of the kafka nodes to
> discover
> > > what
> > > > mode the cluster is in to know which kind of quota is applicable.
> > > >
> > > > I guess there are two ways to think about a feature: one is the
> > increment
> > > > from where we are, and the other is the resulting state after that
> > > > increment is taken. What I am asking is not "is this a low cost step
> > from
> > > > where we are?" which everyone can agree that it is, but rather "does
> > this
> > > > make sense as an end state--i.e. if you were starting fresh with
> > neither
> > > > users nor client ids nor quotas would you end up with this?".
> > > >
> > > 

Re: [VOTE] KIP-55: Secure quotas for authenticated users

[Jay Kreps]

This sounds a lot better to me--hopefully it isn't too much harder! I do
think if it is possible to do this directly that will be better for users
than having an intermediate step since we always have to work through
migrating people who have setup quotas already from the old way to the new
way.

-Jay

On Fri, Jun 10, 2016 at 2:12 PM, Rajini Sivaram <
rajinisiva...@googlemail.com> wrote:

> I do think client-id is a valid and useful grouping for quotas even in
> secure clusters - but only if clientA of user1 is treated as a different
> client-id from clientA of user2. Grouping of clients of a user enables
> users to allocate their quota effectively to their clients (eg. guarantee
> that critical event processing clients are not throttled by auditing
> clients). When the KIP was down-sized to support only user-based quotas, I
> was hoping that we could extend it at a later time to enable hierarchical
> quotas. But I understand that it can be confusing to switch the semantics
> of quotas based on modes set in the brokers. So let me try once again to
> promote the original KIP-55. At the time, I did have a flag to revert to
> the existing client-id behavior to maintain compatibility. But perhaps that
> is not necessary.
>
> How does this sound?
>
>- Quotas may be configured for users. Sub-quotas may be configured for
>client-ids of a user. Quotas may also be configured for client-ids of
> users
>with unlimited quota (Long.MaxValue).
>- Users who don't have a quota override are allocated a configurable
>default quota.
>- Client-ids without a sub-quota override share the remainder of the
>user quota if the user has a quota limit. Default quotas may be defined
> for
>clients of users with unlimited quota.
>- For an insecure or single-user secure cluster, the existing client-id
>based quota semantics can be achieved by configuring unlimited quota for
>the user and sub-quota configuration for client-id that matches the
> current
>client-id quota configuration.
>- For a cluster mixes both secure and insecure access, client-id quotas
>can be set for unauthenticated clients (unlimited quota for ANONYMOUS,
>quotas for client-ids) and user quotas can be set for authenticated
> users.
>- In a multi-user cluster, it is currently possible to define quotas for
>client-ids that span multiple users. This will no longer be supported.
>
>
>
>
> On Fri, Jun 10, 2016 at 6:43 PM, Gwen Shapira  wrote:
>
> > I am not crazy about modes either. An earlier proposal supported both
> > client-ids and users at the same time, and it made more sense to me. I
> > believe it was dropped without proper discussion (Basically, Jun
> > mentioned it is complex and Rajini agreed to drop it). We should
> > probably rethink the complexity of supporting both vs the limitations
> > of "modes".
> >
> > As you said, we will have secure clients authenticating with users and
> > insecure clients authenticating with client-ids at the same time.
> >
> > On Fri, Jun 10, 2016 at 7:19 PM, Jay Kreps  wrote:
> > > Hey Rajini,
> > >
> > > 1. That makes sense to me. Is that reflected in the documentation
> > anywhere
> > > (I couldn't really find it)? Is there a way to discover that
> definition?
> > We
> > > do way better when we right this stuff down so it has an official
> > > definition users and developers can work off of...
> > > 2. If client id is a thing that makes sense even when you have users,
> why
> > > would you not want to quota on it?
> > >
> > > I am not wild about these "modes" where you boot the cluster in mode X
> > and
> > > it behaves in one way and in mode Y and it behaves in another. It is
> > > complex then for users who expect to be able to set quotas but then
> have
> > to
> > > be able to get access to the filesystem of the kafka nodes to discover
> > what
> > > mode the cluster is in to know which kind of quota is applicable.
> > >
> > > I guess there are two ways to think about a feature: one is the
> increment
> > > from where we are, and the other is the resulting state after that
> > > increment is taken. What I am asking is not "is this a low cost step
> from
> > > where we are?" which everyone can agree that it is, but rather "does
> this
> > > make sense as an end state--i.e. if you were starting fresh with
> neither
> > > users nor client ids nor quotas would you end up with this?".
> > >
> > > In terms of use cases, I think that we support mixing secure and
> insecure
> > > access on a single cluster so presumably in that case you would want to
> > be
> > > able to quota insecure users based on client id and secure users based
> on
> > > user, right? Likewise, as you said, client id is a valid grouping even
> in
> > > the presence of users, so it might be the case that several apps that
> are
> > > all part of the same system might access Kafka under a single user, but
> > you
> > > might have different quotas for these 

Re: [VOTE] KIP-55: Secure quotas for authenticated users

[Rajini Sivaram]

I do think client-id is a valid and useful grouping for quotas even in
secure clusters - but only if clientA of user1 is treated as a different
client-id from clientA of user2. Grouping of clients of a user enables
users to allocate their quota effectively to their clients (eg. guarantee
that critical event processing clients are not throttled by auditing
clients). When the KIP was down-sized to support only user-based quotas, I
was hoping that we could extend it at a later time to enable hierarchical
quotas. But I understand that it can be confusing to switch the semantics
of quotas based on modes set in the brokers. So let me try once again to
promote the original KIP-55. At the time, I did have a flag to revert to
the existing client-id behavior to maintain compatibility. But perhaps that
is not necessary.

How does this sound?

   - Quotas may be configured for users. Sub-quotas may be configured for
   client-ids of a user. Quotas may also be configured for client-ids of users
   with unlimited quota (Long.MaxValue).
   - Users who don't have a quota override are allocated a configurable
   default quota.
   - Client-ids without a sub-quota override share the remainder of the
   user quota if the user has a quota limit. Default quotas may be defined for
   clients of users with unlimited quota.
   - For an insecure or single-user secure cluster, the existing client-id
   based quota semantics can be achieved by configuring unlimited quota for
   the user and sub-quota configuration for client-id that matches the current
   client-id quota configuration.
   - For a cluster mixes both secure and insecure access, client-id quotas
   can be set for unauthenticated clients (unlimited quota for ANONYMOUS,
   quotas for client-ids) and user quotas can be set for authenticated users.
   - In a multi-user cluster, it is currently possible to define quotas for
   client-ids that span multiple users. This will no longer be supported.




On Fri, Jun 10, 2016 at 6:43 PM, Gwen Shapira  wrote:

> I am not crazy about modes either. An earlier proposal supported both
> client-ids and users at the same time, and it made more sense to me. I
> believe it was dropped without proper discussion (Basically, Jun
> mentioned it is complex and Rajini agreed to drop it). We should
> probably rethink the complexity of supporting both vs the limitations
> of "modes".
>
> As you said, we will have secure clients authenticating with users and
> insecure clients authenticating with client-ids at the same time.
>
> On Fri, Jun 10, 2016 at 7:19 PM, Jay Kreps  wrote:
> > Hey Rajini,
> >
> > 1. That makes sense to me. Is that reflected in the documentation
> anywhere
> > (I couldn't really find it)? Is there a way to discover that definition?
> We
> > do way better when we right this stuff down so it has an official
> > definition users and developers can work off of...
> > 2. If client id is a thing that makes sense even when you have users, why
> > would you not want to quota on it?
> >
> > I am not wild about these "modes" where you boot the cluster in mode X
> and
> > it behaves in one way and in mode Y and it behaves in another. It is
> > complex then for users who expect to be able to set quotas but then have
> to
> > be able to get access to the filesystem of the kafka nodes to discover
> what
> > mode the cluster is in to know which kind of quota is applicable.
> >
> > I guess there are two ways to think about a feature: one is the increment
> > from where we are, and the other is the resulting state after that
> > increment is taken. What I am asking is not "is this a low cost step from
> > where we are?" which everyone can agree that it is, but rather "does this
> > make sense as an end state--i.e. if you were starting fresh with neither
> > users nor client ids nor quotas would you end up with this?".
> >
> > In terms of use cases, I think that we support mixing secure and insecure
> > access on a single cluster so presumably in that case you would want to
> be
> > able to quota insecure users based on client id and secure users based on
> > user, right? Likewise, as you said, client id is a valid grouping even in
> > the presence of users, so it might be the case that several apps that are
> > all part of the same system might access Kafka under a single user, but
> you
> > might have different quotas for these different apps. Basically if client
> > id is a valid grouping even in the presence of users (willing to debate
> > this point, btw!) then you would want to quota on it.
> >
> > -Jay
> >
> >
> >
> > On Fri, Jun 10, 2016 at 4:49 AM, Rajini Sivaram <
> > rajinisiva...@googlemail.com> wrote:
> >
> >> Jay,
> >>
> >> Thank you for the feedback.
> >>
> >> 1. I think it is good to have a single concept of identity, but multiple
> >> ways of grouping clients for different functions. Client-id is a logical
> >> grouping of clients with a meaningful name that is used in client
> metrics
> 

Re: [VOTE] KIP-55: Secure quotas for authenticated users

[Gwen Shapira]

I am not crazy about modes either. An earlier proposal supported both
client-ids and users at the same time, and it made more sense to me. I
believe it was dropped without proper discussion (Basically, Jun
mentioned it is complex and Rajini agreed to drop it). We should
probably rethink the complexity of supporting both vs the limitations
of "modes".

As you said, we will have secure clients authenticating with users and
insecure clients authenticating with client-ids at the same time.

On Fri, Jun 10, 2016 at 7:19 PM, Jay Kreps  wrote:
> Hey Rajini,
>
> 1. That makes sense to me. Is that reflected in the documentation anywhere
> (I couldn't really find it)? Is there a way to discover that definition? We
> do way better when we right this stuff down so it has an official
> definition users and developers can work off of...
> 2. If client id is a thing that makes sense even when you have users, why
> would you not want to quota on it?
>
> I am not wild about these "modes" where you boot the cluster in mode X and
> it behaves in one way and in mode Y and it behaves in another. It is
> complex then for users who expect to be able to set quotas but then have to
> be able to get access to the filesystem of the kafka nodes to discover what
> mode the cluster is in to know which kind of quota is applicable.
>
> I guess there are two ways to think about a feature: one is the increment
> from where we are, and the other is the resulting state after that
> increment is taken. What I am asking is not "is this a low cost step from
> where we are?" which everyone can agree that it is, but rather "does this
> make sense as an end state--i.e. if you were starting fresh with neither
> users nor client ids nor quotas would you end up with this?".
>
> In terms of use cases, I think that we support mixing secure and insecure
> access on a single cluster so presumably in that case you would want to be
> able to quota insecure users based on client id and secure users based on
> user, right? Likewise, as you said, client id is a valid grouping even in
> the presence of users, so it might be the case that several apps that are
> all part of the same system might access Kafka under a single user, but you
> might have different quotas for these different apps. Basically if client
> id is a valid grouping even in the presence of users (willing to debate
> this point, btw!) then you would want to quota on it.
>
> -Jay
>
>
>
> On Fri, Jun 10, 2016 at 4:49 AM, Rajini Sivaram <
> rajinisiva...@googlemail.com> wrote:
>
>> Jay,
>>
>> Thank you for the feedback.
>>
>> 1. I think it is good to have a single concept of identity, but multiple
>> ways of grouping clients for different functions. Client-id is a logical
>> grouping of clients with a meaningful name that is used in client metrics
>> and logs. User principal is an authenticated user or a grouping of
>> unauthenticated users chosen by the broker and is used for ACLs. In my
>> view, in a multi-user system, there is a hierarchy - user owns zero or more
>> clients. (principal, client-id) defines a safe group, but the shorter
>> unsafe client-id is sufficient in client metrics and logs.
>>
>>
>> 2. KIP-55 was initially written to support hierarchical quotas (quotas for
>> user1-clientA, user2-clientA etc), but Jun's view was that the complexity
>> was not justified since there is no clear requirement for this. The
>> cut-down version of the KIP is clearly a lot simpler. But I think your
>> suggestion is to enable non-hierarchical user quotas and client-id quotas
>> at the same time. Basically treat users and client-ids as distinct entities
>> like topics and allow quotas to be applied to each of these entities. I
>> agree that we want to support quotas simultaneously on different entities
>> like topics and users. I am less convinced of client-id and user being
>>
>> distinct entities that require separate quotas at the same time. And
>> treating client-id and user as distinct entities makes it harder to
>> implement hierarchical quotas in future.
>>
>>
>> A single user system needs only client-id quotas, and multi-tenant system
>> cannot use client-id quotas since we need to guarantee that one tenant's
>> quotas can never be used by another tenant. I suppose a multi-user cluster
>> where users trust each other could apply separate quotas for both clients
>> and users, but I am not sure if there is a usecase that can't be satisfied
>> with just client-id based quotas for this case. Do you have a usecase in
>> mind where you want to apply separate quotas for clients and users
>> simultaneously?
>>
>>
>>
>>
>> On Thu, Jun 9, 2016 at 9:40 PM, Jay Kreps  wrote:
>>
>> > Super sorry to come in late on this one. Rajini, I had two quick
>> questions
>> > I think we should be able to answer:
>> >
>> >1. Do client ids make sense in a world which has users? If not should
>> we
>> >unify them the way Hadoop did (without auth the user is a kind of 

Re: [VOTE] KIP-55: Secure quotas for authenticated users

[Jay Kreps]

Hey Rajini,

1. That makes sense to me. Is that reflected in the documentation anywhere
(I couldn't really find it)? Is there a way to discover that definition? We
do way better when we right this stuff down so it has an official
definition users and developers can work off of...
2. If client id is a thing that makes sense even when you have users, why
would you not want to quota on it?

I am not wild about these "modes" where you boot the cluster in mode X and
it behaves in one way and in mode Y and it behaves in another. It is
complex then for users who expect to be able to set quotas but then have to
be able to get access to the filesystem of the kafka nodes to discover what
mode the cluster is in to know which kind of quota is applicable.

I guess there are two ways to think about a feature: one is the increment
from where we are, and the other is the resulting state after that
increment is taken. What I am asking is not "is this a low cost step from
where we are?" which everyone can agree that it is, but rather "does this
make sense as an end state--i.e. if you were starting fresh with neither
users nor client ids nor quotas would you end up with this?".

In terms of use cases, I think that we support mixing secure and insecure
access on a single cluster so presumably in that case you would want to be
able to quota insecure users based on client id and secure users based on
user, right? Likewise, as you said, client id is a valid grouping even in
the presence of users, so it might be the case that several apps that are
all part of the same system might access Kafka under a single user, but you
might have different quotas for these different apps. Basically if client
id is a valid grouping even in the presence of users (willing to debate
this point, btw!) then you would want to quota on it.

-Jay



On Fri, Jun 10, 2016 at 4:49 AM, Rajini Sivaram <
rajinisiva...@googlemail.com> wrote:

> Jay,
>
> Thank you for the feedback.
>
> 1. I think it is good to have a single concept of identity, but multiple
> ways of grouping clients for different functions. Client-id is a logical
> grouping of clients with a meaningful name that is used in client metrics
> and logs. User principal is an authenticated user or a grouping of
> unauthenticated users chosen by the broker and is used for ACLs. In my
> view, in a multi-user system, there is a hierarchy - user owns zero or more
> clients. (principal, client-id) defines a safe group, but the shorter
> unsafe client-id is sufficient in client metrics and logs.
>
>
> 2. KIP-55 was initially written to support hierarchical quotas (quotas for
> user1-clientA, user2-clientA etc), but Jun's view was that the complexity
> was not justified since there is no clear requirement for this. The
> cut-down version of the KIP is clearly a lot simpler. But I think your
> suggestion is to enable non-hierarchical user quotas and client-id quotas
> at the same time. Basically treat users and client-ids as distinct entities
> like topics and allow quotas to be applied to each of these entities. I
> agree that we want to support quotas simultaneously on different entities
> like topics and users. I am less convinced of client-id and user being
>
> distinct entities that require separate quotas at the same time. And
> treating client-id and user as distinct entities makes it harder to
> implement hierarchical quotas in future.
>
>
> A single user system needs only client-id quotas, and multi-tenant system
> cannot use client-id quotas since we need to guarantee that one tenant's
> quotas can never be used by another tenant. I suppose a multi-user cluster
> where users trust each other could apply separate quotas for both clients
> and users, but I am not sure if there is a usecase that can't be satisfied
> with just client-id based quotas for this case. Do you have a usecase in
> mind where you want to apply separate quotas for clients and users
> simultaneously?
>
>
>
>
> On Thu, Jun 9, 2016 at 9:40 PM, Jay Kreps  wrote:
>
> > Super sorry to come in late on this one. Rajini, I had two quick
> questions
> > I think we should be able to answer:
> >
> >1. Do client ids make sense in a world which has users? If not should
> we
> >unify them the way Hadoop did (without auth the user is a kind of best
> >effort honor system identity). This came up in the discussion thread
> > but I
> >didn't really see a crisp answer. Basically, what is the definition of
> >"client id" and what is the definition of "user" and how do the
> concepts
> >relate?
> >2. If both client ids and users are sensible distinct notions and we
> >want to maintain both, why don't we just support quotas on both? If
> they
> >both make sense then you would have a reason to set quotas at both
> > levels.
> >Why have this "mode" that you set that swaps between only being able
> to
> > use
> >one and the other? I should be able to set quotas at both levels.
> Going
> > 

Re: [VOTE] KIP-55: Secure quotas for authenticated users

[Rajini Sivaram]

Jay,

Thank you for the feedback.

1. I think it is good to have a single concept of identity, but multiple
ways of grouping clients for different functions. Client-id is a logical
grouping of clients with a meaningful name that is used in client metrics
and logs. User principal is an authenticated user or a grouping of
unauthenticated users chosen by the broker and is used for ACLs. In my
view, in a multi-user system, there is a hierarchy - user owns zero or more
clients. (principal, client-id) defines a safe group, but the shorter
unsafe client-id is sufficient in client metrics and logs.


2. KIP-55 was initially written to support hierarchical quotas (quotas for
user1-clientA, user2-clientA etc), but Jun's view was that the complexity
was not justified since there is no clear requirement for this. The
cut-down version of the KIP is clearly a lot simpler. But I think your
suggestion is to enable non-hierarchical user quotas and client-id quotas
at the same time. Basically treat users and client-ids as distinct entities
like topics and allow quotas to be applied to each of these entities. I
agree that we want to support quotas simultaneously on different entities
like topics and users. I am less convinced of client-id and user being

distinct entities that require separate quotas at the same time. And
treating client-id and user as distinct entities makes it harder to
implement hierarchical quotas in future.


A single user system needs only client-id quotas, and multi-tenant system
cannot use client-id quotas since we need to guarantee that one tenant's
quotas can never be used by another tenant. I suppose a multi-user cluster
where users trust each other could apply separate quotas for both clients
and users, but I am not sure if there is a usecase that can't be satisfied
with just client-id based quotas for this case. Do you have a usecase in
mind where you want to apply separate quotas for clients and users
simultaneously?




On Thu, Jun 9, 2016 at 9:40 PM, Jay Kreps  wrote:

> Super sorry to come in late on this one. Rajini, I had two quick questions
> I think we should be able to answer:
>
>1. Do client ids make sense in a world which has users? If not should we
>unify them the way Hadoop did (without auth the user is a kind of best
>effort honor system identity). This came up in the discussion thread
> but I
>didn't really see a crisp answer. Basically, what is the definition of
>"client id" and what is the definition of "user" and how do the concepts
>relate?
>2. If both client ids and users are sensible distinct notions and we
>want to maintain both, why don't we just support quotas on both? If they
>both make sense then you would have a reason to set quotas at both
> levels.
>Why have this "mode" that you set that swaps between only being able to
> use
>one and the other? I should be able to set quotas at both levels. Going
>forward the model we had discussed with quotas was potentially being
> able
>to set quotas for many things independently (say at the topic level),
> and I
>don't think it would make sense to extend this mode approach to those.
>
> -Jay
>
> On Wed, Jun 8, 2016 at 12:56 PM, Rajini Sivaram <
> rajinisiva...@googlemail.com> wrote:
>
> > I would like to initiate the vote for KIP-55.
> >
> > The KIP details are here: KIP-55: Secure quotas for authenticated users
> > <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-55%3A+Secure+Quotas+for+Authenticated+Users
> > >
> > .
> >
> > The JIRA  KAFKA-3492   > >has
> > a draft PR here: https://github.com/apache/kafka/pull/1256.
> >
> > Thank you...
> >
> > Regards,
> >
> > Rajini
> >
>



-- 
Regards,

Rajini

Re: [VOTE] KIP-55: Secure quotas for authenticated users

[Jay Kreps]

Super sorry to come in late on this one. Rajini, I had two quick questions
I think we should be able to answer:

   1. Do client ids make sense in a world which has users? If not should we
   unify them the way Hadoop did (without auth the user is a kind of best
   effort honor system identity). This came up in the discussion thread but I
   didn't really see a crisp answer. Basically, what is the definition of
   "client id" and what is the definition of "user" and how do the concepts
   relate?
   2. If both client ids and users are sensible distinct notions and we
   want to maintain both, why don't we just support quotas on both? If they
   both make sense then you would have a reason to set quotas at both levels.
   Why have this "mode" that you set that swaps between only being able to use
   one and the other? I should be able to set quotas at both levels. Going
   forward the model we had discussed with quotas was potentially being able
   to set quotas for many things independently (say at the topic level), and I
   don't think it would make sense to extend this mode approach to those.

-Jay

On Wed, Jun 8, 2016 at 12:56 PM, Rajini Sivaram <
rajinisiva...@googlemail.com> wrote:

> I would like to initiate the vote for KIP-55.
>
> The KIP details are here: KIP-55: Secure quotas for authenticated users
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-55%3A+Secure+Quotas+for+Authenticated+Users
> >
> .
>
> The JIRA  KAFKA-3492   >has
> a draft PR here: https://github.com/apache/kafka/pull/1256.
>
> Thank you...
>
> Regards,
>
> Rajini
>

Re: [VOTE] KIP-55: Secure quotas for authenticated users

[Tom Crayford]

+1 (non-binding)

On Wed, Jun 8, 2016 at 8:56 PM, Rajini Sivaram  wrote:

> I would like to initiate the vote for KIP-55.
>
> The KIP details are here: KIP-55: Secure quotas for authenticated users
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-55%3A+Secure+Quotas+for+Authenticated+Users
> >
> .
>
> The JIRA  KAFKA-3492   >has
> a draft PR here: https://github.com/apache/kafka/pull/1256.
>
> Thank you...
>
> Regards,
>
> Rajini
>

Re: [VOTE] KIP-55: Secure quotas for authenticated users

[Rajini Sivaram]

Jun,

Oops, sorry, I hadn't realized that the last note was on the discuss
thread. Thank you for pointing it out. I have sent another note for voting.


On Wed, Jun 8, 2016 at 4:30 PM, Jun Rao  wrote:

> Rajini,
>
> Perhaps it will be clearer if you start the voting in a new thread (with
> VOTE in the subject).
>
> Thanks,
>
> Jun
>
> On Tue, Jun 7, 2016 at 1:55 PM, Rajini Sivaram <
> rajinisiva...@googlemail.com
> > wrote:
>
> > I would like to initiate the vote for KIP-55.
> >
> > The KIP details are here: KIP-55: Secure quotas for authenticated users
> > <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-55%3A+Secure+Quotas+for+Authenticated+Users
> > >
> > .
> >
> > The JIRA  KAFKA-3492   > >has
> > a draft PR here: https://github.com/apache/kafka/pull/1256.
> >
> > Thank you...
> >
> >
> > Regards,
> >
> > Rajini
> >
>



-- 
Regards,

Rajini

[VOTE] KIP-55: Secure quotas for authenticated users

[Rajini Sivaram]

I would like to initiate the vote for KIP-55.

The KIP details are here: KIP-55: Secure quotas for authenticated users

.

The JIRA  KAFKA-3492  has
a draft PR here: https://github.com/apache/kafka/pull/1256.

Thank you...

Regards,

Rajini

Re: [VOTE] KIP-55: Secure quotas for authenticated users

[Jun Rao]

Rajini,

Perhaps it will be clearer if you start the voting in a new thread (with
VOTE in the subject).

Thanks,

Jun

On Tue, Jun 7, 2016 at 1:55 PM, Rajini Sivaram  wrote:

> I would like to initiate the vote for KIP-55.
>
> The KIP details are here: KIP-55: Secure quotas for authenticated users
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-55%3A+Secure+Quotas+for+Authenticated+Users
> >
> .
>
> The JIRA  KAFKA-3492   >has
> a draft PR here: https://github.com/apache/kafka/pull/1256.
>
> Thank you...
>
>
> Regards,
>
> Rajini
>

[VOTE] KIP-55: Secure quotas for authenticated users

[Rajini Sivaram]

I would like to initiate the vote for KIP-55.

The KIP details are here: KIP-55: Secure quotas for authenticated users

.

The JIRA  KAFKA-3492  has
a draft PR here: https://github.com/apache/kafka/pull/1256.

Thank you...


Regards,

Rajini