Sorry Gwen, completely misunderstood the question :-).
* Does everyone have the privilege to create a new Group and use it to
consume from Topics he's already privileged on?
Yes in current proposal. I did not see an API to create group but if you
have a READ permission on a TOPIC and WRITE permission on that Group you
are free to join and consume.
* Will the CLI tool be used to manage group membership too?
Yes and I think that means I need to add ―group. Updating the KIP.
Thanks
for pointing this out.
* Groups are kind of ephemeral, right? If all consumers in the group
disconnect the group is gone, AFAIK. Do we preserve the ACLs? Or do we
treat the new group as completely new resource? Can we create ACLs
before the group exists, in anticipation of it getting created?
I have considered any auto delete and auto create as out of scope for
the
first release. So Right now I was going with preserving the acls. Do you
see any issues with this? Auto deleting would mean authorizer will now
have to get into implementation details of kafka which I was trying to
avoid.
Thanks
Parth
On 4/24/15, 11:33 AM, "Gwen Shapira" <gshap...@cloudera.com> wrote:
>We are not talking about same Groups :)
>
>I meant, Groups of consumers (which KIP-11 lists as a separate
>resource in the Privilege table)
>
>On Fri, Apr 24, 2015 at 11:31 AM, Parth Brahmbhatt
><pbrahmbh...@hortonworks.com> wrote:
>> I see Groups as something we can add incrementally in the current model.
>> The acls take principalType: name so groups can be represented as group:
>> groupName. We are not managing group memberships anywhere in kafka and I
>> don’t see the need to do so.
>>
>> So for a topic1 using the CLI an admin can add an acl to grant access to
>> group:kafka-test-users.
>>
>> The authorizer implementation can have a plugin to map authenticated
>>user
>> to groups ( This is how hadoop and storm works). The plugin could be
>> mapping user to linux/ldap/active directory groups but that is again
>>upto
>> the implementation.
>>
>> What we are offering is an interface that is extensible so these
>>features
>> can be added incrementally. I can add support for this in the first
>> release but don’t necessarily see why this would be absolute necessity.
>>
>> Thanks
>> Parth
>>
>> On 4/24/15, 11:00 AM, "Gwen Shapira" <gshap...@cloudera.com> wrote:
>>
>>>Thanks.
>>>
>>>One more thing I'm missing in the KIP is details on the Group resource
>>>(I think we discussed this and it was just not fully updated):
>>>
>>>* Does everyone have the privilege to create a new Group and use it to
>>>consume from Topics he's already privileged on?
>>>* Will the CLI tool be used to manage group membership too?
>>>* Groups are kind of ephemeral, right? If all consumers in the group
>>>disconnect the group is gone, AFAIK. Do we preserve the ACLs? Or do we
>>>treat the new group as completely new resource? Can we create ACLs
>>>before the group exists, in anticipation of it getting created?
>>>
>>>Its all small details, but it will be difficult to implement KIP-11
>>>without knowing the answers :)
>>>
>>>Gwen
>>>
>>>
>>>On Fri, Apr 24, 2015 at 9:58 AM, Parth Brahmbhatt
>>><pbrahmbh...@hortonworks.com> wrote:
>>>> You are right, moved it to the default implementation section.
>>>>
>>>> Thanks
>>>> Parth
>>>>
>>>> On 4/24/15, 9:52 AM, "Gwen Shapira" <gshap...@cloudera.com> wrote:
>>>>
>>>>>Sample ACL JSON and Zookeeper is in public API, but I thought it is
>>>>>part of DefaultAuthorizer (Since Sentry and Argus won't be using
>>>>>Zookeeper).
>>>>>Am I wrong? Or is it the KIP?
>>>>>
>>>>>On Fri, Apr 24, 2015 at 9:49 AM, Parth Brahmbhatt
>>>>><pbrahmbh...@hortonworks.com> wrote:
>>>>>> Thanks for clarifying Gwen, KIP updated.
>>>>>>
>>>>>> I tried to make the distinction by creating a section for all public
>>>>>>APIs
>>>>>>
>>>>>>https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorizat
>>>>>>io
>>>>>>n+
>>>>>>In
>>>>>> terface#KIP-11-AuthorizationInterface-PublicInterfacesandclasses
>>>>>>
>>>>>> Let me know if you think there is a better way to reflect this.
>>>>>>
>>>>>> Thanks
>>>>>> Parth
>>>>>>
>>>>>> On 4/24/15, 9:37 AM, "Gwen Shapira" <gshap...@cloudera.com> wrote:
>>>>>>
>>>>>>>+1 (non-binding)
>>>>>>>
>>>>>>>Two nitpicks for the wiki:
>>>>>>>* Heartbeat is probably a READ and not CLUSTER operation. I'm pretty
>>>>>>>sure new consumers need it to be part of a consumer group.
>>>>>>>* Can you clearly separate which parts are the API (common to every
>>>>>>>Authorizer) and which parts are DefaultAuthorizer implementation? It
>>>>>>>will make reviews and Authorizer implementations a bit easier to
>>>>>>>know
>>>>>>>exactly which is which.
>>>>>>>
>>>>>>>Gwen
>>>>>>>
>>>>>>>On Fri, Apr 24, 2015 at 9:28 AM, Parth Brahmbhatt
>>>>>>><pbrahmbh...@hortonworks.com> wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I would like to open KIP-11 for voting.
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>> Parth
>>>>>>>>
>>>>>>>> On 4/22/15, 1:56 PM, "Parth Brahmbhatt"
>>>>>>>><pbrahmbh...@hortonworks.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>>Hi Jeff,
>>>>>>>>>
>>>>>>>>>Thanks a lot for the review. I think you have a valid point about
>>>>>>>>>acls
>>>>>>>>>being duplicated and the simplest solution would be to modify acls
>>>>>>>>>class
>>>>>>>>>so they hold a set of principals instead of single principal. i.e
>>>>>>>>>
>>>>>>>>><user_a,user_b> has <READ,WRITE,DESCRIBE> Permissions on <Topic1>
>>>>>>>>>from
>>>>>>>>><Host1, Host2, Host3>.
>>>>>>>>>
>>>>>>>>>I think the evaluation order only matters for the permissionType
>>>>>>>>>which
>>>>>>>>>is
>>>>>>>>>Deny acls should be evaluated before allow acls. To give you an
>>>>>>>>>example
>>>>>>>>>suppose we have following acls
>>>>>>>>>
>>>>>>>>>acl1 -> user1 is allowed to READ from all hosts.
>>>>>>>>>acl2 -> host1 is allowed to READ regardless of who is the user.
>>>>>>>>>acl3 -> host2 is allowed to READ regardless of who is the user.
>>>>>>>>>
>>>>>>>>>acl4 -> user1 is denied to READ from host1.
>>>>>>>>>
>>>>>>>>>As stated in the KIP we first evaluate DENY so if user1 tries to
>>>>>>>>>access
>>>>>>>>>from host1 he will be denied(acl4), even though both user1 and
>>>>>>>>>host1
>>>>>>>>>has
>>>>>>>>>acl’s for allow with wildcards (acl1, acl2).
>>>>>>>>>If user1 tried to READ from host2 , the action will be allowed and
>>>>>>>>>it
>>>>>>>>>does
>>>>>>>>>not matter if we match acl3 or acl1 so I don’t think the
>>>>>>>>>evaluation
>>>>>>>>>order
>>>>>>>>>matters here.
>>>>>>>>>
>>>>>>>>>“Will people actually use hosts with users?” I really don’t know
>>>>>>>>>but
>>>>>>>>>given
>>>>>>>>>ACl’s are part of our Public APIs I thought it is better to try
>>>>>>>>>and
>>>>>>>>>cover
>>>>>>>>>more use cases. If others think this extra complexity is not worth
>>>>>>>>>the
>>>>>>>>>value its adding please raise your concerns so we can discuss if
>>>>>>>>>it
>>>>>>>>>should
>>>>>>>>>be removed from the acl structure. Note that even in absence of
>>>>>>>>>hosts
>>>>>>>>>from
>>>>>>>>>ACL users will still be able to whitelist/blacklist host as long
>>>>>>>>>as
>>>>>>>>>we
>>>>>>>>>start supporting principalType = “host”, easy to add and can be an
>>>>>>>>>incremental improvement. They will however loose the ability to
>>>>>>>>>restrict
>>>>>>>>>access to users just from a set of hosts.
>>>>>>>>>
>>>>>>>>>We agreed to offer a CLI to overcome the JSON acl config
>>>>>>>>>https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authori
>>>>>>>>>za
>>>>>>>>>ti
>>>>>>>>>on
>>>>>>>>>+I
>>>>>>>>>n
>>>>>>>>>terface#KIP-11-AuthorizationInterface-AclManagement(CLI). I still
>>>>>>>>>like
>>>>>>>>>Jsons but that probably has something to do with me being a
>>>>>>>>>developer
>>>>>>>>>:-).
>>>>>>>>>
>>>>>>>>>Thanks
>>>>>>>>>Parth
>>>>>>>>>
>>>>>>>>>On 4/22/15, 11:38 AM, "Jeff Holoman" <jholo...@cloudera.com>
>>>>>>>>>wrote:
>>>>>>>>>
>>>>>>>>>>Parth,
>>>>>>>>>>
>>>>>>>>>>This is a long thread, so trying to keep up here, sorry if this
>>>>>>>>>>has
>>>>>>>>>>been
>>>>>>>>>>covered before. First, great job on the KIP proposal and work so
>>>>>>>>>>far.
>>>>>>>>>>
>>>>>>>>>>Are we sure that we want to tie host level access to a given
>>>>>>>>>>user?
>>>>>>>>>>My
>>>>>>>>>>understanding is that the ACL will be (omitting some fields)
>>>>>>>>>>
>>>>>>>>>>user_a, host1, host2, host3
>>>>>>>>>>user_b, host1, host2, host3
>>>>>>>>>>
>>>>>>>>>>So there would potentially be a lot of redundancy in the configs.
>>>>>>>>>>Does
>>>>>>>>>>it
>>>>>>>>>>make sense to have hosts be at the same level as principal in the
>>>>>>>>>>hierarchy? This way you could just blanket the allowed / denied
>>>>>>>>>>hosts
>>>>>>>>>>and
>>>>>>>>>>only have to worry about the users. So if you follow this, then
>>>>>>>>>>
>>>>>>>>>>we can wildcard the user so we can have a separate list of just
>>>>>>>>>>host-based
>>>>>>>>>>access. What's the order that the perms would be evaluated if a
>>>>>>>>>>there
>>>>>>>>>>was
>>>>>>>>>>more than one match on a principal ?
>>>>>>>>>>
>>>>>>>>>>Is the thought that there wouldn't usually be much overlap on
>>>>>>>>>>hosts?
>>>>>>>>>>I
>>>>>>>>>>guess I can imagine a scenario where I want to offline/online
>>>>>>>>>>access
>>>>>>>>>>to a
>>>>>>>>>>particular hosts or set of hosts and if there was overlap, I'm
>>>>>>>>>>doing
>>>>>>>>>>a
>>>>>>>>>>bunch of alter commands for just a single host. Maybe this is too
>>>>>>>>>>contrived
>>>>>>>>>>an example?
>>>>>>>>>>
>>>>>>>>>>I agree that having this level of granularity gives flexibility
>>>>>>>>>>but I
>>>>>>>>>>wonder if people will actually use it and not just * the hosts
>>>>>>>>>>for
>>>>>>>>>>a
>>>>>>>>>>given
>>>>>>>>>>user and create separate "global" list as i mentioned above?
>>>>>>>>>>
>>>>>>>>>>The only other system I know of that ties users with hosts for
>>>>>>>>>>access
>>>>>>>>>>is
>>>>>>>>>>MySql and I don't love that model. Companies usually standardize
>>>>>>>>>>on
>>>>>>>>>>group
>>>>>>>>>>authorization anyway, are we complicating that issue with the
>>>>>>>>>>inclusion
>>>>>>>>>>of
>>>>>>>>>>hosts attached to users? Additionally I worry about the debt of
>>>>>>>>>>big
>>>>>>>>>>JSON
>>>>>>>>>>configs in the first place, most non-developers find them
>>>>>>>>>>non-intuitive
>>>>>>>>>>already, so anything to ease this I think would be beneficial.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>Thanks
>>>>>>>>>>
>>>>>>>>>>Jeff
>>>>>>>>>>
>>>>>>>>>>On Wed, Apr 22, 2015 at 2:22 PM, Parth Brahmbhatt <
>>>>>>>>>>pbrahmbh...@hortonworks.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Sorry I missed your last questions. I am +0 on adding ―host
>>>>>>>>>>>option
>>>>>>>>>>>for
>>>>>>>>>>> ―list, we could add it for symmetry. Again if this is only a
>>>>>>>>>>>CLI
>>>>>>>>>>>change
>>>>>>>>>>>it
>>>>>>>>>>> can be added later if you mean adding this in authorizer
>>>>>>>>>>>interface
>>>>>>>>>>>then
>>>>>>>>>>>we
>>>>>>>>>>> should make a decision now.
>>>>>>>>>>>
>>>>>>>>>>> Given a choice I would like to actually keep only one option
>>>>>>>>>>>which
>>>>>>>>>>>is
>>>>>>>>>>> resource based get (remove even the get based on principal). I
>>>>>>>>>>>see
>>>>>>>>>>>those
>>>>>>>>>>> (getAcl for principal or host) as special filtering case which
>>>>>>>>>>>can
>>>>>>>>>>>easily
>>>>>>>>>>> be achieved by a third party tool by doing "list all topics"
>>>>>>>>>>>and
>>>>>>>>>>>calling
>>>>>>>>>>> getAcls for each topic and applying filtering logic on that. I
>>>>>>>>>>>really
>>>>>>>>>>> don’t see the need to make those first class citizens of the
>>>>>>>>>>>authorizer
>>>>>>>>>>> interface given these kind of queries will be issued outside of
>>>>>>>>>>>broker
>>>>>>>>>>>JVM
>>>>>>>>>>> so they will not benefit from the caching and because the
>>>>>>>>>>>storage
>>>>>>>>>>>will
>>>>>>>>>>>be
>>>>>>>>>>> indexed on resource both these options even as a first class
>>>>>>>>>>>API
>>>>>>>>>>>will
>>>>>>>>>>>just
>>>>>>>>>>> scan all topic acls and apply filtering logic.
>>>>>>>>>>>
>>>>>>>>>>> Thanks
>>>>>>>>>>> Parth
>>>>>>>>>>>
>>>>>>>>>>> On 4/22/15, 11:08 AM, "Parth Brahmbhatt"
>>>>>>>>>>><pbrahmbh...@hortonworks.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> >Please see all the available options here
>>>>>>>>>>> >
>>>>>>>>>>>
>>>>>>>>>>>https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Autho
>>>>>>>>>>>ri
>>>>>>>>>>>za
>>>>>>>>>>>ti
>>>>>>>>>>>on
>>>>>>>>>>>+
>>>>>>>>>>>I
>>>>>>>>>>> >nterface#KIP-11-AuthorizationInterface-AclManagement(CLI) . I
>>>>>>>>>>>think
>>>>>>>>>>>it
>>>>>>>>>>> >covers both hosts and operations and allows to specify a list
>>>>>>>>>>>for
>>>>>>>>>>>both.
>>>>>>>>>>> >
>>>>>>>>>>> >Thanks
>>>>>>>>>>> >Parth
>>>>>>>>>>> >
>>>>>>>>>>> >From: Tom Graves
>>>>>>>>>>><tgraves...@yahoo.com<mailto:tgraves...@yahoo.com>>
>>>>>>>>>>> >Reply-To: Tom Graves
>>>>>>>>>>><tgraves...@yahoo.com<mailto:tgraves...@yahoo.com>>
>>>>>>>>>>> >Date: Wednesday, April 22, 2015 at 11:02 AM
>>>>>>>>>>> >To: Parth Brahmbhatt
>>>>>>>>>>>
>>>>>>>>>>>><pbrahmbh...@hortonworks.com<mailto:pbrahmbh...@hortonworks.com
>>>>>>>>>>>>>>
>>>>>>>>>>>>,
>>>>>>>>>>> >"dev@kafka.apache.org<mailto:dev@kafka.apache.org>"
>>>>>>>>>>> ><dev@kafka.apache.org<mailto:dev@kafka.apache.org>>
>>>>>>>>>>> >Subject: Re: [DISCUSS] KIP-11- Authorization design for kafka
>>>>>>>>>>>security
>>>>>>>>>>> >
>>>>>>>>>>> >Thanks for the explanations Parth.
>>>>>>>>>>> >
>>>>>>>>>>> >On the configs questions, the way I see it is its more likely
>>>>>>>>>>>to
>>>>>>>>>>> >accidentally give everyone access, especially since you have
>>>>>>>>>>>to
>>>>>>>>>>>run
>>>>>>>>>>>a
>>>>>>>>>>> >separate command to change the acls. If there was some config
>>>>>>>>>>>for
>>>>>>>>>>> >defaults, a cluster admin could change that to be nobody or
>>>>>>>>>>>certain
>>>>>>>>>>>set
>>>>>>>>>>> >of users, then grant others permissions. This would also
>>>>>>>>>>>remove
>>>>>>>>>>>the
>>>>>>>>>>>race
>>>>>>>>>>> >between commands. This is something you can always add later
>>>>>>>>>>>though
>>>>>>>>>>>if
>>>>>>>>>>> >people request it.
>>>>>>>>>>> >
>>>>>>>>>>> >So in kafka-acl.sh how do I actually tell it what the
>>>>>>>>>>>operation
>>>>>>>>>>>is?
>>>>>>>>>>> >kafka-acl.sh --topic testtopic --add --grandprincipal
>>>>>>>>>>>user:joe,user:kate
>>>>>>>>>>> >
>>>>>>>>>>> >where does READ, WRITE, etc go? Can specify as a list so I
>>>>>>>>>>>don't
>>>>>>>>>>>have
>>>>>>>>>>>to
>>>>>>>>>>> >run this a bunch of times for each.
>>>>>>>>>>> >
>>>>>>>>>>> >Do you want to have a --host option for --list so that admins
>>>>>>>>>>>could
>>>>>>>>>>>see
>>>>>>>>>>> >what acls apply to specific host(s)?
>>>>>>>>>>> >
>>>>>>>>>>> >Tom
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>> >On Wednesday, April 22, 2015 11:38 AM, Parth Brahmbhatt
>>>>>>>>>>>
>>>>>>>>>>>><pbrahmbh...@hortonworks.com<mailto:pbrahmbh...@hortonworks.com
>>>>>>>>>>>>>>
>>>>>>>>>>>wrote:
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>> >FYI, I have modified the KIP to include group as resource. In
>>>>>>>>>>>order
>>>>>>>>>>>to
>>>>>>>>>>> >access “joinGroup” and “commitOFfset” APIs the user will need
>>>>>>>>>>>a
>>>>>>>>>>>read
>>>>>>>>>>> >permission on topic and WRITE permission on group.
>>>>>>>>>>> >
>>>>>>>>>>> >I plan to open a VOTE thread by noon if there are no more
>>>>>>>>>>>concerns.
>>>>>>>>>>> >
>>>>>>>>>>> >Thanks
>>>>>>>>>>> >Parth
>>>>>>>>>>> >
>>>>>>>>>>> >On 4/22/15, 9:03 AM, "Tom Graves"
>>>>>>>>>>>
>>>>>>>>>>>><tgraves...@yahoo.com.INVALID<mailto:tgraves...@yahoo.com.INVAL
>>>>>>>>>>>>ID
>>>>>>>>>>>>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>> >
>>>>>>>>>>> >>Hey everyone,
>>>>>>>>>>> >>Sorry to jump in on the conversation so late. I'm new to
>>>>>>>>>>>Kafka.
>>>>>>>>>>>I'll
>>>>>>>>>>> >>apologize in advance if you have already covered some of my
>>>>>>>>>>>questions. I
>>>>>>>>>>> >>read through the wiki and had some comments and questions.
>>>>>>>>>>> >>1) public enum Operation needs EDIT changed to ALTER
>>>>>>>>>>> >
>>>>>>>>>>> >> Done.
>>>>>>>>>>> >
>>>>>>>>>>> >>2) Does the Authorizer class need a setAcls? Rather then
>>>>>>>>>>>just
>>>>>>>>>>>add
>>>>>>>>>>>to
>>>>>>>>>>>be
>>>>>>>>>>> >>able to set to explicit list and overwrite what was there? I
>>>>>>>>>>>see
>>>>>>>>>>>the
>>>>>>>>>>> >>kafka-acl.sh lists a removeall so I guess you could do
>>>>>>>>>>>removeall
>>>>>>>>>>>and
>>>>>>>>>>>then
>>>>>>>>>>> >>add. I also don't see a removeall in the Authorizer class,
>>>>>>>>>>>is
>>>>>>>>>>>it
>>>>>>>>>>>going
>>>>>>>>>>> >>to loop through them all to remove each one?
>>>>>>>>>>> >
>>>>>>>>>>> > There is an overloaded version of removeAcls in the
>>>>>>>>>>>interface
>>>>>>>>>>>that
>>>>>>>>>>> >takes
>>>>>>>>>>> >in resource as the only input and as described in the javadoc
>>>>>>>>>>>all
>>>>>>>>>>>the
>>>>>>>>>>>acls
>>>>>>>>>>> >attached to that resource will be deleted. To cover the setAcl
>>>>>>>>>>>use
>>>>>>>>>>>case
>>>>>>>>>>> >the caller can first call remove and then add.
>>>>>>>>>>> >
>>>>>>>>>>> >>3) Can someone tell me what the use case to do acls based on
>>>>>>>>>>>the
>>>>>>>>>>>hosts?
>>>>>>>>>>> >>I can see some possibilities just wondering if we can
>>>>>>>>>>>concrete
>>>>>>>>>>>ones
>>>>>>>>>>>where
>>>>>>>>>>> >>one user is allowed from one host but not another.
>>>>>>>>>>> >
>>>>>>>>>>> > I am not sure if I understand the question given the use
>>>>>>>>>>>case
>>>>>>>>>>>you
>>>>>>>>>>> >described in your question is what we are trying to cover with
>>>>>>>>>>>use
>>>>>>>>>>>of
>>>>>>>>>>> >hosts in Acl. There are some additional use cases like “allow
>>>>>>>>>>>access
>>>>>>>>>>>to
>>>>>>>>>>> >any user from host1,host2” but I think primarily it gives the
>>>>>>>>>>>admins
>>>>>>>>>>>the
>>>>>>>>>>> >ability to define acls at a more granular level.
>>>>>>>>>>> >
>>>>>>>>>>> >>4) I'm a bit unclear how the "resource" works in the
>>>>>>>>>>>Authorizer
>>>>>>>>>>>class.
>>>>>>>>>>> >>From what I see we have 2 resources - topics and cluster.
>>>>>>>>>>>If I
>>>>>>>>>>>want
>>>>>>>>>>>to
>>>>>>>>>>> >>add an acl to allow "joe" to CREATE for the cluster then I
>>>>>>>>>>>call
>>>>>>>>>>>addAcls
>>>>>>>>>>> >>with Acl("user: joe", ALLOW, Set(*), Set(CREATE)) and
>>>>>>>>>>>"cluster"?
>>>>>>>>>>>What
>>>>>>>>>>> >>if I want to call addAcls for DESCRIBE on a topic? Is the
>>>>>>>>>>>resource
>>>>>>>>>>>then
>>>>>>>>>>> >>"topic" or is it the topic name?
>>>>>>>>>>> >
>>>>>>>>>>> > We now have 3 resources(added group), please see the
>>>>>>>>>>>updated
>>>>>>>>>>>doc.
>>>>>>>>>>>The
>>>>>>>>>>> >CREATE acl that you described is correct. For any topic
>>>>>>>>>>>operation
>>>>>>>>>>>you
>>>>>>>>>>> >should use topic name as the resource name and for group the
>>>>>>>>>>>user
>>>>>>>>>>>will
>>>>>>>>>>> >provide groupId as resource name.
>>>>>>>>>>> >
>>>>>>>>>>> >>5) reassigning partitions is a CLUSTER_ACTION or superuser?
>>>>>>>>>>>Its
>>>>>>>>>>>not
>>>>>>>>>>> >>totally clear to me the differences between these. what
>>>>>>>>>>>about
>>>>>>>>>>>increasing
>>>>>>>>>>> >># of partitions?
>>>>>>>>>>> >
>>>>>>>>>>> > I see this as an alter topic operation so it is at topic
>>>>>>>>>>>level
>>>>>>>>>>>and
>>>>>>>>>>>the
>>>>>>>>>>> >user must have alter permissions on topic.
>>>>>>>>>>> >
>>>>>>>>>>> >>6) groups are mentioned, are we supporting right away or is
>>>>>>>>>>>that
>>>>>>>>>>>a
>>>>>>>>>>>follow
>>>>>>>>>>> >>on item? (is there going to be a kafka.supergroups)
>>>>>>>>>>> >
>>>>>>>>>>> > I think it can be a separate jira just for braking down
>>>>>>>>>>>the
>>>>>>>>>>>code
>>>>>>>>>>> >review
>>>>>>>>>>> >in smaller chunk. We will support it in first version but I
>>>>>>>>>>>think
>>>>>>>>>>>if
>>>>>>>>>>>we
>>>>>>>>>>> >can not do it for any reason that should not block a release
>>>>>>>>>>>with
>>>>>>>>>>>all
>>>>>>>>>>>the
>>>>>>>>>>> >other authZ work. We made deliberate design choices (like
>>>>>>>>>>>introducing
>>>>>>>>>>>a
>>>>>>>>>>> >principalType in KafkaPrinciapl) to allow supporting groups as
>>>>>>>>>>>an
>>>>>>>>>>> >incremental change.
>>>>>>>>>>> >
>>>>>>>>>>> >>7) Are there config options for setting acls when I create my
>>>>>>>>>>>topic?
>>>>>>>>>>>Or
>>>>>>>>>>> >>do I have to create my topic and then run the kafka-acl.sh
>>>>>>>>>>>script
>>>>>>>>>>>to
>>>>>>>>>>>set
>>>>>>>>>>> >>them? Although its very small, there would be possible race
>>>>>>>>>>>there
>>>>>>>>>>>that
>>>>>>>>>>> >>someone could start producing to topic before acls are set.
>>>>>>>>>>> >
>>>>>>>>>>> > We discussed this yesterday and we agreed to go with
>>>>>>>>>>>kafka-acl.sh.
>>>>>>>>>>>Yes
>>>>>>>>>>> >there is a very very small window of vulnerability but I think
>>>>>>>>>>>that
>>>>>>>>>>>really
>>>>>>>>>>> >does not warrant to change the decision in this case.
>>>>>>>>>>> >
>>>>>>>>>>> >>8) are there configs for cluster level acl defaults? Or does
>>>>>>>>>>>it
>>>>>>>>>>>default
>>>>>>>>>>> >>to superusers on bringing up new cluster and you have to
>>>>>>>>>>>modify
>>>>>>>>>>>with
>>>>>>>>>>>cli.
>>>>>>>>>>> >>thanks,Tom
>>>>>>>>>>> >
>>>>>>>>>>> > No defaults, the default is superusers will have full
>>>>>>>>>>>access.
>>>>>>>>>>>I
>>>>>>>>>>>don’t
>>>>>>>>>>> >think making assumptions about ones security requirement
>>>>>>>>>>>should
>>>>>>>>>>>be
>>>>>>>>>>>our
>>>>>>>>>>> >burden.
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>> >>
>>>>>>>>>>> >> On Tuesday, April 21, 2015 7:10 PM, Parth Brahmbhatt
>>>>>>>>>>>
>>>>>>>>>>>>><pbrahmbh...@hortonworks.com<mailto:pbrahmbh...@hortonworks.co
>>>>>>>>>>>>>m>
>>>>>>>>>>>>>>
>>>>>>>>>>>wrote:
>>>>>>>>>>> >>
>>>>>>>>>>> >>
>>>>>>>>>>> >> I have added the notes to KIP-11 Open question sections.
>>>>>>>>>>> >>
>>>>>>>>>>> >>Thanks
>>>>>>>>>>> >>Parth
>>>>>>>>>>> >>
>>>>>>>>>>> >>On 4/21/15, 4:49 PM, "Gwen Shapira"
>>>>>>>>>>> >><gshap...@cloudera.com<mailto:gshap...@cloudera.com>> wrote:
>>>>>>>>>>> >>
>>>>>>>>>>> >>>Adding my notes from today's call to the thread:
>>>>>>>>>>> >>>
>>>>>>>>>>> >>>** Deny or Allow all by default? We will add a configuration
>>>>>>>>>>>to
>>>>>>>>>>> >>>control this. The configuration will default to “allow” for
>>>>>>>>>>>backward
>>>>>>>>>>> >>>compatibility. Security admins can set it to "deny"
>>>>>>>>>>> >>>
>>>>>>>>>>> >>>** Storing ACLs for default authorizers: We'll store them in
>>>>>>>>>>>ZK.
>>>>>>>>>>>We'll
>>>>>>>>>>> >>>support pointing the authorizer to any ZK.
>>>>>>>>>>> >>>The use of ZK will be internal to the default authorizer.
>>>>>>>>>>>Authorizer
>>>>>>>>>>> >>>reads ACLs from cache every hour. We proposed having
>>>>>>>>>>>mechanism
>>>>>>>>>>> >>>(possibly via new ZK node) to tell broker to refresh the
>>>>>>>>>>>cache
>>>>>>>>>>> >>>immediately.
>>>>>>>>>>> >>>
>>>>>>>>>>> >>>** Support deny as permission type - we agreed to keep this.
>>>>>>>>>>> >>>
>>>>>>>>>>> >>>** Mapping operations to API: We may need to add Group as a
>>>>>>>>>>>resource,
>>>>>>>>>>> >>>with JoinGroup and OffsetCommit require privilege on the
>>>>>>>>>>>consumer
>>>>>>>>>>> >>>group.
>>>>>>>>>>> >>>This can be something we pass now and authorizers can
>>>>>>>>>>>support
>>>>>>>>>>>in
>>>>>>>>>>> >>>future. - Jay will write specifics to the mailing list
>>>>>>>>>>>discussion.
>>>>>>>>>>> >>>
>>>>>>>>>>> >>>On Tue, Apr 21, 2015 at 4:32 PM, Jay Kreps
>>>>>>>>>>> >>><jay.kr...@gmail.com<mailto:jay.kr...@gmail.com>> wrote:
>>>>>>>>>>> >>>> Following up on the KIP discussion. Two options for
>>>>>>>>>>>authorizing
>>>>>>>>>>> >>>>consumers
>>>>>>>>>>> >>>> to read topic "t" as part of group "g":
>>>>>>>>>>> >>>> 1. READ permission on resource /topic/t
>>>>>>>>>>> >>>> 2. READ permission on resource /topic/t AND WRITE
>>>>>>>>>>>permission
>>>>>>>>>>>on
>>>>>>>>>>> >>>>/group/g
>>>>>>>>>>> >>>>
>>>>>>>>>>> >>>> The advantage of (1) is that it is simpler. The
>>>>>>>>>>>disadvantage
>>>>>>>>>>>is
>>>>>>>>>>>that
>>>>>>>>>>> >>>>any
>>>>>>>>>>> >>>> member of any group that reads from t can commit offsets
>>>>>>>>>>>as
>>>>>>>>>>>any
>>>>>>>>>>>other
>>>>>>>>>>> >>>> member of a different group. This doesn't effect data
>>>>>>>>>>>security
>>>>>>>>>>>(who
>>>>>>>>>>> >>>>can
>>>>>>>>>>> >>>> access what) but it is a bit of a management issue--a
>>>>>>>>>>>malicious
>>>>>>>>>>>person
>>>>>>>>>>> >>>>can
>>>>>>>>>>> >>>> cause data loss or duplicates for another consumer by
>>>>>>>>>>>committing
>>>>>>>>>>> >>>>offset.
>>>>>>>>>>> >>>>
>>>>>>>>>>> >>>> I think I favor (2) but it's worth it to think it through.
>>>>>>>>>>> >>>>
>>>>>>>>>>> >>>> -Jay
>>>>>>>>>>> >>>>
>>>>>>>>>>> >>>> On Tue, Apr 21, 2015 at 2:43 PM, Parth Brahmbhatt <
>>>>>>>>>>> >>>>
>>>>>>>>>>>pbrahmbh...@hortonworks.com<mailto:pbrahmbh...@hortonworks.com>>
>>>>>>>>>>> >>>>wrote:
>>>>>>>>>>> >>>>
>>>>>>>>>>> >>>>> Hey Jun,
>>>>>>>>>>> >>>>>
>>>>>>>>>>> >>>>> Yes and we support wild cards for all acl entities
>>>>>>>>>>>principal,
>>>>>>>>>>>hosts
>>>>>>>>>>> >>>>>and
>>>>>>>>>>> >>>>> operation.
>>>>>>>>>>> >>>>>
>>>>>>>>>>> >>>>> Thanks
>>>>>>>>>>> >>>>> Parth
>>>>>>>>>>> >>>>>
>>>>>>>>>>> >>>>> On 4/21/15, 9:06 AM, "Jun Rao"
>>>>>>>>>>> >>>>><j...@confluent.io<mailto:j...@confluent.io>> wrote:
>>>>>>>>>>> >>>>>
>>>>>>>>>>> >>>>> >Harsha, Parth,
>>>>>>>>>>> >>>>> >
>>>>>>>>>>> >>>>> >Thanks for the clarification. This makes sense. Perhaps
>>>>>>>>>>>we
>>>>>>>>>>>can
>>>>>>>>>>> >>>>>clarify the
>>>>>>>>>>> >>>>> >meaning of those rules in the wiki.
>>>>>>>>>>> >>>>> >
>>>>>>>>>>> >>>>> >Related to this, it seems that we need to support
>>>>>>>>>>>wildcard
>>>>>>>>>>>in
>>>>>>>>>>> >>>>>cli/request
>>>>>>>>>>> >>>>> >protocol for topics?
>>>>>>>>>>> >>>>> >
>>>>>>>>>>> >>>>> >Jun
>>>>>>>>>>> >>>>> >
>>>>>>>>>>> >>>>> >On Mon, Apr 20, 2015 at 9:07 PM, Parth Brahmbhatt <
>>>>>>>>>>> >>>>>
>>>>>>>>>>>>pbrahmbh...@hortonworks.com<mailto:pbrahmbh...@hortonworks.com>
>>>>>>>>>>>>>
>>>>>>>>>>> >>>>>wrote:
>>>>>>>>>>> >>>>> >
>>>>>>>>>>> >>>>> >> The iptables on unix supports the DENY operator, not
>>>>>>>>>>>that
>>>>>>>>>>>it
>>>>>>>>>>> >>>>>should
>>>>>>>>>>> >>>>> >> matter. The deny operator can also be used to specify
>>>>>>>>>>>³allow
>>>>>>>>>>>user1
>>>>>>>>>>> >>>>>to
>>>>>>>>>>> >>>>> >>READ
>>>>>>>>>>> >>>>> >> from topic1 from all hosts but host1,host2². Again we
>>>>>>>>>>>could
>>>>>>>>>>>add a
>>>>>>>>>>> >>>>>host
>>>>>>>>>>> >>>>> >> group semantic and extra complexity around that, not
>>>>>>>>>>>sure
>>>>>>>>>>>if
>>>>>>>>>>>its
>>>>>>>>>>> >>>>>worth
>>>>>>>>>>> >>>>> >>it.
>>>>>>>>>>> >>>>> >> In addition with DENY operator you are now not forced
>>>>>>>>>>>to
>>>>>>>>>>>create a
>>>>>>>>>>> >>>>> >>special
>>>>>>>>>>> >>>>> >> group just to support the authorization use case. I am
>>>>>>>>>>>not
>>>>>>>>>>> >>>>>convinced
>>>>>>>>>>> >>>>> >>that
>>>>>>>>>>> >>>>> >> the operator it self is really all that confusing.
>>>>>>>>>>>There
>>>>>>>>>>>are 3
>>>>>>>>>>> >>>>>practical
>>>>>>>>>>> >>>>> >> use cases:
>>>>>>>>>>> >>>>> >> - Resource with no acl what so ever -> allow access to
>>>>>>>>>>>everyone (
>>>>>>>>>>> >>>>>just
>>>>>>>>>>> >>>>> >>for
>>>>>>>>>>> >>>>> >> backward compatibility, I would much rather fail close
>>>>>>>>>>>and
>>>>>>>>>>>force
>>>>>>>>>>> >>>>>users
>>>>>>>>>>> >>>>> >>to
>>>>>>>>>>> >>>>> >> explicitly grant acls that allows access to all
>>>>>>>>>>>users.)
>>>>>>>>>>> >>>>> >> - Resource with some acl attached -> only users that
>>>>>>>>>>>have
>>>>>>>>>>>a
>>>>>>>>>>> >>>>>matching
>>>>>>>>>>> >>>>> >>allow
>>>>>>>>>>> >>>>> >> acl are allowed (i.e. ³allow READ access to topic1 to
>>>>>>>>>>>user1
>>>>>>>>>>>from
>>>>>>>>>>> >>>>>all
>>>>>>>>>>> >>>>> >> hosts², only user1 has READ access and no other user
>>>>>>>>>>>has
>>>>>>>>>>>access of
>>>>>>>>>>> >>>>>any
>>>>>>>>>>> >>>>> >> kind)
>>>>>>>>>>> >>>>> >> - Resource with some allow and some deny acl attached
>>>>>>>>>>>->
>>>>>>>>>>>users
>>>>>>>>>>>are
>>>>>>>>>>> >>>>> >>allowed
>>>>>>>>>>> >>>>> >> to perform operation only when they satisfy allow acl
>>>>>>>>>>>and
>>>>>>>>>>>do
>>>>>>>>>>>not
>>>>>>>>>>> >>>>>have
>>>>>>>>>>> >>>>> >> conflicting deny acl. Users that have no acl(allow or
>>>>>>>>>>>deny)
>>>>>>>>>>>will
>>>>>>>>>>> >>>>>still
>>>>>>>>>>> >>>>> >>not
>>>>>>>>>>> >>>>> >> have any access. (i.e. ³allow READ access to topic1 to
>>>>>>>>>>>user1
>>>>>>>>>>>from
>>>>>>>>>>> >>>>>all
>>>>>>>>>>> >>>>> >> hosts except host1 and host², only user1 has access
>>>>>>>>>>>but
>>>>>>>>>>>not
>>>>>>>>>>>from
>>>>>>>>>>> >>>>>host1
>>>>>>>>>>> >>>>> >>an
>>>>>>>>>>> >>>>> >> host2)
>>>>>>>>>>> >>>>> >>
>>>>>>>>>>> >>>>> >> I think we need to make a decision on deny primarily
>>>>>>>>>>>because
>>>>>>>>>>>with
>>>>>>>>>>> >>>>> >> introduction of acl management API, Acl is now a
>>>>>>>>>>>public
>>>>>>>>>>>class
>>>>>>>>>>>that
>>>>>>>>>>> >>>>>will
>>>>>>>>>>> >>>>> >>be
>>>>>>>>>>> >>>>> >> used by Ranger/Santry and other authroization
>>>>>>>>>>>providers.
>>>>>>>>>>>In
>>>>>>>>>>> >>>>>Current
>>>>>>>>>>> >>>>> >>design
>>>>>>>>>>> >>>>> >> the acl has a permissionType enum field with possible
>>>>>>>>>>>values
>>>>>>>>>>>of
>>>>>>>>>>> >>>>>Allow
>>>>>>>>>>> >>>>> >>and
>>>>>>>>>>> >>>>> >> Deny. If we chose to remove deny we can assume all
>>>>>>>>>>>acls
>>>>>>>>>>>to
>>>>>>>>>>>be
>>>>>>>>>>>of
>>>>>>>>>>> >>>>>allow
>>>>>>>>>>> >>>>> >> type and remove the permissionType field completely.
>>>>>>>>>>> >>>>> >>
>>>>>>>>>>> >>>>> >> Thanks
>>>>>>>>>>> >>>>> >> Parth
>>>>>>>>>>> >>>>> >>
>>>>>>>>>>> >>>>> >> On 4/20/15, 6:12 PM, "Gwen Shapira"
>>>>>>>>>>> >>>>><gshap...@cloudera.com<mailto:gshap...@cloudera.com>>
>>>>>>>>>>>wrote:
>>>>>>>>>>> >>>>> >>
>>>>>>>>>>> >>>>> >> >I think thats how its done in pretty much any system
>>>>>>>>>>>I
>>>>>>>>>>>can
>>>>>>>>>>>think
>>>>>>>>>>> >>>>>of.
>>>>>>>>>>> >>>>> >> >
>>>>>>>>>>> >>>>> >>
>>>>>>>>>>> >>>>> >>
>>>>>>>>>>> >>>>>
>>>>>>>>>>> >>>>>
>>>>>>>>>>> >>
>>>>>>>>>>> >>
>>>>>>>>>>> >>
>>>>>>>>>>> >>
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>--
>>>>>>>>>>Jeff Holoman
>>>>>>>>>>Systems Engineer
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>
