Hey everyone,
Sorry to jump in on the conversation so late. I'm new to Kafka. I'll apologize
in advance if you have already covered some of my questions. I read through
the wiki and had some comments and questions.
1) public enum Operation needs EDIT changed to ALTER
2) Does the Authorizer
FYI, I have modified the KIP to include group as resource. In order to
access “joinGroup” and “commitOFfset” APIs the user will need a read
permission on topic and WRITE permission on group.
I plan to open a VOTE thread by noon if there are no more concerns.
Thanks
Parth
On 4/22/15, 9:03 AM,
Thanks for the explanations Parth.
On the configs questions, the way I see it is its more likely to accidentally
give everyone access, especially since you have to run a separate command to
change the acls. If there was some config for defaults, a cluster admin could
change that to be nobody or
@kafka.apache.orgmailto:dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-11- Authorization design for kafka security
Thanks for the explanations Parth.
On the configs questions, the way I see it is its more likely to accidentally
give everyone access, especially since you have to run a separate command
.
ThanksParth
From: Tom Graves tgraves...@yahoo.com
Reply-To: Tom Graves tgraves...@yahoo.com
Date: Wednesday, April 22, 2015 at 11:02 AM
To: Parth Brahmbhatt pbrahmbh...@hortonworks.com, dev@kafka.apache.org
dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-11- Authorization design for kafka security
To: Parth Brahmbhatt
pbrahmbh...@hortonworks.commailto:pbrahmbh...@hortonworks.com,
dev@kafka.apache.orgmailto:dev@kafka.apache.org
dev@kafka.apache.orgmailto:dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-11- Authorization design for kafka security
Thanks for the explanations Parth.
On the configs
: Wednesday, April 22, 2015 at 11:02 AM
To: Parth Brahmbhatt
pbrahmbh...@hortonworks.commailto:pbrahmbh...@hortonworks.com,
dev@kafka.apache.orgmailto:dev@kafka.apache.org
dev@kafka.apache.orgmailto:dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-11- Authorization design for kafka security
Thanks
...@yahoo.commailto:tgraves...@yahoo.com
Date: Wednesday, April 22, 2015 at 11:02 AM
To: Parth Brahmbhatt
pbrahmbh...@hortonworks.commailto:pbrahmbh...@hortonworks.com,
dev@kafka.apache.orgmailto:dev@kafka.apache.org
dev@kafka.apache.orgmailto:dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-11
@kafka.apache.orgmailto:dev@kafka.apache.org
dev@kafka.apache.orgmailto:dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-11- Authorization design for kafka security
Thanks for the explanations Parth.
On the configs questions, the way I see it is its more likely to
accidentally give everyone access
@kafka.apache.org
dev@kafka.apache.orgmailto:dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-11- Authorization design for kafka security
Thanks for the explanations Parth.
On the configs questions, the way I see it is its more likely to
accidentally give everyone access, especially since you have to run
22, 2015 at 11:02 AM
To: Parth Brahmbhatt
pbrahmbh...@hortonworks.commailto:pbrahmbh...@hortonworks.com,
dev@kafka.apache.orgmailto:dev@kafka.apache.org
dev@kafka.apache.orgmailto:dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-11- Authorization design for kafka security
Thanks
:Re: [DISCUSS] KIP-11- Authorization design for kafka security
Parth,
This is a long thread, so trying to keep up here, sorry if this has been
covered before. First, great job on the KIP proposal and work so far.
Are we sure that we want to tie host level access to a given user? My
Harsha, Parth,
Thanks for the clarification. This makes sense. Perhaps we can clarify the
meaning of those rules in the wiki.
Related to this, it seems that we need to support wildcard in cli/request
protocol for topics?
Jun
On Mon, Apr 20, 2015 at 9:07 PM, Parth Brahmbhatt
Also, I think I may have missed this but does READ imply you also have
DESCRIBE? A reader will need access to both read offsets (to determine
their own initial position) as well as commit offsets. Currently, though
fetching offsets is under DESCRIBE only and commit offsets is under READ.
If
Changed Edit to Alter.
I did not think about it that way but Sriharsha raised the same point in a
private conversation. I did not think about it that way but I agree it
makes sense. If no one objects I think in default implementation we can
infer that if user have READ or WRITE access he gets
Hey Parth,
Great write-up!
One super minor thing: could we change the EDIT permission to be called
ALTER? The request name in KIP-4 is Alter and the command line tool has
always been alter (or we could go the other way and change those to EDIT).
Not sure that one is any better than the other but
Hey Jun,
Yes and we support wild cards for all acl entities principal, hosts and
operation.
Thanks
Parth
On 4/21/15, 9:06 AM, Jun Rao j...@confluent.io wrote:
Harsha, Parth,
Thanks for the clarification. This makes sense. Perhaps we can clarify the
meaning of those rules in the wiki.
Related
Following up on the KIP discussion. Two options for authorizing consumers
to read topic t as part of group g:
1. READ permission on resource /topic/t
2. READ permission on resource /topic/t AND WRITE permission on /group/g
The advantage of (1) is that it is simpler. The disadvantage is that any
Adding my notes from today's call to the thread:
** Deny or Allow all by default? We will add a configuration to
control this. The configuration will default to “allow” for backward
compatibility. Security admins can set it to deny
** Storing ACLs for default authorizers: We'll store them in ZK.
I have added the notes to KIP-11 Open question sections.
Thanks
Parth
On 4/21/15, 4:49 PM, Gwen Shapira gshap...@cloudera.com wrote:
Adding my notes from today's call to the thread:
** Deny or Allow all by default? We will add a configuration to
control this. The configuration will default to
I admit that I'm also not sure what we gain by having deny rules.
A user is either in the allow list (one way or another) or it will
be denied by default.
I think this is something we can skip for now and add later if needed?
On Mon, Apr 20, 2015 at 5:24 PM, Jun Rao j...@confluent.io wrote:
Here is a pseudo code that explains my current approach:
acls = authorizer.getAcl(resource)
if(acls == null || acls.isEmpty) {
allow all requests for backward compatibility. (any topics that were
created prior to security support will not have acls) This is debatable ,
generally we
According to the pseudo code, if you have a rule deny user1, then it
essentially denies all users?
Thanks,
Jun
On Mon, Apr 20, 2015 at 5:16 PM, Parth Brahmbhatt
pbrahmbh...@hortonworks.com wrote:
Here is a pseudo code that explains my current approach:
acls = authorizer.getAcl(resource)
Hi Parth,
OK, I understand your approach much better, now. I don’t really have a
strong opinion; I guess I just shared Gwen’s concern (in the hangout)
that, with this approach, if a site implements their own authorizer using
their own ACL implementation, it will work, but they’ll have this
According to the pseudo code, if you have a rule deny user1, then it
essentially denies all users?
Gwen,
I see “deny” being useful. I.e you want to allow everyone except few
users. This is probably a common case otherwise without deny you have to add
long list of users who should be
Hmm, I thought the semantics is that if you only have rule deny user2, it
means that everyone except user2 has access?
Thanks,
Jun
On Mon, Apr 20, 2015 at 3:25 PM, Parth Brahmbhatt
pbrahmbh...@hortonworks.com wrote:
user3 does not have access and removing the deny rule does not grant him
or
Wouldn't the everyone except few users be easier to manage through
groups / roles?
I think thats how its done in pretty much any system I can think of.
Do you know any system with deny option?
I don't feel strongly about this (i.e. its an extra feature, people
can use it or not. maybe it will be
Ignore my last; I was catching up on this list hadn’t seen Gwen’s
subsequent proposal.
On 4/15/15, 5:15 PM, Parth Brahmbhatt pbrahmbh...@hortonworks.com
wrote:
Kafka currently stores logConfig overrides specified during topic creation
in zookeeper, its just an instance of java.util.Properties
I tend to agree with Parth's point here. Most ACL systems I run into have deny
and allow. In general, you have a default policy of allow, then you follow your
rules stopping at the first line that matches. If you would like a default deny
policy, you have a bunch of allow rules and your last
The iptables on unix supports the DENY operator, not that it should
matter. The deny operator can also be used to specify ³allow user1 to READ
from topic1 from all hosts but host1,host2². Again we could add a host
group semantic and extra complexity around that, not sure if its worth it.
In
Thanks for clarifying the logic.
I'm +0 on the deny thing.
IMO, its not really needed, but if you think its important, I don't
object to having it in.
Gwen
On Mon, Apr 20, 2015 at 7:07 PM, Parth Brahmbhatt
pbrahmbh...@hortonworks.com wrote:
The iptables on unix supports the DENY operator, not
user3 does not have access and removing the deny rule does not grant him
or user2 access. user2 even without the deny rule will not have access.
Thanks
Parth
On 4/20/15, 12:03 PM, Jun Rao j...@confluent.io wrote:
Just a followup question. Suppose there are two rules. Rule1 allows user1
and
Hi Parth,
Nice work on this KIP. I did another read through and had a few more
comments (with edits after I went through the thread). Many of these
comments were brought up by others as well, so it appears that the KIP
would benefit from an update at this point to incorporate comments
from the
Just a followup question. Suppose there are two rules. Rule1 allows user1
and rule2 denies user2. Does user3 have access? If not, does removing rule1
enable user3 access?
Thanks,
Jun
On Mon, Apr 20, 2015 at 1:34 PM, Parth Brahmbhatt
pbrahmbh...@hortonworks.com wrote:
Hi Joel,
Thanks for
Hi Joel,
Thanks for the review and I plan to update the KIP today with all the updated
info. My comments in line below.
Thanks
Parth
On 4/20/15, 10:07 AM, Joel Koshy
jjkosh...@gmail.commailto:jjkosh...@gmail.com wrote:
Hi Parth,
Nice work on this KIP. I did another read through and had a
Currently the authorizer does not perform any dns lookups and uses the
hostname it receives as part of request.session as is. So in a way we are
allowing only ip addresses. The only match is equality based so no ip
ranges yet but that is easy to add.
However, I think it is ok to allow for both ip
Hi Michael,
There is code in kafka codebase that reads and interprets the topic config JSON
which has acls, owner and logconfig properties. There are 3 use cases that we
are supporting with current proposal:
* You use out of box simpleAcl authorizer which is tied to the acl stored
in
Parth,
If one wants to use his or her own access control including
authentication system, with this design what will be needed to be done? Can
one completely turn this off so that the system behaves exactly same as it
is today?
Thanks.
Tong
Sent from my iPhone
On Apr 15, 2015, at 1:51
Hi Parth,
I’m a little confused: why would Kafka need to interpret the JSON? IIRC
KIP-11 even says that the TopicConfigData will just store the JSON. I’m
not really making a design recommendation here, just trying to understand
what you’re proposing.
On 4/15/15, 11:20 AM, Parth Brahmbhatt
Kafka currently stores logConfig overrides specified during topic creation
in zookeeper, its just an instance of java.util.Properties converted to
json. I am proposing in addition to that we store acls and owner as well
as part of same Properties map.
There is some infrastructure around reading
I have added the following to list of open questions based on the hangout
discussion:
* The owner field of a topic in current proposal is set to the user who
created the topic and this user has all access to the topic. There was
suggestion on making this a list of users who can share ownership.
Hi Parth,
Sorry to chime in so late, but I’ve got a minor question on the KIP.
Several methods take a parameter named “host” of type String. Is that
intended to be a hostname, or an IP address? If the former, I’m curious as
to how that’s found (in my experience, when accepting an incoming socket
I also feel, having just IP would be more appropriate. Host lookup will
unnecessary slow things down and would be insecure as you pointed out.
With IP, it will be also able to setup policies (in future if needed) with
ranges or netmasks and it would be more scalable.
Bosco
On 4/14/15, 1:40 PM,
Thanks for the writeup. A few more comments.
20. I agree that it would be better to do this after KIP-4 (admin commands)
is done. With KIP-4, all admin operations will be sent as requests to the
brokers instead of accessing ZK directly. This will make authorization
easier.
21. Operation: What
21. Operation: What about other types of requests not covered in the list,
such as committing and fetching offsets, list topics, fetching consumer
metadata, heartbeat, join group, etc?
Would “CONFIGURE”, “DESCRIBE”, etc take care of this? Or should we add
high level grouping like “ADMIN”,
Thanks for reviewing, comments inline:
On 3/31/15, 9:21 AM, Jun Rao j...@confluent.iomailto:j...@confluent.io
wrote:
Thanks for the writeup. A few more comments.
20. I agree that it would be better to do this after KIP-4 (admin commands)
is done. With KIP-4, all admin operations will be sent
From the design doc , one of the added config:
* kafka.superusers: list of users that will be given superuser access.
These users will have access to everything. Users should set this to the
user kafka broker processes are running as to avoid duplicate
configuration for every single topic like
Related interesting question:
Since a broker is a consumer (of lead replicas), how do we handle the
broker level of permissions? Do we hardcode a broker-principal name
and automatically authorize brokers to do anything? Or is there a
cleaner way?
I feel, in Kerberos environment, “kafka” keytab
Related interesting question:
Since a broker is a consumer (of lead replicas), how do we handle the
broker level of permissions? Do we hardcode a broker-principal name
and automatically authorize brokers to do anything? Or is there a
cleaner way?
On Tue, Mar 31, 2015 at 10:17 AM, Don Bosco Durai
Yes in case of kerberos we will use superACL and this will be equivalent to
kafka broker’s principal name.
But in SSL as two-way auth is not mandatory the only option if we want enforce
authorizer in case of ssl is to force two-way auth.
Again this can be an issue on client side , lets say if a
21. What you suggested makes sense. Could you include the categorization of
each request in the wiki?
24. We have a jira (KAFKA-1595) to look for a better json parser. However,
it depends on dropping the scala 2.9.x support, which is being discussed.
Thanks,
Jun
On Tue, Mar 31, 2015 at 10:56
Hi Gwen,
Thanks a lot for taking the time to review this. I have tried to address all
your questions below.
Thanks
Parth
On 3/28/15, 8:08 PM, Gwen Shapira
gshap...@cloudera.commailto:gshap...@cloudera.com wrote:
Preparing for Tuesday meeting, I went over the KIP :)
First, Parth did an
Preparing for Tuesday meeting, I went over the KIP :)
First, Parth did an amazing job, the KIP is fantastic - detailed and
readable. Thank you!
Second, I have a lng list of questions :) No objections, just some
things I'm unclear on and random minor comments. In general, I like
the design, I
Hi all,
I have modified the KIP to reflect the recent change request from the
reviewers. I have been working on the code and I have the server side code
for authorization ready. I am now modifying the command line utilities. I
would really appreciate if some of the committers can spend sometime
Parth,
We can make some 15 mins or so to discuss this at the next KIP hangout.
Thanks,
Neha
On Wed, Mar 25, 2015 at 1:07 PM, Parth Brahmbhatt
pbrahmbh...@hortonworks.com wrote:
Hi all,
I have modified the KIP to reflect the recent change request from the
reviewers. I have been working on
Hi Michael,
Thanks for taking the time to review. Currently I did not plan on adding
“Deny” but I guess it can’t hurt except for adding more constructs would
probably make the acls more complex.
When a topic is created with no acls provided , I was planning to add a
default ACL which would allow
Hi Parth,
Thanks! A few questions:
1. Do you want to permit rules in your ACLs that DENY access as well as
ALLOW? This can be handy setting up rules that have exceptions. E.g.
“Allow principal P to READ resource R from all hosts” with “Deny principal
P READ access to resource R from host H1” in
Parth
Overall it is looking good. Couple of questions
- Can you give an example how the policies will look like in the default
implementation?
- In the operations, can we support ³CONNECT² also? This can be used
during Session connection
- Regarding access control for ³Topic Creation², since we
Hi Bosco,
Thanks for taking the time to review. I will update the doc with a policy
example and I will add CONNECT as an operation.
For Admin APIs, I agree the best thing to do right now is just wait for
KIP-4 to be submitted. I will update the doc to reflect the same.
Thanks
Parth
On 3/9/15,
Hi Parth,
Thanks for putting this together. Overall it looks good to
me. Although AdminUtils is a concern KIP-4 can probably fix
that part.
Thanks,
Harsha
On Thu, Mar 5, 2015, at 10:39 AM, Parth Brahmbhatt wrote:
Forgot to add links to wiki and jira.
Link
Forgot to add links to wiki and jira.
Link to wiki:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorization+Interface
Link to Jira: https://issues.apache.org/jira/browse/KAFKA-1688
Thanks
Parth
From: Parth Brahmbhatt
61 matches
Mail list logo
