RE: Security JIRAS
I'm looking at Kafka Brokers authentication with ZooKeeper since this looks independent of other tasks. [AM] 1) Is authentication required only between kafka broker and zookeeper? Can we assume world read so that consumers don't have to be authenticated (I believe in any case kafka is planning to change in such that consumers don't have to interact with zk)? In this case I assume kafka broker can I think easily create the znode with appropriate acl list - broker can be admin. 2) Zookeeper supports Kerberos authentication. Zookeeper supports SSL connections (version 3.4 or later) but I don't see an x509 authentication provider. Do we want to support x509 cert based authentication for zk? - Arvind
Re: Security JIRAS
For the moment, consumers still need to write under the /consumers tree. Even if they are committing offsets to Kafka instead of ZK, they will need to write owner information there when they are balancing. Eventually, you are correct, this is going away with the new consumer. -Todd On Fri, Oct 17, 2014 at 10:09 AM, Arvind Mani am...@linkedin.com.invalid wrote: I'm looking at Kafka Brokers authentication with ZooKeeper since this looks independent of other tasks. [AM] 1) Is authentication required only between kafka broker and zookeeper? Can we assume world read so that consumers don't have to be authenticated (I believe in any case kafka is planning to change in such that consumers don't have to interact with zk)? In this case I assume kafka broker can I think easily create the znode with appropriate acl list - broker can be admin. 2) Zookeeper supports Kerberos authentication. Zookeeper supports SSL connections (version 3.4 or later) but I don't see an x509 authentication provider. Do we want to support x509 cert based authentication for zk? - Arvind
Re: Security JIRAS
Yes, I think we can focus on Broker to Zookeeper communication only. At least for initial stage. Gwen On Fri, Oct 17, 2014 at 2:10 PM, Todd Palino tpal...@gmail.com wrote: For the moment, consumers still need to write under the /consumers tree. Even if they are committing offsets to Kafka instead of ZK, they will need to write owner information there when they are balancing. Eventually, you are correct, this is going away with the new consumer. -Todd On Fri, Oct 17, 2014 at 10:09 AM, Arvind Mani am...@linkedin.com.invalid wrote: I'm looking at Kafka Brokers authentication with ZooKeeper since this looks independent of other tasks. [AM] 1) Is authentication required only between kafka broker and zookeeper? Can we assume world read so that consumers don't have to be authenticated (I believe in any case kafka is planning to change in such that consumers don't have to interact with zk)? In this case I assume kafka broker can I think easily create the znode with appropriate acl list - broker can be admin. 2) Zookeeper supports Kerberos authentication. Zookeeper supports SSL connections (version 3.4 or later) but I don't see an x509 authentication provider. Do we want to support x509 cert based authentication for zk? - Arvind
Re: Security JIRAS
Thanks, Jay. I¹m new to the project, and I¹m wondering how things proceed from hereŠ are folks working on these tasks, or do they get assigned, orŠ? On 10/7/14, 5:15 PM, Jay Kreps jay.kr...@gmail.com wrote: Hey guys, As promised, I added a tree of JIRAs for the stuff in the security wiki ( https://cwiki.apache.org/confluence/display/KAFKA/Security): https://issues.apache.org/jira/browse/KAFKA-1682 I tried to break it into reasonably standalone pieces. I think many of the tickets could actually be done in parallel. Since there were many people interested in this area this may help parallelize the work a bit. I added some strawman details on implementation to each ticket. We can discuss and refine further on the individual tickets. Please take a look and let me know if this breakdown seems reasonable. Cheers, -Jay
Re: Security JIRAS
Wondering the same here :) I think there are some parallel threads here (SSL is independent of Kerberos, as far as I can see). Kerberos work is blocked on https://issues.apache.org/jira/browse/KAFKA-1683 - Implement a session concept in the socket server. So there's no point in picking up other tasks before this is assigned (and at least designed). I'm looking at Kafka Brokers authentication with ZooKeeper since this looks independent of other tasks. Gwen On Thu, Oct 16, 2014 at 4:23 PM, Michael Herstine mherst...@linkedin.com.invalid wrote: Thanks, Jay. I¹m new to the project, and I¹m wondering how things proceed from hereŠ are folks working on these tasks, or do they get assigned, orŠ? On 10/7/14, 5:15 PM, Jay Kreps jay.kr...@gmail.com wrote: Hey guys, As promised, I added a tree of JIRAs for the stuff in the security wiki ( https://cwiki.apache.org/confluence/display/KAFKA/Security): https://issues.apache.org/jira/browse/KAFKA-1682 I tried to break it into reasonably standalone pieces. I think many of the tickets could actually be done in parallel. Since there were many people interested in this area this may help parallelize the work a bit. I added some strawman details on implementation to each ticket. We can discuss and refine further on the individual tickets. Please take a look and let me know if this breakdown seems reasonable. Cheers, -Jay
Re: Security JIRAS
I'd vote for accepting every major change with the relevant system tests. We didn't do this for major features in the past that lead to weak coverage and a great deal of work for someone else to add tests for features that were done in the past. I'm guilty of this myself :-( On Thu, Oct 9, 2014 at 6:45 PM, Gwen Shapira gshap...@cloudera.com wrote: Added some details on delegation tokens. I hope it at least clarifies some of the scope. I'm working on a more detailed design doc. On Thu, Oct 9, 2014 at 1:44 PM, Jay Kreps jay.kr...@gmail.com wrote: Hey Gwen, Your absolutely right about these. I added the ticket for ZK authentication and Hadoop delegation tokens. For the Hadoop case I actually don't understand Hadoop security very well. Maybe you could fill in some of the details on what needs to happen for that to work? For testing, we should probably discuss the best way to test security. I think this is a fairly critical thing, if we are going to say we have security we really need to have good tests in place to ensure we do. This will require some thought. I think we should be able to test TLS fairly easily using junit integration test that just starts the server and connects using TLS. For Kerberos though it isn't clear to me how to do good integration testing since we need a KDC to test against and it isn't clear how that happens in the test environment except possibly manually (which is not ideal). How do other projects handle this? -Jay On Tue, Oct 7, 2014 at 5:25 PM, Gwen Shapira gshap...@cloudera.com wrote: I think we need to add: * Authentication of Kafka brokers with a secured ZooKeeper * Kafka should be able to generate delegation tokens for MapReduce / Spark / Yarn jobs. * Extend systest framework to allow testing secured kafka Gwen On Tue, Oct 7, 2014 at 5:15 PM, Jay Kreps jay.kr...@gmail.com wrote: Hey guys, As promised, I added a tree of JIRAs for the stuff in the security wiki ( https://cwiki.apache.org/confluence/display/KAFKA/Security): https://issues.apache.org/jira/browse/KAFKA-1682 I tried to break it into reasonably standalone pieces. I think many of the tickets could actually be done in parallel. Since there were many people interested in this area this may help parallelize the work a bit. I added some strawman details on implementation to each ticket. We can discuss and refine further on the individual tickets. Please take a look and let me know if this breakdown seems reasonable. Cheers, -Jay
Re: Security JIRAS
I would be strong +1 on that. I’ve seen a lot of regressions on other projects when new functionality cause regressions when running in secure mode. Jarcec On Oct 10, 2014, at 9:43 AM, Neha Narkhede neha.narkh...@gmail.com wrote: I'd vote for accepting every major change with the relevant system tests. We didn't do this for major features in the past that lead to weak coverage and a great deal of work for someone else to add tests for features that were done in the past. I'm guilty of this myself :-( On Thu, Oct 9, 2014 at 6:45 PM, Gwen Shapira gshap...@cloudera.com wrote: Added some details on delegation tokens. I hope it at least clarifies some of the scope. I'm working on a more detailed design doc. On Thu, Oct 9, 2014 at 1:44 PM, Jay Kreps jay.kr...@gmail.com wrote: Hey Gwen, Your absolutely right about these. I added the ticket for ZK authentication and Hadoop delegation tokens. For the Hadoop case I actually don't understand Hadoop security very well. Maybe you could fill in some of the details on what needs to happen for that to work? For testing, we should probably discuss the best way to test security. I think this is a fairly critical thing, if we are going to say we have security we really need to have good tests in place to ensure we do. This will require some thought. I think we should be able to test TLS fairly easily using junit integration test that just starts the server and connects using TLS. For Kerberos though it isn't clear to me how to do good integration testing since we need a KDC to test against and it isn't clear how that happens in the test environment except possibly manually (which is not ideal). How do other projects handle this? -Jay On Tue, Oct 7, 2014 at 5:25 PM, Gwen Shapira gshap...@cloudera.com wrote: I think we need to add: * Authentication of Kafka brokers with a secured ZooKeeper * Kafka should be able to generate delegation tokens for MapReduce / Spark / Yarn jobs. * Extend systest framework to allow testing secured kafka Gwen On Tue, Oct 7, 2014 at 5:15 PM, Jay Kreps jay.kr...@gmail.com wrote: Hey guys, As promised, I added a tree of JIRAs for the stuff in the security wiki ( https://cwiki.apache.org/confluence/display/KAFKA/Security): https://issues.apache.org/jira/browse/KAFKA-1682 I tried to break it into reasonably standalone pieces. I think many of the tickets could actually be done in parallel. Since there were many people interested in this area this may help parallelize the work a bit. I added some strawman details on implementation to each ticket. We can discuss and refine further on the individual tickets. Please take a look and let me know if this breakdown seems reasonable. Cheers, -Jay
Re: Security JIRAS
Hey Gwen, Your absolutely right about these. I added the ticket for ZK authentication and Hadoop delegation tokens. For the Hadoop case I actually don't understand Hadoop security very well. Maybe you could fill in some of the details on what needs to happen for that to work? For testing, we should probably discuss the best way to test security. I think this is a fairly critical thing, if we are going to say we have security we really need to have good tests in place to ensure we do. This will require some thought. I think we should be able to test TLS fairly easily using junit integration test that just starts the server and connects using TLS. For Kerberos though it isn't clear to me how to do good integration testing since we need a KDC to test against and it isn't clear how that happens in the test environment except possibly manually (which is not ideal). How do other projects handle this? -Jay On Tue, Oct 7, 2014 at 5:25 PM, Gwen Shapira gshap...@cloudera.com wrote: I think we need to add: * Authentication of Kafka brokers with a secured ZooKeeper * Kafka should be able to generate delegation tokens for MapReduce / Spark / Yarn jobs. * Extend systest framework to allow testing secured kafka Gwen On Tue, Oct 7, 2014 at 5:15 PM, Jay Kreps jay.kr...@gmail.com wrote: Hey guys, As promised, I added a tree of JIRAs for the stuff in the security wiki ( https://cwiki.apache.org/confluence/display/KAFKA/Security): https://issues.apache.org/jira/browse/KAFKA-1682 I tried to break it into reasonably standalone pieces. I think many of the tickets could actually be done in parallel. Since there were many people interested in this area this may help parallelize the work a bit. I added some strawman details on implementation to each ticket. We can discuss and refine further on the individual tickets. Please take a look and let me know if this breakdown seems reasonable. Cheers, -Jay
Re: Security JIRAS
For Kerberos though it isn't clear to me how to do good integration testing since we need a KDC to test against and it isn't clear how that happens in the test environment except possibly manually (which is not ideal). How do other projects handle this? Actually it’s not that hard. Hadoop is providing small KDC implementation for testing purpose called MiniKdc. It’s super easy to use, check it out in Sentry: https://github.com/apache/incubator-sentry/blob/master/sentry-provider/sentry-provider-db/src/test/java/org/apache/sentry/provider/db/service/thrift/SentryMiniKdcTestcase.java Jarcec On Oct 9, 2014, at 1:44 PM, Jay Kreps jay.kr...@gmail.com wrote: Hey Gwen, Your absolutely right about these. I added the ticket for ZK authentication and Hadoop delegation tokens. For the Hadoop case I actually don't understand Hadoop security very well. Maybe you could fill in some of the details on what needs to happen for that to work? For testing, we should probably discuss the best way to test security. I think this is a fairly critical thing, if we are going to say we have security we really need to have good tests in place to ensure we do. This will require some thought. I think we should be able to test TLS fairly easily using junit integration test that just starts the server and connects using TLS. For Kerberos though it isn't clear to me how to do good integration testing since we need a KDC to test against and it isn't clear how that happens in the test environment except possibly manually (which is not ideal). How do other projects handle this? -Jay On Tue, Oct 7, 2014 at 5:25 PM, Gwen Shapira gshap...@cloudera.com wrote: I think we need to add: * Authentication of Kafka brokers with a secured ZooKeeper * Kafka should be able to generate delegation tokens for MapReduce / Spark / Yarn jobs. * Extend systest framework to allow testing secured kafka Gwen On Tue, Oct 7, 2014 at 5:15 PM, Jay Kreps jay.kr...@gmail.com wrote: Hey guys, As promised, I added a tree of JIRAs for the stuff in the security wiki ( https://cwiki.apache.org/confluence/display/KAFKA/Security): https://issues.apache.org/jira/browse/KAFKA-1682 I tried to break it into reasonably standalone pieces. I think many of the tickets could actually be done in parallel. Since there were many people interested in this area this may help parallelize the work a bit. I added some strawman details on implementation to each ticket. We can discuss and refine further on the individual tickets. Please take a look and let me know if this breakdown seems reasonable. Cheers, -Jay
Re: Security JIRAS
Added some details on delegation tokens. I hope it at least clarifies some of the scope. I'm working on a more detailed design doc. On Thu, Oct 9, 2014 at 1:44 PM, Jay Kreps jay.kr...@gmail.com wrote: Hey Gwen, Your absolutely right about these. I added the ticket for ZK authentication and Hadoop delegation tokens. For the Hadoop case I actually don't understand Hadoop security very well. Maybe you could fill in some of the details on what needs to happen for that to work? For testing, we should probably discuss the best way to test security. I think this is a fairly critical thing, if we are going to say we have security we really need to have good tests in place to ensure we do. This will require some thought. I think we should be able to test TLS fairly easily using junit integration test that just starts the server and connects using TLS. For Kerberos though it isn't clear to me how to do good integration testing since we need a KDC to test against and it isn't clear how that happens in the test environment except possibly manually (which is not ideal). How do other projects handle this? -Jay On Tue, Oct 7, 2014 at 5:25 PM, Gwen Shapira gshap...@cloudera.com wrote: I think we need to add: * Authentication of Kafka brokers with a secured ZooKeeper * Kafka should be able to generate delegation tokens for MapReduce / Spark / Yarn jobs. * Extend systest framework to allow testing secured kafka Gwen On Tue, Oct 7, 2014 at 5:15 PM, Jay Kreps jay.kr...@gmail.com wrote: Hey guys, As promised, I added a tree of JIRAs for the stuff in the security wiki ( https://cwiki.apache.org/confluence/display/KAFKA/Security): https://issues.apache.org/jira/browse/KAFKA-1682 I tried to break it into reasonably standalone pieces. I think many of the tickets could actually be done in parallel. Since there were many people interested in this area this may help parallelize the work a bit. I added some strawman details on implementation to each ticket. We can discuss and refine further on the individual tickets. Please take a look and let me know if this breakdown seems reasonable. Cheers, -Jay
Re: Security JIRAS
I think we need to add: * Authentication of Kafka brokers with a secured ZooKeeper * Kafka should be able to generate delegation tokens for MapReduce / Spark / Yarn jobs. * Extend systest framework to allow testing secured kafka Gwen On Tue, Oct 7, 2014 at 5:15 PM, Jay Kreps jay.kr...@gmail.com wrote: Hey guys, As promised, I added a tree of JIRAs for the stuff in the security wiki ( https://cwiki.apache.org/confluence/display/KAFKA/Security): https://issues.apache.org/jira/browse/KAFKA-1682 I tried to break it into reasonably standalone pieces. I think many of the tickets could actually be done in parallel. Since there were many people interested in this area this may help parallelize the work a bit. I added some strawman details on implementation to each ticket. We can discuss and refine further on the individual tickets. Please take a look and let me know if this breakdown seems reasonable. Cheers, -Jay
