[jira] [Commented] (KAFKA-1683) Implement a session concept in the socket server
[ https://issues.apache.org/jira/browse/KAFKA-1683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14715489#comment-14715489 ] ASF GitHub Bot commented on KAFKA-1683: --- Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/155 Implement a session concept in the socket server -- Key: KAFKA-1683 URL: https://issues.apache.org/jira/browse/KAFKA-1683 Project: Kafka Issue Type: Sub-task Components: security Affects Versions: 0.8.2.1 Reporter: Jay Kreps Assignee: Gwen Shapira Fix For: 0.8.3 Attachments: KAFKA-1683.patch, KAFKA-1683.patch To implement authentication we need a way to keep track of some things between requests. The initial use for this would be remembering the authenticated user/principle info, but likely more uses would come up (for example we will also need to remember whether and which encryption or integrity measures are in place on the socket so we can wrap and unwrap writes and reads). I was thinking we could just add a Session object that might have a user field. The session object would need to get added to RequestChannel.Request so it is passed down to the API layer with each request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1683) Implement a session concept in the socket server
[ https://issues.apache.org/jira/browse/KAFKA-1683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14704649#comment-14704649 ] Ismael Juma commented on KAFKA-1683: [~gwenshap], KAFKA-1690 has been merged (as you know), so it may be time to start thinking about this again. :) Implement a session concept in the socket server -- Key: KAFKA-1683 URL: https://issues.apache.org/jira/browse/KAFKA-1683 Project: Kafka Issue Type: Sub-task Components: security Affects Versions: 0.9.0 Reporter: Jay Kreps Assignee: Gwen Shapira Fix For: 0.8.3 Attachments: KAFKA-1683.patch, KAFKA-1683.patch To implement authentication we need a way to keep track of some things between requests. The initial use for this would be remembering the authenticated user/principle info, but likely more uses would come up (for example we will also need to remember whether and which encryption or integrity measures are in place on the socket so we can wrap and unwrap writes and reads). I was thinking we could just add a Session object that might have a user field. The session object would need to get added to RequestChannel.Request so it is passed down to the API layer with each request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1683) Implement a session concept in the socket server
[ https://issues.apache.org/jira/browse/KAFKA-1683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14706103#comment-14706103 ] Sriharsha Chintalapani commented on KAFKA-1683: --- [~gwenshap] got your point . Agree on not throwing exception and returning ANONYMOUS. Implement a session concept in the socket server -- Key: KAFKA-1683 URL: https://issues.apache.org/jira/browse/KAFKA-1683 Project: Kafka Issue Type: Sub-task Components: security Affects Versions: 0.9.0 Reporter: Jay Kreps Assignee: Gwen Shapira Fix For: 0.8.3 Attachments: KAFKA-1683.patch, KAFKA-1683.patch To implement authentication we need a way to keep track of some things between requests. The initial use for this would be remembering the authenticated user/principle info, but likely more uses would come up (for example we will also need to remember whether and which encryption or integrity measures are in place on the socket so we can wrap and unwrap writes and reads). I was thinking we could just add a Session object that might have a user field. The session object would need to get added to RequestChannel.Request so it is passed down to the API layer with each request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1683) Implement a session concept in the socket server
[ https://issues.apache.org/jira/browse/KAFKA-1683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14706096#comment-14706096 ] Sriharsha Chintalapani commented on KAFKA-1683: --- [~gwenshap] Hmm if the handshake establishes than they are authenticated technically. In this case it will give the hostname and we do have pluggable principal builder if the user's inclined to read X509Cert and makeup their own principal. Do you see any issues with this. Implement a session concept in the socket server -- Key: KAFKA-1683 URL: https://issues.apache.org/jira/browse/KAFKA-1683 Project: Kafka Issue Type: Sub-task Components: security Affects Versions: 0.9.0 Reporter: Jay Kreps Assignee: Gwen Shapira Fix For: 0.8.3 Attachments: KAFKA-1683.patch, KAFKA-1683.patch To implement authentication we need a way to keep track of some things between requests. The initial use for this would be remembering the authenticated user/principle info, but likely more uses would come up (for example we will also need to remember whether and which encryption or integrity measures are in place on the socket so we can wrap and unwrap writes and reads). I was thinking we could just add a Session object that might have a user field. The session object would need to get added to RequestChannel.Request so it is passed down to the API layer with each request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1683) Implement a session concept in the socket server
[ https://issues.apache.org/jira/browse/KAFKA-1683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14706135#comment-14706135 ] Parth Brahmbhatt commented on KAFKA-1683: - [~gwenshap] I agree, the original KIP also assumes that when no authentication is done we will still set the Principal as ANONYMOUS. Only suggestion is , lets declare this constant in some Config in case we need to access the value from somewhere else in the code. Implement a session concept in the socket server -- Key: KAFKA-1683 URL: https://issues.apache.org/jira/browse/KAFKA-1683 Project: Kafka Issue Type: Sub-task Components: security Affects Versions: 0.9.0 Reporter: Jay Kreps Assignee: Gwen Shapira Fix For: 0.8.3 Attachments: KAFKA-1683.patch, KAFKA-1683.patch To implement authentication we need a way to keep track of some things between requests. The initial use for this would be remembering the authenticated user/principle info, but likely more uses would come up (for example we will also need to remember whether and which encryption or integrity measures are in place on the socket so we can wrap and unwrap writes and reads). I was thinking we could just add a Session object that might have a user field. The session object would need to get added to RequestChannel.Request so it is passed down to the API layer with each request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1683) Implement a session concept in the socket server
[ https://issues.apache.org/jira/browse/KAFKA-1683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14706099#comment-14706099 ] Gwen Shapira commented on KAFKA-1683: - [~harsha_ch] mmm... I'm talking about returning ANONYMOUS instead of throwing unauthenticated exception. You can check my pull request to see what I mean. Since we need to have a session object anyway, I need to put some principal there, so if getPrincipal throws I'll need to handle it when creating a session. I think it is cleaner not to throw, but I may be missing SSL context. Implement a session concept in the socket server -- Key: KAFKA-1683 URL: https://issues.apache.org/jira/browse/KAFKA-1683 Project: Kafka Issue Type: Sub-task Components: security Affects Versions: 0.9.0 Reporter: Jay Kreps Assignee: Gwen Shapira Fix For: 0.8.3 Attachments: KAFKA-1683.patch, KAFKA-1683.patch To implement authentication we need a way to keep track of some things between requests. The initial use for this would be remembering the authenticated user/principle info, but likely more uses would come up (for example we will also need to remember whether and which encryption or integrity measures are in place on the socket so we can wrap and unwrap writes and reads). I was thinking we could just add a Session object that might have a user field. The session object would need to get added to RequestChannel.Request so it is passed down to the API layer with each request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1683) Implement a session concept in the socket server
[ https://issues.apache.org/jira/browse/KAFKA-1683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14706097#comment-14706097 ] ASF GitHub Bot commented on KAFKA-1683: --- GitHub user gwenshap opened a pull request: https://github.com/apache/kafka/pull/155 KAFKA-1683: persisting session information in Requests You can merge this pull request into a Git repository by running: $ git pull https://github.com/gwenshap/kafka KAFKA-1683 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/155.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #155 commit d97449d3626c1a209eefa2eb01e011ecdec4f147 Author: Gwen Shapira csh...@gmail.com Date: 2015-08-21T01:46:49Z persisting session information in Requests Implement a session concept in the socket server -- Key: KAFKA-1683 URL: https://issues.apache.org/jira/browse/KAFKA-1683 Project: Kafka Issue Type: Sub-task Components: security Affects Versions: 0.9.0 Reporter: Jay Kreps Assignee: Gwen Shapira Fix For: 0.8.3 Attachments: KAFKA-1683.patch, KAFKA-1683.patch To implement authentication we need a way to keep track of some things between requests. The initial use for this would be remembering the authenticated user/principle info, but likely more uses would come up (for example we will also need to remember whether and which encryption or integrity measures are in place on the socket so we can wrap and unwrap writes and reads). I was thinking we could just add a Session object that might have a user field. The session object would need to get added to RequestChannel.Request so it is passed down to the API layer with each request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1683) Implement a session concept in the socket server
[ https://issues.apache.org/jira/browse/KAFKA-1683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14705538#comment-14705538 ] Parth Brahmbhatt commented on KAFKA-1683: - [~gwenshap] Given this blocks KAFKA-2210, if you are busy I can submit a PR for this one. Implement a session concept in the socket server -- Key: KAFKA-1683 URL: https://issues.apache.org/jira/browse/KAFKA-1683 Project: Kafka Issue Type: Sub-task Components: security Affects Versions: 0.9.0 Reporter: Jay Kreps Assignee: Gwen Shapira Fix For: 0.8.3 Attachments: KAFKA-1683.patch, KAFKA-1683.patch To implement authentication we need a way to keep track of some things between requests. The initial use for this would be remembering the authenticated user/principle info, but likely more uses would come up (for example we will also need to remember whether and which encryption or integrity measures are in place on the socket so we can wrap and unwrap writes and reads). I was thinking we could just add a Session object that might have a user field. The session object would need to get added to RequestChannel.Request so it is passed down to the API layer with each request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1683) Implement a session concept in the socket server
[ https://issues.apache.org/jira/browse/KAFKA-1683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14705540#comment-14705540 ] Gwen Shapira commented on KAFKA-1683: - I think I'm good, expect a patch by Friday. Implement a session concept in the socket server -- Key: KAFKA-1683 URL: https://issues.apache.org/jira/browse/KAFKA-1683 Project: Kafka Issue Type: Sub-task Components: security Affects Versions: 0.9.0 Reporter: Jay Kreps Assignee: Gwen Shapira Fix For: 0.8.3 Attachments: KAFKA-1683.patch, KAFKA-1683.patch To implement authentication we need a way to keep track of some things between requests. The initial use for this would be remembering the authenticated user/principle info, but likely more uses would come up (for example we will also need to remember whether and which encryption or integrity measures are in place on the socket so we can wrap and unwrap writes and reads). I was thinking we could just add a Session object that might have a user field. The session object would need to get added to RequestChannel.Request so it is passed down to the API layer with each request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1683) Implement a session concept in the socket server
[ https://issues.apache.org/jira/browse/KAFKA-1683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14693676#comment-14693676 ] Eugene Miretsky commented on KAFKA-1683: My apologies, didn't word the question properly. I think that KAFKA-1686 solvers it - Kerberos support will allow authenticating as a specific user, and storing the user identity in a session for later authorization. Implement a session concept in the socket server -- Key: KAFKA-1683 URL: https://issues.apache.org/jira/browse/KAFKA-1683 Project: Kafka Issue Type: Sub-task Components: security Affects Versions: 0.9.0 Reporter: Jay Kreps Assignee: Gwen Shapira Fix For: 0.8.3 Attachments: KAFKA-1683.patch, KAFKA-1683.patch To implement authentication we need a way to keep track of some things between requests. The initial use for this would be remembering the authenticated user/principle info, but likely more uses would come up (for example we will also need to remember whether and which encryption or integrity measures are in place on the socket so we can wrap and unwrap writes and reads). I was thinking we could just add a Session object that might have a user field. The session object would need to get added to RequestChannel.Request so it is passed down to the API layer with each request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1683) Implement a session concept in the socket server
[ https://issues.apache.org/jira/browse/KAFKA-1683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14692317#comment-14692317 ] Parth Brahmbhatt commented on KAFKA-1683: - [~eugenstud] I believe this patch will set the foundation for the authorization. It will introduce the concept of a session where session will capture identity of the client so authorization layer can use that identity to authorize against some acl store. The Authorizer it self is being reviewed as part of KAFKA-2210. I am not sure what you mean by as different users. can you elaborate? Implement a session concept in the socket server -- Key: KAFKA-1683 URL: https://issues.apache.org/jira/browse/KAFKA-1683 Project: Kafka Issue Type: Sub-task Components: security Affects Versions: 0.9.0 Reporter: Jay Kreps Assignee: Gwen Shapira Fix For: 0.8.3 Attachments: KAFKA-1683.patch, KAFKA-1683.patch To implement authentication we need a way to keep track of some things between requests. The initial use for this would be remembering the authenticated user/principle info, but likely more uses would come up (for example we will also need to remember whether and which encryption or integrity measures are in place on the socket so we can wrap and unwrap writes and reads). I was thinking we could just add a Session object that might have a user field. The session object would need to get added to RequestChannel.Request so it is passed down to the API layer with each request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1683) Implement a session concept in the socket server
[ https://issues.apache.org/jira/browse/KAFKA-1683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14681149#comment-14681149 ] Eugene Miretsky commented on KAFKA-1683: Would this patch include the ability to authorizer as different users? Or will it be handled in another JIRA? Implement a session concept in the socket server -- Key: KAFKA-1683 URL: https://issues.apache.org/jira/browse/KAFKA-1683 Project: Kafka Issue Type: Sub-task Components: security Affects Versions: 0.9.0 Reporter: Jay Kreps Assignee: Gwen Shapira Fix For: 0.8.3 Attachments: KAFKA-1683.patch, KAFKA-1683.patch To implement authentication we need a way to keep track of some things between requests. The initial use for this would be remembering the authenticated user/principle info, but likely more uses would come up (for example we will also need to remember whether and which encryption or integrity measures are in place on the socket so we can wrap and unwrap writes and reads). I was thinking we could just add a Session object that might have a user field. The session object would need to get added to RequestChannel.Request so it is passed down to the API layer with each request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1683) Implement a session concept in the socket server
[ https://issues.apache.org/jira/browse/KAFKA-1683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14654435#comment-14654435 ] Gwen Shapira commented on KAFKA-1683: - I started looking at this again, and I think the right timing is immediately after KAFKA-1690. It looks like the correct place to expose the principal will be in the Channel interface that KAFKA-1690 adds. I'll upload a patch the depends on KAFKA-1690 for review. Implement a session concept in the socket server -- Key: KAFKA-1683 URL: https://issues.apache.org/jira/browse/KAFKA-1683 Project: Kafka Issue Type: Sub-task Components: security Affects Versions: 0.9.0 Reporter: Jay Kreps Assignee: Gwen Shapira Attachments: KAFKA-1683.patch, KAFKA-1683.patch To implement authentication we need a way to keep track of some things between requests. The initial use for this would be remembering the authenticated user/principle info, but likely more uses would come up (for example we will also need to remember whether and which encryption or integrity measures are in place on the socket so we can wrap and unwrap writes and reads). I was thinking we could just add a Session object that might have a user field. The session object would need to get added to RequestChannel.Request so it is passed down to the API layer with each request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1683) Implement a session concept in the socket server
[ https://issues.apache.org/jira/browse/KAFKA-1683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14644396#comment-14644396 ] Ismael Juma commented on KAFKA-1683: [~gwenshap], what is the status of this? Is it something that is still needed or has it been superseded by another JIRA? Implement a session concept in the socket server -- Key: KAFKA-1683 URL: https://issues.apache.org/jira/browse/KAFKA-1683 Project: Kafka Issue Type: Sub-task Components: security Affects Versions: 0.9.0 Reporter: Jay Kreps Assignee: Gwen Shapira Attachments: KAFKA-1683.patch, KAFKA-1683.patch To implement authentication we need a way to keep track of some things between requests. The initial use for this would be remembering the authenticated user/principle info, but likely more uses would come up (for example we will also need to remember whether and which encryption or integrity measures are in place on the socket so we can wrap and unwrap writes and reads). I was thinking we could just add a Session object that might have a user field. The session object would need to get added to RequestChannel.Request so it is passed down to the API layer with each request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1683) Implement a session concept in the socket server
[ https://issues.apache.org/jira/browse/KAFKA-1683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14644823#comment-14644823 ] Ismael Juma commented on KAFKA-1683: So merging the Authorizer will effectively include this work too? Implement a session concept in the socket server -- Key: KAFKA-1683 URL: https://issues.apache.org/jira/browse/KAFKA-1683 Project: Kafka Issue Type: Sub-task Components: security Affects Versions: 0.9.0 Reporter: Jay Kreps Assignee: Gwen Shapira Attachments: KAFKA-1683.patch, KAFKA-1683.patch To implement authentication we need a way to keep track of some things between requests. The initial use for this would be remembering the authenticated user/principle info, but likely more uses would come up (for example we will also need to remember whether and which encryption or integrity measures are in place on the socket so we can wrap and unwrap writes and reads). I was thinking we could just add a Session object that might have a user field. The session object would need to get added to RequestChannel.Request so it is passed down to the API layer with each request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1683) Implement a session concept in the socket server
[ https://issues.apache.org/jira/browse/KAFKA-1683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14644805#comment-14644805 ] Parth Brahmbhatt commented on KAFKA-1683: - FYI, I relied on this Session object as part of Authorizer design and implementation. Implement a session concept in the socket server -- Key: KAFKA-1683 URL: https://issues.apache.org/jira/browse/KAFKA-1683 Project: Kafka Issue Type: Sub-task Components: security Affects Versions: 0.9.0 Reporter: Jay Kreps Assignee: Gwen Shapira Attachments: KAFKA-1683.patch, KAFKA-1683.patch To implement authentication we need a way to keep track of some things between requests. The initial use for this would be remembering the authenticated user/principle info, but likely more uses would come up (for example we will also need to remember whether and which encryption or integrity measures are in place on the socket so we can wrap and unwrap writes and reads). I was thinking we could just add a Session object that might have a user field. The session object would need to get added to RequestChannel.Request so it is passed down to the API layer with each request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1683) Implement a session concept in the socket server
[ https://issues.apache.org/jira/browse/KAFKA-1683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14644830#comment-14644830 ] Parth Brahmbhatt commented on KAFKA-1683: - I did not include this as part of Authorizer work, I just applied this patch and developed against it. The pull request for authorizer does not have this patch, it assumes it is being reviewed in its own PR and will be merged separately. Implement a session concept in the socket server -- Key: KAFKA-1683 URL: https://issues.apache.org/jira/browse/KAFKA-1683 Project: Kafka Issue Type: Sub-task Components: security Affects Versions: 0.9.0 Reporter: Jay Kreps Assignee: Gwen Shapira Attachments: KAFKA-1683.patch, KAFKA-1683.patch To implement authentication we need a way to keep track of some things between requests. The initial use for this would be remembering the authenticated user/principle info, but likely more uses would come up (for example we will also need to remember whether and which encryption or integrity measures are in place on the socket so we can wrap and unwrap writes and reads). I was thinking we could just add a Session object that might have a user field. The session object would need to get added to RequestChannel.Request so it is passed down to the API layer with each request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1683) Implement a session concept in the socket server
[ https://issues.apache.org/jira/browse/KAFKA-1683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14644840#comment-14644840 ] Ismael Juma commented on KAFKA-1683: OK, it depends on it. Understood, thanks. Implement a session concept in the socket server -- Key: KAFKA-1683 URL: https://issues.apache.org/jira/browse/KAFKA-1683 Project: Kafka Issue Type: Sub-task Components: security Affects Versions: 0.9.0 Reporter: Jay Kreps Assignee: Gwen Shapira Attachments: KAFKA-1683.patch, KAFKA-1683.patch To implement authentication we need a way to keep track of some things between requests. The initial use for this would be remembering the authenticated user/principle info, but likely more uses would come up (for example we will also need to remember whether and which encryption or integrity measures are in place on the socket so we can wrap and unwrap writes and reads). I was thinking we could just add a Session object that might have a user field. The session object would need to get added to RequestChannel.Request so it is passed down to the API layer with each request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1683) Implement a session concept in the socket server
[ https://issues.apache.org/jira/browse/KAFKA-1683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14184682#comment-14184682 ] Arvind Mani commented on KAFKA-1683: Hi [~joestein], Regarding revocation - are you talking about revoking access of individual client (kafka consumer, client, or mirror maker), revoking server (kafka broker) identity, or revoking the issuer of client/server identities (KDC or CA). Revoking client access is not hard provided ACL list maintained by authz is fresh and ACLs are checked per request. If we revoke server identity, then considering the circumstances this calls for taking the compromised server offline and pointing clients at another server. Revocation of issuer is tricky and the implementation may be dependent on authentication mechanism (e.g., PKI vs kerberos). Broadly there are several options: 1) Periodically disconnect forcing re-authentication 2) Periodically check revocation - e.g., OCSP for certs but note that this is typically done just at time of full TLS handshake. 3) Renegotiation within existing session - this is tricky to implement correctly. TLS supports renegotiation but is often subject to bugs. In addition there will be changes to configuration on client and server - e.g., new trust store or KDC. If you are talking about dealing with issuer revocation then it might be worth opening a separate tickets for PKI and for Kerberos. - Arvind Implement a session concept in the socket server -- Key: KAFKA-1683 URL: https://issues.apache.org/jira/browse/KAFKA-1683 Project: Kafka Issue Type: Sub-task Components: security Affects Versions: 0.9.0 Reporter: Jay Kreps Assignee: Gwen Shapira Attachments: KAFKA-1683.patch, KAFKA-1683.patch To implement authentication we need a way to keep track of some things between requests. The initial use for this would be remembering the authenticated user/principle info, but likely more uses would come up (for example we will also need to remember whether and which encryption or integrity measures are in place on the socket so we can wrap and unwrap writes and reads). I was thinking we could just add a Session object that might have a user field. The session object would need to get added to RequestChannel.Request so it is passed down to the API layer with each request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1683) Implement a session concept in the socket server
[ https://issues.apache.org/jira/browse/KAFKA-1683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14184402#comment-14184402 ] Gwen Shapira commented on KAFKA-1683: - Created reviewboard https://reviews.apache.org/r/27203/diff/ against branch origin/trunk Implement a session concept in the socket server -- Key: KAFKA-1683 URL: https://issues.apache.org/jira/browse/KAFKA-1683 Project: Kafka Issue Type: Sub-task Components: security Affects Versions: 0.9.0 Reporter: Jay Kreps Assignee: Gwen Shapira Attachments: KAFKA-1683.patch To implement authentication we need a way to keep track of some things between requests. The initial use for this would be remembering the authenticated user/principle info, but likely more uses would come up (for example we will also need to remember whether and which encryption or integrity measures are in place on the socket so we can wrap and unwrap writes and reads). I was thinking we could just add a Session object that might have a user field. The session object would need to get added to RequestChannel.Request so it is passed down to the API layer with each request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1683) Implement a session concept in the socket server
[ https://issues.apache.org/jira/browse/KAFKA-1683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14184404#comment-14184404 ] Gwen Shapira commented on KAFKA-1683: - As mentioned earlier - we need to enrich the SocketServer with security related information (such as Principal) and actions (security handshakes when accepting connections and such). This patch adds the extra information by attaching an object to the SelectedKeys. This is inspired by similar implementation in ZooKeeper and is different than the method used by KAFKA-1684 patch, which extends SocketChannel. Both methods have merits, but we need to decide on one before proceeding, so TLS and SASL implementations will be at least somewhat similar. Implement a session concept in the socket server -- Key: KAFKA-1683 URL: https://issues.apache.org/jira/browse/KAFKA-1683 Project: Kafka Issue Type: Sub-task Components: security Affects Versions: 0.9.0 Reporter: Jay Kreps Assignee: Gwen Shapira Attachments: KAFKA-1683.patch To implement authentication we need a way to keep track of some things between requests. The initial use for this would be remembering the authenticated user/principle info, but likely more uses would come up (for example we will also need to remember whether and which encryption or integrity measures are in place on the socket so we can wrap and unwrap writes and reads). I was thinking we could just add a Session object that might have a user field. The session object would need to get added to RequestChannel.Request so it is passed down to the API layer with each request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1683) Implement a session concept in the socket server
[ https://issues.apache.org/jira/browse/KAFKA-1683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14184411#comment-14184411 ] Gwen Shapira commented on KAFKA-1683: - Created reviewboard https://reviews.apache.org/r/27204/diff/ against branch origin/trunk Implement a session concept in the socket server -- Key: KAFKA-1683 URL: https://issues.apache.org/jira/browse/KAFKA-1683 Project: Kafka Issue Type: Sub-task Components: security Affects Versions: 0.9.0 Reporter: Jay Kreps Assignee: Gwen Shapira Attachments: KAFKA-1683.patch, KAFKA-1683.patch To implement authentication we need a way to keep track of some things between requests. The initial use for this would be remembering the authenticated user/principle info, but likely more uses would come up (for example we will also need to remember whether and which encryption or integrity measures are in place on the socket so we can wrap and unwrap writes and reads). I was thinking we could just add a Session object that might have a user field. The session object would need to get added to RequestChannel.Request so it is passed down to the API layer with each request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1683) Implement a session concept in the socket server
[ https://issues.apache.org/jira/browse/KAFKA-1683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14183089#comment-14183089 ] Gwen Shapira commented on KAFKA-1683: - Reviewed Zookeeper SASL code to get better handle on where we are heading. Since our high-level design is inspired by ZK, I encourage everyone involved to take a look too. One thing it clarified for me - SASL does not (necessarily) maintain its own information on the socket. We need to attach something extra that will be able to retrieve authenticated identity and provide some of the security-protocol specific implementation. In ZK, this something is a ServerCnxn instance that gets attached to SelectionKey (nice trick that I was unaware of) and optionally references a ZooKeeperSaslServer. In Ivan's patch for KAFKA-1684, we instead extend SocketChannel with a protocol-specific wrapper and use this to maintain authentication state. As far as I can see, both solutions are valid, and both allow us to attach authentication information to a socket/channel and maintain it there - with the benefit that it can easily match the socket lifecycle. I need to have one of these exist for the no authentication case for this patch. I'm going to go with Cnxn instance attached to keys and not the SocketChannel extension, simply because its less code to merge into KAFKA-1684 later :) For same reason, the Cnxn will be a clear mock - i.e. will not contain any of the functionality we actually need from a security-protocol-specific object except authId getters and setters. If we decide to go with Cnxn objects, the functionality we actually need (handshakes, wrapping) will get implemented in followup JIRAs and if we decide to keep the SocketChannels, this will go away anyway. Implement a session concept in the socket server -- Key: KAFKA-1683 URL: https://issues.apache.org/jira/browse/KAFKA-1683 Project: Kafka Issue Type: Sub-task Components: security Affects Versions: 0.9.0 Reporter: Jay Kreps Assignee: Gwen Shapira To implement authentication we need a way to keep track of some things between requests. The initial use for this would be remembering the authenticated user/principle info, but likely more uses would come up (for example we will also need to remember whether and which encryption or integrity measures are in place on the socket so we can wrap and unwrap writes and reads). I was thinking we could just add a Session object that might have a user field. The session object would need to get added to RequestChannel.Request so it is passed down to the API layer with each request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1683) Implement a session concept in the socket server
[ https://issues.apache.org/jira/browse/KAFKA-1683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14181323#comment-14181323 ] Joe Stein commented on KAFKA-1683: -- [~gwenshap] We need to be careful with auth once only constant session. Revoking credentials would not be picked up until a new session was created. So we need some balance between always re-auth and session for scale. All of this down to deploy requirements many/most should be ok I think with the re-auth only once every X timeframe or always (levers/knobs). Or some other part of the code/system that is constantly checking auth changes and notifying the server of that change expiring the session then (making it async in nature). I really like it to be in the KafkaApi that is where all the good stuff is and this should mixed in cleanly with that IMHO. Implement a session concept in the socket server -- Key: KAFKA-1683 URL: https://issues.apache.org/jira/browse/KAFKA-1683 Project: Kafka Issue Type: Sub-task Components: security Affects Versions: 0.9.0 Reporter: Jay Kreps Assignee: Gwen Shapira To implement authentication we need a way to keep track of some things between requests. The initial use for this would be remembering the authenticated user/principle info, but likely more uses would come up (for example we will also need to remember whether and which encryption or integrity measures are in place on the socket so we can wrap and unwrap writes and reads). I was thinking we could just add a Session object that might have a user field. The session object would need to get added to RequestChannel.Request so it is passed down to the API layer with each request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1683) Implement a session concept in the socket server
[ https://issues.apache.org/jira/browse/KAFKA-1683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14181569#comment-14181569 ] Jay Kreps commented on KAFKA-1683: -- I had imagined the auth happening in the network layer, though I agree that may be somewhat mixing concerns and your approach sounds like it could potentially better. There are a few details to work out, though. In the Kerberos case this is straight-forward since the request is processed in the API layer anyway. But how will this work for TLS? I was imagining that TLS would use a separate acceptor in the network layer that after accepting a connection and completing the handshake would populate the appropriate session fields. However if the sessions are maintained in the API layer then somehow we will have to work out how the result of the authentication gets to the api layer. Another issue is how we remove sessions when the connection closes. We would need to garbage collect these as connections were closed, and, perhaps more importantly, we need to ensure that it isn't possible to trick your way into that hash table. I.e. if the notion of equality for SelectorKey is comparing the underlying file descriptor, and file descriptors are reused periodically there could be problems. The implementation I was something like this. Currently we attach the current request to each socket in the socket server. Instead of this we would attach some object that could hold the session, the current request, and any other future things we need. For TLS the user information would be populated after the handshake but before the acceptor hands the connection over to the processor thread. For Kerberos the session would be populated in KafkaApis.handleAuthenticate (or whatever we call it). The only advantage to this approach that I see is that the session lifetime is easily tied to the connection (which is what we want) and it will be easy to validate this. I was thinking of the session concept as being a somewhat generic feature of the network layer to maintain per-connection state so in that sense it isn't too terrible to have it be part of the networkapi layer contract, though I agree in general that the simpler we can keep the network stuff the better (it would be really easy for more and more business logic to creep in once we are doing auth stuff). Implement a session concept in the socket server -- Key: KAFKA-1683 URL: https://issues.apache.org/jira/browse/KAFKA-1683 Project: Kafka Issue Type: Sub-task Components: security Affects Versions: 0.9.0 Reporter: Jay Kreps Assignee: Gwen Shapira To implement authentication we need a way to keep track of some things between requests. The initial use for this would be remembering the authenticated user/principle info, but likely more uses would come up (for example we will also need to remember whether and which encryption or integrity measures are in place on the socket so we can wrap and unwrap writes and reads). I was thinking we could just add a Session object that might have a user field. The session object would need to get added to RequestChannel.Request so it is passed down to the API layer with each request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1683) Implement a session concept in the socket server
[ https://issues.apache.org/jira/browse/KAFKA-1683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14181696#comment-14181696 ] Gwen Shapira commented on KAFKA-1683: - Very good points [~jkreps] and [~joestein]. The way I read this - we should not rely on Session to maintain security. This is too high of a risk and very much re-inventing the wheel. We should use Session to maintain a generic user information, common to SASL, TLS and delegation tokens, that the API handlers know how to use. This will basically serve as the basis for the authorization layer. The way GSS + JAAS work (and I'm pretty sure we want to use these for Kerberos) - the client and server do a little authentication dance when the secured connection is first established, and the client is left with input and output streams that are secured. Everything sent and received on that socket afterwards will be encrypted with the server key and include the client session token. Decrypting the message and validating the client session token doesn't require communicating with Kerberos, so while there is certain overhead, it doesn't involve the network. If we use this token to populate our Session object with every request, we can be certain that the information is relevant to current connection and is unexpired (It will be the client responsibility to renew the token with KRB occasionally, this is per protocol). This will be easier to do if we use a separate port for Kerberos connections, they way we do for TLS, otherwise we'll need another way to figure out whether to perform the little song-and-dance when accepting the connection. But we can figure out this part later. Meanwhile, Session object, populated by SocketServer when creating the request, sent down to API layer. Since I like having the requestKey in the API, I'll add the new Session object as an addition, not a replacement. ... and that was a very roundabout way to go back to the original request, but I had to make sure this still makes sense to me :) Implement a session concept in the socket server -- Key: KAFKA-1683 URL: https://issues.apache.org/jira/browse/KAFKA-1683 Project: Kafka Issue Type: Sub-task Components: security Affects Versions: 0.9.0 Reporter: Jay Kreps Assignee: Gwen Shapira To implement authentication we need a way to keep track of some things between requests. The initial use for this would be remembering the authenticated user/principle info, but likely more uses would come up (for example we will also need to remember whether and which encryption or integrity measures are in place on the socket so we can wrap and unwrap writes and reads). I was thinking we could just add a Session object that might have a user field. The session object would need to get added to RequestChannel.Request so it is passed down to the API layer with each request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1683) Implement a session concept in the socket server
[ https://issues.apache.org/jira/browse/KAFKA-1683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14182024#comment-14182024 ] Jay Kreps commented on KAFKA-1683: -- Here is what I was imagining, not sure if this is what you are proposing or not. It might be worth while to chat quickly to hash it out. 1. We add Session to the request passed from the socket server to the api layer. Session.authenticatedUser is the user that we will use in the authorization layer to check permissions. Session.securityCodec is the codec used to wrap and unwrap messages sent by the socket layer. If the securityCodec is null, no wrapping or unwrapping is done. We will need an SSL and SASL implementation of the security codec. The SSL version will wrap the SSLEngine instance and the SASL version will wrap the SaslServer instance, both of which provide a wrap and unwrap version. 2. Currently in the socketserver the only state we maintain per connection is the current request we are reading. This is attached to the channel. We would replace this with some variable which could hold both the in-progress request and the session. 3. We would add a new TlsAcceptor that would listen on the TLS port and handle the handshake for new TLS connections. When the handshake is complete the TlsAcceptor would grab the user information and populate that field in the session along with the securityCodec. 4. A similar thing would happen in KafkaApis.handleAuthenticate for Kerberos. This api would take the next step in the SASL challenge/response cycle and if it is complete it would populate the authenticatedUser and securityCodec fields in the session. 5. All authorization checks would be done using Session.authenticatedUser. 6. When the socket server is writing data it will always first run it through securityCodec.wrap() prior to sending it to make sure it is encrypted. When it is reading data it will read a complete size delimited message and then do securityCodec.unwrap() prior to handing that buffer off to the api layer. If securityCodec is null it will skip this step. A couple of assumptions here that I'm not sure were explicit: a. Gwen as you point out the notion of user/subject/principle isn't totally clear. Can this just be a string we extract rather than a java Principle/Subject? b. Is the protocol for encrypted or otherwise integrity checked packets a 4 byte plaintext size followed by an N byte packet? Or is the size also encrypted? If the later the unwrapping will have to happen in tandem with the reading which is slightly more complex but perhaps more clean. Implement a session concept in the socket server -- Key: KAFKA-1683 URL: https://issues.apache.org/jira/browse/KAFKA-1683 Project: Kafka Issue Type: Sub-task Components: security Affects Versions: 0.9.0 Reporter: Jay Kreps Assignee: Gwen Shapira To implement authentication we need a way to keep track of some things between requests. The initial use for this would be remembering the authenticated user/principle info, but likely more uses would come up (for example we will also need to remember whether and which encryption or integrity measures are in place on the socket so we can wrap and unwrap writes and reads). I was thinking we could just add a Session object that might have a user field. The session object would need to get added to RequestChannel.Request so it is passed down to the API layer with each request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1683) Implement a session concept in the socket server
[ https://issues.apache.org/jira/browse/KAFKA-1683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14180983#comment-14180983 ] Gwen Shapira commented on KAFKA-1683: - We need to map SelectorKeys (representing connections) to a Session object (containing user and other identifiers). This can be done in a map that will get populated when an AUTH request appears and used by the rest of the APIs. Since the Request that appears in the API contains the SelectorKeys (RequestKey field, currently unused, but I'm glad someone had the foresight to add it), I think it makes much more sense to manage this mapping in KafkaApis, rather than in SocketServer. This can be internal to the API layer - create the Session object and add the mapping (Request.requestKey.hashcode()-Session) when an AuthRequest happens and have the various handlers use it to map the requests with the keys to sessions. An alternative design would be to maintain this mapping in the SocketServer and have the processor add this information to Request when it creates the request. I don't like the idea of moving this responsibility to the network layer when it can be done on a higher level, which it can - because the request object already has a session identifier. So unless someone objects - I'm going with the modification to KafkaApis. [~jkreps][~joestein] - any issues with this? As a side note, other systems (pretty much anything HTTP-based, REST, Thrift, etc) send the Kerberos session ticket in every request and use it to re-authenticate and provide the identity rather than maintain a stable session. This can be an alternative design, although I'd think its one with higher overhead. Implement a session concept in the socket server -- Key: KAFKA-1683 URL: https://issues.apache.org/jira/browse/KAFKA-1683 Project: Kafka Issue Type: Sub-task Components: security Affects Versions: 0.9.0 Reporter: Jay Kreps Assignee: Gwen Shapira To implement authentication we need a way to keep track of some things between requests. The initial use for this would be remembering the authenticated user/principle info, but likely more uses would come up (for example we will also need to remember whether and which encryption or integrity measures are in place on the socket so we can wrap and unwrap writes and reads). I was thinking we could just add a Session object that might have a user field. The session object would need to get added to RequestChannel.Request so it is passed down to the API layer with each request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
