[jira] [Updated] (HADOOP-9621) Document/analyze current Hadoop security model
[ https://issues.apache.org/jira/browse/HADOOP-9621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kyle Leckie updated HADOOP-9621: Attachment: HadoopSecurityAnalysis-20130624.pdf Document/analyze current Hadoop security model -- Key: HADOOP-9621 URL: https://issues.apache.org/jira/browse/HADOOP-9621 Project: Hadoop Common Issue Type: Task Components: security Reporter: Brian Swan Priority: Minor Labels: documentation Attachments: HadoopSecurityAnalysis-20130612.pdf, HadoopSecurityAnalysis-20130614.pdf, HadoopSecurityAnalysis-20130624.pdf, ThreatsforToken-basedAuthN-20130619.pdf Original Estimate: 336h Remaining Estimate: 336h In light of the proposed changes to Hadoop security in Hadoop-9533 and Hadoop-9392, having a common, detailed understanding (in the form of a document) of the benefits/drawbacks of the current security model and how it works would be useful. The document should address all security principals, their authentication mechanisms, and handling of shared secrets through the lens of the following principles: Minimize attack surface area, Establish secure defaults, Principle of Least privilege, Principle of Defense in depth, Fail securely, Don’t trust services, Separation of duties, Avoid security by obscurity, Keep security simple, Fix security issues correctly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9621) Document/analyze current Hadoop security model
[ https://issues.apache.org/jira/browse/HADOOP-9621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13692617#comment-13692617 ] Kyle Leckie commented on HADOOP-9621: - Public url for document in progress is: https://docs.google.com/document/d/1POyKfDxZaMNVJi-4c2mpJUfuSBch1arW-pc5xvUKNno/edit?usp=sharing See attached PDFs for history of document. Please comment in this JIRA with any suggestions or corrections. Document/analyze current Hadoop security model -- Key: HADOOP-9621 URL: https://issues.apache.org/jira/browse/HADOOP-9621 Project: Hadoop Common Issue Type: Task Components: security Reporter: Brian Swan Priority: Minor Labels: documentation Attachments: HadoopSecurityAnalysis-20130612.pdf, HadoopSecurityAnalysis-20130614.pdf, HadoopSecurityAnalysis-20130624.pdf, ThreatsforToken-basedAuthN-20130619.pdf Original Estimate: 336h Remaining Estimate: 336h In light of the proposed changes to Hadoop security in Hadoop-9533 and Hadoop-9392, having a common, detailed understanding (in the form of a document) of the benefits/drawbacks of the current security model and how it works would be useful. The document should address all security principals, their authentication mechanisms, and handling of shared secrets through the lens of the following principles: Minimize attack surface area, Establish secure defaults, Principle of Least privilege, Principle of Defense in depth, Fail securely, Don’t trust services, Separation of duties, Avoid security by obscurity, Keep security simple, Fix security issues correctly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9533) Centralized Hadoop SSO/Token Server
[ https://issues.apache.org/jira/browse/HADOOP-9533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13677835#comment-13677835 ] Kyle Leckie commented on HADOOP-9533: - Hi Daryn, I have assumed that the issues with TLS will be: 1) Key management 2) Possible performance degradation With the intensive use and reliance on the JDKs implementations of TLS I don't expect any known unpatched issues. Older versions of the protocol have weaknesses but we can enforce TLS 1.1+. -- Kyle Centralized Hadoop SSO/Token Server --- Key: HADOOP-9533 URL: https://issues.apache.org/jira/browse/HADOOP-9533 Project: Hadoop Common Issue Type: New Feature Components: security Reporter: Larry McCay Attachments: HSSO-Interaction-Overview-rev-1.docx, HSSO-Interaction-Overview-rev-1.pdf This is an umbrella Jira filing to oversee a set of proposals for introducing a new master service for Hadoop Single Sign On (HSSO). There is an increasing need for pluggable authentication providers that authenticate both users and services as well as validate tokens in order to federate identities authenticated by trusted IDPs. These IDPs may be deployed within the enterprise or third-party IDPs that are external to the enterprise. These needs speak to a specific pain point: which is a narrow integration path into the enterprise identity infrastructure. Kerberos is a fine solution for those that already have it in place or are willing to adopt its use but there remains a class of user that finds this unacceptable and needs to integrate with a wider variety of identity management solutions. Another specific pain point is that of rolling and distributing keys. A related and integral part of the HSSO server is library called the Credential Management Framework (CMF), which will be a common library for easing the management of secrets, keys and credentials. Initially, the existing delegation, block access and job tokens will continue to be utilized. There may be some changes required to leverage a PKI based signature facility rather than shared secrets. This is a means to simplify the solution for the pain point of distributing shared secrets. This project will primarily centralize the responsibility of authentication and federation into a single service that is trusted across the Hadoop cluster and optionally across multiple clusters. This greatly simplifies a number of things in the Hadoop ecosystem: 1.a single token format that is used across all of Hadoop regardless of authentication method 2.a single service to have pluggable providers instead of all services 3.a single token authority that would be trusted across the cluster/s and through PKI encryption be able to easily issue cryptographically verifiable tokens 4.automatic rolling of the token authority’s keys and publishing of the public key for easy access by those parties that need to verify incoming tokens 5.use of PKI for signatures eliminates the need for securely sharing and distributing shared secrets In addition to serving as the internal Hadoop SSO service this service will be leveraged by the Knox Gateway from the cluster perimeter in order to acquire the Hadoop cluster tokens. The same token mechanism that is used for internal services will be used to represent user identities. Providing for interesting scenarios such as SSO across Hadoop clusters within an enterprise and/or into the cloud. The HSSO service will be comprised of three major components and capabilities: 1.Federating IDP – authenticates users/services and issues the common Hadoop token 2.Federating SP – validates the token of trusted external IDPs and issues the common Hadoop token 3.Token Authority – management of the common Hadoop tokens – including: a.Issuance b.Renewal c.Revocation As this is a meta Jira for tracking this overall effort, the details of the individual efforts will be submitted along with the child Jira filings. Hadoop-Common would seem to be the most appropriate home for such a service and its related common facilities. We will also leverage and extend existing common mechanisms as appropriate. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9621) Document/analyze current Hadoop security model
[ https://issues.apache.org/jira/browse/HADOOP-9621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13677837#comment-13677837 ] Kyle Leckie commented on HADOOP-9621: - I think this is needed. I have not yet seen an analysis wrt the common security principles. Document/analyze current Hadoop security model -- Key: HADOOP-9621 URL: https://issues.apache.org/jira/browse/HADOOP-9621 Project: Hadoop Common Issue Type: Task Components: security Reporter: Brian Swan Priority: Minor Labels: documentation Original Estimate: 336h Remaining Estimate: 336h In light of the proposed changes to Hadoop security in Hadoop-9533 and Hadoop-9392, having a common, detailed understanding (in the form of a document) of the benefits/drawbacks of the current security model and how it works would be useful. The document should address all security principals, their authentication mechanisms, and handling of shared secrets through the lens of the following principles: Minimize attack surface area, Establish secure defaults, Principle of Least privilege, Principle of Defense in depth, Fail securely, Don’t trust services, Separation of duties, Avoid security by obscurity, Keep security simple, Fix security issues correctly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9392) Token based authentication and Single Sign On
[ https://issues.apache.org/jira/browse/HADOOP-9392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13674055#comment-13674055 ] Kyle Leckie commented on HADOOP-9392: - Thanks for your thorough response Kai, 1) I agree on having support for tokens with pluggable token validation. Having the token contain an audience property in order to limit its scope should not add significant overhead but I take your point about having an initial implementation and progressing from there on an as needed basis. 2)3) Thanks for the clarification. It seems that supporting pluggable token validation is a significant feature in itself and the TAS work can be layered on top. What do you think of having the token validation and transmission as a separate JIRA? -- Kyle Token based authentication and Single Sign On - Key: HADOOP-9392 URL: https://issues.apache.org/jira/browse/HADOOP-9392 Project: Hadoop Common Issue Type: New Feature Components: security Reporter: Kai Zheng Assignee: Kai Zheng Fix For: 3.0.0 Attachments: token-based-authn-plus-sso.pdf This is an umbrella entry for one of project Rhino’s topic, for details of project Rhino, please refer to https://github.com/intel-hadoop/project-rhino/. The major goal for this entry as described in project Rhino was “Core, HDFS, ZooKeeper, and HBase currently support Kerberos authentication at the RPC layer, via SASL. However this does not provide valuable attributes such as group membership, classification level, organizational identity, or support for user defined attributes. Hadoop components must interrogate external resources for discovering these attributes and at scale this is problematic. There is also no consistent delegation model. HDFS has a simple delegation capability, and only Oozie can take limited advantage of it. We will implement a common token based authentication framework to decouple internal user and service authentication from external mechanisms used to support it (like Kerberos)” We’d like to start our work from Hadoop-Common and try to provide common facilities by extending existing authentication framework which support: 1.Pluggable token provider interface 2.Pluggable token verification protocol and interface 3.Security mechanism to distribute secrets in cluster nodes 4.Delegation model of user authentication -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9533) Centralized Hadoop SSO/Token Server
[ https://issues.apache.org/jira/browse/HADOOP-9533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13674075#comment-13674075 ] Kyle Leckie commented on HADOOP-9533: - Hi Daryn, Yes SASL could occur over SSL. With SSL we get protection from eavesdropping, tampering and possibly server authentication. With that we can pass the a bearer token over the network. Performing the SASL exchange would only slow down a request. In addition the JAVA SASL mechanisms seem out of date. (see: Moving DIGEST-MD5 to Historic http://tools.ietf.org/html/rfc6331). This also describes the issues with downgrade attacks. If we are going to bet on a piece of code that needs to be updated, performant and promptly patched I would bet on the TLS code. The SGT will only be handed to the HSSO and not the services such as the NN. The NN would get an NN specific token. -- Kyle Centralized Hadoop SSO/Token Server --- Key: HADOOP-9533 URL: https://issues.apache.org/jira/browse/HADOOP-9533 Project: Hadoop Common Issue Type: New Feature Components: security Reporter: Larry McCay Attachments: HSSO-Interaction-Overview-rev-1.docx, HSSO-Interaction-Overview-rev-1.pdf This is an umbrella Jira filing to oversee a set of proposals for introducing a new master service for Hadoop Single Sign On (HSSO). There is an increasing need for pluggable authentication providers that authenticate both users and services as well as validate tokens in order to federate identities authenticated by trusted IDPs. These IDPs may be deployed within the enterprise or third-party IDPs that are external to the enterprise. These needs speak to a specific pain point: which is a narrow integration path into the enterprise identity infrastructure. Kerberos is a fine solution for those that already have it in place or are willing to adopt its use but there remains a class of user that finds this unacceptable and needs to integrate with a wider variety of identity management solutions. Another specific pain point is that of rolling and distributing keys. A related and integral part of the HSSO server is library called the Credential Management Framework (CMF), which will be a common library for easing the management of secrets, keys and credentials. Initially, the existing delegation, block access and job tokens will continue to be utilized. There may be some changes required to leverage a PKI based signature facility rather than shared secrets. This is a means to simplify the solution for the pain point of distributing shared secrets. This project will primarily centralize the responsibility of authentication and federation into a single service that is trusted across the Hadoop cluster and optionally across multiple clusters. This greatly simplifies a number of things in the Hadoop ecosystem: 1.a single token format that is used across all of Hadoop regardless of authentication method 2.a single service to have pluggable providers instead of all services 3.a single token authority that would be trusted across the cluster/s and through PKI encryption be able to easily issue cryptographically verifiable tokens 4.automatic rolling of the token authority’s keys and publishing of the public key for easy access by those parties that need to verify incoming tokens 5.use of PKI for signatures eliminates the need for securely sharing and distributing shared secrets In addition to serving as the internal Hadoop SSO service this service will be leveraged by the Knox Gateway from the cluster perimeter in order to acquire the Hadoop cluster tokens. The same token mechanism that is used for internal services will be used to represent user identities. Providing for interesting scenarios such as SSO across Hadoop clusters within an enterprise and/or into the cloud. The HSSO service will be comprised of three major components and capabilities: 1.Federating IDP – authenticates users/services and issues the common Hadoop token 2.Federating SP – validates the token of trusted external IDPs and issues the common Hadoop token 3.Token Authority – management of the common Hadoop tokens – including: a.Issuance b.Renewal c.Revocation As this is a meta Jira for tracking this overall effort, the details of the individual efforts will be submitted along with the child Jira filings. Hadoop-Common would seem to be the most appropriate home for such a service and its related common facilities. We will also leverage and extend existing common mechanisms as appropriate. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more
[jira] [Commented] (HADOOP-9392) Token based authentication and Single Sign On
[ https://issues.apache.org/jira/browse/HADOOP-9392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13660843#comment-13660843 ] Kyle Leckie commented on HADOOP-9392: - Kai - Thanks for the proposal. A few questions: 1) Is it correct that the identity token is not restricted to a particular service and the same token is valid for all services that trust the token realm? 2) Is is correct that the identity token is utilized as a bearer token and not a shared secret? 3) You have repeatedly grouped LDAP and AD as the same form of authentication. Can you further clarify on the meaning of this grouping? Token based authentication and Single Sign On - Key: HADOOP-9392 URL: https://issues.apache.org/jira/browse/HADOOP-9392 Project: Hadoop Common Issue Type: New Feature Components: security Reporter: Kai Zheng Assignee: Kai Zheng Fix For: 3.0.0 Attachments: token-based-authn-plus-sso.pdf This is an umbrella entry for one of project Rhino’s topic, for details of project Rhino, please refer to https://github.com/intel-hadoop/project-rhino/. The major goal for this entry as described in project Rhino was “Core, HDFS, ZooKeeper, and HBase currently support Kerberos authentication at the RPC layer, via SASL. However this does not provide valuable attributes such as group membership, classification level, organizational identity, or support for user defined attributes. Hadoop components must interrogate external resources for discovering these attributes and at scale this is problematic. There is also no consistent delegation model. HDFS has a simple delegation capability, and only Oozie can take limited advantage of it. We will implement a common token based authentication framework to decouple internal user and service authentication from external mechanisms used to support it (like Kerberos)” We’d like to start our work from Hadoop-Common and try to provide common facilities by extending existing authentication framework which support: 1.Pluggable token provider interface 2.Pluggable token verification protocol and interface 3.Security mechanism to distribute secrets in cluster nodes 4.Delegation model of user authentication -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9533) Centralized Hadoop SSO/Token Server
[ https://issues.apache.org/jira/browse/HADOOP-9533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13660855#comment-13660855 ] Kyle Leckie commented on HADOOP-9533: - Daryn Great point on negotiation. One concern I have with SASL or other forms of mechanism negotiation is that the authenticity of the mechanism can be confirmed in order to avoid downgrade attacks. Would this assume TLS or some other mechanism? Centralized Hadoop SSO/Token Server --- Key: HADOOP-9533 URL: https://issues.apache.org/jira/browse/HADOOP-9533 Project: Hadoop Common Issue Type: New Feature Components: security Reporter: Larry McCay Attachments: HSSO-Interaction-Overview-rev-1.docx, HSSO-Interaction-Overview-rev-1.pdf This is an umbrella Jira filing to oversee a set of proposals for introducing a new master service for Hadoop Single Sign On (HSSO). There is an increasing need for pluggable authentication providers that authenticate both users and services as well as validate tokens in order to federate identities authenticated by trusted IDPs. These IDPs may be deployed within the enterprise or third-party IDPs that are external to the enterprise. These needs speak to a specific pain point: which is a narrow integration path into the enterprise identity infrastructure. Kerberos is a fine solution for those that already have it in place or are willing to adopt its use but there remains a class of user that finds this unacceptable and needs to integrate with a wider variety of identity management solutions. Another specific pain point is that of rolling and distributing keys. A related and integral part of the HSSO server is library called the Credential Management Framework (CMF), which will be a common library for easing the management of secrets, keys and credentials. Initially, the existing delegation, block access and job tokens will continue to be utilized. There may be some changes required to leverage a PKI based signature facility rather than shared secrets. This is a means to simplify the solution for the pain point of distributing shared secrets. This project will primarily centralize the responsibility of authentication and federation into a single service that is trusted across the Hadoop cluster and optionally across multiple clusters. This greatly simplifies a number of things in the Hadoop ecosystem: 1.a single token format that is used across all of Hadoop regardless of authentication method 2.a single service to have pluggable providers instead of all services 3.a single token authority that would be trusted across the cluster/s and through PKI encryption be able to easily issue cryptographically verifiable tokens 4.automatic rolling of the token authority’s keys and publishing of the public key for easy access by those parties that need to verify incoming tokens 5.use of PKI for signatures eliminates the need for securely sharing and distributing shared secrets In addition to serving as the internal Hadoop SSO service this service will be leveraged by the Knox Gateway from the cluster perimeter in order to acquire the Hadoop cluster tokens. The same token mechanism that is used for internal services will be used to represent user identities. Providing for interesting scenarios such as SSO across Hadoop clusters within an enterprise and/or into the cloud. The HSSO service will be comprised of three major components and capabilities: 1.Federating IDP – authenticates users/services and issues the common Hadoop token 2.Federating SP – validates the token of trusted external IDPs and issues the common Hadoop token 3.Token Authority – management of the common Hadoop tokens – including: a.Issuance b.Renewal c.Revocation As this is a meta Jira for tracking this overall effort, the details of the individual efforts will be submitted along with the child Jira filings. Hadoop-Common would seem to be the most appropriate home for such a service and its related common facilities. We will also leverage and extend existing common mechanisms as appropriate. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
