[jira] [Comment Edited] (HADOOP-13836) Securing Hadoop RPC using SSL
[ https://issues.apache.org/jira/browse/HADOOP-13836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15833959#comment-15833959 ] kartheek muthyala edited comment on HADOOP-13836 at 1/23/17 6:24 AM: - Version consists of fix for some of the failed unit tests. I have to still get around with a way of adding the Certificate files Server.jks, Client.jks and TrustedCerts.jks to the patch which are required by some of the unit tests. All the whitespaces failed checks are also due to this issue. But, for some reason git apply patch is not adding it to the jenkins workspace properly. May be because I created this patch on windows environment. Can anyone help me in creating a patch with binary files on Windows environment?. was (Author: kartheek): Version consists of fix for some of the failed unit tests. I have to still get around with passing a way of adding the Certificate files Server.jks, Client.jks and TrustedCerts.jks which are required by some of the unit tests. All the whitespaces checks are also due to this issue. But, for some reason git apply patch is not adding it to the jenkins workspace properly. May be because I created this patch on windows environment. Can anyone help me in creating a patch with binary files on Windows environment?. > Securing Hadoop RPC using SSL > - > > Key: HADOOP-13836 > URL: https://issues.apache.org/jira/browse/HADOOP-13836 > Project: Hadoop Common > Issue Type: New Feature > Components: ipc >Reporter: kartheek muthyala >Assignee: kartheek muthyala > Attachments: HADOOP-13836.patch, HADOOP-13836-v2.patch, > HADOOP-13836-v3.patch, Secure IPC OSS Proposal-1.pdf, SecureIPC Performance > Analysis-OSS.pdf > > > Today, RPC connections in Hadoop are encrypted using Simple Authentication & > Security Layer (SASL), with the Kerberos ticket based authentication or > Digest-md5 checksum based authentication protocols. This proposal is about > enhancing this cipher suite with SSL/TLS based encryption and authentication. > SSL/TLS is a proposed Internet Engineering Task Force (IETF) standard, that > provides data security and integrity across two different end points in a > network. This protocol has made its way to a number of applications such as > web browsing, email, internet faxing, messaging, VOIP etc. And supporting > this cipher suite at the core of Hadoop would give a good synergy with the > applications on top and also bolster industry adoption of Hadoop. > The Server and Client code in Hadoop IPC should support the following modes > of communication > 1.Plain > 2. SASL encryption with an underlying authentication > 3. SSL based encryption and authentication (x509 certificate) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13836) Securing Hadoop RPC using SSL
[ https://issues.apache.org/jira/browse/HADOOP-13836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15820360#comment-15820360 ] kartheek muthyala edited comment on HADOOP-13836 at 1/12/17 7:23 AM: - Hi all, I am posting the performance analysis of SSL feature implemented using the current patch. We ran Teragen and Terasort to compare the overhead with SSL on a small cluster. was (Author: kartheek): Hi all, I am posting the performance analysis of SSL feature implemented using the current patch, using Teragen and Terasort ran on a small cluster. > Securing Hadoop RPC using SSL > - > > Key: HADOOP-13836 > URL: https://issues.apache.org/jira/browse/HADOOP-13836 > Project: Hadoop Common > Issue Type: New Feature > Components: ipc >Reporter: kartheek muthyala >Assignee: kartheek muthyala > Attachments: HADOOP-13836.patch, Secure IPC OSS Proposal-1.pdf, > SecureIPC Performance Analysis-OSS.pdf > > > Today, RPC connections in Hadoop are encrypted using Simple Authentication & > Security Layer (SASL), with the Kerberos ticket based authentication or > Digest-md5 checksum based authentication protocols. This proposal is about > enhancing this cipher suite with SSL/TLS based encryption and authentication. > SSL/TLS is a proposed Internet Engineering Task Force (IETF) standard, that > provides data security and integrity across two different end points in a > network. This protocol has made its way to a number of applications such as > web browsing, email, internet faxing, messaging, VOIP etc. And supporting > this cipher suite at the core of Hadoop would give a good synergy with the > applications on top and also bolster industry adoption of Hadoop. > The Server and Client code in Hadoop IPC should support the following modes > of communication > 1.Plain > 2. SASL encryption with an underlying authentication > 3. SSL based encryption and authentication (x509 certificate) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13836) Securing Hadoop RPC using SSL
[ https://issues.apache.org/jira/browse/HADOOP-13836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15820325#comment-15820325 ] kartheek muthyala edited comment on HADOOP-13836 at 1/12/17 7:05 AM: - Hi all, Please find attached the first version of enable SSL to Hadoop RPC design draft. [~kaizh], [~daryn] and [~ste...@apache.org], Kindly provide your comments on the same. was (Author: kartheek): Secure IPC design draft - v1 > Securing Hadoop RPC using SSL > - > > Key: HADOOP-13836 > URL: https://issues.apache.org/jira/browse/HADOOP-13836 > Project: Hadoop Common > Issue Type: New Feature > Components: ipc >Reporter: kartheek muthyala >Assignee: kartheek muthyala > Attachments: HADOOP-13836.patch, Secure IPC OSS Proposal-1.pdf > > > Today, RPC connections in Hadoop are encrypted using Simple Authentication & > Security Layer (SASL), with the Kerberos ticket based authentication or > Digest-md5 checksum based authentication protocols. This proposal is about > enhancing this cipher suite with SSL/TLS based encryption and authentication. > SSL/TLS is a proposed Internet Engineering Task Force (IETF) standard, that > provides data security and integrity across two different end points in a > network. This protocol has made its way to a number of applications such as > web browsing, email, internet faxing, messaging, VOIP etc. And supporting > this cipher suite at the core of Hadoop would give a good synergy with the > applications on top and also bolster industry adoption of Hadoop. > The Server and Client code in Hadoop IPC should support the following modes > of communication > 1.Plain > 2. SASL encryption with an underlying authentication > 3. SSL based encryption and authentication (x509 certificate) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13836) Securing Hadoop RPC using SSL
[ https://issues.apache.org/jira/browse/HADOOP-13836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15706026#comment-15706026 ] Arun Suresh edited comment on HADOOP-13836 at 11/29/16 6:02 PM: Thanks for pointing out IPv6 [~steve_l] I assume it should just work, considering this comes into play only at Socket creation. My understanding is that SSL/TLS certificate authentication is based on the DNS names of the entities involved, so again, I assume it should just work, if DNS resolution works correctly. But yes, we will try to verify it. My understanding is that the HADOOP-11890 branch had most of the IPv6 based changes. If this branch is uptodate, [~kartheek], we can maybe test it against that branch. Any more pointers to verifying if this would work with IPv6 is welcome. (cc: [~eclark], [~nkedel]) was (Author: asuresh): Thanks for pointing out IPv6 [~steve_l] I assume it should just work, considering this comes into play only at Socket creation. My understanding is that SSL/TLS certification authentication is based on the DNS names of the entities involved, so again, I assume it should just work, if DNS resolution works correctly. But yes, we will try to verify it. My understanding is that the HADOOP-11890 branch had most of the IPv6 based changes. If this branch is uptodate, [~kartheek], we can maybe test it against that branch. Any more pointers to verifying if this would work with IPv6 is welcome. (cc: [~eclark], [~nkedel]) > Securing Hadoop RPC using SSL > - > > Key: HADOOP-13836 > URL: https://issues.apache.org/jira/browse/HADOOP-13836 > Project: Hadoop Common > Issue Type: New Feature > Components: ipc >Reporter: kartheek muthyala >Assignee: kartheek muthyala > Attachments: HADOOP-13836.patch > > > Today, RPC connections in Hadoop are encrypted using Simple Authentication & > Security Layer (SASL), with the Kerberos ticket based authentication or > Digest-md5 checksum based authentication protocols. This proposal is about > enhancing this cipher suite with SSL/TLS based encryption and authentication. > SSL/TLS is a proposed Internet Engineering Task Force (IETF) standard, that > provides data security and integrity across two different end points in a > network. This protocol has made its way to a number of applications such as > web browsing, email, internet faxing, messaging, VOIP etc. And supporting > this cipher suite at the core of Hadoop would give a good synergy with the > applications on top and also bolster industry adoption of Hadoop. > The Server and Client code in Hadoop IPC should support the following modes > of communication > 1.Plain > 2. SASL encryption with an underlying authentication > 3. SSL based encryption and authentication (x509 certificate) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13836) Securing Hadoop RPC using SSL
[ https://issues.apache.org/jira/browse/HADOOP-13836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15702593#comment-15702593 ] Antonios Kouzoupis edited comment on HADOOP-13836 at 11/28/16 5:41 PM: --- [~kartheek] I took a quick look on your patch. I think it's more reasonable to use the "hadoop.rpc.socket.factory.class.default" configuration key to load the desired socket factory. At the moment the StandardSocketFactory it's been used but you may provide your own factory with ssl/tls support. Also, it might be better to reuse org.apache.hadoop.security.ssl.SSLFactory for the SSLEngine creation. was (Author: antkou): [~kartheek] I took a quick look on your patch. I think it's more reasonable to use the "hadoop.rpc.socket.factory.class.default" configuration key to load the desired socket factory. At the moment the StandardSocketFactory it's been used but you may provide your own factory with ssl/tls support. > Securing Hadoop RPC using SSL > - > > Key: HADOOP-13836 > URL: https://issues.apache.org/jira/browse/HADOOP-13836 > Project: Hadoop Common > Issue Type: New Feature > Components: ipc >Reporter: kartheek muthyala > Attachments: HADOOP-13836.patch > > > Today, RPC connections in Hadoop are encrypted using Simple Authentication & > Security Layer (SASL), with the Kerberos ticket based authentication or > Digest-md5 checksum based authentication protocols. This proposal is about > enhancing this cipher suite with SSL/TLS based encryption and authentication. > SSL/TLS is a proposed Internet Engineering Task Force (IETF) standard, that > provides data security and integrity across two different end points in a > network. This protocol has made its way to a number of applications such as > web browsing, email, internet faxing, messaging, VOIP etc. And supporting > this cipher suite at the core of Hadoop would give a good synergy with the > applications on top and also bolster industry adoption of Hadoop. > The Server and Client code in Hadoop IPC should support the following modes > of communication > 1.Plain > 2. SASL encryption with an underlying authentication > 3. SSL based encryption and authentication (x509 certificate) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13836) Securing Hadoop RPC using SSL
[ https://issues.apache.org/jira/browse/HADOOP-13836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15702498#comment-15702498 ] kartheek muthyala edited comment on HADOOP-13836 at 11/28/16 5:03 PM: -- Yes,[~asuresh], that is exactly what we are doing here. The proposal intends to implement an SSL layer on top of existing Hadoop RPC. It introduces SSLEngine in Server to encode and decode messages, and uses Java's javax.net.ssl library to encode and decode on the Client side. We have relied on niossl library for the server side implementation of SSLEngine. Because, this implementation sits on top of SSLSocket channel implementation, we can still keep the channels open as before, and just encode and decode messages using the existing cipher keys. But, as [~ste...@apache.org]pointed out, this introduces an overhead of additional handshakes between Server and Client for different reasons like certificate exchange, validation etc. We can trade off this performance hit with the security that we will be enhancing. This will improve the usage of secure IPC in large systems. We have been running this patch internally with some long running jobs and the performance seems to be decent. I don't have the exact numbers right away, but I will post them soon. was (Author: kartheek): Yes,[~asuresh], that is exactly what we are doing here. The proposal intends to implement an SSL layer on top of existing Hadoop RPC. It introduces SSLEngine in Server to encode and decode messages, and Java's javax.net.ssl library to encode and decode on the Client side. We have relied on niossl library for the server side implementation of SSLEngine. Because, this implementation sits on top of SSLSocket channel implementation, we can still keep the channels open as before, and just encode and decode messages using the existing cipher keys. But, as [~ste...@apache.org]pointed out, this introduces an overhead of additional handshakes between Server and Client for different reasons like certificate exchange, validation etc. We can trade off this performance hit with the security that we will be enhancing. This will improve the usage of secure IPC in large systems. We have been running this patch internally with some long running jobs and the performance seems to be decent. I don't have the exact numbers right away, but I will post them soon. > Securing Hadoop RPC using SSL > - > > Key: HADOOP-13836 > URL: https://issues.apache.org/jira/browse/HADOOP-13836 > Project: Hadoop Common > Issue Type: New Feature > Components: ipc >Reporter: kartheek muthyala > Attachments: HADOOP-13836.patch > > > Today, RPC connections in Hadoop are encrypted using Simple Authentication & > Security Layer (SASL), with the Kerberos ticket based authentication or > Digest-md5 checksum based authentication protocols. This proposal is about > enhancing this cipher suite with SSL/TLS based encryption and authentication. > SSL/TLS is a proposed Internet Engineering Task Force (IETF) standard, that > provides data security and integrity across two different end points in a > network. This protocol has made its way to a number of applications such as > web browsing, email, internet faxing, messaging, VOIP etc. And supporting > this cipher suite at the core of Hadoop would give a good synergy with the > applications on top and also bolster industry adoption of Hadoop. > The Server and Client code in Hadoop IPC should support the following modes > of communication > 1.Plain > 2. SASL encryption with an underlying authentication > 3. SSL based encryption and authentication (x509 certificate) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13836) Securing Hadoop RPC using SSL
[ https://issues.apache.org/jira/browse/HADOOP-13836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15702209#comment-15702209 ] kartheek muthyala edited comment on HADOOP-13836 at 11/28/16 3:32 PM: -- Hi all, We are submitting an initial version of the patch for a preliminary review. We have tested this patch on a bunch of long running jobs, and the performance is decent. We will publish some performance numbers soon. Feel free to enhance the patch. This patch contains 1. Reorganization of IPC Server and Client classes to make them more extensible. The changes are a. A new ListenerFactory class that can dynamically instantiate appropriate listener based on the configuration. b. A new AbstractListener class that abstracts the common functionalities of different listeners. c. ConnectionFactory class to instantiate an appropriate connection class in Server and Client classes 2. Implementation of SSL layer in Server.java class 3. Implementation of SSL Client in Client.java using javax.net.ssl library. 4. Unit testing of SSL implementation. was (Author: kartheek): Hi all, We are submitting an initial version of the patch for a preliminary review. We have tested this patch on a bunch of long running jobs, and the performance is decent. We will publish some performance numbers soon. Feel free to enhance the patch. This patch contains 1. Reorganization of IPC Server and Client classes to make them more extensible. The changes are a. A new ListenerFactory class that can dynamically instantiate appropriate listener based on the configuration. b. A new AbstractListener class that abstracts the common functionalities of different listeners. c. ConnectionFactory class to instantiate an appropriate connection class in Server and Client classes 2. Implementation of SSL layer in Server.java class 3. Implementation on SSL Client in Client.java using javax.net.ssl library. 4. Unit testing of SSL implementation. > Securing Hadoop RPC using SSL > - > > Key: HADOOP-13836 > URL: https://issues.apache.org/jira/browse/HADOOP-13836 > Project: Hadoop Common > Issue Type: New Feature > Components: ipc >Reporter: kartheek muthyala > Attachments: HADOOP-13836.patch > > > Today, RPC connections in Hadoop are encrypted using Simple Authentication & > Security Layer (SASL), with the Kerberos ticket based authentication or > Digest-md5 checksum based authentication protocols. This proposal is about > enhancing this cipher suite with SSL/TLS based encryption and authentication. > SSL/TLS is a proposed Internet Engineering Task Force (IETF) standard, that > provides data security and integrity across two different end points in a > network. This protocol has made its way to a number of applications such as > web browsing, email, internet faxing, messaging, VOIP etc. And supporting > this cipher suite at the core of Hadoop would give a good synergy with the > applications on top and also bolster industry adoption of Hadoop. > The Server and Client code in Hadoop IPC should support the following modes > of communication > 1.Plain > 2. SASL encryption with an underlying authentication > 3. SSL based encryption and authentication (x509 certificate) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13836) Securing Hadoop RPC using SSL
[ https://issues.apache.org/jira/browse/HADOOP-13836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15702209#comment-15702209 ] kartheek muthyala edited comment on HADOOP-13836 at 11/28/16 3:31 PM: -- Hi all, We are submitting an initial version of the patch for a preliminary review. We have tested this patch on a bunch of long running jobs, and the performance is decent. We will publish some performance numbers soon. Feel free to enhance the patch. This patch contains 1. Reorganization of IPC Server and Client classes to make them more extensible. The changes are a. A new ListenerFactory class that can dynamically instantiate appropriate listener based on the configuration. b. A new AbstractListener class that abstracts the common functionalities of different listeners. c. ConnectionFactory class to instantiate an appropriate connection class in Server and Client classes 2. Implementation of SSL layer in Server.java class 3. Implementation on SSL Client in Client.java using javax.net.ssl library. 4. Unit testing of SSL implementation. was (Author: kartheek): Hi all, We are submitting an initial version of the patch for a preliminary review. We have tested this patch on a bunch of long running jobs, and the performance is decent. We will publish some performance numbers soon. Feel free to enhance the patch. This patch contains 1. Reorganization of IPC Server and Client classes to make them more extensible. The changes are a. A new ListenerFactory class that can dynamically instantiate appropriate listener based on the configuration. b. A new AbstractListener class that abstracts the common functionalities of different listeners. c. ConnectionFactory class to instantiate an appropriate connection class in Server and Client classes 2. Implementation of SSL layer in Server.java class 3. Implementation on Client.java uses javax.net.ssl library to make SSL connections 4. Unit testing of SSL implementation. > Securing Hadoop RPC using SSL > - > > Key: HADOOP-13836 > URL: https://issues.apache.org/jira/browse/HADOOP-13836 > Project: Hadoop Common > Issue Type: New Feature > Components: ipc >Reporter: kartheek muthyala > Attachments: HADOOP-13836.patch > > > Today, RPC connections in Hadoop are encrypted using Simple Authentication & > Security Layer (SASL), with the Kerberos ticket based authentication or > Digest-md5 checksum based authentication protocols. This proposal is about > enhancing this cipher suite with SSL/TLS based encryption and authentication. > SSL/TLS is a proposed Internet Engineering Task Force (IETF) standard, that > provides data security and integrity across two different end points in a > network. This protocol has made its way to a number of applications such as > web browsing, email, internet faxing, messaging, VOIP etc. And supporting > this cipher suite at the core of Hadoop would give a good synergy with the > applications on top and also bolster industry adoption of Hadoop. > The Server and Client code in Hadoop IPC should support the following modes > of communication > 1.Plain > 2. SASL encryption with an underlying authentication > 3. SSL based encryption and authentication (x509 certificate) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
