[jira] [Commented] (HADOOP-13836) Securing Hadoop RPC using SSL
[ https://issues.apache.org/jira/browse/HADOOP-13836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949827#comment-16949827 ] Wei-Chiu Chuang commented on HADOOP-13836: -- I think this work is superseded by HADOOP-15977 where Daryn has made good progress. > Securing Hadoop RPC using SSL > - > > Key: HADOOP-13836 > URL: https://issues.apache.org/jira/browse/HADOOP-13836 > Project: Hadoop Common > Issue Type: New Feature > Components: ipc >Reporter: kartheek muthyala >Assignee: kartheek muthyala >Priority: Major > Attachments: HADOOP-13836-v2.patch, HADOOP-13836-v3.patch, > HADOOP-13836-v4.patch, HADOOP-13836.patch, Secure IPC OSS Proposal-1.pdf, > SecureIPC Performance Analysis-OSS.pdf > > > Today, RPC connections in Hadoop are encrypted using Simple Authentication & > Security Layer (SASL), with the Kerberos ticket based authentication or > Digest-md5 checksum based authentication protocols. This proposal is about > enhancing this cipher suite with SSL/TLS based encryption and authentication. > SSL/TLS is a proposed Internet Engineering Task Force (IETF) standard, that > provides data security and integrity across two different end points in a > network. This protocol has made its way to a number of applications such as > web browsing, email, internet faxing, messaging, VOIP etc. And supporting > this cipher suite at the core of Hadoop would give a good synergy with the > applications on top and also bolster industry adoption of Hadoop. > The Server and Client code in Hadoop IPC should support the following modes > of communication > 1.Plain > 2. SASL encryption with an underlying authentication > 3. SSL based encryption and authentication (x509 certificate) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13836) Securing Hadoop RPC using SSL
[ https://issues.apache.org/jira/browse/HADOOP-13836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949579#comment-16949579 ] hirik commented on HADOOP-13836: [~kartheek] is there any release timeline for this feature? > Securing Hadoop RPC using SSL > - > > Key: HADOOP-13836 > URL: https://issues.apache.org/jira/browse/HADOOP-13836 > Project: Hadoop Common > Issue Type: New Feature > Components: ipc >Reporter: kartheek muthyala >Assignee: kartheek muthyala >Priority: Major > Attachments: HADOOP-13836-v2.patch, HADOOP-13836-v3.patch, > HADOOP-13836-v4.patch, HADOOP-13836.patch, Secure IPC OSS Proposal-1.pdf, > SecureIPC Performance Analysis-OSS.pdf > > > Today, RPC connections in Hadoop are encrypted using Simple Authentication & > Security Layer (SASL), with the Kerberos ticket based authentication or > Digest-md5 checksum based authentication protocols. This proposal is about > enhancing this cipher suite with SSL/TLS based encryption and authentication. > SSL/TLS is a proposed Internet Engineering Task Force (IETF) standard, that > provides data security and integrity across two different end points in a > network. This protocol has made its way to a number of applications such as > web browsing, email, internet faxing, messaging, VOIP etc. And supporting > this cipher suite at the core of Hadoop would give a good synergy with the > applications on top and also bolster industry adoption of Hadoop. > The Server and Client code in Hadoop IPC should support the following modes > of communication > 1.Plain > 2. SASL encryption with an underlying authentication > 3. SSL based encryption and authentication (x509 certificate) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13836) Securing Hadoop RPC using SSL
[
https://issues.apache.org/jira/browse/HADOOP-13836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16922569#comment-16922569
]
Hadoop QA commented on HADOOP-13836:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 7s{color}
| {color:red} HADOOP-13836 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-13836 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12848944/HADOOP-13836-v4.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16515/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Securing Hadoop RPC using SSL
> -
>
> Key: HADOOP-13836
> URL: https://issues.apache.org/jira/browse/HADOOP-13836
> Project: Hadoop Common
> Issue Type: New Feature
> Components: ipc
>Reporter: kartheek muthyala
>Assignee: kartheek muthyala
>Priority: Major
> Attachments: HADOOP-13836-v2.patch, HADOOP-13836-v3.patch,
> HADOOP-13836-v4.patch, HADOOP-13836.patch, Secure IPC OSS Proposal-1.pdf,
> SecureIPC Performance Analysis-OSS.pdf
>
>
> Today, RPC connections in Hadoop are encrypted using Simple Authentication &
> Security Layer (SASL), with the Kerberos ticket based authentication or
> Digest-md5 checksum based authentication protocols. This proposal is about
> enhancing this cipher suite with SSL/TLS based encryption and authentication.
> SSL/TLS is a proposed Internet Engineering Task Force (IETF) standard, that
> provides data security and integrity across two different end points in a
> network. This protocol has made its way to a number of applications such as
> web browsing, email, internet faxing, messaging, VOIP etc. And supporting
> this cipher suite at the core of Hadoop would give a good synergy with the
> applications on top and also bolster industry adoption of Hadoop.
> The Server and Client code in Hadoop IPC should support the following modes
> of communication
> 1.Plain
> 2. SASL encryption with an underlying authentication
> 3. SSL based encryption and authentication (x509 certificate)
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13836) Securing Hadoop RPC using SSL
[ https://issues.apache.org/jira/browse/HADOOP-13836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16921534#comment-16921534 ] Igal Flegmann commented on HADOOP-13836: Hi it seems that this feature is mostly for the encryption, is there anyone working on the x.509 Certificates? my team is looking at contributing to this and would like to collaborate with anyone that is working on this as well. > Securing Hadoop RPC using SSL > - > > Key: HADOOP-13836 > URL: https://issues.apache.org/jira/browse/HADOOP-13836 > Project: Hadoop Common > Issue Type: New Feature > Components: ipc >Reporter: kartheek muthyala >Assignee: kartheek muthyala >Priority: Major > Attachments: HADOOP-13836-v2.patch, HADOOP-13836-v3.patch, > HADOOP-13836-v4.patch, HADOOP-13836.patch, Secure IPC OSS Proposal-1.pdf, > SecureIPC Performance Analysis-OSS.pdf > > > Today, RPC connections in Hadoop are encrypted using Simple Authentication & > Security Layer (SASL), with the Kerberos ticket based authentication or > Digest-md5 checksum based authentication protocols. This proposal is about > enhancing this cipher suite with SSL/TLS based encryption and authentication. > SSL/TLS is a proposed Internet Engineering Task Force (IETF) standard, that > provides data security and integrity across two different end points in a > network. This protocol has made its way to a number of applications such as > web browsing, email, internet faxing, messaging, VOIP etc. And supporting > this cipher suite at the core of Hadoop would give a good synergy with the > applications on top and also bolster industry adoption of Hadoop. > The Server and Client code in Hadoop IPC should support the following modes > of communication > 1.Plain > 2. SASL encryption with an underlying authentication > 3. SSL based encryption and authentication (x509 certificate) -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13836) Securing Hadoop RPC using SSL
[
https://issues.apache.org/jira/browse/HADOOP-13836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16089922#comment-16089922
]
Hadoop QA commented on HADOOP-13836:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s{color}
| {color:red} HADOOP-13836 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-13836 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12848944/HADOOP-13836-v4.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12802/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Securing Hadoop RPC using SSL
> -
>
> Key: HADOOP-13836
> URL: https://issues.apache.org/jira/browse/HADOOP-13836
> Project: Hadoop Common
> Issue Type: New Feature
> Components: ipc
>Reporter: kartheek muthyala
>Assignee: kartheek muthyala
> Attachments: HADOOP-13836.patch, HADOOP-13836-v2.patch,
> HADOOP-13836-v3.patch, HADOOP-13836-v4.patch, Secure IPC OSS Proposal-1.pdf,
> SecureIPC Performance Analysis-OSS.pdf
>
>
> Today, RPC connections in Hadoop are encrypted using Simple Authentication &
> Security Layer (SASL), with the Kerberos ticket based authentication or
> Digest-md5 checksum based authentication protocols. This proposal is about
> enhancing this cipher suite with SSL/TLS based encryption and authentication.
> SSL/TLS is a proposed Internet Engineering Task Force (IETF) standard, that
> provides data security and integrity across two different end points in a
> network. This protocol has made its way to a number of applications such as
> web browsing, email, internet faxing, messaging, VOIP etc. And supporting
> this cipher suite at the core of Hadoop would give a good synergy with the
> applications on top and also bolster industry adoption of Hadoop.
> The Server and Client code in Hadoop IPC should support the following modes
> of communication
> 1.Plain
> 2. SASL encryption with an underlying authentication
> 3. SSL based encryption and authentication (x509 certificate)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13836) Securing Hadoop RPC using SSL
[ https://issues.apache.org/jira/browse/HADOOP-13836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16089906#comment-16089906 ] Antonios Kouzoupis commented on HADOOP-13836: - Hello [~kartheek], do you have any update on this JIRA? I suppose it's not top priority but we are particularly interested and we have implemented our own PoC for TLS encryption on the IPC/RPC layer. > Securing Hadoop RPC using SSL > - > > Key: HADOOP-13836 > URL: https://issues.apache.org/jira/browse/HADOOP-13836 > Project: Hadoop Common > Issue Type: New Feature > Components: ipc >Reporter: kartheek muthyala >Assignee: kartheek muthyala > Attachments: HADOOP-13836.patch, HADOOP-13836-v2.patch, > HADOOP-13836-v3.patch, HADOOP-13836-v4.patch, Secure IPC OSS Proposal-1.pdf, > SecureIPC Performance Analysis-OSS.pdf > > > Today, RPC connections in Hadoop are encrypted using Simple Authentication & > Security Layer (SASL), with the Kerberos ticket based authentication or > Digest-md5 checksum based authentication protocols. This proposal is about > enhancing this cipher suite with SSL/TLS based encryption and authentication. > SSL/TLS is a proposed Internet Engineering Task Force (IETF) standard, that > provides data security and integrity across two different end points in a > network. This protocol has made its way to a number of applications such as > web browsing, email, internet faxing, messaging, VOIP etc. And supporting > this cipher suite at the core of Hadoop would give a good synergy with the > applications on top and also bolster industry adoption of Hadoop. > The Server and Client code in Hadoop IPC should support the following modes > of communication > 1.Plain > 2. SASL encryption with an underlying authentication > 3. SSL based encryption and authentication (x509 certificate) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13836) Securing Hadoop RPC using SSL
[ https://issues.apache.org/jira/browse/HADOOP-13836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15923489#comment-15923489 ] kartheek muthyala commented on HADOOP-13836: [~daryn], Got busy with some other work. Will work on the raised concerns and post an update soon. Thanks > Securing Hadoop RPC using SSL > - > > Key: HADOOP-13836 > URL: https://issues.apache.org/jira/browse/HADOOP-13836 > Project: Hadoop Common > Issue Type: New Feature > Components: ipc >Reporter: kartheek muthyala >Assignee: kartheek muthyala > Attachments: HADOOP-13836.patch, HADOOP-13836-v2.patch, > HADOOP-13836-v3.patch, HADOOP-13836-v4.patch, Secure IPC OSS Proposal-1.pdf, > SecureIPC Performance Analysis-OSS.pdf > > > Today, RPC connections in Hadoop are encrypted using Simple Authentication & > Security Layer (SASL), with the Kerberos ticket based authentication or > Digest-md5 checksum based authentication protocols. This proposal is about > enhancing this cipher suite with SSL/TLS based encryption and authentication. > SSL/TLS is a proposed Internet Engineering Task Force (IETF) standard, that > provides data security and integrity across two different end points in a > network. This protocol has made its way to a number of applications such as > web browsing, email, internet faxing, messaging, VOIP etc. And supporting > this cipher suite at the core of Hadoop would give a good synergy with the > applications on top and also bolster industry adoption of Hadoop. > The Server and Client code in Hadoop IPC should support the following modes > of communication > 1.Plain > 2. SASL encryption with an underlying authentication > 3. SSL based encryption and authentication (x509 certificate) -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13836) Securing Hadoop RPC using SSL
[ https://issues.apache.org/jira/browse/HADOOP-13836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15899515#comment-15899515 ] Daryn Sharp commented on HADOOP-13836: -- I think the suggested course of action is good. Any luck with addressing the issues? > Securing Hadoop RPC using SSL > - > > Key: HADOOP-13836 > URL: https://issues.apache.org/jira/browse/HADOOP-13836 > Project: Hadoop Common > Issue Type: New Feature > Components: ipc >Reporter: kartheek muthyala >Assignee: kartheek muthyala > Attachments: HADOOP-13836.patch, HADOOP-13836-v2.patch, > HADOOP-13836-v3.patch, HADOOP-13836-v4.patch, Secure IPC OSS Proposal-1.pdf, > SecureIPC Performance Analysis-OSS.pdf > > > Today, RPC connections in Hadoop are encrypted using Simple Authentication & > Security Layer (SASL), with the Kerberos ticket based authentication or > Digest-md5 checksum based authentication protocols. This proposal is about > enhancing this cipher suite with SSL/TLS based encryption and authentication. > SSL/TLS is a proposed Internet Engineering Task Force (IETF) standard, that > provides data security and integrity across two different end points in a > network. This protocol has made its way to a number of applications such as > web browsing, email, internet faxing, messaging, VOIP etc. And supporting > this cipher suite at the core of Hadoop would give a good synergy with the > applications on top and also bolster industry adoption of Hadoop. > The Server and Client code in Hadoop IPC should support the following modes > of communication > 1.Plain > 2. SASL encryption with an underlying authentication > 3. SSL based encryption and authentication (x509 certificate) -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13836) Securing Hadoop RPC using SSL
[ https://issues.apache.org/jira/browse/HADOOP-13836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15867474#comment-15867474 ] kartheek muthyala commented on HADOOP-13836: Sorry for the delayed response. [~daryn], given that most of your concerns are related to the blocking implementation of readAndProcess, let me break this task to 2 items 1. Validate if we can implement the same functionality without having a separate readAndProcess method for SSL implementation. This will most probably push the buffered reader implementation to SSLServerSocketChannel layer. 2. Address the performance concerns of the implementation. For achieving task1, I will make changes in the existing patch to check if we can achieve it, without making major changes to readAndProcess. Task2 might require more iterations to identify the bugs, deadlocks, blocking readers etc. Given that at this point we have a clear separation of SSL implementation from default, we should be safe to commit it and work on the issues as they get identified. Please suggest. " kartheek muthyala, please use a profiler to check for a hot spot or highly contended sync point" - Sure, when I do the next level of testing, I will attach a profiler and see if there are any hot spots. > Securing Hadoop RPC using SSL > - > > Key: HADOOP-13836 > URL: https://issues.apache.org/jira/browse/HADOOP-13836 > Project: Hadoop Common > Issue Type: New Feature > Components: ipc >Reporter: kartheek muthyala >Assignee: kartheek muthyala > Attachments: HADOOP-13836.patch, HADOOP-13836-v2.patch, > HADOOP-13836-v3.patch, HADOOP-13836-v4.patch, Secure IPC OSS Proposal-1.pdf, > SecureIPC Performance Analysis-OSS.pdf > > > Today, RPC connections in Hadoop are encrypted using Simple Authentication & > Security Layer (SASL), with the Kerberos ticket based authentication or > Digest-md5 checksum based authentication protocols. This proposal is about > enhancing this cipher suite with SSL/TLS based encryption and authentication. > SSL/TLS is a proposed Internet Engineering Task Force (IETF) standard, that > provides data security and integrity across two different end points in a > network. This protocol has made its way to a number of applications such as > web browsing, email, internet faxing, messaging, VOIP etc. And supporting > this cipher suite at the core of Hadoop would give a good synergy with the > applications on top and also bolster industry adoption of Hadoop. > The Server and Client code in Hadoop IPC should support the following modes > of communication > 1.Plain > 2. SASL encryption with an underlying authentication > 3. SSL based encryption and authentication (x509 certificate) -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13836) Securing Hadoop RPC using SSL
[
https://issues.apache.org/jira/browse/HADOOP-13836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15856797#comment-15856797
]
Daryn Sharp commented on HADOOP-13836:
--
I understand the difficulties of handling ssl partial reads/writes, reads
wanting to write, vice versa. I'm interested in this feature but these issues
I outlined are blockers - no nio pun intended. :)
{quote}
bq. Multi-threaded clients generating requests faster than read will
indefinitely tie up a reader
I am not sure if it gets indefinitely tied up, but they will get processed
eventually.
{quote}
Yes, maybe, probably, but it's classic indefinite postponement which is not
acceptable.
{quote}
bq. Clients sending a slow trickle of bytes will tie up a reader until a
request is fully read.
This is a problem that exists still today, when large data packets are sent and
we use ChannelIO on the server to process this.
{quote}
Incorrect. ChannelIO does loop using a nio optimal buffer size, but will
read/write at most 1 call or until the non-blocking op returns less than a full
buffer.
{quote}
bq. Clients stalled mid-request will cause the reader to go into a spin loop.
The connection timeout on the stalled clients, would lead to closure of channel
and the spin loop breaks
{quote}
There's no acceptable justification for a spin loop...
bq. Note that SSL over the current protocol is not wire-compatible anyway, I
would argue that it might make sense to build a new protocol on top of HTTP/2
and to leverage great implementation available today (e.g., Netty 4.1 / gRPC).
[~wheat9] Given that EZ has lower performance impact, I do agree something is
very amiss. [~kartheek], please use a profiler to check for a hot spot or
highly contended sync point. It may be correlated with increased object
allocation/copying levels causing an increase in young gen gc frequency.
Unfortunately I have not seen good benchmarks for java gRPC. Given the
atrocious garbage generation rates of PB and guava, I have low confidence gRPC
would be performant. Webhdfs is the poster child for the horrors of a java
REST protocol at scale. Even after all my attempts to tame webhdfs, even when
capped with iptables to 5-10k connections max, a flood of perhaps ~10k ops/sec
will blow up the heap and cause a full gc or come dangerously close. For
comparison, we can now handle storms of rpc call rates exceeding 100k/sec.
> Securing Hadoop RPC using SSL
> -
>
> Key: HADOOP-13836
> URL: https://issues.apache.org/jira/browse/HADOOP-13836
> Project: Hadoop Common
> Issue Type: New Feature
> Components: ipc
>Reporter: kartheek muthyala
>Assignee: kartheek muthyala
> Attachments: HADOOP-13836.patch, HADOOP-13836-v2.patch,
> HADOOP-13836-v3.patch, HADOOP-13836-v4.patch, Secure IPC OSS Proposal-1.pdf,
> SecureIPC Performance Analysis-OSS.pdf
>
>
> Today, RPC connections in Hadoop are encrypted using Simple Authentication &
> Security Layer (SASL), with the Kerberos ticket based authentication or
> Digest-md5 checksum based authentication protocols. This proposal is about
> enhancing this cipher suite with SSL/TLS based encryption and authentication.
> SSL/TLS is a proposed Internet Engineering Task Force (IETF) standard, that
> provides data security and integrity across two different end points in a
> network. This protocol has made its way to a number of applications such as
> web browsing, email, internet faxing, messaging, VOIP etc. And supporting
> this cipher suite at the core of Hadoop would give a good synergy with the
> applications on top and also bolster industry adoption of Hadoop.
> The Server and Client code in Hadoop IPC should support the following modes
> of communication
> 1.Plain
> 2. SASL encryption with an underlying authentication
> 3. SSL based encryption and authentication (x509 certificate)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13836) Securing Hadoop RPC using SSL
[ https://issues.apache.org/jira/browse/HADOOP-13836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15855409#comment-15855409 ] kartheek muthyala commented on HADOOP-13836: [~daryn], Thank you for the insightful feedback. :) When SSL encrypts the databuffers, the length of the data packets differ from the actual data sent. For example, if we have a 10 byte data packet, after encryption - the data packet can grow up to 16 byte depending on the algorithm used for encryption. So, when a hadoop RPC is sent on a channel, we read the data length to get to know the data to be read in advance. So, in the current readAndProcess, when we replace the socket channel with SSLServerSocketChannel, the channelRead might read partial data, which might not be able to sense the data length or data. For example, when we call SSLSocketChannel.read() might yield only 3 bytes, even though it has read 8 bytes on the channel. These 3 bytes won't be able to decode the data length, because today we use 4 bytes to understand the data length. So this nature of varying datalength on the channel, made me to modify the readAndProcess to continuously loop until we have enough data. This can probably be simplified by having another class which extends SSLServerSocketChannel and buffers at a layer under readAndProcess. That might avoid the extra readAndProcess. I will create an improvement on top of this jira to verify if that abstraction is possible. But even with this extra interface, we still have to loop for the data because of the same data length issues. Multi-threaded clients generating requests faster than read will indefinitely tie up a reader - I am not sure if it gets indefinitely tied up, but they will get processed eventually. Clients sending a slow trickle of bytes will tie up a reader until a request is fully read. - This is a problem that exists still today, when large data packets are sent and we use ChannelIO on the server to process this. Clients stalled mid-request will cause the reader to go into a spin loop. - The connection timeout on the stalled clients, would lead to closure of channel and the spin loop breaks. [~wheat9], The performance study quoted in the link occurs on a setup where clients are interfacing with frontend machines which support HTTPS. They pointed out that "On our production frontend machines, SSL/TLS accounts for less than 1% of the CPU load, less than 10KB of memory per connection and less than 2% of network overhead.", so it is an overall 3% overall for them too including network overhead due to handshaking. I am not sure if this is an Apple to Apple comparison with the setup on which I have taken performance numbers. The CPU processing speed in decoding and encoding, SSL protocol used, network bandwidth between the machines and workload characteristics etc.. might have varied in both the setups. > Securing Hadoop RPC using SSL > - > > Key: HADOOP-13836 > URL: https://issues.apache.org/jira/browse/HADOOP-13836 > Project: Hadoop Common > Issue Type: New Feature > Components: ipc >Reporter: kartheek muthyala >Assignee: kartheek muthyala > Attachments: HADOOP-13836.patch, HADOOP-13836-v2.patch, > HADOOP-13836-v3.patch, HADOOP-13836-v4.patch, Secure IPC OSS Proposal-1.pdf, > SecureIPC Performance Analysis-OSS.pdf > > > Today, RPC connections in Hadoop are encrypted using Simple Authentication & > Security Layer (SASL), with the Kerberos ticket based authentication or > Digest-md5 checksum based authentication protocols. This proposal is about > enhancing this cipher suite with SSL/TLS based encryption and authentication. > SSL/TLS is a proposed Internet Engineering Task Force (IETF) standard, that > provides data security and integrity across two different end points in a > network. This protocol has made its way to a number of applications such as > web browsing, email, internet faxing, messaging, VOIP etc. And supporting > this cipher suite at the core of Hadoop would give a good synergy with the > applications on top and also bolster industry adoption of Hadoop. > The Server and Client code in Hadoop IPC should support the following modes > of communication > 1.Plain > 2. SASL encryption with an underlying authentication > 3. SSL based encryption and authentication (x509 certificate) -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13836) Securing Hadoop RPC using SSL
[ https://issues.apache.org/jira/browse/HADOOP-13836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15836483#comment-15836483 ] Daryn Sharp commented on HADOOP-13836: -- SSL is notoriously hard for non-blocking io. The added org.baswerc.niossl library appears dead and unsupported. V0.2 was posted 1.5y ago. There are a handful of open bugs about memory leaks, selectors not being reliable, etc. No responses to the bugs. Not encouraging. Regarding the Connection subclasses. The readAndProcess method is already a bit dicey. It embodies authentication handshake and general rpc message reading and queuing. I'm hesitant of two different impls because it’s likely lead to unintended divergence, as illustrated below, but also increasing the chance of security holes. Ideally the ssl channel impl should be transparent and not require changes to readAndProcess. Verifying correctness of partial reads is a bit difficult. The position within the byte[] appBufBytes, which is extracted from ByteBuffer appBuf, is being tracked via appBuf’s position and repeatedly updated. The extraction seems unnecessary and explicitly position update seem unnecessary. It’d be easier to follow if applicationBufferRead() took a source/dest byte buffer, copied up to dest’s remaining from the source, updated the position. That said… The ssl readAndProcess behavior isn’t equivalent to the current NIO behavior: read only what’s available, process request when fully read. If the ssl version encounters a partial payload, it loops until at least the full payload is read into appBuf. If appBuf isn’t fully consumed it loops again. This causes problems that NIO is avoiding: * Multi-threaded clients generating requests faster than read will indefinitely tie up a reader. * Clients sending a slow trickle of bytes will tie up a reader until a request is fully read. * Clients stalled mid-request will cause the reader to go into a spin loop. When the reader loops on a connection, the reader’s other established connections are starved. The reader also isn’t consuming new connections queued by the listener. Eventually the listener will block and stop accepting. The result is the ipc layer going into a series of seizures that severely degrade performance. This may be partly responsible for the performance degradation. Another issue is the all or nothing requirement for enabling ssl. I’d be potentially interested in using ssl if I could configure which hosts require ssl, ie. intra-colo. Sasl qop impl allows the server to selectively control if clients are forced to encrypt. The same would be nice for ssl. Lastly, the 11-14% performance is unacceptable for production use. I would have expected terasort’s heavy cpu usage to eclipse the ssl rpc load. It didn’t, which implies that less cpu-intensive jobs will have a more pronounced hit? > Securing Hadoop RPC using SSL > - > > Key: HADOOP-13836 > URL: https://issues.apache.org/jira/browse/HADOOP-13836 > Project: Hadoop Common > Issue Type: New Feature > Components: ipc >Reporter: kartheek muthyala >Assignee: kartheek muthyala > Attachments: HADOOP-13836.patch, HADOOP-13836-v2.patch, > HADOOP-13836-v3.patch, HADOOP-13836-v4.patch, Secure IPC OSS Proposal-1.pdf, > SecureIPC Performance Analysis-OSS.pdf > > > Today, RPC connections in Hadoop are encrypted using Simple Authentication & > Security Layer (SASL), with the Kerberos ticket based authentication or > Digest-md5 checksum based authentication protocols. This proposal is about > enhancing this cipher suite with SSL/TLS based encryption and authentication. > SSL/TLS is a proposed Internet Engineering Task Force (IETF) standard, that > provides data security and integrity across two different end points in a > network. This protocol has made its way to a number of applications such as > web browsing, email, internet faxing, messaging, VOIP etc. And supporting > this cipher suite at the core of Hadoop would give a good synergy with the > applications on top and also bolster industry adoption of Hadoop. > The Server and Client code in Hadoop IPC should support the following modes > of communication > 1.Plain > 2. SASL encryption with an underlying authentication > 3. SSL based encryption and authentication (x509 certificate) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13836) Securing Hadoop RPC using SSL
[ https://issues.apache.org/jira/browse/HADOOP-13836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15834797#comment-15834797 ] Daryn Sharp commented on HADOOP-13836: -- I'll try to review/comment in the next few days. The cited performance hit is rather concerning though. > Securing Hadoop RPC using SSL > - > > Key: HADOOP-13836 > URL: https://issues.apache.org/jira/browse/HADOOP-13836 > Project: Hadoop Common > Issue Type: New Feature > Components: ipc >Reporter: kartheek muthyala >Assignee: kartheek muthyala > Attachments: HADOOP-13836.patch, HADOOP-13836-v2.patch, > HADOOP-13836-v3.patch, Secure IPC OSS Proposal-1.pdf, SecureIPC Performance > Analysis-OSS.pdf > > > Today, RPC connections in Hadoop are encrypted using Simple Authentication & > Security Layer (SASL), with the Kerberos ticket based authentication or > Digest-md5 checksum based authentication protocols. This proposal is about > enhancing this cipher suite with SSL/TLS based encryption and authentication. > SSL/TLS is a proposed Internet Engineering Task Force (IETF) standard, that > provides data security and integrity across two different end points in a > network. This protocol has made its way to a number of applications such as > web browsing, email, internet faxing, messaging, VOIP etc. And supporting > this cipher suite at the core of Hadoop would give a good synergy with the > applications on top and also bolster industry adoption of Hadoop. > The Server and Client code in Hadoop IPC should support the following modes > of communication > 1.Plain > 2. SASL encryption with an underlying authentication > 3. SSL based encryption and authentication (x509 certificate) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13836) Securing Hadoop RPC using SSL
[ https://issues.apache.org/jira/browse/HADOOP-13836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15834427#comment-15834427 ] Antonios Kouzoupis commented on HADOOP-13836: - [~kartheek] you can use org.apache.hadoop.security.ssl.KeyStoreTestUtils to create all the necessary cryptographic material before running your JUnit tests instead of shipping binaries. > Securing Hadoop RPC using SSL > - > > Key: HADOOP-13836 > URL: https://issues.apache.org/jira/browse/HADOOP-13836 > Project: Hadoop Common > Issue Type: New Feature > Components: ipc >Reporter: kartheek muthyala >Assignee: kartheek muthyala > Attachments: HADOOP-13836.patch, HADOOP-13836-v2.patch, > HADOOP-13836-v3.patch, Secure IPC OSS Proposal-1.pdf, SecureIPC Performance > Analysis-OSS.pdf > > > Today, RPC connections in Hadoop are encrypted using Simple Authentication & > Security Layer (SASL), with the Kerberos ticket based authentication or > Digest-md5 checksum based authentication protocols. This proposal is about > enhancing this cipher suite with SSL/TLS based encryption and authentication. > SSL/TLS is a proposed Internet Engineering Task Force (IETF) standard, that > provides data security and integrity across two different end points in a > network. This protocol has made its way to a number of applications such as > web browsing, email, internet faxing, messaging, VOIP etc. And supporting > this cipher suite at the core of Hadoop would give a good synergy with the > applications on top and also bolster industry adoption of Hadoop. > The Server and Client code in Hadoop IPC should support the following modes > of communication > 1.Plain > 2. SASL encryption with an underlying authentication > 3. SSL based encryption and authentication (x509 certificate) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13836) Securing Hadoop RPC using SSL
[
https://issues.apache.org/jira/browse/HADOOP-13836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15834089#comment-15834089
]
Hadoop QA commented on HADOOP-13836:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 20m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 17m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
29s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m
22s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m
44s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 40s{color} | {color:orange} root: The patch generated 46 new + 429 unchanged
- 16 fixed = 475 total (was 445) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 8m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 19 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m
29s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 16m 2s{color}
| {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
37s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}144m 40s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ipc.TestSSLIPC |
| | hadoop.ipc.TestSSLSocketFactory |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HADOOP-13836 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12848837/HADOOP-13836-v3.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit xml findbugs checkstyle |
| uname | Linux b1b951ed5040 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / a847903 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| checkstyle |
[jira] [Commented] (HADOOP-13836) Securing Hadoop RPC using SSL
[
https://issues.apache.org/jira/browse/HADOOP-13836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15833663#comment-15833663
]
Hadoop QA commented on HADOOP-13836:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
56s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 10m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
11s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m
10s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
16s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 40s{color} | {color:orange} root: The patch generated 46 new + 402 unchanged
- 16 fixed = 448 total (was 418) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 9m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 19 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m
26s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 16m 8s{color}
| {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
37s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}133m 25s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ipc.TestSSLIPC |
| | hadoop.ipc.TestRPC |
| | hadoop.ipc.TestIPC |
| | hadoop.ipc.TestSSLSocketFactory |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HADOOP-13836 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12848784/HADOOP-13836-v2.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit xml findbugs checkstyle |
| uname | Linux badc885ece4d 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / a847903 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| checkstyle |
[jira] [Commented] (HADOOP-13836) Securing Hadoop RPC using SSL
[ https://issues.apache.org/jira/browse/HADOOP-13836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15833623#comment-15833623 ] kartheek muthyala commented on HADOOP-13836: Thank you @Kai Zheng and @Steve Loughran for the initial feedback on the performance numbers. My focus was to find out the impact of SSL implementation with respect to Plain socket implementation, I restrained from configuring the cluster for SASL. If I get some time, I will try to get the cluster configured for SASL and run the same test suite again. [~antkou], Thank you for the feedback on the initial patch. I have included the changes suggested by you in version 2. [~asuresh], [~daryn] [~sanjay.radia] [~owen.omalley], can you guys please review this patch. > Securing Hadoop RPC using SSL > - > > Key: HADOOP-13836 > URL: https://issues.apache.org/jira/browse/HADOOP-13836 > Project: Hadoop Common > Issue Type: New Feature > Components: ipc >Reporter: kartheek muthyala >Assignee: kartheek muthyala > Attachments: HADOOP-13836.patch, HADOOP-13836-v2.patch, Secure IPC > OSS Proposal-1.pdf, SecureIPC Performance Analysis-OSS.pdf > > > Today, RPC connections in Hadoop are encrypted using Simple Authentication & > Security Layer (SASL), with the Kerberos ticket based authentication or > Digest-md5 checksum based authentication protocols. This proposal is about > enhancing this cipher suite with SSL/TLS based encryption and authentication. > SSL/TLS is a proposed Internet Engineering Task Force (IETF) standard, that > provides data security and integrity across two different end points in a > network. This protocol has made its way to a number of applications such as > web browsing, email, internet faxing, messaging, VOIP etc. And supporting > this cipher suite at the core of Hadoop would give a good synergy with the > applications on top and also bolster industry adoption of Hadoop. > The Server and Client code in Hadoop IPC should support the following modes > of communication > 1.Plain > 2. SASL encryption with an underlying authentication > 3. SSL based encryption and authentication (x509 certificate) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13836) Securing Hadoop RPC using SSL
[ https://issues.apache.org/jira/browse/HADOOP-13836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15827893#comment-15827893 ] Steve Loughran commented on HADOOP-13836: - I don't personally know whether that 12% would go down well or not; I presume it'd be up to the user. I do know SASL symmetric encryption has a performance hit all of its own, which is why it is underused. It might be interesting to do a test run with SASL=encrypt to see what the numbers show up there —maybe they are even worse than the SSL values. I'm not in a position to review the RPC code itself, as I'm scared of it. It's one of those critical-part pieces of code which everyone treads with caution around. I think you'll need [~daryn] [~sanjay.radia] and [~owen.omalley] to look at it there. That said, the UML doc you've added does imply that this work does include some cleanup of today's codebase ... this can only be welcome > Securing Hadoop RPC using SSL > - > > Key: HADOOP-13836 > URL: https://issues.apache.org/jira/browse/HADOOP-13836 > Project: Hadoop Common > Issue Type: New Feature > Components: ipc >Reporter: kartheek muthyala >Assignee: kartheek muthyala > Attachments: HADOOP-13836.patch, Secure IPC OSS Proposal-1.pdf, > SecureIPC Performance Analysis-OSS.pdf > > > Today, RPC connections in Hadoop are encrypted using Simple Authentication & > Security Layer (SASL), with the Kerberos ticket based authentication or > Digest-md5 checksum based authentication protocols. This proposal is about > enhancing this cipher suite with SSL/TLS based encryption and authentication. > SSL/TLS is a proposed Internet Engineering Task Force (IETF) standard, that > provides data security and integrity across two different end points in a > network. This protocol has made its way to a number of applications such as > web browsing, email, internet faxing, messaging, VOIP etc. And supporting > this cipher suite at the core of Hadoop would give a good synergy with the > applications on top and also bolster industry adoption of Hadoop. > The Server and Client code in Hadoop IPC should support the following modes > of communication > 1.Plain > 2. SASL encryption with an underlying authentication > 3. SSL based encryption and authentication (x509 certificate) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13836) Securing Hadoop RPC using SSL
[ https://issues.apache.org/jira/browse/HADOOP-13836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15827676#comment-15827676 ] Kai Zheng commented on HADOOP-13836: Thanks for the benchmark report. The ~12% overhead incurred by SSL in the Terasort workload looks good. I'm not sure how it would mean for IPC heavy servers like NameNode, though, particularly considering NN might be already a hot spot. The design doc looks very nice and I will go thru it later. > Securing Hadoop RPC using SSL > - > > Key: HADOOP-13836 > URL: https://issues.apache.org/jira/browse/HADOOP-13836 > Project: Hadoop Common > Issue Type: New Feature > Components: ipc >Reporter: kartheek muthyala >Assignee: kartheek muthyala > Attachments: HADOOP-13836.patch, Secure IPC OSS Proposal-1.pdf, > SecureIPC Performance Analysis-OSS.pdf > > > Today, RPC connections in Hadoop are encrypted using Simple Authentication & > Security Layer (SASL), with the Kerberos ticket based authentication or > Digest-md5 checksum based authentication protocols. This proposal is about > enhancing this cipher suite with SSL/TLS based encryption and authentication. > SSL/TLS is a proposed Internet Engineering Task Force (IETF) standard, that > provides data security and integrity across two different end points in a > network. This protocol has made its way to a number of applications such as > web browsing, email, internet faxing, messaging, VOIP etc. And supporting > this cipher suite at the core of Hadoop would give a good synergy with the > applications on top and also bolster industry adoption of Hadoop. > The Server and Client code in Hadoop IPC should support the following modes > of communication > 1.Plain > 2. SASL encryption with an underlying authentication > 3. SSL based encryption and authentication (x509 certificate) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13836) Securing Hadoop RPC using SSL
[ https://issues.apache.org/jira/browse/HADOOP-13836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15827664#comment-15827664 ] Antonios Kouzoupis commented on HADOOP-13836: - One comment regarding the proposed design that I have also pointed out before is to use the "hadoop.rpc.socket.factory.class.default" property to specify the socket factory, in that case it would create a secure socket. One side-effect that we encountered with this was that secure sockets were been created in places that, for the moment, we don't want to. A solution to that is to use NetUtilis#getSocketFactory along with a configuration property instead of NetUtils#getDefaultSocketFactory > Securing Hadoop RPC using SSL > - > > Key: HADOOP-13836 > URL: https://issues.apache.org/jira/browse/HADOOP-13836 > Project: Hadoop Common > Issue Type: New Feature > Components: ipc >Reporter: kartheek muthyala >Assignee: kartheek muthyala > Attachments: HADOOP-13836.patch, Secure IPC OSS Proposal-1.pdf, > SecureIPC Performance Analysis-OSS.pdf > > > Today, RPC connections in Hadoop are encrypted using Simple Authentication & > Security Layer (SASL), with the Kerberos ticket based authentication or > Digest-md5 checksum based authentication protocols. This proposal is about > enhancing this cipher suite with SSL/TLS based encryption and authentication. > SSL/TLS is a proposed Internet Engineering Task Force (IETF) standard, that > provides data security and integrity across two different end points in a > network. This protocol has made its way to a number of applications such as > web browsing, email, internet faxing, messaging, VOIP etc. And supporting > this cipher suite at the core of Hadoop would give a good synergy with the > applications on top and also bolster industry adoption of Hadoop. > The Server and Client code in Hadoop IPC should support the following modes > of communication > 1.Plain > 2. SASL encryption with an underlying authentication > 3. SSL based encryption and authentication (x509 certificate) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13836) Securing Hadoop RPC using SSL
[ https://issues.apache.org/jira/browse/HADOOP-13836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15826330#comment-15826330 ] Arun Suresh commented on HADOOP-13836: -- [~drankye], [~daryn], [~steve_l], wondering if you folks had a chance to look at the attached design doc and perf numbers. If we are ok with the design, I can start with reviewing the patch. > Securing Hadoop RPC using SSL > - > > Key: HADOOP-13836 > URL: https://issues.apache.org/jira/browse/HADOOP-13836 > Project: Hadoop Common > Issue Type: New Feature > Components: ipc >Reporter: kartheek muthyala >Assignee: kartheek muthyala > Attachments: HADOOP-13836.patch, Secure IPC OSS Proposal-1.pdf, > SecureIPC Performance Analysis-OSS.pdf > > > Today, RPC connections in Hadoop are encrypted using Simple Authentication & > Security Layer (SASL), with the Kerberos ticket based authentication or > Digest-md5 checksum based authentication protocols. This proposal is about > enhancing this cipher suite with SSL/TLS based encryption and authentication. > SSL/TLS is a proposed Internet Engineering Task Force (IETF) standard, that > provides data security and integrity across two different end points in a > network. This protocol has made its way to a number of applications such as > web browsing, email, internet faxing, messaging, VOIP etc. And supporting > this cipher suite at the core of Hadoop would give a good synergy with the > applications on top and also bolster industry adoption of Hadoop. > The Server and Client code in Hadoop IPC should support the following modes > of communication > 1.Plain > 2. SASL encryption with an underlying authentication > 3. SSL based encryption and authentication (x509 certificate) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13836) Securing Hadoop RPC using SSL
[ https://issues.apache.org/jira/browse/HADOOP-13836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15706026#comment-15706026 ] Arun Suresh commented on HADOOP-13836: -- Thanks for pointing out IPv6 [~steve_l] I assume it should just work, considering this comes into play only at Socket creation. My understanding is that SSL/TLS certification authentication is based on the DNS names of the entities involved, so again, I assume it should just work, if DNS resolution works correctly. But yes, we will try to verify it. My understanding is that the HADOOP-11890 branch had most of the IPv6 based changes. If this branch is uptodate, [~kartheek], we can maybe test it against that branch. Any more pointers to verifying if this would work with IPv6 is welcome. (cc: [~eclark], [~nkedel]) > Securing Hadoop RPC using SSL > - > > Key: HADOOP-13836 > URL: https://issues.apache.org/jira/browse/HADOOP-13836 > Project: Hadoop Common > Issue Type: New Feature > Components: ipc >Reporter: kartheek muthyala >Assignee: kartheek muthyala > Attachments: HADOOP-13836.patch > > > Today, RPC connections in Hadoop are encrypted using Simple Authentication & > Security Layer (SASL), with the Kerberos ticket based authentication or > Digest-md5 checksum based authentication protocols. This proposal is about > enhancing this cipher suite with SSL/TLS based encryption and authentication. > SSL/TLS is a proposed Internet Engineering Task Force (IETF) standard, that > provides data security and integrity across two different end points in a > network. This protocol has made its way to a number of applications such as > web browsing, email, internet faxing, messaging, VOIP etc. And supporting > this cipher suite at the core of Hadoop would give a good synergy with the > applications on top and also bolster industry adoption of Hadoop. > The Server and Client code in Hadoop IPC should support the following modes > of communication > 1.Plain > 2. SASL encryption with an underlying authentication > 3. SSL based encryption and authentication (x509 certificate) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13836) Securing Hadoop RPC using SSL
[ https://issues.apache.org/jira/browse/HADOOP-13836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15705730#comment-15705730 ] Steve Loughran commented on HADOOP-13836: - This is all going to work once the IPV6 patch is merged in, right? > Securing Hadoop RPC using SSL > - > > Key: HADOOP-13836 > URL: https://issues.apache.org/jira/browse/HADOOP-13836 > Project: Hadoop Common > Issue Type: New Feature > Components: ipc >Reporter: kartheek muthyala >Assignee: kartheek muthyala > Attachments: HADOOP-13836.patch > > > Today, RPC connections in Hadoop are encrypted using Simple Authentication & > Security Layer (SASL), with the Kerberos ticket based authentication or > Digest-md5 checksum based authentication protocols. This proposal is about > enhancing this cipher suite with SSL/TLS based encryption and authentication. > SSL/TLS is a proposed Internet Engineering Task Force (IETF) standard, that > provides data security and integrity across two different end points in a > network. This protocol has made its way to a number of applications such as > web browsing, email, internet faxing, messaging, VOIP etc. And supporting > this cipher suite at the core of Hadoop would give a good synergy with the > applications on top and also bolster industry adoption of Hadoop. > The Server and Client code in Hadoop IPC should support the following modes > of communication > 1.Plain > 2. SASL encryption with an underlying authentication > 3. SSL based encryption and authentication (x509 certificate) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13836) Securing Hadoop RPC using SSL
[ https://issues.apache.org/jira/browse/HADOOP-13836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15705693#comment-15705693 ] kartheek muthyala commented on HADOOP-13836: [~antkou], I liked your suggestion of using "hadoop.rpc.socket.factory.class.default" to specify our own SSLSocketFactory class on the Client side. For this I just have to move the existing SSL socket creation related changes from Client.java to a separate file. Will consider this in my next patch. We have considered org.apache.hadoop.security.ssl.SSLFactory for creating SSLEngine on Server. But it doesn't has SSLSocketChannel on top of SSLEngine, like niossl library does. Having this support would make us stick to the existing SocketChannel Server design. > Securing Hadoop RPC using SSL > - > > Key: HADOOP-13836 > URL: https://issues.apache.org/jira/browse/HADOOP-13836 > Project: Hadoop Common > Issue Type: New Feature > Components: ipc >Reporter: kartheek muthyala >Assignee: kartheek muthyala > Attachments: HADOOP-13836.patch > > > Today, RPC connections in Hadoop are encrypted using Simple Authentication & > Security Layer (SASL), with the Kerberos ticket based authentication or > Digest-md5 checksum based authentication protocols. This proposal is about > enhancing this cipher suite with SSL/TLS based encryption and authentication. > SSL/TLS is a proposed Internet Engineering Task Force (IETF) standard, that > provides data security and integrity across two different end points in a > network. This protocol has made its way to a number of applications such as > web browsing, email, internet faxing, messaging, VOIP etc. And supporting > this cipher suite at the core of Hadoop would give a good synergy with the > applications on top and also bolster industry adoption of Hadoop. > The Server and Client code in Hadoop IPC should support the following modes > of communication > 1.Plain > 2. SASL encryption with an underlying authentication > 3. SSL based encryption and authentication (x509 certificate) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13836) Securing Hadoop RPC using SSL
[ https://issues.apache.org/jira/browse/HADOOP-13836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15705614#comment-15705614 ] kartheek muthyala commented on HADOOP-13836: [~drankye] , very good questions. Here are some of my responses. Correct me if I am wrong. I will soon post a design doc with all these details What's the scenarios, requirements and use cases you have in mind for this support (other than Kerberos)? - Avoiding man in the middle attacks, through proper SSL Connection handshake before even exchanging the data. - Better encryption over the wire. - Ability to get trusted third party validation through Versign, goDaddy etc, which improves industrial adoption for sensitive data exchange. - We can extend the same cipher suite to encrypt data on flight and rest. What interfaces will be taken care of by this: RPC/commands, REST, web, JDBC and etc. - For now we are supporting interfaces that derive from the hadoop.ipc.Server and hadoop.ipc.Client classes. So, primarily RPC. How authentication will be considered? Still simple or some mechanisms over SSL/TLS? - Today we enabled client to authenticate with the servers on connection. So configuring keystore is a must on the server. And with a configured KeyManager we can decide on what authentication credentials should be sent to the remote host for authentication during SSL handshake. How would you manage credentials (X.509 certificates) for Hadoop services and maybe clients? - The current work requires both server and client to be installed with keystore and truststore, and configured through ssl-client.xml and ssl-server.xml. What's the exact SSL/TLS versions to support and how to configure such with the cipher suite options? - Currently we are supporting TLSv1.2 as a default. Because the cipher suite hasn't changed between TLSv1 and TLSv1.2, TLSv1 also should be supported. Given that the SSLContext varies for different versions of SSL/TLS, we can provide an interface for deriving this SSLContext depending upon the version of TLS/SSL configured. > Securing Hadoop RPC using SSL > - > > Key: HADOOP-13836 > URL: https://issues.apache.org/jira/browse/HADOOP-13836 > Project: Hadoop Common > Issue Type: New Feature > Components: ipc >Reporter: kartheek muthyala >Assignee: kartheek muthyala > Attachments: HADOOP-13836.patch > > > Today, RPC connections in Hadoop are encrypted using Simple Authentication & > Security Layer (SASL), with the Kerberos ticket based authentication or > Digest-md5 checksum based authentication protocols. This proposal is about > enhancing this cipher suite with SSL/TLS based encryption and authentication. > SSL/TLS is a proposed Internet Engineering Task Force (IETF) standard, that > provides data security and integrity across two different end points in a > network. This protocol has made its way to a number of applications such as > web browsing, email, internet faxing, messaging, VOIP etc. And supporting > this cipher suite at the core of Hadoop would give a good synergy with the > applications on top and also bolster industry adoption of Hadoop. > The Server and Client code in Hadoop IPC should support the following modes > of communication > 1.Plain > 2. SASL encryption with an underlying authentication > 3. SSL based encryption and authentication (x509 certificate) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13836) Securing Hadoop RPC using SSL
[ https://issues.apache.org/jira/browse/HADOOP-13836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15704241#comment-15704241 ] Kai Zheng commented on HADOOP-13836: It's good to see this. Some quick questions for now: * What's the scenarios, requirements and use cases you have in mind for this support (other than Kerberos)? * What interfaces will be taken care of by this: RPC/commands, REST, web, JDBC and etc. * How authentication will be considered? Still simple or some mechanisms over SSL/TLS? * How would you manage credentials (X.509 certificates) for Hadoop services and maybe clients? * What's the exact SSL/TLS versions to support and how to configure such with the cipher suite options? We may need a design doc to document these. Thanks. > Securing Hadoop RPC using SSL > - > > Key: HADOOP-13836 > URL: https://issues.apache.org/jira/browse/HADOOP-13836 > Project: Hadoop Common > Issue Type: New Feature > Components: ipc >Reporter: kartheek muthyala > Attachments: HADOOP-13836.patch > > > Today, RPC connections in Hadoop are encrypted using Simple Authentication & > Security Layer (SASL), with the Kerberos ticket based authentication or > Digest-md5 checksum based authentication protocols. This proposal is about > enhancing this cipher suite with SSL/TLS based encryption and authentication. > SSL/TLS is a proposed Internet Engineering Task Force (IETF) standard, that > provides data security and integrity across two different end points in a > network. This protocol has made its way to a number of applications such as > web browsing, email, internet faxing, messaging, VOIP etc. And supporting > this cipher suite at the core of Hadoop would give a good synergy with the > applications on top and also bolster industry adoption of Hadoop. > The Server and Client code in Hadoop IPC should support the following modes > of communication > 1.Plain > 2. SASL encryption with an underlying authentication > 3. SSL based encryption and authentication (x509 certificate) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13836) Securing Hadoop RPC using SSL
[ https://issues.apache.org/jira/browse/HADOOP-13836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15702593#comment-15702593 ] Antonios Kouzoupis commented on HADOOP-13836: - [~kartheek] I took a quick look on your patch. I think it's more reasonable to use the "hadoop.rpc.socket.factory.class.default" configuration key to load the desired socket factory. At the moment the StandardSocketFactory it's been used but you may provide your own factory with ssl/tls support. > Securing Hadoop RPC using SSL > - > > Key: HADOOP-13836 > URL: https://issues.apache.org/jira/browse/HADOOP-13836 > Project: Hadoop Common > Issue Type: New Feature > Components: ipc >Reporter: kartheek muthyala > Attachments: HADOOP-13836.patch > > > Today, RPC connections in Hadoop are encrypted using Simple Authentication & > Security Layer (SASL), with the Kerberos ticket based authentication or > Digest-md5 checksum based authentication protocols. This proposal is about > enhancing this cipher suite with SSL/TLS based encryption and authentication. > SSL/TLS is a proposed Internet Engineering Task Force (IETF) standard, that > provides data security and integrity across two different end points in a > network. This protocol has made its way to a number of applications such as > web browsing, email, internet faxing, messaging, VOIP etc. And supporting > this cipher suite at the core of Hadoop would give a good synergy with the > applications on top and also bolster industry adoption of Hadoop. > The Server and Client code in Hadoop IPC should support the following modes > of communication > 1.Plain > 2. SASL encryption with an underlying authentication > 3. SSL based encryption and authentication (x509 certificate) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13836) Securing Hadoop RPC using SSL
[ https://issues.apache.org/jira/browse/HADOOP-13836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15702498#comment-15702498 ] kartheek muthyala commented on HADOOP-13836: Yes,[~asuresh], that is exactly what we are doing here. The proposal intends to implement an SSL layer on top of existing Hadoop RPC. It introduces SSLEngine in Server to encode and decode messages, and Java's javax.net.ssl library to encode and decode on the Client side. We have relied on niossl library for the server side implementation of SSLEngine. Because, this implementation sits on top of SSLSocket channel implementation, we can still keep the channels open as before, and just encode and decode messages using the existing cipher keys. But, as [~ste...@apache.org]pointed out, this introduces an overhead of additional handshakes between Server and Client for different reasons like certificate exchange, validation etc. We can trade off this performance hit with the security that we will be enhancing. This will improve the usage of secure IPC in large systems. We have been running this patch internally with some long running jobs and the performance seems to be decent. I don't have the exact numbers right away, but I will post them soon. > Securing Hadoop RPC using SSL > - > > Key: HADOOP-13836 > URL: https://issues.apache.org/jira/browse/HADOOP-13836 > Project: Hadoop Common > Issue Type: New Feature > Components: ipc >Reporter: kartheek muthyala > Attachments: HADOOP-13836.patch > > > Today, RPC connections in Hadoop are encrypted using Simple Authentication & > Security Layer (SASL), with the Kerberos ticket based authentication or > Digest-md5 checksum based authentication protocols. This proposal is about > enhancing this cipher suite with SSL/TLS based encryption and authentication. > SSL/TLS is a proposed Internet Engineering Task Force (IETF) standard, that > provides data security and integrity across two different end points in a > network. This protocol has made its way to a number of applications such as > web browsing, email, internet faxing, messaging, VOIP etc. And supporting > this cipher suite at the core of Hadoop would give a good synergy with the > applications on top and also bolster industry adoption of Hadoop. > The Server and Client code in Hadoop IPC should support the following modes > of communication > 1.Plain > 2. SASL encryption with an underlying authentication > 3. SSL based encryption and authentication (x509 certificate) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13836) Securing Hadoop RPC using SSL
[ https://issues.apache.org/jira/browse/HADOOP-13836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15702466#comment-15702466 ] Arun Suresh commented on HADOOP-13836: -- bq. wire encryption can only be good, though the cost of negotiating secure HTTPS connections can be high; I don't know if this proposal will have the same problem. [~steve_l], From my initial glance of the patch, it looks like it is replacing the socket used for the RPC with an SSL Socket. In which case, It should be technically possible to replace the standard JSSE SSLEngine with OpenSSL's JNI based codecs for improved performance (maybe as a later patch), like what Tomcat does. [~kartheek], do you have some numbers that quantify the performance degradation ? > Securing Hadoop RPC using SSL > - > > Key: HADOOP-13836 > URL: https://issues.apache.org/jira/browse/HADOOP-13836 > Project: Hadoop Common > Issue Type: New Feature > Components: ipc >Reporter: kartheek muthyala > Attachments: HADOOP-13836.patch > > > Today, RPC connections in Hadoop are encrypted using Simple Authentication & > Security Layer (SASL), with the Kerberos ticket based authentication or > Digest-md5 checksum based authentication protocols. This proposal is about > enhancing this cipher suite with SSL/TLS based encryption and authentication. > SSL/TLS is a proposed Internet Engineering Task Force (IETF) standard, that > provides data security and integrity across two different end points in a > network. This protocol has made its way to a number of applications such as > web browsing, email, internet faxing, messaging, VOIP etc. And supporting > this cipher suite at the core of Hadoop would give a good synergy with the > applications on top and also bolster industry adoption of Hadoop. > The Server and Client code in Hadoop IPC should support the following modes > of communication > 1.Plain > 2. SASL encryption with an underlying authentication > 3. SSL based encryption and authentication (x509 certificate) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13836) Securing Hadoop RPC using SSL
[
https://issues.apache.org/jira/browse/HADOOP-13836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15702398#comment-15702398
]
Hadoop QA commented on HADOOP-13836:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m
11s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 40s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 50 new + 402 unchanged - 16 fixed = 452 total (was 418) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 8 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
4s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
26s{color} | {color:red} hadoop-common-project/hadoop-common generated 1 new +
0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 10m 3s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
30s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 60m 51s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-common-project/hadoop-common |
| | Uninitialized read of backlogLength in new
org.apache.hadoop.ipc.AbstractListener(String, int, int, int, String,
Configuration, Server$ConnectionManager) At AbstractListener.java:new
org.apache.hadoop.ipc.AbstractListener(String, int, int, int, String,
Configuration, Server$ConnectionManager) At AbstractListener.java:[line 71] |
| Failed junit tests | hadoop.ipc.TestSSLIPC |
| | hadoop.ipc.TestRPC |
| | hadoop.ipc.TestIPC |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HADOOP-13836 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12840656/HADOOP-13836.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit xml findbugs checkstyle |
| uname | Linux b1c57be4d725 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 5d5614f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
[jira] [Commented] (HADOOP-13836) Securing Hadoop RPC using SSL
[ https://issues.apache.org/jira/browse/HADOOP-13836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15702238#comment-15702238 ] kartheek muthyala commented on HADOOP-13836: Hey [~antkou], Good to know that you are also working on the similar feature. We have submitted an initial version of the patch. Kindly review it and let us know your feedback. > Securing Hadoop RPC using SSL > - > > Key: HADOOP-13836 > URL: https://issues.apache.org/jira/browse/HADOOP-13836 > Project: Hadoop Common > Issue Type: New Feature > Components: ipc >Reporter: kartheek muthyala > Attachments: HADOOP-13836.patch > > > Today, RPC connections in Hadoop are encrypted using Simple Authentication & > Security Layer (SASL), with the Kerberos ticket based authentication or > Digest-md5 checksum based authentication protocols. This proposal is about > enhancing this cipher suite with SSL/TLS based encryption and authentication. > SSL/TLS is a proposed Internet Engineering Task Force (IETF) standard, that > provides data security and integrity across two different end points in a > network. This protocol has made its way to a number of applications such as > web browsing, email, internet faxing, messaging, VOIP etc. And supporting > this cipher suite at the core of Hadoop would give a good synergy with the > applications on top and also bolster industry adoption of Hadoop. > The Server and Client code in Hadoop IPC should support the following modes > of communication > 1.Plain > 2. SASL encryption with an underlying authentication > 3. SSL based encryption and authentication (x509 certificate) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13836) Securing Hadoop RPC using SSL
[ https://issues.apache.org/jira/browse/HADOOP-13836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15701980#comment-15701980 ] Steve Loughran commented on HADOOP-13836: - wire encryption can only be good, though the cost of negotiating secure HTTPS connections can be high; I don't know if this proposal will have the same problem. At least with Hadoop RPC the channel can be kept open for a while, a luxury we don't get so much with the object store clients. one thing to consider is that although SASL supports encryption, it rarely gets used in large systems —even when the CPUs have opcodes for encryption. Why? The performance of the standard Java APIs are pretty bad; the API always allocates new buffers for the encrypted result, rather than allowing buffer re-use. It may be necessary to achieve necessary performance though some more native code in the hadoop native libs, and having the clients optionally use it. > Securing Hadoop RPC using SSL > - > > Key: HADOOP-13836 > URL: https://issues.apache.org/jira/browse/HADOOP-13836 > Project: Hadoop Common > Issue Type: New Feature > Components: ipc >Reporter: kartheek muthyala > > Today, RPC connections in Hadoop are encrypted using Simple Authentication & > Security Layer (SASL), with the Kerberos ticket based authentication or > Digest-md5 checksum based authentication protocols. This proposal is about > enhancing this cipher suite with SSL/TLS based encryption and authentication. > SSL/TLS is a proposed Internet Engineering Task Force (IETF) standard, that > provides data security and integrity across two different end points in a > network. This protocol has made its way to a number of applications such as > web browsing, email, internet faxing, messaging, VOIP etc. And supporting > this cipher suite at the core of Hadoop would give a good synergy with the > applications on top and also bolster industry adoption of Hadoop. > The Server and Client code in Hadoop IPC should support the following modes > of communication > 1.Plain > 2. SASL encryption with an underlying authentication > 3. SSL based encryption and authentication (x509 certificate) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13836) Securing Hadoop RPC using SSL
[ https://issues.apache.org/jira/browse/HADOOP-13836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15701896#comment-15701896 ] Antonios Kouzoupis commented on HADOOP-13836: - Hello. I've been working on this feature and I can submit a preliminary patch when I clean up my code a little bit. We could also collaborate on this. > Securing Hadoop RPC using SSL > - > > Key: HADOOP-13836 > URL: https://issues.apache.org/jira/browse/HADOOP-13836 > Project: Hadoop Common > Issue Type: New Feature > Components: ipc >Reporter: kartheek muthyala > > Today, RPC connections in Hadoop are encrypted using Simple Authentication & > Security Layer (SASL), with the Kerberos ticket based authentication or > Digest-md5 checksum based authentication protocols. This proposal is about > enhancing this cipher suite with SSL/TLS based encryption and authentication. > SSL/TLS is a proposed Internet Engineering Task Force (IETF) standard, that > provides data security and integrity across two different end points in a > network. This protocol has made its way to a number of applications such as > web browsing, email, internet faxing, messaging, VOIP etc. And supporting > this cipher suite at the core of Hadoop would give a good synergy with the > applications on top and also bolster industry adoption of Hadoop. > The Server and Client code in Hadoop IPC should support the following modes > of communication > 1.Plain > 2. SASL encryption with an underlying authentication > 3. SSL based encryption and authentication (x509 certificate) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13836) Securing Hadoop RPC using SSL
[ https://issues.apache.org/jira/browse/HADOOP-13836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15701781#comment-15701781 ] kartheek muthyala commented on HADOOP-13836: I will come up with a preliminary patch for this in a few days. Keeping this open for general discussion and interest around this topic. > Securing Hadoop RPC using SSL > - > > Key: HADOOP-13836 > URL: https://issues.apache.org/jira/browse/HADOOP-13836 > Project: Hadoop Common > Issue Type: New Feature > Components: ipc >Reporter: kartheek muthyala > > Today, RPC connections in Hadoop are encrypted using Simple Authentication & > Security Layer (SASL), with the Kerberos ticket based authentication or > Digest-md5 checksum based authentication protocols. This proposal is about > enhancing this cipher suite with SSL/TLS based encryption and authentication. > SSL/TLS is a proposed Internet Engineering Task Force (IETF) standard, that > provides data security and integrity across two different end points in a > network. This protocol has made its way to a number of applications such as > web browsing, email, internet faxing, messaging, VOIP etc. And supporting > this cipher suite at the core of Hadoop would give a good synergy with the > applications on top and also bolster industry adoption of Hadoop. > The Server and Client code in Hadoop IPC should support the following modes > of communication > 1.Plain > 2. SASL encryption with an underlying authentication > 3. SSL based encryption and authentication (x509 certificate) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
