[
https://issues.apache.org/jira/browse/HADOOP-10768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16500669#comment-16500669
]
Daryn Sharp commented on HADOOP-10768:
--------------------------------------
{quote}Yes, shouldAuthenticateOverKrb() would throw NPE in some cases, we
should catch it.
{quote}
No, catching and ignoring all exceptions from shouldAuthenticateOverKrb is
completely wrong.
I appreciate all the work here. However I have a number of security concerns.
I didn't care too much before other than not breaking the protocol and not
making simple mistakes. Times change. I care a lot now.
I don't fully understand what's going on in the key negotiation, don't feel
qualified to determine if it's secure (although it doesn't appear to be), doubt
few in our community are either, now and in the future. I don't like being
tied to AES/CTR+custom-hmac when other ciphers like AES/GCM/ECDHE may be a
requirement.
An actual SSL implementation written by experts that we don't maintain and is
tuned for performance is a far preferable approach. To that end, I've made
surprising small tweaks to the ipc server to use netty (configurable). Goal is
dropping in netty's SSL handlers.
Note that hadoop's ipc is pretty darn optimized. I was surprised to find netty
initially incurred a 20% performance hit. I've since closed the gap to <5%.
Note that one of the challenges was avoiding array copies and allocations. I
see so much in the current patch that it's likely what's sapping performance so
much.
I'm finishing up debugging a few failed test cases. Should post on a different
Jira this afternoon. Then we can work on the ipc client.
> Optimize Hadoop RPC encryption performance
> ------------------------------------------
>
> Key: HADOOP-10768
> URL: https://issues.apache.org/jira/browse/HADOOP-10768
> Project: Hadoop Common
> Issue Type: Improvement
> Components: performance, security
> Affects Versions: 3.0.0-alpha1
> Reporter: Yi Liu
> Assignee: Dapeng Sun
> Priority: Major
> Attachments: HADOOP-10768.001.patch, HADOOP-10768.002.patch,
> HADOOP-10768.003.patch, HADOOP-10768.004.patch, HADOOP-10768.005.patch,
> HADOOP-10768.006.patch, HADOOP-10768.007.patch, HADOOP-10768.008.patch,
> HADOOP-10768.009.patch, HADOOP-10768.010.patch, HADOOP-10768.011.patch,
> Optimize Hadoop RPC encryption performance.pdf,
> cpu_profile_RPC_encryption_AES.png,
> cpu_profile_rpc_encryption_optimize_calculateHMAC.png
>
>
> Hadoop RPC encryption is enabled by setting {{hadoop.rpc.protection}} to
> "privacy". It utilized SASL {{GSSAPI}} and {{DIGEST-MD5}} mechanisms for
> secure authentication and data protection. Even {{GSSAPI}} supports using
> AES, but without AES-NI support by default, so the encryption is slow and
> will become bottleneck.
> After discuss with [~atm], [~tucu00] and [~umamaheswararao], we can do the
> same optimization as in HDFS-6606. Use AES-NI with more than *20x* speedup.
> On the other hand, RPC message is small, but RPC is frequent and there may be
> lots of RPC calls in one connection, we needs to setup benchmark to see real
> improvement and then make a trade-off.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
