[
https://issues.apache.org/jira/browse/HADOOP-10768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16252892#comment-16252892
]
Dapeng Sun commented on HADOOP-10768:
-------------------------------------
Hi, Daryn Sharp, Aaron T. Myers, I have updated the patch, please help to
review if you time allow.
{quote}Add some unit tests and results of measuring the performance improvement
{quote}
I have finished the initial Benchmark with {{RPC Call Benchmark}} and also
added two UTs in the latest patch, the benchmark result shows the patch improve
{{~4X}} on the throughput of RPC call, now the performance of PRIVACY is close
to INTEGRITY’s, here is the detail data of total calls per second:
||SASL||r:4 c:1 s:1 m:1024|| r:4 c:30 s:30 m:1024||
|NONE|17102|233726|
|INTEGRITY|11693|114303|
|PRIVACY (AES with HadoopOpensslCodec)|9194|112763|
|PRIVACY (AES with HadoopJceCodec)|7718|111807|
|PRIVACY (Original DES)|1872|34007|
{quote}Why not use javax cipher libraries? Any number of ciphers could be used
now and in the future w/o code change. The aes ciphers are supposed to use
aes-ni intrinsics when available.{quote}
JDK cipher doesn’t take advantage of Intel AES-NI very well until JDK 9
([JDK-8143925|https://bugs.openjdk.java.net/browse/JDK-8143925] Improve JDK 9
AES-CTR with 4~6x gain), so I think using Crypto Stream in Hadoop should be a
good choice.
{quote}Should use a custom sasl client/server that delegates to the actual sasl
instance. The ipc layer changes would be minimal and easier to maintain.{quote}
It is hard to separate current logic to independent sasl client/server with
minimal changes, I have move the logic to separate method for minimizing the
changes.
{quote}The cipher options appears to be present in every packet. If so, it
should only be in the negotiate/initiate messages.{quote}
The cipher option only appears in SASL negotiate/initiate.
Thanks.
> Optimize Hadoop RPC encryption performance
> ------------------------------------------
>
> Key: HADOOP-10768
> URL: https://issues.apache.org/jira/browse/HADOOP-10768
> Project: Hadoop Common
> Issue Type: Improvement
> Components: performance, security
> Affects Versions: 3.0.0-alpha1
> Reporter: Yi Liu
> Assignee: Dapeng Sun
> Attachments: HADOOP-10768.001.patch, HADOOP-10768.002.patch,
> HADOOP-10768.003.patch, HADOOP-10768.004.patch, HADOOP-10768.005.patch,
> HADOOP-10768.006.patch, Optimize Hadoop RPC encryption performance.pdf
>
>
> Hadoop RPC encryption is enabled by setting {{hadoop.rpc.protection}} to
> "privacy". It utilized SASL {{GSSAPI}} and {{DIGEST-MD5}} mechanisms for
> secure authentication and data protection. Even {{GSSAPI}} supports using
> AES, but without AES-NI support by default, so the encryption is slow and
> will become bottleneck.
> After discuss with [~atm], [~tucu00] and [~umamaheswararao], we can do the
> same optimization as in HDFS-6606. Use AES-NI with more than *20x* speedup.
> On the other hand, RPC message is small, but RPC is frequent and there may be
> lots of RPC calls in one connection, we needs to setup benchmark to see real
> improvement and then make a trade-off.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
