[
https://issues.apache.org/jira/browse/HADOOP-16011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16804354#comment-16804354
]
Siyao Meng commented on HADOOP-16011:
-------------------------------------
[~jojochuang] I see no reference of OsSecureRandom in core-default.xml, only
OpensslAesCtrCryptoCodec here:
{code:xml}
<property>
<name>hadoop.security.crypto.codec.classes.aes.ctr.nopadding</name>
<value>org.apache.hadoop.crypto.OpensslAesCtrCryptoCodec,
org.apache.hadoop.crypto.JceAesCtrCryptoCodec</value>
<description>
Comma-separated list of crypto codec implementations for AES/CTR/NoPadding.
The first implementation will be used if available, others are fallbacks.
</description>
</property>
{code}
What change do we need here?
> OsSecureRandom very slow compared to other SecureRandom implementations
> -----------------------------------------------------------------------
>
> Key: HADOOP-16011
> URL: https://issues.apache.org/jira/browse/HADOOP-16011
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
> Reporter: Todd Lipcon
> Assignee: Siyao Meng
> Priority: Major
> Attachments: HADOOP-16011.001.patch, MyBenchmark.java
>
>
> In looking at performance of a workload which creates a lot of short-lived
> remote connections to a secured DN, [~philip] and I found very high system
> CPU usage. We tracked it down to reads from /dev/random, which are incurred
> by the DN using CryptoCodec.generateSecureRandom to generate a transient
> session key and IV for AES encryption.
> In the case that the OpenSSL codec is not enabled, the above code falls
> through to the JDK SecureRandom implementation, which performs reasonably.
> However, OpenSSLCodec defaults to using OsSecureRandom, which reads all
> random data from /dev/random rather than doing something more efficient like
> initializing a CSPRNG from a small seed.
> I wrote a simple JMH benchmark to compare various approaches when running
> with concurrency 10:
> testHadoop - using CryptoCodec
> testNewSecureRandom - using 'new SecureRandom()' each iteration
> testSha1PrngNew - using the SHA1PRNG explicitly, new instance each iteration
> testSha1PrngShared - using a single shared instance of SHA1PRNG
> testSha1PrngThread - using a thread-specific instance of SHA1PRNG
> {code:java}
> Benchmark Mode Cnt Score Error Units
> MyBenchmark.testHadoop thrpt 1293.000 ops/s
> [with libhadoop.so]
> MyBenchmark.testHadoop thrpt 461515.697 ops/s
> [without libhadoop.so]
> MyBenchmark.testNewSecureRandom thrpt 43413.640 ops/s
> MyBenchmark.testSha1PrngNew thrpt 395515.000 ops/s
> MyBenchmark.testSha1PrngShared thrpt 164488.713 ops/s
> MyBenchmark.testSha1PrngThread thrpt 4295123.210 ops/s
> {code}
> In other words, the presence of the OpenSSL acceleration slows down this code
> path by 356x. And, compared to the optimal (thread-local Sha1Prng) it's 3321x
> slower.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
