[
https://issues.apache.org/jira/browse/HADOOP-16011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16809960#comment-16809960
]
Siyao Meng commented on HADOOP-16011:
-------------------------------------
I've tried to run [~tlipcon]'s benchmark w/ and w/o HADOOP-16011 on my Mac. But
failed to find any difference, not sure if that's because there's something
wrong with my steps or it's due to the Hadoop Configuration() class. JDK
1.8.0_181
Result: 3.2.0 jar (non-native)
{code}
Benchmark Mode Cnt Score Error Units
MyBenchmark.testHadoop thrpt 25 119383.456 ± 1424.871 ops/s
MyBenchmark.testNewSecureRandom thrpt 25 267809.626 ± 4098.208 ops/s
MyBenchmark.testSha1PrngNew thrpt 25 492695.425 ± 5068.229 ops/s
MyBenchmark.testSha1PrngShared thrpt 25 809413.172 ± 7708.147 ops/s
MyBenchmark.testSha1PrngThread thrpt 25 805343.533 ± 7990.329 ops/s
{code}
Result: 3.2.0 + HADOOP-16011 compiled, non-native
{code}
Benchmark Mode Cnt Score Error Units
MyBenchmark.testHadoop thrpt 25 113706.696 ± 951.552 ops/s
MyBenchmark.testNewSecureRandom thrpt 25 251700.547 ± 983.088 ops/s
MyBenchmark.testSha1PrngNew thrpt 25 462082.417 ± 3810.001 ops/s
MyBenchmark.testSha1PrngShared thrpt 25 818583.812 ± 7127.007 ops/s
MyBenchmark.testSha1PrngThread thrpt 25 822274.714 ± 3985.425 ops/s
{code}
My steps to generate the 3.2.0 + HADOOP-16011 jar is as following:
{code}
# In my local hadoop asf repo
git checkout branch-3.2.0
gcp -x e62cbcbc83026a7af43eac6223fe53f9de963d91
mvn install -Pdist -DskipTests -e -Dmaven.javadoc.skip=true
-Denforcer.skip=true -DskipShade
# In benchmark project repo
mvn package # This should pick up locally cached Hadoop 3.2.0 jars that I've
just compiled and installed.
java -jar target/benchmarks.jar
{code}
> OsSecureRandom very slow compared to other SecureRandom implementations
> -----------------------------------------------------------------------
>
> Key: HADOOP-16011
> URL: https://issues.apache.org/jira/browse/HADOOP-16011
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
> Reporter: Todd Lipcon
> Assignee: Siyao Meng
> Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-16011.001.patch, HADOOP-16011.002.patch,
> MyBenchmark.java
>
>
> In looking at performance of a workload which creates a lot of short-lived
> remote connections to a secured DN, [~philip] and I found very high system
> CPU usage. We tracked it down to reads from /dev/random, which are incurred
> by the DN using CryptoCodec.generateSecureRandom to generate a transient
> session key and IV for AES encryption.
> In the case that the OpenSSL codec is not enabled, the above code falls
> through to the JDK SecureRandom implementation, which performs reasonably.
> However, OpenSSLCodec defaults to using OsSecureRandom, which reads all
> random data from /dev/random rather than doing something more efficient like
> initializing a CSPRNG from a small seed.
> I wrote a simple JMH benchmark to compare various approaches when running
> with concurrency 10:
> testHadoop - using CryptoCodec
> testNewSecureRandom - using 'new SecureRandom()' each iteration
> testSha1PrngNew - using the SHA1PRNG explicitly, new instance each iteration
> testSha1PrngShared - using a single shared instance of SHA1PRNG
> testSha1PrngThread - using a thread-specific instance of SHA1PRNG
> {code:java}
> Benchmark Mode Cnt Score Error Units
> MyBenchmark.testHadoop thrpt 1293.000 ops/s
> [with libhadoop.so]
> MyBenchmark.testHadoop thrpt 461515.697 ops/s
> [without libhadoop.so]
> MyBenchmark.testNewSecureRandom thrpt 43413.640 ops/s
> MyBenchmark.testSha1PrngNew thrpt 395515.000 ops/s
> MyBenchmark.testSha1PrngShared thrpt 164488.713 ops/s
> MyBenchmark.testSha1PrngThread thrpt 4295123.210 ops/s
> {code}
> In other words, the presence of the OpenSSL acceleration slows down this code
> path by 356x. And, compared to the optimal (thread-local Sha1Prng) it's 3321x
> slower.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
