[ https://issues.apache.org/jira/browse/HADOOP-16011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16809296#comment-16809296 ]
Hudson commented on HADOOP-16011: --------------------------------- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16343 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/16343/]) HADOOP-16011. OsSecureRandom very slow compared to other SecureRandom (weichiu: rev e62cbcbc83026a7af43eac6223fe53f9de963d91) * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/OpensslAesCtrCryptoCodec.java * (edit) hadoop-common-project/hadoop-common/src/main/resources/core-default.xml > OsSecureRandom very slow compared to other SecureRandom implementations > ----------------------------------------------------------------------- > > Key: HADOOP-16011 > URL: https://issues.apache.org/jira/browse/HADOOP-16011 > Project: Hadoop Common > Issue Type: Bug > Components: security > Reporter: Todd Lipcon > Assignee: Siyao Meng > Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-16011.001.patch, HADOOP-16011.002.patch, > MyBenchmark.java > > > In looking at performance of a workload which creates a lot of short-lived > remote connections to a secured DN, [~philip] and I found very high system > CPU usage. We tracked it down to reads from /dev/random, which are incurred > by the DN using CryptoCodec.generateSecureRandom to generate a transient > session key and IV for AES encryption. > In the case that the OpenSSL codec is not enabled, the above code falls > through to the JDK SecureRandom implementation, which performs reasonably. > However, OpenSSLCodec defaults to using OsSecureRandom, which reads all > random data from /dev/random rather than doing something more efficient like > initializing a CSPRNG from a small seed. > I wrote a simple JMH benchmark to compare various approaches when running > with concurrency 10: > testHadoop - using CryptoCodec > testNewSecureRandom - using 'new SecureRandom()' each iteration > testSha1PrngNew - using the SHA1PRNG explicitly, new instance each iteration > testSha1PrngShared - using a single shared instance of SHA1PRNG > testSha1PrngThread - using a thread-specific instance of SHA1PRNG > {code:java} > Benchmark Mode Cnt Score Error Units > MyBenchmark.testHadoop thrpt 1293.000 ops/s > [with libhadoop.so] > MyBenchmark.testHadoop thrpt 461515.697 ops/s > [without libhadoop.so] > MyBenchmark.testNewSecureRandom thrpt 43413.640 ops/s > MyBenchmark.testSha1PrngNew thrpt 395515.000 ops/s > MyBenchmark.testSha1PrngShared thrpt 164488.713 ops/s > MyBenchmark.testSha1PrngThread thrpt 4295123.210 ops/s > {code} > In other words, the presence of the OpenSSL acceleration slows down this code > path by 356x. And, compared to the optimal (thread-local Sha1Prng) it's 3321x > slower. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org