[jira] [Commented] (HADOOP-16011) OsSecureRandom very slow compared to other SecureRandom implementations
[
https://issues.apache.org/jira/browse/HADOOP-16011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815359#comment-16815359
]
Siyao Meng commented on HADOOP-16011:
-
And yes, pre-patch performs miserably with OpenSSL:
{code:java}
Benchmark Mode Cnt Score Error Units
MyBenchmark.testHadoop thrpt31557.764 ± 446.665 ops/s [ w/
OpenSSL, pre-patch ]
{code}
I'm using Hadoop 3.1.0 jars.
There's definitely room for further improvement.
> OsSecureRandom very slow compared to other SecureRandom implementations
> ---
>
> Key: HADOOP-16011
> URL: https://issues.apache.org/jira/browse/HADOOP-16011
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Reporter: Todd Lipcon
>Assignee: Siyao Meng
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-16011.001.patch, HADOOP-16011.002.patch,
> MyBenchmark.java
>
>
> In looking at performance of a workload which creates a lot of short-lived
> remote connections to a secured DN, [~philip] and I found very high system
> CPU usage. We tracked it down to reads from /dev/random, which are incurred
> by the DN using CryptoCodec.generateSecureRandom to generate a transient
> session key and IV for AES encryption.
> In the case that the OpenSSL codec is not enabled, the above code falls
> through to the JDK SecureRandom implementation, which performs reasonably.
> However, OpenSSLCodec defaults to using OsSecureRandom, which reads all
> random data from /dev/random rather than doing something more efficient like
> initializing a CSPRNG from a small seed.
> I wrote a simple JMH benchmark to compare various approaches when running
> with concurrency 10:
> testHadoop - using CryptoCodec
> testNewSecureRandom - using 'new SecureRandom()' each iteration
> testSha1PrngNew - using the SHA1PRNG explicitly, new instance each iteration
> testSha1PrngShared - using a single shared instance of SHA1PRNG
> testSha1PrngThread - using a thread-specific instance of SHA1PRNG
> {code:java}
> Benchmark Mode CntScore Error Units
> MyBenchmark.testHadoop thrpt 1293.000 ops/s
> [with libhadoop.so]
> MyBenchmark.testHadoop thrpt461515.697 ops/s
> [without libhadoop.so]
> MyBenchmark.testNewSecureRandom thrpt 43413.640 ops/s
> MyBenchmark.testSha1PrngNew thrpt395515.000 ops/s
> MyBenchmark.testSha1PrngShared thrpt164488.713 ops/s
> MyBenchmark.testSha1PrngThread thrpt 4295123.210 ops/s
> {code}
> In other words, the presence of the OpenSSL acceleration slows down this code
> path by 356x. And, compared to the optimal (thread-local Sha1Prng) it's 3321x
> slower.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16011) OsSecureRandom very slow compared to other SecureRandom implementations
[
https://issues.apache.org/jira/browse/HADOOP-16011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815356#comment-16815356
]
Siyao Meng commented on HADOOP-16011:
-
Thanks [~tlipcon]. I've also tried to run this w/ native on a Ubuntu 16.04 node.
It seems Hadoop 3.1/3.2 already tries to use OpensslAesCtrCryptoCodec as the
first choice, it falls back when it can't find its path. After setting log4j
level to TRACE I can see its attempt to load native-hadoop lib:
{code}
DEBUG [org.sample.MyBenchmark.testHadoop-jmh-worker-1] Trying to load the
custom-built native-hadoop library...
DEBUG [org.sample.MyBenchmark.testHadoop-jmh-worker-1] Failed to load
native-hadoop with error: java.lang.UnsatisfiedLinkError: no hadoop in
java.library.path
DEBUG [org.sample.MyBenchmark.testHadoop-jmh-worker-1]
java.library.path=/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
WARN [org.sample.MyBenchmark.testHadoop-jmh-worker-1] Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
DEBUG [org.sample.MyBenchmark.testHadoop-jmh-worker-1] Failed to load OpenSSL
Cipher.
{code}
So I manually specified the native lib path:
{code:bash}
java
-Djava.library.path=/home/systest/trunk/hadoop-dist/target/hadoop-3.3.0-SNAPSHOT/lib/native/
-jar target/benchmarks.jar
{code}
Results with 3.3.0-SNAPSHOT:
{code}
Benchmark Mode Cnt Score Error Units
MyBenchmark.testHadoop thrpt3 231204.042 ± 6968.536 ops/s [ w/
OpenSSL ]
MyBenchmark.testHadoop thrpt3 102696.569 ± 1289.506 ops/s [
w/o OpenSSL, uses JceAesCtrCryptoCodec, JDK 1.8.0_191 ]
MyBenchmark.testNewSecureRandom thrpt3 104188.125 ± 15255.362 ops/s
MyBenchmark.testSha1PrngNew thrpt3 343648.417 ± 20911.516 ops/s
MyBenchmark.testSha1PrngShared thrpt3 583999.658 ± 8519.252 ops/s
MyBenchmark.testSha1PrngThread thrpt3 581220.206 ± 13029.829 ops/s
{code}
> OsSecureRandom very slow compared to other SecureRandom implementations
> ---
>
> Key: HADOOP-16011
> URL: https://issues.apache.org/jira/browse/HADOOP-16011
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Reporter: Todd Lipcon
>Assignee: Siyao Meng
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-16011.001.patch, HADOOP-16011.002.patch,
> MyBenchmark.java
>
>
> In looking at performance of a workload which creates a lot of short-lived
> remote connections to a secured DN, [~philip] and I found very high system
> CPU usage. We tracked it down to reads from /dev/random, which are incurred
> by the DN using CryptoCodec.generateSecureRandom to generate a transient
> session key and IV for AES encryption.
> In the case that the OpenSSL codec is not enabled, the above code falls
> through to the JDK SecureRandom implementation, which performs reasonably.
> However, OpenSSLCodec defaults to using OsSecureRandom, which reads all
> random data from /dev/random rather than doing something more efficient like
> initializing a CSPRNG from a small seed.
> I wrote a simple JMH benchmark to compare various approaches when running
> with concurrency 10:
> testHadoop - using CryptoCodec
> testNewSecureRandom - using 'new SecureRandom()' each iteration
> testSha1PrngNew - using the SHA1PRNG explicitly, new instance each iteration
> testSha1PrngShared - using a single shared instance of SHA1PRNG
> testSha1PrngThread - using a thread-specific instance of SHA1PRNG
> {code:java}
> Benchmark Mode CntScore Error Units
> MyBenchmark.testHadoop thrpt 1293.000 ops/s
> [with libhadoop.so]
> MyBenchmark.testHadoop thrpt461515.697 ops/s
> [without libhadoop.so]
> MyBenchmark.testNewSecureRandom thrpt 43413.640 ops/s
> MyBenchmark.testSha1PrngNew thrpt395515.000 ops/s
> MyBenchmark.testSha1PrngShared thrpt164488.713 ops/s
> MyBenchmark.testSha1PrngThread thrpt 4295123.210 ops/s
> {code}
> In other words, the presence of the OpenSSL acceleration slows down this code
> path by 356x. And, compared to the optimal (thread-local Sha1Prng) it's 3321x
> slower.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16011) OsSecureRandom very slow compared to other SecureRandom implementations
[
https://issues.apache.org/jira/browse/HADOOP-16011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16813580#comment-16813580
]
Todd Lipcon commented on HADOOP-16011:
--
Re-ran the benchmarks locally (laptop, JDK10, not server hardware) and verified
this has the expected improvement.
{code}
Benchmark Mode CntScore Error Units
MyBenchmark.testHadoop thrpt321125.844 ops/s [
without libhadoop ]
MyBenchmark.testHadoop thrpt 19819.205 ops/s [ with
libhadoop, pre-patch ]
MyBenchmark.testHadoop thrpt267370.701 ops/s [ with
libhadoop + patch ]
MyBenchmark.testNewSecureRandom thrpt362266.143 ops/s
MyBenchmark.testSha1PrngNew thrpt782896.217 ops/s
MyBenchmark.testSha1PrngShared thrpt483401.829 ops/s
MyBenchmark.testSha1PrngThread thrpt 3099035.965 ops/s
{code}
It's unfortunate that libhadoop still slows down from the non-libhadoop patch,
and certainly seems like we could get a big improvement using other mechanisms,
but this probably isn't a bottleneck anymore.
> OsSecureRandom very slow compared to other SecureRandom implementations
> ---
>
> Key: HADOOP-16011
> URL: https://issues.apache.org/jira/browse/HADOOP-16011
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Reporter: Todd Lipcon
>Assignee: Siyao Meng
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-16011.001.patch, HADOOP-16011.002.patch,
> MyBenchmark.java
>
>
> In looking at performance of a workload which creates a lot of short-lived
> remote connections to a secured DN, [~philip] and I found very high system
> CPU usage. We tracked it down to reads from /dev/random, which are incurred
> by the DN using CryptoCodec.generateSecureRandom to generate a transient
> session key and IV for AES encryption.
> In the case that the OpenSSL codec is not enabled, the above code falls
> through to the JDK SecureRandom implementation, which performs reasonably.
> However, OpenSSLCodec defaults to using OsSecureRandom, which reads all
> random data from /dev/random rather than doing something more efficient like
> initializing a CSPRNG from a small seed.
> I wrote a simple JMH benchmark to compare various approaches when running
> with concurrency 10:
> testHadoop - using CryptoCodec
> testNewSecureRandom - using 'new SecureRandom()' each iteration
> testSha1PrngNew - using the SHA1PRNG explicitly, new instance each iteration
> testSha1PrngShared - using a single shared instance of SHA1PRNG
> testSha1PrngThread - using a thread-specific instance of SHA1PRNG
> {code:java}
> Benchmark Mode CntScore Error Units
> MyBenchmark.testHadoop thrpt 1293.000 ops/s
> [with libhadoop.so]
> MyBenchmark.testHadoop thrpt461515.697 ops/s
> [without libhadoop.so]
> MyBenchmark.testNewSecureRandom thrpt 43413.640 ops/s
> MyBenchmark.testSha1PrngNew thrpt395515.000 ops/s
> MyBenchmark.testSha1PrngShared thrpt164488.713 ops/s
> MyBenchmark.testSha1PrngThread thrpt 4295123.210 ops/s
> {code}
> In other words, the presence of the OpenSSL acceleration slows down this code
> path by 356x. And, compared to the optimal (thread-local Sha1Prng) it's 3321x
> slower.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16011) OsSecureRandom very slow compared to other SecureRandom implementations
[
https://issues.apache.org/jira/browse/HADOOP-16011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16810368#comment-16810368
]
Todd Lipcon commented on HADOOP-16011:
--
Right, this code will only kick in if you're running with libhadoop.so present
and loadable, along with OpenSSL support. I don't know if that works on Mac.
Might have to run the benchmark on Linux
> OsSecureRandom very slow compared to other SecureRandom implementations
> ---
>
> Key: HADOOP-16011
> URL: https://issues.apache.org/jira/browse/HADOOP-16011
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Reporter: Todd Lipcon
>Assignee: Siyao Meng
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-16011.001.patch, HADOOP-16011.002.patch,
> MyBenchmark.java
>
>
> In looking at performance of a workload which creates a lot of short-lived
> remote connections to a secured DN, [~philip] and I found very high system
> CPU usage. We tracked it down to reads from /dev/random, which are incurred
> by the DN using CryptoCodec.generateSecureRandom to generate a transient
> session key and IV for AES encryption.
> In the case that the OpenSSL codec is not enabled, the above code falls
> through to the JDK SecureRandom implementation, which performs reasonably.
> However, OpenSSLCodec defaults to using OsSecureRandom, which reads all
> random data from /dev/random rather than doing something more efficient like
> initializing a CSPRNG from a small seed.
> I wrote a simple JMH benchmark to compare various approaches when running
> with concurrency 10:
> testHadoop - using CryptoCodec
> testNewSecureRandom - using 'new SecureRandom()' each iteration
> testSha1PrngNew - using the SHA1PRNG explicitly, new instance each iteration
> testSha1PrngShared - using a single shared instance of SHA1PRNG
> testSha1PrngThread - using a thread-specific instance of SHA1PRNG
> {code:java}
> Benchmark Mode CntScore Error Units
> MyBenchmark.testHadoop thrpt 1293.000 ops/s
> [with libhadoop.so]
> MyBenchmark.testHadoop thrpt461515.697 ops/s
> [without libhadoop.so]
> MyBenchmark.testNewSecureRandom thrpt 43413.640 ops/s
> MyBenchmark.testSha1PrngNew thrpt395515.000 ops/s
> MyBenchmark.testSha1PrngShared thrpt164488.713 ops/s
> MyBenchmark.testSha1PrngThread thrpt 4295123.210 ops/s
> {code}
> In other words, the presence of the OpenSSL acceleration slows down this code
> path by 356x. And, compared to the optimal (thread-local Sha1Prng) it's 3321x
> slower.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16011) OsSecureRandom very slow compared to other SecureRandom implementations
[
https://issues.apache.org/jira/browse/HADOOP-16011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809960#comment-16809960
]
Siyao Meng commented on HADOOP-16011:
-
I've tried to run [~tlipcon]'s benchmark w/ and w/o HADOOP-16011 on my Mac. But
failed to find any difference, not sure if that's because there's something
wrong with my steps or it's due to the Hadoop Configuration() class. JDK
1.8.0_181
Result: 3.2.0 jar (non-native)
{code}
Benchmark Mode Cnt Score Error Units
MyBenchmark.testHadoop thrpt 25 119383.456 ± 1424.871 ops/s
MyBenchmark.testNewSecureRandom thrpt 25 267809.626 ± 4098.208 ops/s
MyBenchmark.testSha1PrngNew thrpt 25 492695.425 ± 5068.229 ops/s
MyBenchmark.testSha1PrngShared thrpt 25 809413.172 ± 7708.147 ops/s
MyBenchmark.testSha1PrngThread thrpt 25 805343.533 ± 7990.329 ops/s
{code}
Result: 3.2.0 + HADOOP-16011 compiled, non-native
{code}
Benchmark Mode Cnt Score Error Units
MyBenchmark.testHadoop thrpt 25 113706.696 ± 951.552 ops/s
MyBenchmark.testNewSecureRandom thrpt 25 251700.547 ± 983.088 ops/s
MyBenchmark.testSha1PrngNew thrpt 25 462082.417 ± 3810.001 ops/s
MyBenchmark.testSha1PrngShared thrpt 25 818583.812 ± 7127.007 ops/s
MyBenchmark.testSha1PrngThread thrpt 25 822274.714 ± 3985.425 ops/s
{code}
My steps to generate the 3.2.0 + HADOOP-16011 jar is as following:
{code}
# In my local hadoop asf repo
git checkout branch-3.2.0
gcp -x e62cbcbc83026a7af43eac6223fe53f9de963d91
mvn install -Pdist -DskipTests -e -Dmaven.javadoc.skip=true
-Denforcer.skip=true -DskipShade
# In benchmark project repo
mvn package # This should pick up locally cached Hadoop 3.2.0 jars that I've
just compiled and installed.
java -jar target/benchmarks.jar
{code}
> OsSecureRandom very slow compared to other SecureRandom implementations
> ---
>
> Key: HADOOP-16011
> URL: https://issues.apache.org/jira/browse/HADOOP-16011
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Reporter: Todd Lipcon
>Assignee: Siyao Meng
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-16011.001.patch, HADOOP-16011.002.patch,
> MyBenchmark.java
>
>
> In looking at performance of a workload which creates a lot of short-lived
> remote connections to a secured DN, [~philip] and I found very high system
> CPU usage. We tracked it down to reads from /dev/random, which are incurred
> by the DN using CryptoCodec.generateSecureRandom to generate a transient
> session key and IV for AES encryption.
> In the case that the OpenSSL codec is not enabled, the above code falls
> through to the JDK SecureRandom implementation, which performs reasonably.
> However, OpenSSLCodec defaults to using OsSecureRandom, which reads all
> random data from /dev/random rather than doing something more efficient like
> initializing a CSPRNG from a small seed.
> I wrote a simple JMH benchmark to compare various approaches when running
> with concurrency 10:
> testHadoop - using CryptoCodec
> testNewSecureRandom - using 'new SecureRandom()' each iteration
> testSha1PrngNew - using the SHA1PRNG explicitly, new instance each iteration
> testSha1PrngShared - using a single shared instance of SHA1PRNG
> testSha1PrngThread - using a thread-specific instance of SHA1PRNG
> {code:java}
> Benchmark Mode CntScore Error Units
> MyBenchmark.testHadoop thrpt 1293.000 ops/s
> [with libhadoop.so]
> MyBenchmark.testHadoop thrpt461515.697 ops/s
> [without libhadoop.so]
> MyBenchmark.testNewSecureRandom thrpt 43413.640 ops/s
> MyBenchmark.testSha1PrngNew thrpt395515.000 ops/s
> MyBenchmark.testSha1PrngShared thrpt164488.713 ops/s
> MyBenchmark.testSha1PrngThread thrpt 4295123.210 ops/s
> {code}
> In other words, the presence of the OpenSSL acceleration slows down this code
> path by 356x. And, compared to the optimal (thread-local Sha1Prng) it's 3321x
> slower.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16011) OsSecureRandom very slow compared to other SecureRandom implementations
[
https://issues.apache.org/jira/browse/HADOOP-16011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809296#comment-16809296
]
Hudson commented on HADOOP-16011:
-
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16343 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/16343/])
HADOOP-16011. OsSecureRandom very slow compared to other SecureRandom (weichiu:
rev e62cbcbc83026a7af43eac6223fe53f9de963d91)
* (edit)
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/OpensslAesCtrCryptoCodec.java
* (edit) hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
> OsSecureRandom very slow compared to other SecureRandom implementations
> ---
>
> Key: HADOOP-16011
> URL: https://issues.apache.org/jira/browse/HADOOP-16011
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Reporter: Todd Lipcon
>Assignee: Siyao Meng
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-16011.001.patch, HADOOP-16011.002.patch,
> MyBenchmark.java
>
>
> In looking at performance of a workload which creates a lot of short-lived
> remote connections to a secured DN, [~philip] and I found very high system
> CPU usage. We tracked it down to reads from /dev/random, which are incurred
> by the DN using CryptoCodec.generateSecureRandom to generate a transient
> session key and IV for AES encryption.
> In the case that the OpenSSL codec is not enabled, the above code falls
> through to the JDK SecureRandom implementation, which performs reasonably.
> However, OpenSSLCodec defaults to using OsSecureRandom, which reads all
> random data from /dev/random rather than doing something more efficient like
> initializing a CSPRNG from a small seed.
> I wrote a simple JMH benchmark to compare various approaches when running
> with concurrency 10:
> testHadoop - using CryptoCodec
> testNewSecureRandom - using 'new SecureRandom()' each iteration
> testSha1PrngNew - using the SHA1PRNG explicitly, new instance each iteration
> testSha1PrngShared - using a single shared instance of SHA1PRNG
> testSha1PrngThread - using a thread-specific instance of SHA1PRNG
> {code:java}
> Benchmark Mode CntScore Error Units
> MyBenchmark.testHadoop thrpt 1293.000 ops/s
> [with libhadoop.so]
> MyBenchmark.testHadoop thrpt461515.697 ops/s
> [without libhadoop.so]
> MyBenchmark.testNewSecureRandom thrpt 43413.640 ops/s
> MyBenchmark.testSha1PrngNew thrpt395515.000 ops/s
> MyBenchmark.testSha1PrngShared thrpt164488.713 ops/s
> MyBenchmark.testSha1PrngThread thrpt 4295123.210 ops/s
> {code}
> In other words, the presence of the OpenSSL acceleration slows down this code
> path by 356x. And, compared to the optimal (thread-local Sha1Prng) it's 3321x
> slower.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16011) OsSecureRandom very slow compared to other SecureRandom implementations
[
https://issues.apache.org/jira/browse/HADOOP-16011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16808695#comment-16808695
]
Wei-Chiu Chuang commented on HADOOP-16011:
--
+1
The change makes perfect sense to me. When OpenSSL is available and
OpensslAesCtrCryptoCodec is used, OpenSslSecureRandom should also be used.
[~smeng] any chance you can offer an updated benchmark number after this change?
> OsSecureRandom very slow compared to other SecureRandom implementations
> ---
>
> Key: HADOOP-16011
> URL: https://issues.apache.org/jira/browse/HADOOP-16011
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Reporter: Todd Lipcon
>Assignee: Siyao Meng
>Priority: Major
> Attachments: HADOOP-16011.001.patch, HADOOP-16011.002.patch,
> MyBenchmark.java
>
>
> In looking at performance of a workload which creates a lot of short-lived
> remote connections to a secured DN, [~philip] and I found very high system
> CPU usage. We tracked it down to reads from /dev/random, which are incurred
> by the DN using CryptoCodec.generateSecureRandom to generate a transient
> session key and IV for AES encryption.
> In the case that the OpenSSL codec is not enabled, the above code falls
> through to the JDK SecureRandom implementation, which performs reasonably.
> However, OpenSSLCodec defaults to using OsSecureRandom, which reads all
> random data from /dev/random rather than doing something more efficient like
> initializing a CSPRNG from a small seed.
> I wrote a simple JMH benchmark to compare various approaches when running
> with concurrency 10:
> testHadoop - using CryptoCodec
> testNewSecureRandom - using 'new SecureRandom()' each iteration
> testSha1PrngNew - using the SHA1PRNG explicitly, new instance each iteration
> testSha1PrngShared - using a single shared instance of SHA1PRNG
> testSha1PrngThread - using a thread-specific instance of SHA1PRNG
> {code:java}
> Benchmark Mode CntScore Error Units
> MyBenchmark.testHadoop thrpt 1293.000 ops/s
> [with libhadoop.so]
> MyBenchmark.testHadoop thrpt461515.697 ops/s
> [without libhadoop.so]
> MyBenchmark.testNewSecureRandom thrpt 43413.640 ops/s
> MyBenchmark.testSha1PrngNew thrpt395515.000 ops/s
> MyBenchmark.testSha1PrngShared thrpt164488.713 ops/s
> MyBenchmark.testSha1PrngThread thrpt 4295123.210 ops/s
> {code}
> In other words, the presence of the OpenSSL acceleration slows down this code
> path by 356x. And, compared to the optimal (thread-local Sha1Prng) it's 3321x
> slower.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16011) OsSecureRandom very slow compared to other SecureRandom implementations
[
https://issues.apache.org/jira/browse/HADOOP-16011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16808211#comment-16808211
]
Hadoop QA commented on HADOOP-16011:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
26s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 23m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 55s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
12s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 22m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 22m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 40s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
43s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
45s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}122m 5s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HADOOP-16011 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12964617/HADOOP-16011.002.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux fa4fb672e1b6 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / cf26811 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16106/testReport/ |
| Max. process+thread count | 1347 (vs. ulimit of 1) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
[jira] [Commented] (HADOOP-16011) OsSecureRandom very slow compared to other SecureRandom implementations
[
https://issues.apache.org/jira/browse/HADOOP-16011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805929#comment-16805929
]
Wei-Chiu Chuang commented on HADOOP-16011:
--
This one
{code}
hadoop.security.secure.random.impl
Implementation of secure random.
{code}
> OsSecureRandom very slow compared to other SecureRandom implementations
> ---
>
> Key: HADOOP-16011
> URL: https://issues.apache.org/jira/browse/HADOOP-16011
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Reporter: Todd Lipcon
>Assignee: Siyao Meng
>Priority: Major
> Attachments: HADOOP-16011.001.patch, MyBenchmark.java
>
>
> In looking at performance of a workload which creates a lot of short-lived
> remote connections to a secured DN, [~philip] and I found very high system
> CPU usage. We tracked it down to reads from /dev/random, which are incurred
> by the DN using CryptoCodec.generateSecureRandom to generate a transient
> session key and IV for AES encryption.
> In the case that the OpenSSL codec is not enabled, the above code falls
> through to the JDK SecureRandom implementation, which performs reasonably.
> However, OpenSSLCodec defaults to using OsSecureRandom, which reads all
> random data from /dev/random rather than doing something more efficient like
> initializing a CSPRNG from a small seed.
> I wrote a simple JMH benchmark to compare various approaches when running
> with concurrency 10:
> testHadoop - using CryptoCodec
> testNewSecureRandom - using 'new SecureRandom()' each iteration
> testSha1PrngNew - using the SHA1PRNG explicitly, new instance each iteration
> testSha1PrngShared - using a single shared instance of SHA1PRNG
> testSha1PrngThread - using a thread-specific instance of SHA1PRNG
> {code:java}
> Benchmark Mode CntScore Error Units
> MyBenchmark.testHadoop thrpt 1293.000 ops/s
> [with libhadoop.so]
> MyBenchmark.testHadoop thrpt461515.697 ops/s
> [without libhadoop.so]
> MyBenchmark.testNewSecureRandom thrpt 43413.640 ops/s
> MyBenchmark.testSha1PrngNew thrpt395515.000 ops/s
> MyBenchmark.testSha1PrngShared thrpt164488.713 ops/s
> MyBenchmark.testSha1PrngThread thrpt 4295123.210 ops/s
> {code}
> In other words, the presence of the OpenSSL acceleration slows down this code
> path by 356x. And, compared to the optimal (thread-local Sha1Prng) it's 3321x
> slower.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16011) OsSecureRandom very slow compared to other SecureRandom implementations
[
https://issues.apache.org/jira/browse/HADOOP-16011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16804354#comment-16804354
]
Siyao Meng commented on HADOOP-16011:
-
[~jojochuang] I see no reference of OsSecureRandom in core-default.xml, only
OpensslAesCtrCryptoCodec here:
{code:xml}
hadoop.security.crypto.codec.classes.aes.ctr.nopadding
org.apache.hadoop.crypto.OpensslAesCtrCryptoCodec,
org.apache.hadoop.crypto.JceAesCtrCryptoCodec
Comma-separated list of crypto codec implementations for AES/CTR/NoPadding.
The first implementation will be used if available, others are fallbacks.
{code}
What change do we need here?
> OsSecureRandom very slow compared to other SecureRandom implementations
> ---
>
> Key: HADOOP-16011
> URL: https://issues.apache.org/jira/browse/HADOOP-16011
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Reporter: Todd Lipcon
>Assignee: Siyao Meng
>Priority: Major
> Attachments: HADOOP-16011.001.patch, MyBenchmark.java
>
>
> In looking at performance of a workload which creates a lot of short-lived
> remote connections to a secured DN, [~philip] and I found very high system
> CPU usage. We tracked it down to reads from /dev/random, which are incurred
> by the DN using CryptoCodec.generateSecureRandom to generate a transient
> session key and IV for AES encryption.
> In the case that the OpenSSL codec is not enabled, the above code falls
> through to the JDK SecureRandom implementation, which performs reasonably.
> However, OpenSSLCodec defaults to using OsSecureRandom, which reads all
> random data from /dev/random rather than doing something more efficient like
> initializing a CSPRNG from a small seed.
> I wrote a simple JMH benchmark to compare various approaches when running
> with concurrency 10:
> testHadoop - using CryptoCodec
> testNewSecureRandom - using 'new SecureRandom()' each iteration
> testSha1PrngNew - using the SHA1PRNG explicitly, new instance each iteration
> testSha1PrngShared - using a single shared instance of SHA1PRNG
> testSha1PrngThread - using a thread-specific instance of SHA1PRNG
> {code:java}
> Benchmark Mode CntScore Error Units
> MyBenchmark.testHadoop thrpt 1293.000 ops/s
> [with libhadoop.so]
> MyBenchmark.testHadoop thrpt461515.697 ops/s
> [without libhadoop.so]
> MyBenchmark.testNewSecureRandom thrpt 43413.640 ops/s
> MyBenchmark.testSha1PrngNew thrpt395515.000 ops/s
> MyBenchmark.testSha1PrngShared thrpt164488.713 ops/s
> MyBenchmark.testSha1PrngThread thrpt 4295123.210 ops/s
> {code}
> In other words, the presence of the OpenSSL acceleration slows down this code
> path by 356x. And, compared to the optimal (thread-local Sha1Prng) it's 3321x
> slower.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16011) OsSecureRandom very slow compared to other SecureRandom implementations
[
https://issues.apache.org/jira/browse/HADOOP-16011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16804267#comment-16804267
]
Hadoop QA commented on HADOOP-16011:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 57s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 55s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
36s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
43s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 90m 14s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HADOOP-16011 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12964057/HADOOP-16011.001.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux b2114a3453b3 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 49b02d4 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16082/testReport/ |
| Max. process+thread count | 1451 (vs. ulimit of 1) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16082/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> OsSecureRandom very slow
[jira] [Commented] (HADOOP-16011) OsSecureRandom very slow compared to other SecureRandom implementations
[
https://issues.apache.org/jira/browse/HADOOP-16011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16804166#comment-16804166
]
Wei-Chiu Chuang commented on HADOOP-16011:
--
I think you'll also need to update core-default.xml
also, update the release note too.
> OsSecureRandom very slow compared to other SecureRandom implementations
> ---
>
> Key: HADOOP-16011
> URL: https://issues.apache.org/jira/browse/HADOOP-16011
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Reporter: Todd Lipcon
>Assignee: Siyao Meng
>Priority: Major
> Attachments: HADOOP-16011.001.patch, MyBenchmark.java
>
>
> In looking at performance of a workload which creates a lot of short-lived
> remote connections to a secured DN, [~philip] and I found very high system
> CPU usage. We tracked it down to reads from /dev/random, which are incurred
> by the DN using CryptoCodec.generateSecureRandom to generate a transient
> session key and IV for AES encryption.
> In the case that the OpenSSL codec is not enabled, the above code falls
> through to the JDK SecureRandom implementation, which performs reasonably.
> However, OpenSSLCodec defaults to using OsSecureRandom, which reads all
> random data from /dev/random rather than doing something more efficient like
> initializing a CSPRNG from a small seed.
> I wrote a simple JMH benchmark to compare various approaches when running
> with concurrency 10:
> testHadoop - using CryptoCodec
> testNewSecureRandom - using 'new SecureRandom()' each iteration
> testSha1PrngNew - using the SHA1PRNG explicitly, new instance each iteration
> testSha1PrngShared - using a single shared instance of SHA1PRNG
> testSha1PrngThread - using a thread-specific instance of SHA1PRNG
> {code:java}
> Benchmark Mode CntScore Error Units
> MyBenchmark.testHadoop thrpt 1293.000 ops/s
> [with libhadoop.so]
> MyBenchmark.testHadoop thrpt461515.697 ops/s
> [without libhadoop.so]
> MyBenchmark.testNewSecureRandom thrpt 43413.640 ops/s
> MyBenchmark.testSha1PrngNew thrpt395515.000 ops/s
> MyBenchmark.testSha1PrngShared thrpt164488.713 ops/s
> MyBenchmark.testSha1PrngThread thrpt 4295123.210 ops/s
> {code}
> In other words, the presence of the OpenSSL acceleration slows down this code
> path by 356x. And, compared to the optimal (thread-local Sha1Prng) it's 3321x
> slower.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16011) OsSecureRandom very slow compared to other SecureRandom implementations
[
https://issues.apache.org/jira/browse/HADOOP-16011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16722048#comment-16722048
]
Todd Lipcon commented on HADOOP-16011:
--
I think that would probably make sense. I didn't get a chance to benchmark that
implementation but I would guess it is much faster!
> OsSecureRandom very slow compared to other SecureRandom implementations
> ---
>
> Key: HADOOP-16011
> URL: https://issues.apache.org/jira/browse/HADOOP-16011
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Reporter: Todd Lipcon
>Priority: Major
> Attachments: MyBenchmark.java
>
>
> In looking at performance of a workload which creates a lot of short-lived
> remote connections to a secured DN, [~philip] and I found very high system
> CPU usage. We tracked it down to reads from /dev/random, which are incurred
> by the DN using CryptoCodec.generateSecureRandom to generate a transient
> session key and IV for AES encryption.
> In the case that the OpenSSL codec is not enabled, the above code falls
> through to the JDK SecureRandom implementation, which performs reasonably.
> However, OpenSSLCodec defaults to using OsSecureRandom, which reads all
> random data from /dev/random rather than doing something more efficient like
> initializing a CSPRNG from a small seed.
> I wrote a simple JMH benchmark to compare various approaches when running
> with concurrency 10:
> testHadoop - using CryptoCodec
> testNewSecureRandom - using 'new SecureRandom()' each iteration
> testSha1PrngNew - using the SHA1PRNG explicitly, new instance each iteration
> testSha1PrngShared - using a single shared instance of SHA1PRNG
> testSha1PrngThread - using a thread-specific instance of SHA1PRNG
> {code:java}
> Benchmark Mode CntScore Error Units
> MyBenchmark.testHadoop thrpt 1293.000 ops/s
> [with libhadoop.so]
> MyBenchmark.testHadoop thrpt461515.697 ops/s
> [without libhadoop.so]
> MyBenchmark.testNewSecureRandom thrpt 43413.640 ops/s
> MyBenchmark.testSha1PrngNew thrpt395515.000 ops/s
> MyBenchmark.testSha1PrngShared thrpt164488.713 ops/s
> MyBenchmark.testSha1PrngThread thrpt 4295123.210 ops/s
> {code}
> In other words, the presence of the OpenSSL acceleration slows down this code
> path by 356x. And, compared to the optimal (thread-local Sha1Prng) it's 3321x
> slower.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16011) OsSecureRandom very slow compared to other SecureRandom implementations
[
https://issues.apache.org/jira/browse/HADOOP-16011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16722035#comment-16722035
]
Wei-Chiu Chuang commented on HADOOP-16011:
--
Hey Todd, I noticed that too before, and I don't get why OpenSSLCodec uses
OsSecureRandom by default. Should we simply default OpenSSLCodec to to use
OpensslSecureRandom?
> OsSecureRandom very slow compared to other SecureRandom implementations
> ---
>
> Key: HADOOP-16011
> URL: https://issues.apache.org/jira/browse/HADOOP-16011
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Reporter: Todd Lipcon
>Priority: Major
> Attachments: MyBenchmark.java
>
>
> In looking at performance of a workload which creates a lot of short-lived
> remote connections to a secured DN, [~philip] and I found very high system
> CPU usage. We tracked it down to reads from /dev/random, which are incurred
> by the DN using CryptoCodec.generateSecureRandom to generate a transient
> session key and IV for AES encryption.
> In the case that the OpenSSL codec is not enabled, the above code falls
> through to the JDK SecureRandom implementation, which performs reasonably.
> However, OpenSSLCodec defaults to using OsSecureRandom, which reads all
> random data from /dev/random rather than doing something more efficient like
> initializing a CSPRNG from a small seed.
> I wrote a simple JMH benchmark to compare various approaches when running
> with concurrency 10:
> testHadoop - using CryptoCodec
> testNewSecureRandom - using 'new SecureRandom()' each iteration
> testSha1PrngNew - using the SHA1PRNG explicitly, new instance each iteration
> testSha1PrngShared - using a single shared instance of SHA1PRNG
> testSha1PrngThread - using a thread-specific instance of SHA1PRNG
> {code:java}
> Benchmark Mode CntScore Error Units
> MyBenchmark.testHadoop thrpt 1293.000 ops/s
> [with libhadoop.so]
> MyBenchmark.testHadoop thrpt461515.697 ops/s
> [without libhadoop.so]
> MyBenchmark.testNewSecureRandom thrpt 43413.640 ops/s
> MyBenchmark.testSha1PrngNew thrpt395515.000 ops/s
> MyBenchmark.testSha1PrngShared thrpt164488.713 ops/s
> MyBenchmark.testSha1PrngThread thrpt 4295123.210 ops/s
> {code}
> In other words, the presence of the OpenSSL acceleration slows down this code
> path by 356x. And, compared to the optimal (thread-local Sha1Prng) it's 3321x
> slower.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
