Re: RFR 8066397 Remove network-related seed initialization code in ThreadLocal/SplittableRandom
the Microsoft C Runtime Library makes use of this function in its implementation of rand_s, so it's removal would break a lot of programs. I think this is a relative guarantee that the function is here to stay. Ok. What are the fallbacks for SystemRandomImpl if /dev/urandom or the rtlGenRandomFN/CryptGenRandom aren't available? Is that something you'll bake into TLR or will you do it here? I think it's better to leave it to consumers (TLR/SplittableRandom) as they know what's good-enough for them. The API allows for arbitrary number of bytes to be generated and I don't have an easy means of generating more than 8 random bytes just from System.nanoTime() and System.currentTimeMillis() short of using SecureRandom as a fall-back. webrev.03 only has new code, no changes yet to TLR/SplittableRandom, so I'm not yet quite following where TLR/SR will be changing. Also, what is proposed for platforms that aren't Unix/Windows? Should there be a generic fallback mechanism like ThreadedSeedGenerator? (Note, I'm not suggesting using it, it's rather...SSLLLOOO...) Are there other non-Unix and non-Windows platforms? I'm not an expert in non-Oracle Java offerings, but what about the various Java ME devices? Some of the embedded devices are Linux/Windows, but are there others? JavaCard? I know IBM has a large number of OS's they support, for example IBM i. http://en.wikipedia.org/wiki/IBM_i https://www.ibm.com/developerworks/community/wikis/home?lang=en#/wiki/IBM%20i%20Technology%20Updates/page/Java%20on%20IBM%20i I know IBM supports Java on z/OS, but apparently that uses a Linux-style filesystem via UNIX_System_Services. http://en.wikipedia.org/?title=Z/OS Not sure about others. I saw a couple of other non-unix/windows OSs in a wikipedia comparison of JVM's, but most of them seem to be discontinued. I think the planned fall-back for TLR/SplittableRandom is to just use System.currentTimeMillis() System.nanoTime() - these are the defaults now unless SecureRandom is requested. Ok. Since this isn't security-critical, IMHO there just needs to be a reasonable fall-back option in case /dev/urandom or rtlGenRandomFN can't be used. I don't always closely monitor core-libs-dev, so please cc me if further discussion occurs in a different thread. Cheers, and HNY to you and everyone else here, Brad
Re: RFR 8066397 Remove network-related seed initialization code in ThreadLocal/SplittableRandom
Hi Brad, On 01/01/2015 02:46 AM, Bradford Wetmore wrote: To the actual proposal: http://cr.openjdk.java.net/~plevart/jdk9-dev/SystemRandom/webrev.03/ Overall, I'm ok with what's proposed. This is more straightforward to parse/understand than trying to adjust NativeSeedGenerator to create/call directly (e.g. UNIX: new NativeSeedGenerator(/dev/urandom) or Windows: new NativeSeedGenerator()). But I'd still like to understand why you moved away from this. One concern is that you're duplicating native libraries in java.base, and it would be the third JDK library overall with this type of call. There's one in libjava (for java.base/WinCAPISeedGenerator for sun.security.provider.NativeSeedGenerator) and sunmscapi (for jdk.crypto.mscapi/SunMSCAPI/sun.security.mscapi). Would it work to tweak the WinCAPISeedGenerator so you don't have to create a new dll for java.base? The SystemRandom JNI bindings for Windows are located in: java.base/windows/native/libjava/SystemRandomImpl_md.c ...so as I understand they are also part of libjava. No new DLL here. True. My thought should have been about having very similar code duplicated in libjava. I'm ok with this, though it's not really clear if/when MS will drop support for ADVAPI32!RtlGenRandom. This always makes me nervous because whatever is put in will likely never change until some MS change breaks it. the Microsoft C Runtime Library makes use of this function in its implementation of rand_s, so it's removal would break a lot of programs. I think this is a relative guarantee that the function is here to stay. What are the fallbacks for SystemRandomImpl if /dev/urandom or the rtlGenRandomFN/CryptGenRandom aren't available? Is that something you'll bake into TLR or will you do it here? I think it's better to leave it to consumers (TLR/SplittableRandom) as they know what's good-enough for them. The API allows for arbitrary number of bytes to be generated and I don't have an easy means of generating more than 8 random bytes just from System.nanoTime() and System.currentTimeMillis() short of using SecureRandom as a fall-back. webrev.03 only has new code, no changes yet to TLR/SplittableRandom, so I'm not yet quite following where TLR/SR will be changing. Also, what is proposed for platforms that aren't Unix/Windows? Should there be a generic fallback mechanism like ThreadedSeedGenerator? (Note, I'm not suggesting using it, it's rather...SSLLLOOO...) Are there other non-Unix and non-Windows platforms? I think the planned fall-back for TLR/SplittableRandom is to just use System.currentTimeMillis() System.nanoTime() - these are the defaults now unless SecureRandom is requested. Regards, Peter
Re: RFR 8066397 Remove network-related seed initialization code in ThreadLocal/SplittableRandom
To the actual proposal: http://cr.openjdk.java.net/~plevart/jdk9-dev/SystemRandom/webrev.03/ Overall, I'm ok with what's proposed. This is more straightforward to parse/understand than trying to adjust NativeSeedGenerator to create/call directly (e.g. UNIX: new NativeSeedGenerator(/dev/urandom) or Windows: new NativeSeedGenerator()). But I'd still like to understand why you moved away from this. One concern is that you're duplicating native libraries in java.base, and it would be the third JDK library overall with this type of call. There's one in libjava (for java.base/WinCAPISeedGenerator for sun.security.provider.NativeSeedGenerator) and sunmscapi (for jdk.crypto.mscapi/SunMSCAPI/sun.security.mscapi). Would it work to tweak the WinCAPISeedGenerator so you don't have to create a new dll for java.base? The SystemRandom JNI bindings for Windows are located in: java.base/windows/native/libjava/SystemRandomImpl_md.c ...so as I understand they are also part of libjava. No new DLL here. True. My thought should have been about having very similar code duplicated in libjava. I'm ok with this, though it's not really clear if/when MS will drop support for ADVAPI32!RtlGenRandom. This always makes me nervous because whatever is put in will likely never change until some MS change breaks it. I'm not familiar with what Alan/Mandy/company have in mind down the road, but I haven't heard of libjava splitting. What are the fallbacks for SystemRandomImpl if /dev/urandom or the rtlGenRandomFN/CryptGenRandom aren't available? Is that something you'll bake into TLR or will you do it here? I think it's better to leave it to consumers (TLR/SplittableRandom) as they know what's good-enough for them. The API allows for arbitrary number of bytes to be generated and I don't have an easy means of generating more than 8 random bytes just from System.nanoTime() and System.currentTimeMillis() short of using SecureRandom as a fall-back. webrev.03 only has new code, no changes yet to TLR/SplittableRandom, so I'm not yet quite following where TLR/SR will be changing. Also, what is proposed for platforms that aren't Unix/Windows? Should there be a generic fallback mechanism like ThreadedSeedGenerator? (Note, I'm not suggesting using it, it's rather...SSLLLOOO...) The problem is also how to make access to this functionality for different consumers that are located in different packages (java.util, java.util.concurrent) and make it somehow usable also for external access. There is a desire to use this also from stand-alone builds of java.util.concurrent utilities. That's why my initial approach for SystemRandom used a public API in java.util. The approach used with sun.misc.Unsafe is probably not going to work for user code in JDK9 with modules, as sun.misc will not be globally exported. Are any non J2SE packages going to be globally exported? I see jdk and jdk.net are already mentioned as such globally exported packages in modules.xml... That's a good question for Alan/Mandy/company. Martin wrote: https://bugs.openjdk.java.net/browse/JDK-8047769 If you've been following this bug, I've figured why the NativePRNG$* classes are initing and thus opening the /dev/random,urandom. This definitely needs some adjustment. Something like the following could be used in NativePRNG and URLSeedGenerator: http://cr.openjdk.java.net/~plevart/misc/FileInputStreamPool/FileInputStreamPool.java (See the other active thread in core-libs-dev.) Brad
Re: RFR 8066397 Remove network-related seed initialization code in ThreadLocal/SplittableRandom
Hi Brad, I am glad that you came across all the threads on this topic. I agree with your evaluation of my comments from an early thread. I have gone through security providers infrastructure multiple times since then and I can see where my comments were incorrect. The confusion was mostly based around mixing SecureRandom.generateSeed () with SecureRandom.nextBytes () and their underlying implementations for different providers. Now to answer inline... On 12/17/2014 02:36 AM, Bradford Wetmore wrote: Various comments for this thread from June/July/November/December. Some of the comments I'm responding to may already be better understood than when they were originally written. Peter wrote in response to a suggestion to use /dev/random: Although the approach would cause some more classes to load, no arbitrary providers should be initialized. I think this is waht you get when you set java.util.secureRandomSeed system property to true. TLR uses java.security.SecureRandom.getSeed(8) in this case. For the no arbitrary provider part, that may not be quite correct. getSeed() creates/pulls from the default SecureRandom impl (i.e. new SecureRandom().generateSeed()), so it pulls in the Security Provider mechanism to determine the most preferred implementation, which could initialize additional higher-priority providers until an instance of SecureRandom is found. For example, ucrypto on Solaris doesn't have a SecureRandom impl, so it would then fall back to PKCS11. As has been pointed out, the various Oracle SecureRandom implementations and their preference order are a twisty maze of passages, somewhat but not exactly alike. (With apologies to the Colossal Cave.) The default preference order is: Solaris (sparc/sparcv9/x86/x64) PKCS11 - SunPKCS11 NativePRNG - Sun SHA1PRNG - Sun NativePRNGBlocking - Sun NativePRNGNonBlocking - Sun Linux (x86/x64)/MacOS NativePRNG - Sun SHA1PRNG - Sun NativePRNGBlocking - Sun NativePRNGNonBlocking - Sun Windows (x86/x64) SHA1PRNG - Sun Windows-PRNG - SunMSCAPI Here's a few impl details for seeding calls. PKCS11: --- generateSeed() routes to engineNextBytes(), which goes to the underlying PKCS11. NativePRNG: (Unix-only) --- generateSeed() by default routes to /dev/random, unless the System Entropy Gathering Device (EGD) (set via a Security/System property) routes to something else. (FYI: nextBytes() uses /dev/urandom.) NativePRNG$BLOCKING: (Unix-only) generateSeed() always routes to /dev/random. (FYI: nextBytes() uses /dev/random.) NativePRNG$NONBLOCKING: (Unix-only) --- generateSeed() always routes to /dev/urandom. (FYI: nextBytes() uses /dev/urandom.) SHA1PRNG: - generateSeed() depends on the value of the EGD: default EGD is: /dev/random Note: if string /dev/urandom is set, urandom is used instead. Unix: generateSeed() routes to /dev/random (NativeSeedGenerator: pure java) Win: generateSeed() routes to CryptGenRandom (NativeSeedGenerator + libjava native) non-default (not /dev/random//dev/urandom): URL is specified UNIX/Win: routes to URL If both above fail: falls back to ThreadedSeedGenerator (Pure Java) Windows-PRNG generateSeed() routes to mscapi.PRNG/CryptGenRandom Note uses libmscapi, not libjava. That's right. Different defaults on different platforms and a possibility to configure preferential custom providers makes the choices for implementation of SecureRandom.getSeed() static method (which uses 1st SecureRandom provider in the providers list) quite diverse. The problem I see here is different default behaviour depending on platform. A user can make a choice which SecureRandom algorithm the application code uses by explicitly requesting it (with SecureRandom.getInstance(algorithm)), but she can't choose the algorithm when she decides to use SecureRandom for initial seeding of TLR/SplittableRandom. By default on Unix you get a /dev/random kind of implementation for generateSeed() which is blocking on Linux. On Linux one would probably want to use NativePRNGNonBlocking which uses /dev/urandom for generateSeed(). One way to solve this is to extend the meaning of java.util.secureRandomSeed system property - besides true which would choose the 1st provider, one could specify the algorithm name. For example, on Unix one would choose: java.util.secureRandomSeed=NativePRNGNonBlocking to get a /dev/urandom based initial seed for TLR/SplittableRandom. There's also a possibility to hard-code an explicit lookup for particular algorithms and use the one with highest preference that is available with a fall-back on 1st (default) provider. For initial seeding of TLR/SplittableRandom, security is not important, but initialization latency is, so the preference order for choosing SecureRandom algorithm is different for
Re: RFR 8066397 Remove network-related seed initialization code in ThreadLocal/SplittableRandom
On 12/17/2014 06:37 PM, Peter Levart wrote: Martin wrote: https://bugs.openjdk.java.net/browse/JDK-8047769 If you've been following this bug, I've figured why the NativePRNG$* classes are initing and thus opening the /dev/random,urandom. This definitely needs some adjustment. Something like the following could be used in NativePRNG and URLSeedGenerator: http://cr.openjdk.java.net/~plevart/misc/FileInputStreamPool/FileInputStreamPool.java Note that this code has a nasty bug - don't use it. I'll post a fixed source later... Peter
Re: RFR 8066397 Remove network-related seed initialization code in ThreadLocal/SplittableRandom
Various comments for this thread from June/July/November/December. Some of the comments I'm responding to may already be better understood than when they were originally written. Peter wrote in response to a suggestion to use /dev/random: Although the approach would cause some more classes to load, no arbitrary providers should be initialized. I think this is waht you get when you set java.util.secureRandomSeed system property to true. TLR uses java.security.SecureRandom.getSeed(8) in this case. For the no arbitrary provider part, that may not be quite correct. getSeed() creates/pulls from the default SecureRandom impl (i.e. new SecureRandom().generateSeed()), so it pulls in the Security Provider mechanism to determine the most preferred implementation, which could initialize additional higher-priority providers until an instance of SecureRandom is found. For example, ucrypto on Solaris doesn't have a SecureRandom impl, so it would then fall back to PKCS11. As has been pointed out, the various Oracle SecureRandom implementations and their preference order are a twisty maze of passages, somewhat but not exactly alike. (With apologies to the Colossal Cave.) The default preference order is: Solaris (sparc/sparcv9/x86/x64) PKCS11 - SunPKCS11 NativePRNG - Sun SHA1PRNG - Sun NativePRNGBlocking - Sun NativePRNGNonBlocking - Sun Linux (x86/x64)/MacOS NativePRNG - Sun SHA1PRNG - Sun NativePRNGBlocking - Sun NativePRNGNonBlocking - Sun Windows (x86/x64) SHA1PRNG - Sun Windows-PRNG - SunMSCAPI Here's a few impl details for seeding calls. PKCS11: --- generateSeed() routes to engineNextBytes(), which goes to the underlying PKCS11. NativePRNG: (Unix-only) --- generateSeed() by default routes to /dev/random, unless the System Entropy Gathering Device (EGD) (set via a Security/System property) routes to something else. (FYI: nextBytes() uses /dev/urandom.) NativePRNG$BLOCKING: (Unix-only) generateSeed() always routes to /dev/random. (FYI: nextBytes() uses /dev/random.) NativePRNG$NONBLOCKING: (Unix-only) --- generateSeed() always routes to /dev/urandom. (FYI: nextBytes() uses /dev/urandom.) SHA1PRNG: - generateSeed() depends on the value of the EGD: default EGD is: /dev/random Note: if string /dev/urandom is set, urandom is used instead. Unix: generateSeed() routes to /dev/random (NativeSeedGenerator: pure java) Win: generateSeed() routes to CryptGenRandom (NativeSeedGenerator + libjava native) non-default (not /dev/random//dev/urandom): URL is specified UNIX/Win: routes to URL If both above fail: falls back to ThreadedSeedGenerator (Pure Java) Windows-PRNG generateSeed() routes to mscapi.PRNG/CryptGenRandom Note uses libmscapi, not libjava. Peter wrote: The most problematic one is the default on Windows platform (the platform that does not have the /dev/urandom special file and would be used as a fall-back by your proposal) - sun.security.provider.SecureRandom. This one seeds itself by constructing an instance of itself with the result returned from SeedGenerator.getSystemEntropy() method. This method, among other things, uses networking code to gather system entropy: SeedGenerator.getSystemEntropy() that includes the Network interfaces is only called when you need to seed the SHA1PRNG internal seeder in order to generate nextBytes(). generateSeed() doesn't trigger that internal seeder initialization. So as you can see above, SecureRandom.generateSeed(int) is really variable in what you might end up with, and how much cruft comes with it. For sure, the provider mechanism will be dragged in which is fairly substantial. Peter wrote: So by default yes, plain NativePRNG (the default on UNIX-es) is using /dev/urandom for nextBytes(), but this can be changed by defining java.security.egd or securerandom.source system property. EGD really only affects where Seed byes are obtained from, IIRC, nextBytes() is not generally affected by this value. It does tweak which implementation is most preferred within the Sun provider, but the majority of the effect is in Native/SHA1PRNG choice of generateSeed. The original suggestion back in June: http://mail.openjdk.java.net/pipermail/core-libs-dev/2014-June/027389.html http://cr.openjdk.java.net/~plevart/jdk9-dev/TLR_SR_SeedGenerator/webrev.01/ for directly calling into NativeSeedGenerator makes more sense if you want to avoid duplicating existing code and creating a new native libraries as in the current proposal (webrev.03). Your data shows that this approach pulls in a much smaller subset of classes than using the full SecureRandom.getInstance().generateSeed() API. I've gone through the threads a couple times now: somehow I've missed the rationale for why you're moving away from this (.01) for webrev.03. To
Re: RFR 8066397 Remove network-related seed initialization code in ThreadLocal/SplittableRandom
Hi Doug, On 12/06/2014 05:14 PM, Doug Lea wrote: On 12/04/2014 07:22 PM, Doug Lea wrote: Because Random, SplittableRandom, and ThreadLocalRandom all use the same basic approach, they can/should use the same mechanism. In other words, to establish a common default-constructor-seed generator internal to the JDK. So we only need one environmental seed to start it off, and can remove some redundancies. In jdk9 with modules you could move the seed generator into a public class in a package that is not exported from java.base module (for example: java.util.concurrent.internal). Standalone j.u.c builds can be unchanged then. Placing the mechanics in TLR seems to work out best. Well, except that on Windows, we'd still need to create a .dll just for the sake of calling CryptGenRandom (or RtlGenRandom?), which seems like a bad move all around. If we are trying to reduce initialization costs, it would be better and faster if this were integrated with other platform-specific JVM startup. (Requiring a separate .dll also makes it impossible for us to create standalone j.u.c builds for previews etc.) Which brings us back to establishing some means of getting from the JVM, only once, 8 bytes of environmentally-derived bits for a within-JDK caller. Doing this might be a good opportunity for exploiting any upcoming modularity support. Or short of that, placing it in an existing JVM interface class like Unsafe. Any thoughts? What about if we change some existing J2SE API to be more resource/initialization-friendly. java.util.UUID currently uses SecureRandom. It could be changed to use /dev/urandom or RtlGenRandom instead (with a fall-back to non-blocking SecureRandom). Each random UUID instance contains 16 cryptographically strong random bytes (minus 6 bits that are overwritten with UUID variant/version bits). Regards, Peter (Also, at some point we might reconsider our cowardice about not improving the internal java.util.Random algorithm. j.u.Random is much more commonly used, and does not fare well on quality tests. On the other hand, the more that users instead choose to use SR or TLR, the better.) The main problem is code (not just JDK test code) that hardwires expected Random.next* output under given seeds. Which might be enough reason to leave it alone. Do any CCC members have an opinion? As a lesser but still worthwhile target though, I can't think of a reason not to change java.lang.Math.random() to use TLR to get faster and better-quality values. (Because there is no way to explicitly seed Math.random, there is no hardwiring problem.) Some existing javadoc for it saying that it creates a new java.util.Random() should be deleted, although I don't think it has any user-visible impact. http://docs.oracle.com/javase/8/docs/api/java/lang/Math.html#random-- -Doug
Re: RFR 8066397 Remove network-related seed initialization code in ThreadLocal/SplittableRandom
On 12/04/2014 07:22 PM, Doug Lea wrote: Because Random, SplittableRandom, and ThreadLocalRandom all use the same basic approach, they can/should use the same mechanism. In other words, to establish a common default-constructor-seed generator internal to the JDK. So we only need one environmental seed to start it off, and can remove some redundancies. Placing the mechanics in TLR seems to work out best. Well, except that on Windows, we'd still need to create a .dll just for the sake of calling CryptGenRandom (or RtlGenRandom?), which seems like a bad move all around. If we are trying to reduce initialization costs, it would be better and faster if this were integrated with other platform-specific JVM startup. (Requiring a separate .dll also makes it impossible for us to create standalone j.u.c builds for previews etc.) Which brings us back to establishing some means of getting from the JVM, only once, 8 bytes of environmentally-derived bits for a within-JDK caller. Doing this might be a good opportunity for exploiting any upcoming modularity support. Or short of that, placing it in an existing JVM interface class like Unsafe. Any thoughts? (Also, at some point we might reconsider our cowardice about not improving the internal java.util.Random algorithm. j.u.Random is much more commonly used, and does not fare well on quality tests. On the other hand, the more that users instead choose to use SR or TLR, the better.) The main problem is code (not just JDK test code) that hardwires expected Random.next* output under given seeds. Which might be enough reason to leave it alone. Do any CCC members have an opinion? As a lesser but still worthwhile target though, I can't think of a reason not to change java.lang.Math.random() to use TLR to get faster and better-quality values. (Because there is no way to explicitly seed Math.random, there is no hardwiring problem.) Some existing javadoc for it saying that it creates a new java.util.Random() should be deleted, although I don't think it has any user-visible impact. http://docs.oracle.com/javase/8/docs/api/java/lang/Math.html#random-- -Doug
Re: RFR 8066397 Remove network-related seed initialization code in ThreadLocal/SplittableRandom
On 12/04/2014 12:32 AM, Martin Buchholz wrote: On Wed, Dec 3, 2014 at 2:15 PM, Doug Lea d...@cs.oswego.edu wrote: No public API because systemSeed need only be implemented inside TLR, for its initial seed. Then the others can get their seeds using ThreadLocalRandom.current().nextLong(), unless java.util.secureRandomSeed is set (which I didn't illustrate above). In other words, across all non-secure generators, you only need one system-generated seed. That's good enough for seeding other non-cryptographically secure PRNGs, but if you want each caller to get a cryptographically secure random number, you need to avoid correlations between them that would arise when you use a non-CS PRNG to generate them from a single CS seed. Unless java.util.Random is retrofitted to allocate new cryptographically secure seed for each new instance. In such case expression: new java.util.Random().nextLong() ...could be used to gather secure seed. Perhaps even SplittableRandom could allocate new seed from secure source for each new instance (only in public constructor - not when it is split()ed)... What makes those workarounds unsuitable is failure mode. Gathering secure seed is inherently coupled with possible failure which must be communicated explicitly to the consumer. But one must not be bothered with failure in situations where security is not a necessary ingredient. So for non-CS PRNGs, cross-seeding is a possible solution, but it's nicer for all of them to just use a common (internal) API. As far as public API is concerned, there already is one: SecureRandom.generateSeed(). It's as good as SystemRandom (uses same primary means) and has a nice cross-platform fallback (ThreadedSeedGenerator). It's only drawback is that it comes with all the baggage of security providers (Java and MS Crypto API on Windows). But that's ok for user code perhaps. So what we have here is two desires: - we want a resource-friendly / with as little dependencies as possible way to generate some unique seed, with implicit fall-back which need not be secure - we want a resource-friendly / with as little dependencies as possible way to generate secure random bytes that can be pseudo-random, but still secure, with explicit failure mode I'll try to address this dichotomy in the next iteration of the API. Regards, Peter P.S. Is anyone interested in generating truly random bytes? http://www.cryptography.com/public/pdf/Intel_TRNG_Report_20120312.pdf https://software.intel.com/en-us/tags/35970
Re: RFR 8066397 Remove network-related seed initialization code in ThreadLocal/SplittableRandom
On 12/04/2014 09:20 AM, Peter Levart wrote: P.S. Is anyone interested in generating truly random bytes? http://www.cryptography.com/public/pdf/Intel_TRNG_Report_20120312.pdf https://software.intel.com/en-us/tags/35970 Maybe not a good idea: http://en.wikipedia.org/wiki/RdRand#Reception
Re: RFR 8066397 Remove network-related seed initialization code in ThreadLocal/SplittableRandom
Hi, I think we may be over-rotating a little on this. There is already a mechanism to create a cryptographically strong seed by setting a system property and using SecureRandom. That has a high initialization cost, but i think under those circumstances that cost is acceptable (it may well be possible to reduce that cost but i consider that to be a separate issue). For the default case what we need is something with low initialization cost that is *good enough* for the majority of cases e.g. documents the lower bound of cryptographic strength. I like Doug's approach to hide this under TLR, i think that fits well with the good enough aspect (cross seeding can be documented as an @implNote). But I still marginally prefer a public static method as well, and SecureRandom seems an appropriate choice (although it could easily be exposed on System or Runtime). Paul. On Dec 4, 2014, at 9:20 AM, Peter Levart peter.lev...@gmail.com wrote: On 12/04/2014 12:32 AM, Martin Buchholz wrote: On Wed, Dec 3, 2014 at 2:15 PM, Doug Lea d...@cs.oswego.edu wrote: No public API because systemSeed need only be implemented inside TLR, for its initial seed. Then the others can get their seeds using ThreadLocalRandom.current().nextLong(), unless java.util.secureRandomSeed is set (which I didn't illustrate above). In other words, across all non-secure generators, you only need one system-generated seed. That's good enough for seeding other non-cryptographically secure PRNGs, but if you want each caller to get a cryptographically secure random number, you need to avoid correlations between them that would arise when you use a non-CS PRNG to generate them from a single CS seed. Unless java.util.Random is retrofitted to allocate new cryptographically secure seed for each new instance. In such case expression: new java.util.Random().nextLong() ...could be used to gather secure seed. Perhaps even SplittableRandom could allocate new seed from secure source for each new instance (only in public constructor - not when it is split()ed)... What makes those workarounds unsuitable is failure mode. Gathering secure seed is inherently coupled with possible failure which must be communicated explicitly to the consumer. But one must not be bothered with failure in situations where security is not a necessary ingredient. So for non-CS PRNGs, cross-seeding is a possible solution, but it's nicer for all of them to just use a common (internal) API. As far as public API is concerned, there already is one: SecureRandom.generateSeed(). It's as good as SystemRandom (uses same primary means) and has a nice cross-platform fallback (ThreadedSeedGenerator). It's only drawback is that it comes with all the baggage of security providers (Java and MS Crypto API on Windows). But that's ok for user code perhaps. So what we have here is two desires: - we want a resource-friendly / with as little dependencies as possible way to generate some unique seed, with implicit fall-back which need not be secure - we want a resource-friendly / with as little dependencies as possible way to generate secure random bytes that can be pseudo-random, but still secure, with explicit failure mode I'll try to address this dichotomy in the next iteration of the API. Regards, Peter P.S. Is anyone interested in generating truly random bytes? http://www.cryptography.com/public/pdf/Intel_TRNG_Report_20120312.pdf https://software.intel.com/en-us/tags/35970
Re: RFR 8066397 Remove network-related seed initialization code in ThreadLocal/SplittableRandom
On 12/04/2014 10:40 AM, Paul Sandoz wrote: Hi, I think we may be over-rotating a little on this. There is already a mechanism to create a cryptographically strong seed by setting a system property and using SecureRandom. That has a high initialization cost, but i think under those circumstances that cost is acceptable (it may well be possible to reduce that cost but i consider that to be a separate issue). For the default case what we need is something with low initialization cost that is *good enough* for the majority of cases e.g. documents the lower bound of cryptographic strength. I like Doug's approach to hide this under TLR, i think that fits well with the good enough aspect (cross seeding can be documented as an @implNote). But I still marginally prefer a public static method as well, and SecureRandom seems an appropriate choice (although it could easily be exposed on System or Runtime). Paul. Right. I think the cleanest is to provide an internal public API (modules will effectively hide it from user code) and then expose it if desired by delegation or just use it internally. For JDK8u, the approach to disable the API for user code could use similar trick as Unsafe. If anyone wants to bother with reflection, let him do it. We can't prevent that in JDK8 no matter how deep the implementation is buried. Here's the 3rd iteration of SystemRandom: http://cr.openjdk.java.net/~plevart/jdk9-dev/SystemRandom/webrev.03/ I managed to make LoadLibrary work (I used LoadLibraryA - the ANSI variant). The API is back to getBytes() instance method and a getInstance() static method that lazily initializes a singleton instance. This way the initialization is clearly separated from use. Initialization is expected to fail if the system does not support the interface (/dev/urandom or ADVAPI32!RtlGenRandom), but use should always succeed unless there's something wrong with the state of the operating system (testing will show if my assumptions are correct). The enclosed test results (on 64 bit Linux, i7 PC): SystemRandomTest... (8 bytes / invocation) 1st invocation: 116368 ns, result: [47, -68, 15, -28, 76, -11, -70, -10] Following 100 invocations: 0.624066896 s, (624 ns/invocation) On 32 bit Windows 7 (VirtualBox guest on the same Linux host): SystemRandomTest... (8 bytes / invocation) 1st invocation: 1329219 ns, result: [31, -5, 62, -34, -82, -69, 69, 80] Following 100 invocations: 1.137231408 s, (1137 ns/invocation) The initialization latency on Windows is higher that with code in webrev.02, but sill lower that when using MS Crypto API. I think that's because now the ADVAPI32.DLL is loaded lazily as part of singleton initialization, in webrev.02 it was loaded as part of libjava.dll, since it was linked with it, so the measurement did not include the ADVAPI32.DLL loading time. We can now play in Java and use a strategy that releases the singleton after some time (thinking WeakReference) and ADVAPI32.DLL will be effectively unloaded and unmapped from the process address space. The same goes with UNIX variant and it's file descriptor open from /dev/urandom file... I think webrev.03 is ready for testing. Regards, Peter On Dec 4, 2014, at 9:20 AM, Peter Levart peter.lev...@gmail.com wrote: On 12/04/2014 12:32 AM, Martin Buchholz wrote: On Wed, Dec 3, 2014 at 2:15 PM, Doug Lea d...@cs.oswego.edu wrote: No public API because systemSeed need only be implemented inside TLR, for its initial seed. Then the others can get their seeds using ThreadLocalRandom.current().nextLong(), unless java.util.secureRandomSeed is set (which I didn't illustrate above). In other words, across all non-secure generators, you only need one system-generated seed. That's good enough for seeding other non-cryptographically secure PRNGs, but if you want each caller to get a cryptographically secure random number, you need to avoid correlations between them that would arise when you use a non-CS PRNG to generate them from a single CS seed. Unless java.util.Random is retrofitted to allocate new cryptographically secure seed for each new instance. In such case expression: new java.util.Random().nextLong() ...could be used to gather secure seed. Perhaps even SplittableRandom could allocate new seed from secure source for each new instance (only in public constructor - not when it is split()ed)... What makes those workarounds unsuitable is failure mode. Gathering secure seed is inherently coupled with possible failure which must be communicated explicitly to the consumer. But one must not be bothered with failure in situations where security is not a necessary ingredient. So for non-CS PRNGs, cross-seeding is a possible solution, but it's nicer for all of them to just use a common (internal) API. As far as public API is concerned, there already is one: SecureRandom.generateSeed(). It's as good as SystemRandom (uses same primary means) and has a nice cross-platform
Re: RFR 8066397 Remove network-related seed initialization code in ThreadLocal/SplittableRandom
On 12/04/2014 09:20 AM, Peter Levart wrote: So what we have here is two desires: - we want a resource-friendly / with as little dependencies as possible way to generate some unique seed, with implicit fall-back which need not be secure - we want a resource-friendly / with as little dependencies as possible way to generate secure random bytes that can be pseudo-random, but still secure, with explicit failure mode I'll try to address this dichotomy in the next iteration of the API. I tried and found out that it is not that easy. The fall-back strategy is dependent on the use case and is better left to the consumer of SystemRandom which is just an interface to OS provided secure PRNG. The fact that we want to use it for seeding other PRNGs is just a coincidence. Regarding the fall-back strategy: Besides System.currentTime() and System.nanoTime(), the following could be used in addition, to spice it up, and don't seem like dependencies that couldn't be handled: Runtime.getRuntime().freeMemory() Unsafe.getUnsafe().getLoadAverage() Thread.currentThread().getId() Regards, Peter
Re: RFR 8066397 Remove network-related seed initialization code in ThreadLocal/SplittableRandom
On 12/04/2014 04:40 AM, Paul Sandoz wrote: There is already a mechanism to create a cryptographically strong seed by setting a system property and using SecureRandom. That has a high initialization cost, but i think under those circumstances that cost is acceptable (it may well be possible to reduce that cost but i consider that to be a separate issue). For the default case what we need is something with low initialization cost that is *good enough* for the majority of cases e.g. documents the lower bound of cryptographic strength. Further, in these cases, we are not concerned about cryptographic strength in terms of algorithmic unpredictability by attackers. The default constructor methods promise only to use a seed that is highly likely to differ across invocations, including those of different programs running on different machines. Using initial system seeds that include some environmental entropy is the best way to get the first seed. It just so happens that crypto-based RNGs need to do this as well, so we use the cheapest forms of them that we can (or just use a time based seed on failure). Good-quality PRNG techniques produce more default-constructor seeds after the first one. Because Random, SplittableRandom, and ThreadLocalRandom all use the same basic approach, they can/should use the same mechanism. Placing the mechanics in TLR seems to work out best. In which case only TLR needs a static atomically updated seed initialized from system seed, further reducing static init overhead. I think I've convinced myself that this is the most promising approach, so hope to try it out based on Peter's versions some time in the next few days. Relately, Guy Steele and I have been sporadically investigating minor algorithmic and implementation improvements for SR and TLR (including systematic tests of more than 2500 variants). I suspect that we will settle on one of these sometime in the next few months. (Also, at some point we might reconsider our cowardice about not improving the internal java.util.Random algorithm. j.u.Random is much more commonly used, and does not fare well on quality tests. On the other hand, the more that users instead choose to use SR or TLR, the better.) -Doug
Re: RFR 8066397 Remove network-related seed initialization code in ThreadLocal/SplittableRandom
On 02/12/2014 22:34, Martin Buchholz wrote: : Martin's pet peeve: use readFully (why doesn't InputStream support that yet?!) copy-paste from elsewhere in the jdk. I agree, I think this should be #2 on the list to look at as part of the I/O convenience methods thread. -Alan
Re: RFR 8066397 Remove network-related seed initialization code in ThreadLocal/SplittableRandom
Hi, On 12/02/2014 01:02 PM, Paul Sandoz wrote: Please find below a patch to remove the networking code computing a seed in ThreadLocal/SplittableRandom. Looks good. We thought it a good idea at the time :-) but subsequently on certain platforms this results in very high initalization costs that can propagate to other classes such as ConcurrentSkipList*. Yes, that irked me a lot at the time we introduced this. -Aleksey.
Re: RFR 8066397 Remove network-related seed initialization code in ThreadLocal/SplittableRandom
Hi Peter,
A few questions and comments:
- Can/should this function be fit into one of the existing classes?
- Is more than one instance needed?
Seed material seems to be needed only as a one-shot so a simpler
implementation that opens,
uses and closes would leave fewer leftovers (and not introduce a new
finalizer use).
- The Windows native code could be simpler if the hCryptProv
was returned from the init function and passed as an argument where
needed in getBytes and close .
- The static checking tool we use will complain about JNI functions that may
throw exceptions if those exceptions are not checked for.
For example, SystemRandomImpl_md.c:80.
The macros in jni_util.h like CHECK_NULL, CHECK_NULL_RETURN can be used.
Thanks, Roger
On 12/2/2014 11:42 AM, Peter Levart wrote:
On 12/02/2014 11:02 AM, Paul Sandoz wrote:
Hi,
Please find below a patch to remove the networking code computing a
seed in ThreadLocal/SplittableRandom.
We thought it a good idea at the time :-) but subsequently on certain
platforms this results in very high initalization costs that can
propagate to other classes such as ConcurrentSkipList*.
The short-term solution is to remove this code and fallback just
using current system time. This needs to be back-ported to 8u40.
A longer term solution is to provide a simple public API to get
access to some seed bytes that is optimal for the underlying
platform, for example, based on Peter's investigations. For linux
/dev/urandom is sufficient as a source of bytes. The main problem
seems to be Windows. It would also be nice to back-port to say 8u60
using a private API and update TLR/SR.
Hi,
Here's a proof of concept for an API that just delegates to
system-provided cryptographically secure (as declared by the
system(s)) pseudo random number generator:
http://cr.openjdk.java.net/~plevart/jdk9-dev/SystemRandom/webrev.01/
On UNIX-es this uses /dev/urandom (which is non-blocking and uses
system entropy at least for it's seed):
http://en.wikipedia.org/?title=/dev/random
On Windows it uses MS Crypto API's function CryptGenRandom (the JNI
code is ripped from the sun.security.provider.NativeSeedGenerator),
which also seeds from various system sources of entropy:
http://en.wikipedia.org/wiki/CryptGenRandom
http://msdn.microsoft.com/en-us/library/windows/desktop/aa379942%28v=vs.85%29.aspx
The initialization overhead is low on UNIX (the enclosed test run on
64 bit Fedora 20, i7 PC):
SystemRandomTest... (8 bytes / invocation)
1st invocation: 112315 ns, result: [25, 61, -12, -106, 75, -7, -73, -55]
Following 100 invocations: 0.636644474 s, (636 ns/invocation)
The same test run on 32 bit Windows 7 (as VirtualBox guest on the same
machine):
SystemRandomTest... (8 bytes / invocation)
1st invocation: 4880788 ns, result: [-32, 53, -31, 62, 51, 83, 9, -5]
Following 100 invocations: 1.761087512 s, (1761 ns/invocation)
I think the initialization on Windows has an initial latency of approx
5ms because it has to initialize the whole MS Crypto API with it's
providers. But CryptGenRandom, which is part of this API, actually
delegates it's work to RtlGenRandom function:
http://msdn.microsoft.com/en-us/library/windows/desktop/aa387694%28v=vs.85%29.aspx
...which might have lower initialization costs. Unfortunately, the
wording in the Microsoft document states that it might be removed in
the future. Perhaps we could try to use it and fallback to
CryptGenRandom if it is not available...
Regards, Peter
Paul.
[1] https://bugs.openjdk.java.net/browse/JDK-8060435
diff -r 1b599b4755bd
src/java.base/share/classes/java/util/SplittableRandom.java
--- a/src/java.base/share/classes/java/util/SplittableRandom.java Mon
Dec 01 17:59:39 2014 -0800
+++ b/src/java.base/share/classes/java/util/SplittableRandom.java Tue
Dec 02 10:53:47 2014 +0100
@@ -237,34 +237,7 @@
s = (s 8) | ((long)(seedBytes[i]) 0xffL);
return s;
}
-long h = 0L;
-try {
-EnumerationNetworkInterface ifcs =
-NetworkInterface.getNetworkInterfaces();
-boolean retry = false; // retry once if
getHardwareAddress is null
-while (ifcs.hasMoreElements()) {
-NetworkInterface ifc = ifcs.nextElement();
-if (!ifc.isVirtual()) { // skip fake addresses
-byte[] bs = ifc.getHardwareAddress();
-if (bs != null) {
-int n = bs.length;
-int m = Math.min(n 1, 4);
-for (int i = 0; i m; ++i)
-h = (h 16) ^ (bs[i] 8) ^ bs[n-1-i];
-if (m 4)
-h = (h 8) ^ bs[n-1-m];
-h = mix64(h);
-break;
-}
-else if (!retry)
-retry = true;
-
Re: RFR 8066397 Remove network-related seed initialization code in ThreadLocal/SplittableRandom
On 12/02/2014 05:34 PM, Martin Buchholz wrote: I support Peter's initiative and am willing to help review if we have general consensus about the direction. Peter's implementation scheme of using Unix /dev/urandom or the Windows Crypto API (or something else on failure) seems like the best options. I don't see why we shouldn't place this as a jdk9 public static method in one of the existing random classes though (Random, SplittableRandom, ThreadLocalRandom, SecureRandom), rather than as separate entity. (If desired, for jdk8 backport, it could be non-public, with cheats to access.) -Doug http://cr.openjdk.java.net/~plevart/jdk9-dev/SystemRandom/webrev.01/ The API looks reasonable to me too, I'm just not sure that java.util is the right place and whether it needs to be a Java SE API. -Alan
Re: RFR 8066397 Remove network-related seed initialization code in ThreadLocal/SplittableRandom
On Wed, Dec 3, 2014 at 7:26 AM, Doug Lea d...@cs.oswego.edu wrote: On 12/02/2014 05:34 PM, Martin Buchholz wrote: I support Peter's initiative and am willing to help review if we have general consensus about the direction. Peter's implementation scheme of using Unix /dev/urandom or the Windows Crypto API (or something else on failure) seems like the best options. I don't see why we shouldn't place this as a jdk9 public static method in one of the existing random classes though (Random, SplittableRandom, ThreadLocalRandom, SecureRandom), rather than as separate entity. (If desired, for jdk8 backport, it could be non-public, with cheats to access.) I don't think such a general purpose utility belongs in SplittableRandom or ThreadLocalRandom - those are the clients. Random is a slightly better fit, but still not great. There is a generalized fear with initialization-time utilities that we want as few dependencies as possible, to avoid a clinit method that we are not yet ready to execute, or may never need to execute. That is my fear with SecureRandom, which does provider-y stuff I don't understand. We don't actually need java crypto yet.
Re: RFR 8066397 Remove network-related seed initialization code in ThreadLocal/SplittableRandom
On 12/03/2014 12:20 PM, Martin Buchholz wrote: I don't think such a general purpose utility belongs in SplittableRandom or ThreadLocalRandom - those are the clients. Random is a slightly better fit, but still not great. Perhaps one of these could be made to be a good fit. As of now, this new method (systemSeed) would be called at most twice ever -- during static init of SplittableRandom and ThreadLocalRandom. Either of these could be reworked to use a high-quality derived seed from the other as its initial seed. In which case systemSeed would be called only once. Of the two choices, housing the code in ThreadLocalRandom seems logistically a bit easier. Then SplittableRandom could use private static final AtomicLong defaultGen = new AtomicLong(ThreadLocalRandom.current().nextLong()); (For that matter, java.util.Random could use the same idea for its initial seed.) -Doug
Re: RFR 8066397 Remove network-related seed initialization code in ThreadLocal/SplittableRandom
Hi all, There seems to be a consensus that a simpler static method would be just fine, since it will be used only to gather secure seed for other java based PRNG implementations. UNIX file as well as Windows Crypro API implementation are naturally exposed as open/use/close API, so I did that at first. But when experimenting further with Windows, I replaced Crypro API's CryptGenRandom function with underlying implementation which is RtlGenRandom. This function does not need any crypto provider context to be initialized upfront and then closed. So here's the 2nd approach exposing just a static method: http://cr.openjdk.java.net/~plevart/jdk9-dev/SystemRandom/webrev.02/ I moved it to sun.misc as a public class/method. In JDK9 it could be exposed via some standard API if desired, by delegating to it. In UNIX version I played a little with short-term caching of open file in case several invocations are performed one after the other. Here are the results of the test on 64bit Linux: SystemRandomTest... (8 bytes / invocation) 1st invocation: 135356 ns, result: [19, -107, -121, -54, 98, 28, -6, 36] Following 100 invocations: 0.648048516 s, (648 ns/invocation) When open file caching is not used (file is opened/closed for each invocation): SystemRandomTest... (8 bytes / invocation) 1st invocation: 438672 ns, result: [-84, -71, -106, -126, 21, -42, 48, 0] Following 100 invocations: 3.348318113 s, (3348 ns/invocation) And the results on 32bit Windows 7 (VirtualBox guest on the same Linux host): SystemRandomTest... (8 bytes / invocation) 1st invocation: 388038 ns, result: [-121, -18, -54, 116, -31, 3, 40, -7] Following 100 invocations: 1.027277209 s, (1027 ns/invocation) The initialization latency on Windows is much lower now. What we need now is to try this on as much environments as possible. The Windows version needs Windows XP/Windows Server 2003 at least (RtlGenRandom has been introduced at that time). If the function is not found, JVM most probably crashes. The native code should be changed to use LoadLibrary/GetProcAddress to load and get the address of the function dynamically, so that an exception can be thrown if this fails. But I couldn't get a sample program to load the library (Advapi32.dll) successfully (although I could find the .dll file in Windows folder). Regards, Peter On 12/03/2014 04:26 PM, Doug Lea wrote: On 12/02/2014 05:34 PM, Martin Buchholz wrote: I support Peter's initiative and am willing to help review if we have general consensus about the direction. Peter's implementation scheme of using Unix /dev/urandom or the Windows Crypto API (or something else on failure) seems like the best options. I don't see why we shouldn't place this as a jdk9 public static method in one of the existing random classes though (Random, SplittableRandom, ThreadLocalRandom, SecureRandom), rather than as separate entity. (If desired, for jdk8 backport, it could be non-public, with cheats to access.) -Doug http://cr.openjdk.java.net/~plevart/jdk9-dev/SystemRandom/webrev.01/ The API looks reasonable to me too, I'm just not sure that java.util is the right place and whether it needs to be a Java SE API. -Alan
Re: RFR 8066397 Remove network-related seed initialization code in ThreadLocal/SplittableRandom
Hi Roger, Thanks for looking at my baby steps in the JNI / Windows worlds... Answers inline... On 12/03/2014 03:14 PM, roger riggs wrote: Hi Peter, A few questions and comments: - Can/should this function be fit into one of the existing classes? As a static method perhaps, yes. The webrev.02 uses this approach. - Is more than one instance needed? Seed material seems to be needed only as a one-shot so a simpler implementation that opens, uses and closes would leave fewer leftovers (and not introduce a new finalizer use). If it is to be used only for seeding cryptographicaly insecure PRNGs then this would be simpler, yes. I have one concern though. If the cryptographically secure promise of the function is started to be exploited for seeding cryptographically secure PRNGs with less secure backup plans, then an attacker could force using the backup plan by exhausting the resources which would make the function fail (in UNIX variant for example, the attacker could just open max. # of allowed files). The open/use/eventually-close variant of the API can be used in a setting where the resource is pre-allocated at the startup of the JVM (the file is opened), so the use part can happen anytime later without the worry to be prevented by an attacker. - The Windows native code could be simpler if the hCryptProv was returned from the init function and passed as an argument where needed in getBytes and close . It occurred to me, yes. No need for staticInit0() then. Native methods can all be static too... - The static checking tool we use will complain about JNI functions that may throw exceptions if those exceptions are not checked for. For example, SystemRandomImpl_md.c:80. The macros in jni_util.h like CHECK_NULL, CHECK_NULL_RETURN can be used. I'll take a look at how these are used elsewhere. Thanks for comments. Regards, Peter Thanks, Roger On 12/2/2014 11:42 AM, Peter Levart wrote: On 12/02/2014 11:02 AM, Paul Sandoz wrote: Hi, Please find below a patch to remove the networking code computing a seed in ThreadLocal/SplittableRandom. We thought it a good idea at the time :-) but subsequently on certain platforms this results in very high initalization costs that can propagate to other classes such as ConcurrentSkipList*. The short-term solution is to remove this code and fallback just using current system time. This needs to be back-ported to 8u40. A longer term solution is to provide a simple public API to get access to some seed bytes that is optimal for the underlying platform, for example, based on Peter's investigations. For linux /dev/urandom is sufficient as a source of bytes. The main problem seems to be Windows. It would also be nice to back-port to say 8u60 using a private API and update TLR/SR. Hi, Here's a proof of concept for an API that just delegates to system-provided cryptographically secure (as declared by the system(s)) pseudo random number generator: http://cr.openjdk.java.net/~plevart/jdk9-dev/SystemRandom/webrev.01/ On UNIX-es this uses /dev/urandom (which is non-blocking and uses system entropy at least for it's seed): http://en.wikipedia.org/?title=/dev/random On Windows it uses MS Crypto API's function CryptGenRandom (the JNI code is ripped from the sun.security.provider.NativeSeedGenerator), which also seeds from various system sources of entropy: http://en.wikipedia.org/wiki/CryptGenRandom http://msdn.microsoft.com/en-us/library/windows/desktop/aa379942%28v=vs.85%29.aspx The initialization overhead is low on UNIX (the enclosed test run on 64 bit Fedora 20, i7 PC): SystemRandomTest... (8 bytes / invocation) 1st invocation: 112315 ns, result: [25, 61, -12, -106, 75, -7, -73, -55] Following 100 invocations: 0.636644474 s, (636 ns/invocation) The same test run on 32 bit Windows 7 (as VirtualBox guest on the same machine): SystemRandomTest... (8 bytes / invocation) 1st invocation: 4880788 ns, result: [-32, 53, -31, 62, 51, 83, 9, -5] Following 100 invocations: 1.761087512 s, (1761 ns/invocation) I think the initialization on Windows has an initial latency of approx 5ms because it has to initialize the whole MS Crypto API with it's providers. But CryptGenRandom, which is part of this API, actually delegates it's work to RtlGenRandom function: http://msdn.microsoft.com/en-us/library/windows/desktop/aa387694%28v=vs.85%29.aspx ...which might have lower initialization costs. Unfortunately, the wording in the Microsoft document states that it might be removed in the future. Perhaps we could try to use it and fallback to CryptGenRandom if it is not available... Regards, Peter Paul. [1] https://bugs.openjdk.java.net/browse/JDK-8060435 diff -r 1b599b4755bd src/java.base/share/classes/java/util/SplittableRandom.java --- a/src/java.base/share/classes/java/util/SplittableRandom.java Mon Dec 01 17:59:39 2014 -0800 +++
Re: RFR 8066397 Remove network-related seed initialization code in ThreadLocal/SplittableRandom
On Wed, Dec 3, 2014 at 9:38 AM, Doug Lea d...@cs.oswego.edu wrote: On 12/03/2014 12:20 PM, Martin Buchholz wrote: I don't think such a general purpose utility belongs in SplittableRandom or ThreadLocalRandom - those are the clients. Random is a slightly better fit, but still not great. Perhaps one of these could be made to be a good fit. As of now, this new method (systemSeed) would be called at most twice ever -- during static init of SplittableRandom and ThreadLocalRandom. Either of these could be reworked to use a high-quality derived seed from the other as its initial seed. In which case systemSeed would be called only once. Of the two choices, housing the code in ThreadLocalRandom seems logistically a bit easier. Then SplittableRandom could use private static final AtomicLong defaultGen = new AtomicLong(ThreadLocalRandom.current().nextLong()); (For that matter, java.util.Random could use the same idea for its initial seed.) Oh, I think I see - you're saying we don't need a new public API, because ThreadLocalRandom.current() is already exactly the source of random seed what we want? That seems pretty good ... except that the quality of TLR's PRNG is (expected to be) much lower than what we get from the system - /dev/urandom should be at least a CSPRNG, but TLR is not. So ThreadLocalRandom.current().nextLong() is more predictable than some users might like? I'm still thinking a public API that promises cryptographically secure bits without blocking seems worthwhile.
Re: RFR 8066397 Remove network-related seed initialization code in ThreadLocal/SplittableRandom
On 12/03/2014 02:03 PM, Martin Buchholz wrote: On Wed, Dec 3, 2014 at 9:38 AM, Doug Lea d...@cs.oswego.edu wrote: ... Of the two choices, housing the code in ThreadLocalRandom seems logistically a bit easier. Then SplittableRandom could use private static final AtomicLong defaultGen = new AtomicLong(ThreadLocalRandom.current().nextLong()); Oh, I think I see - you're saying we don't need a new public API, because ThreadLocalRandom.current() is already exactly the source of random seed what we want? No public API because systemSeed need only be implemented inside TLR, for its initial seed. Then the others can get their seeds using ThreadLocalRandom.current().nextLong(), unless java.util.secureRandomSeed is set (which I didn't illustrate above). In other words, across all non-secure generators, you only need one system-generated seed. -Doug
Re: RFR 8066397 Remove network-related seed initialization code in ThreadLocal/SplittableRandom
On Wed, Dec 3, 2014 at 2:15 PM, Doug Lea d...@cs.oswego.edu wrote: No public API because systemSeed need only be implemented inside TLR, for its initial seed. Then the others can get their seeds using ThreadLocalRandom.current().nextLong(), unless java.util.secureRandomSeed is set (which I didn't illustrate above). In other words, across all non-secure generators, you only need one system-generated seed. That's good enough for seeding other non-cryptographically secure PRNGs, but if you want each caller to get a cryptographically secure random number, you need to avoid correlations between them that would arise when you use a non-CS PRNG to generate them from a single CS seed.
Re: RFR 8066397 Remove network-related seed initialization code in ThreadLocal/SplittableRandom
On 02/12/2014 10:02, Paul Sandoz wrote: Hi, Please find below a patch to remove the networking code computing a seed in ThreadLocal/SplittableRandom. We thought it a good idea at the time :-) but subsequently on certain platforms this results in very high initalization costs that can propagate to other classes such as ConcurrentSkipList*. The short-term solution is to remove this code and fallback just using current system time. This needs to be back-ported to 8u40. A longer term solution is to provide a simple public API to get access to some seed bytes that is optimal for the underlying platform, for example, based on Peter's investigations. For linux /dev/urandom is sufficient as a source of bytes. The main problem seems to be Windows. It would also be nice to back-port to say 8u60 using a private API and update TLR/SR. Paul. The approach seems pragmatic and the change looks okay to me (you might have some unused imports to removed too). -Alan.
Re: RFR 8066397 Remove network-related seed initialization code in ThreadLocal/SplittableRandom
On Dec 2, 2014, at 11:10 AM, Alan Bateman alan.bate...@oracle.com wrote: On 02/12/2014 10:02, Paul Sandoz wrote: Hi, Please find below a patch to remove the networking code computing a seed in ThreadLocal/SplittableRandom. We thought it a good idea at the time :-) but subsequently on certain platforms this results in very high initalization costs that can propagate to other classes such as ConcurrentSkipList*. The short-term solution is to remove this code and fallback just using current system time. This needs to be back-ported to 8u40. A longer term solution is to provide a simple public API to get access to some seed bytes that is optimal for the underlying platform, for example, based on Peter's investigations. For linux /dev/urandom is sufficient as a source of bytes. The main problem seems to be Windows. It would also be nice to back-port to say 8u60 using a private API and update TLR/SR. Paul. The approach seems pragmatic and the change looks okay to me Thanks. (you might have some unused imports to removed too). Doh! yes, removed. Paul.
Re: RFR 8066397 Remove network-related seed initialization code in ThreadLocal/SplittableRandom
On 12/02/2014 05:02 AM, Paul Sandoz wrote: Hi, Please find below a patch to remove the networking code computing a seed in ThreadLocal/SplittableRandom. Thanks for pushing this through. (I also adjusted 166 sources accordingly.) One more touch-up for SplittableRandom is to remove mention of network interface in internal comments: *** *** 140,150 * other cases, this split must be performed in a thread-safe * manner, so we use an AtomicLong to represent the seed rather * than use an explicit SplittableRandom. To bootstrap the ! * defaultGen, we start off using a seed based on current time and ! * network interface address unless the java.util.secureRandomSeed ! * property is set. This serves as a slimmed-down (and insecure) ! * variant of SecureRandom that also avoids stalls that may occur ! * when using /dev/random. * * It is a relatively simple matter to apply the basic design here * to use 128 bit seeds. However, emulating 128bit arithmetic and --- 139,148 * other cases, this split must be performed in a thread-safe * manner, so we use an AtomicLong to represent the seed rather * than use an explicit SplittableRandom. To bootstrap the ! * defaultGen, we start off using a seed based on current time ! * unless the java.util.secureRandomSeed property is set. This ! * serves as a slimmed-down (and insecure) variant of SecureRandom ! * that also avoids stalls that may occur when using /dev/random. * * It is a relatively simple matter to apply the basic design here * to use 128 bit seeds. However, emulating 128bit arithmetic and ***
Re: RFR 8066397 Remove network-related seed initialization code in ThreadLocal/SplittableRandom
On Dec 2, 2014, at 1:17 PM, Doug Lea d...@cs.oswego.edu wrote: On 12/02/2014 05:02 AM, Paul Sandoz wrote: Hi, Please find below a patch to remove the networking code computing a seed in ThreadLocal/SplittableRandom. Thanks for pushing this through. (I also adjusted 166 sources accordingly.) One more touch-up for SplittableRandom is to remove mention of network interface in internal comments: *** *** 140,150 * other cases, this split must be performed in a thread-safe * manner, so we use an AtomicLong to represent the seed rather * than use an explicit SplittableRandom. To bootstrap the ! * defaultGen, we start off using a seed based on current time and ! * network interface address unless the java.util.secureRandomSeed ! * property is set. This serves as a slimmed-down (and insecure) ! * variant of SecureRandom that also avoids stalls that may occur ! * when using /dev/random. * * It is a relatively simple matter to apply the basic design here * to use 128 bit seeds. However, emulating 128bit arithmetic and --- 139,148 * other cases, this split must be performed in a thread-safe * manner, so we use an AtomicLong to represent the seed rather * than use an explicit SplittableRandom. To bootstrap the ! * defaultGen, we start off using a seed based on current time ! * unless the java.util.secureRandomSeed property is set. This ! * serves as a slimmed-down (and insecure) variant of SecureRandom ! * that also avoids stalls that may occur when using /dev/random. * * It is a relatively simple matter to apply the basic design here * to use 128 bit seeds. However, emulating 128bit arithmetic and *** Thanks, updated my copy, Paul.
Re: RFR 8066397 Remove network-related seed initialization code in ThreadLocal/SplittableRandom
On Dec 2, 2014, at 5:42 PM, Peter Levart peter.lev...@gmail.com wrote: On 12/02/2014 11:02 AM, Paul Sandoz wrote: Hi, Please find below a patch to remove the networking code computing a seed in ThreadLocal/SplittableRandom. We thought it a good idea at the time :-) but subsequently on certain platforms this results in very high initalization costs that can propagate to other classes such as ConcurrentSkipList*. The short-term solution is to remove this code and fallback just using current system time. This needs to be back-ported to 8u40. A longer term solution is to provide a simple public API to get access to some seed bytes that is optimal for the underlying platform, for example, based on Peter's investigations. For linux /dev/urandom is sufficient as a source of bytes. The main problem seems to be Windows. It would also be nice to back-port to say 8u60 using a private API and update TLR/SR. Hi, Here's a proof of concept for an API that just delegates to system-provided cryptographically secure (as declared by the system(s)) pseudo random number generator: http://cr.openjdk.java.net/~plevart/jdk9-dev/SystemRandom/webrev.01/ That's rather good. I updated the bug with a link to your email. Paul.
Re: RFR 8066397 Remove network-related seed initialization code in ThreadLocal/SplittableRandom
On 02/12/2014 16:42, Peter Levart wrote: : Here's a proof of concept for an API that just delegates to system-provided cryptographically secure (as declared by the system(s)) pseudo random number generator: http://cr.openjdk.java.net/~plevart/jdk9-dev/SystemRandom/webrev.01/ The API looks reasonable to me too, I'm just not sure that java.util is the right place and whether it needs to be a Java SE API. -Alan
Re: RFR 8066397 Remove network-related seed initialization code in ThreadLocal/SplittableRandom
I support Peter's initiative and am willing to help review if we have
general consensus about the direction.
From superficial review:
+int nread = randomStream.read(bytes);
+if (nread != bytes.length) {
+throw new InternalError(Short read from: + RANDOM_FILE);
Martin's pet peeve: use readFully (why doesn't InputStream support
that yet?!) copy-paste from elsewhere in the jdk.
---
I'm not sure we need instances that hold on to system resources. any
particular call site is likely to do a single read of a small number
of random bytes in a clinit method.
On Tue, Dec 2, 2014 at 9:36 AM, Alan Bateman alan.bate...@oracle.com wrote:
On 02/12/2014 16:42, Peter Levart wrote:
:
Here's a proof of concept for an API that just delegates to
system-provided cryptographically secure (as declared by the system(s))
pseudo random number generator:
http://cr.openjdk.java.net/~plevart/jdk9-dev/SystemRandom/webrev.01/
The API looks reasonable to me too, I'm just not sure that java.util is the
right place and whether it needs to be a Java SE API.
-Alan
