Re: How to use /dev/srandom
> > -d 1 Diehard OPERM5 Test Suspect > > -d 14Diehard Sums TestDo Not Use And from the site: Note that a few tests appear to have stubborn bugs. In particular, the diehard operm5 test seems to fail all generators in dieharder. and: Similarly, the diehard sums test appears to produce a systematically non-flat distribution of p-values for all rngs tested, in particular for the "gold standard" cryptographic generators aes and threefish, as well as for the "good" generators in the GSL (mt19937, taus, gfsr4). It seems very unlikely that all of these generators would be flawed in the same way, so this test also should not be used to test your rng. Enjoy your windmill tilting.
Re: How to use /dev/srandom
Janne Johansson wrote: > List of the CURRENT fully implemented tests (as of the 08/18/08 snapshot): > > #=# > # dieharder version 3.29.4beta Copyright 2003 Robert G. Brown > # > #=# > Installed dieharder tests: > Test Number Test NameTest Reliability > === > -d 0Diehard Birthdays Test Good > -d 1 Diehard OPERM5 Test Suspect > -d 2Diehard 32x32 Binary Rank Test Good > -d 3 Diehard 6x8 Binary Rank Test Good > -d 4Diehard Bitstream Test Good > -d 5 Diehard OPSO Good > -d 6 Diehard OQSO Test Good > -d 7 Diehard DNA Test Good > -d 8Diehard Count the 1s (stream) Test Good > -d 9 Diehard Count the 1s Test (byte) Good > -d 10 Diehard Parking Lot Test Good > -d 11 Diehard Minimum Distance (2d Circle) Test Good > -d 12 Diehard 3d Sphere (Minimum Distance) Test Good > -d 13 Diehard Squeeze Test Good > -d 14Diehard Sums TestDo Not Use > -d 15Diehard Runs Test Good > -d 16 Diehard Craps Test Good > -d 17 Marsaglia and Tsang GCD Test Good > -d 100STS Monobit Test Good > -d 101 STS Runs Test Good > -d 102 STS Serial Test (Generalized) Good > -d 200 RGB Bit Distribution Test Good > -d 201 RGB Generalized Minimum Distance Test Good > -d 202 RGB Permutations Test Good > -d 203 RGB Lagged Sum Test Good > -d 204RGB Kolmogorov-Smirnov Test Test Good Interesting. Looks like ent with more tests. You should submit a port.
Re: How to use /dev/srandom
2010/10/4 Brad Tilley > Janne Johansson wrote: > > > What I meant was that one can complain of that the NIST programs (diehard > > and > > dieharder springs to mind) only do certain tests, > > Check out ent (it's in ports) it does chi-square, entropy, and a few > other tests to grade the data stream. Not perfect, but about the best > you'll do for now. > > List of the CURRENT fully implemented tests (as of the 08/18/08 snapshot): #=# # dieharder version 3.29.4beta Copyright 2003 Robert G. Brown# #=# Installed dieharder tests: Test Number Test NameTest Reliability === -d 0Diehard Birthdays Test Good -d 1 Diehard OPERM5 Test Suspect -d 2Diehard 32x32 Binary Rank Test Good -d 3 Diehard 6x8 Binary Rank Test Good -d 4Diehard Bitstream Test Good -d 5 Diehard OPSO Good -d 6 Diehard OQSO Test Good -d 7 Diehard DNA Test Good -d 8Diehard Count the 1s (stream) Test Good -d 9 Diehard Count the 1s Test (byte) Good -d 10 Diehard Parking Lot Test Good -d 11 Diehard Minimum Distance (2d Circle) Test Good -d 12 Diehard 3d Sphere (Minimum Distance) Test Good -d 13 Diehard Squeeze Test Good -d 14Diehard Sums TestDo Not Use -d 15Diehard Runs Test Good -d 16 Diehard Craps Test Good -d 17 Marsaglia and Tsang GCD Test Good -d 100STS Monobit Test Good -d 101 STS Runs Test Good -d 102 STS Serial Test (Generalized) Good -d 200 RGB Bit Distribution Test Good -d 201 RGB Generalized Minimum Distance Test Good -d 202 RGB Permutations Test Good -d 203 RGB Lagged Sum Test Good -d 204RGB Kolmogorov-Smirnov Test Test Good -- To our sweethearts and wives. May they never meet. -- 19th century toast
Re: How to use /dev/srandom
Janne Johansson wrote: > What I meant was that one can complain of that the NIST programs (diehard > and > dieharder springs to mind) only do certain tests, but that is just because > noone > can make a short program that _proves_ a certain stream is random. The only > thing available seems to be a series of tests against a defined set of > properties a > random stream shouldnt have, but that list isnt conclusive, nor finished. Check out ent (it's in ports) it does chi-square, entropy, and a few other tests to grade the data stream. Not perfect, but about the best you'll do for now. Brad
Re: How to use /dev/srandom
2010/10/4 Kevin Chadwick > >Then of course the tiiiny tiiiny problem of defining in code how to > >_prove_ that the input > >is random. Proving some input is skewed in one of 123 ways is easy and > >relatively fast, > >but proving that the input data will never fail a statistical test is.. > >Hard. > > If a situation is possible where a certain device starts doing a ton of > work in a highly regular fashion that the entropy gathering code > doesn't dismiss and so affects the entropy, then I can see this being > useful, but if that was possible which I doubt, then maybe the entropy > gathering should be improved. > > What I meant was that one can complain of that the NIST programs (diehard and dieharder springs to mind) only do certain tests, but that is just because noone can make a short program that _proves_ a certain stream is random. The only thing available seems to be a series of tests against a defined set of properties a random stream shouldnt have, but that list isnt conclusive, nor finished. And it probably never will. Its just among the best options available right now, and it takes lots of time to run and it can only disprove certain inputs, not prove randomness in the others. -- To our sweethearts and wives. May they never meet. -- 19th century toast
Re: How to use /dev/srandom
>Then of course the tiiiny tiiiny problem of defining in code how to >_prove_ that the input >is random. Proving some input is skewed in one of 123 ways is easy and >relatively fast, >but proving that the input data will never fail a statistical test is.. >Hard. If a situation is possible where a certain device starts doing a ton of work in a highly regular fashion that the entropy gathering code doesn't dismiss and so affects the entropy, then I can see this being useful, but if that was possible which I doubt, then maybe the entropy gathering should be improved. Or do you mean a tool that can alert and so pause actions like ssl if highly sensitive, which may be useful but it was stated that arandom is like a duracell bunny on john smiths bitter and won't drain the entropy. >>It is more efficient. There is almost always enough entropy for >>arandom, and if there isn't, you would have a hard time detecting >>that. I would be interested what effect an attacker purposefully draining the entropy could have (Ted's comment suggests little, but you never know) and if your proposed tool could detect and warn of that.
Re: How to use /dev/srandom
Kevin Chadwick writes: > First I'd ask how well can anyone prove that the NIST statistical test > suite can reliably judge randomness? It can't; it can only weed out weak generators but could not distinguish an entropic generator from, say, MD5. See http://lcamtuf.coredump.cx/soft/stompy.tgz for some fun. -- http://noncombatant.org/
Re: How to use /dev/srandom
2010/10/4 Kevin Chadwick > > I do love all this considerations. Just wondering by on earth entropy > > doesn't get much attention in a world where people seems so worried > > about security and privacy. > > Do you mean the world in general or the OpenBSD world. > > I presume you've read the OpenBSD crypto papers that talk about how > impossible it is to create a true random generator. > > First I'd ask how well can anyone prove that the NIST statistical test > suite can reliably judge randomness? > > It just tries to prove the opposite. If the data has patterns it can find, its not random. Proving something is random is insanely much harder. -- To our sweethearts and wives. May they never meet. -- 19th century toast
Re: How to use /dev/srandom
I do love all this considerations. Just wondering by on earth entropy doesn't get much attention in a world where people seems so worried about security and privacy. Have you ever used any specific method to measure the randomness quality of the numbers generated by the kernel when randomness pool goes low? By means of the NIST Statistical Test Suite or anything like that. Maybe it could be possible to maintain a 'randomness quality factor' variable updated in the kernel to be able to estimate, in a given time, the randomness available. Just thinking loud! I'd take a look to that. El 29/09/2010 19:16, Theo de Raadt escribis: On Wed, Sep 29, 2010 at 12:49 PM, Kevin Chadwick wrote: And isn't srandom sometimes (very rarely!) appropriate? E.g. for generating encryption keys? If arandom is somehow not appropriate for generating keys, it should be fixed. I'd be interested to hear more. For those who don't want to go read the code, the algorith on the very back end is roughly this: (a) collect entropy until there is a big enough buffer (b) fold it into the srandom buffer, eventually That is just like the past. But the front end is different. From the kernel side: (1) grab a srandom buffer and start a arc4 stream cipher on it (discarding the first bit, of course) (2) now the kernel starts taking data from this on every packet it sends, to modulate this, to modulate that, who knows. (3) lots of other subsystems get small chunks of random from the stream; deeply unpredictable when (4) on very interrupt, based on quality, the kernel injects something into (a) (5) re-seed the buffer as stated in (1) when needed Simultaneously, userland programs need random data: (i) libc does a sysctl to get a chunk from the rc4 buffer (ii) starts a arc4 buffer of it's own, in that program (iii) feeds data to the program, and re-seeds the buffer when needed The arc4 stream ciphers get new entropy when they need. But the really neat architecture here is that a single stream cipher is *unpredictably* having entropy taken out of it, by hundreds of consumers. In regular unix operating systems, there are only a few entropy consumers. In OpenBSD there are hundreds and hundreds. The entire system is full of random number readers, at every level. That is why this works so well. I notice arandom doesn't pause. Is arandom always better or only when there's enough entropy? It is more efficient. There is almost always enough entropy for arandom, and if there isn't, you would have a hard time detecting that. There is always enough. The generator will keep moving, until it has fetched too much, or too much time has gone by. Then it reseeds; though I think it fundamentally does not care if the srandom buffer it feeds from is full or not.
Re: How to use /dev/srandom
On Thu, 30 Sep 2010 11:37:14 +0200 Daniel Gracia wrote: > I do love all this considerations. Just wondering by on earth entropy > doesn't get much attention in a world where people seems so worried > about security and privacy. Do you mean the world in general or the OpenBSD world. I presume you've read the OpenBSD crypto papers that talk about how impossible it is to create a true random generator. First I'd ask how well can anyone prove that the NIST statistical test suite can reliably judge randomness?
Re: How to use /dev/srandom
On Mon, 4 Oct 2010 13:33:00 +0200 Janne Johansson wrote: > 2010/10/4 Kevin Chadwick > > > > I do love all this considerations. Just wondering by on earth entropy > > > doesn't get much attention in a world where people seems so worried > > > about security and privacy. > > > > Do you mean the world in general or the OpenBSD world. > > > > I presume you've read the OpenBSD crypto papers that talk about how > > impossible it is to create a true random generator. > > > > First I'd ask how well can anyone prove that the NIST statistical test > > suite can reliably judge randomness? > > > > > It just tries to prove the opposite. If the data has patterns it can find, > its not random. > Proving something is random is insanely much harder. > > -- > To our sweethearts and wives. May they never meet. -- 19th century toast > Thought about that but surely you'd need a lab to do that well as you'd need a ridiculous amount of processing power and/or would be helping any attacker do his job. Plus truly random data could very occasionally have short lived random patterns. I imagine the current system monitors the input and output of the entropy pool, which would seem like the logical thing to do, but I wouldn't know. If you can improve the current codes info or accuracy, then cool.
Re: How to use /dev/srandom
On Wed, 29 Sep 2010 13:02:41 -0400 Ted Unangst wrote: > On Wed, Sep 29, 2010 at 12:49 PM, Kevin Chadwick wrote: > >> > And isn't srandom sometimes (very rarely!) appropriate? E.g. for > >> > generating encryption keys? > > If arandom is somehow not appropriate for generating keys, it should > be fixed. I'd be interested to hear more. > > > I notice arandom doesn't pause. Is arandom always better or only when > > there's enough entropy? > > It is more efficient. There is almost always enough entropy for > arandom, and if there isn't, you would have a hard time detecting > that. I love it when you get something that's more secure and more functional. It strikes in the face of sweeping and simplistic statements about security. :C>
Re: How to use /dev/srandom
On Fri, Oct 01, 2010 at 10:45:30AM +0200, Massimo Lusetti wrote: > On Wed, 29 Sep 2010 Theo de Raadt wrote: > > [Ted Unangst wrote: -- Joachim Schipper] > > > [/dev/arandom] is more efficient. There is almost always enough entropy > > > for > > > arandom, and if there isn't, you would have a hard time detecting > > > that. > > > > There is always enough. The generator will keep moving, until it has > ^^^ > > Like "64K will be enough for everyone" ? ;) ... please put it in theo.c No, as in "always enough". (A)RC4 is a pseudorandom generator/stream cipher, which means[1] that it turns a small chunk of random data into an infinite[2] stream of (pseudo-)random data. And if we're going to add stuff to theo.c, I'd be more partial to "oh, but linux people told you it was the best.", a few message upthread. Joachim [1] Well, the mathematical object it's instantiating has this property (by definition). We hope that (A)RC4 does too; so far, nobody has been able to break (A)RC4 (with modern countermeasures like discarding the first part of the output.) [2] For all practical purposes, at least. Like any algorithm with finite state, (A)RC4 will eventually enter a (long!) cycle. Note that /dev/arandom is also re-seeded with fresh entropy, so you could indeed consider it infinite. -- TFMotD: arithmetic (6) - quiz on simple arithmetic http://www.joachimschipper.nl/
Re: How to use /dev/srandom
On Wed, 29 Sep 2010 11:16:53 -0600 Theo de Raadt wrote: > > It is more efficient. There is almost always enough entropy for > > arandom, and if there isn't, you would have a hard time detecting > > that. > > There is always enough. The generator will keep moving, until it has ^^^ Like "64K will be enough for everyone" ? ;) ... please put it in theo.c -- Massimo
Re: How to use /dev/srandom
> On Wed, Sep 29, 2010 at 12:49 PM, Kevin Chadwick > wrote: > >> > And isn't srandom sometimes (very rarely!) appropriate? E.g. for > >> > generating encryption keys? > > If arandom is somehow not appropriate for generating keys, it should > be fixed. I'd be interested to hear more. For those who don't want to go read the code, the algorith on the very back end is roughly this: (a) collect entropy until there is a big enough buffer (b) fold it into the srandom buffer, eventually That is just like the past. But the front end is different. From the kernel side: (1) grab a srandom buffer and start a arc4 stream cipher on it (discarding the first bit, of course) (2) now the kernel starts taking data from this on every packet it sends, to modulate this, to modulate that, who knows. (3) lots of other subsystems get small chunks of random from the stream; deeply unpredictable when (4) on very interrupt, based on quality, the kernel injects something into (a) (5) re-seed the buffer as stated in (1) when needed Simultaneously, userland programs need random data: (i) libc does a sysctl to get a chunk from the rc4 buffer (ii) starts a arc4 buffer of it's own, in that program (iii) feeds data to the program, and re-seeds the buffer when needed The arc4 stream ciphers get new entropy when they need. But the really neat architecture here is that a single stream cipher is *unpredictably* having entropy taken out of it, by hundreds of consumers. In regular unix operating systems, there are only a few entropy consumers. In OpenBSD there are hundreds and hundreds. The entire system is full of random number readers, at every level. That is why this works so well. > > I notice arandom doesn't pause. Is arandom always better or only when > > there's enough entropy? > > It is more efficient. There is almost always enough entropy for > arandom, and if there isn't, you would have a hard time detecting > that. There is always enough. The generator will keep moving, until it has fetched too much, or too much time has gone by. Then it reseeds; though I think it fundamentally does not care if the srandom buffer it feeds from is full or not.
Re: How to use /dev/srandom
> On Wed, Sep 29, 2010 at 11:39 AM, Theo de Raadt w= > rote: > >> Independent of other problems, I don't think you should be using > >> srandom. =A0We should just take that interface away, people see it and > >> then they want to use it, but it doesn't work the way they want. > > > > Taking it away would first require an extensive audit of the ports > > tree -- to make sure that the applications in there don't end up > > choosing something even *worse* than srandom... > > I was just going to make it a symlink to arandom. :) Ah! That's a good idea.
Re: How to use /dev/srandom
On Wed, Sep 29, 2010 at 11:39 AM, Theo de Raadt wrote: >> Independent of other problems, I don't think you should be using >> srandom. We should just take that interface away, people see it and >> then they want to use it, but it doesn't work the way they want. > > Taking it away would first require an extensive audit of the ports > tree -- to make sure that the applications in there don't end up > choosing something even *worse* than srandom... I was just going to make it a symlink to arandom. :)
Re: How to use /dev/srandom
On Wed, Sep 29, 2010 at 12:49 PM, Kevin Chadwick wrote: >> > And isn't srandom sometimes (very rarely!) appropriate? E.g. for >> > generating encryption keys? If arandom is somehow not appropriate for generating keys, it should be fixed. I'd be interested to hear more. > I notice arandom doesn't pause. Is arandom always better or only when > there's enough entropy? It is more efficient. There is almost always enough entropy for arandom, and if there isn't, you would have a hard time detecting that.
Re: How to use /dev/srandom
On Wed, 29 Sep 2010 10:02:16 -0600 Theo de Raadt wrote: > > And isn't srandom sometimes (very rarely!) appropriate? E.g. for > > generating encryption keys? > > hell no! > > srandom is definately worse than the arc4random generator. > > oh, but linux people told you it was the best. I get it. > I notice arandom doesn't pause. Is arandom always better or only when there's enough entropy?
Re: How to use /dev/srandom
> On Wed, Sep 29, 2010 at 09:39:06AM -0600, Theo de Raadt wrote: > > > On Wed, Sep 29, 2010 at 9:57 AM, Simon Perreault > > > wrote: > > > > I'm trying to use /dev/srandom, but I can't get even a single byte out > > > > of it. > > > > > > Independent of other problems, I don't think you should be using > > > srandom. We should just take that interface away, people see it and > > > then they want to use it, but it doesn't work the way they want. > > > > Taking it away would first require an extensive audit of the ports > > tree -- to make sure that the applications in there don't end up > > choosing something even *worse* than srandom... > > And isn't srandom sometimes (very rarely!) appropriate? E.g. for > generating encryption keys? hell no! srandom is definately worse than the arc4random generator. oh, but linux people told you it was the best. I get it.
Re: How to use /dev/srandom
On Wed, Sep 29, 2010 at 09:39:06AM -0600, Theo de Raadt wrote: > > On Wed, Sep 29, 2010 at 9:57 AM, Simon Perreault > > wrote: > > > I'm trying to use /dev/srandom, but I can't get even a single byte out > > > of it. > > > > Independent of other problems, I don't think you should be using > > srandom. We should just take that interface away, people see it and > > then they want to use it, but it doesn't work the way they want. > > Taking it away would first require an extensive audit of the ports > tree -- to make sure that the applications in there don't end up > choosing something even *worse* than srandom... And isn't srandom sometimes (very rarely!) appropriate? E.g. for generating encryption keys? Joachim
Re: How to use /dev/srandom
> On Wed, Sep 29, 2010 at 9:57 AM, Simon Perreault > wrote: > > I'm trying to use /dev/srandom, but I can't get even a single byte out > > of it. > > Independent of other problems, I don't think you should be using > srandom. We should just take that interface away, people see it and > then they want to use it, but it doesn't work the way they want. Taking it away would first require an extensive audit of the ports tree -- to make sure that the applications in there don't end up choosing something even *worse* than srandom...
Re: How to use /dev/srandom
On Wed, Sep 29, 2010 at 9:57 AM, Simon Perreault wrote: > I'm trying to use /dev/srandom, but I can't get even a single byte out > of it. Independent of other problems, I don't think you should be using srandom. We should just take that interface away, people see it and then they want to use it, but it doesn't work the way they want.
Re: How to use /dev/srandom
On 2010-09-29 10:49, Theo de Raadt wrote:
> Perhaps a posix weenie can look into making hexdump use setvbuf and
> adjusting the read requirements for fread() when the length (-n
> argument) is specified as being short of the blocksize.
How about this weenie?
Index: display.c
===
RCS file: /cvs/src/usr.bin/hexdump/display.c,v
retrieving revision 1.18
diff -u -p -r1.18 display.c
--- display.c 27 Oct 2009 23:59:39 - 1.18
+++ display.c 29 Sep 2010 15:03:11 -
@@ -300,6 +300,8 @@ next(char **argv)
++_argv;
continue;
}
+ if (length > 0 && length < BUFSIZ)
+ setvbuf(stdin, NULL, _IONBF, 0);
statok = done = 1;
} else {
if (done++)
--
NAT64/DNS64 open-source --> http://ecdysis.viagenie.ca
STUN/TURN server--> http://numb.viagenie.ca
vCard 4.0 --> http://www.vcarddav.org
Re: How to use /dev/srandom
> > it is hanging because: > > > > 23208 hexdump CALL read(0,0x81ffc000,0x1) > > > > It is trying to read too much. A whole buffer, into stdio. > > > > So it empties the pool it can have, and then has to wait for more. > > eventually it does get data, and print 1 char. > > Thanks! I was using the much slower "add printf()s" debugging method... > > > I am susprised that hexdump doesn't decide to read less based on the -n > > argument. > > Me too! > > Thanks a lot for your help, that fixes my issue. Perhaps a posix weenie can look into making hexdump use setvbuf and adjusting the read requirements for fread() when the length (-n argument) is specified as being short of the blocksize.
Re: How to use /dev/srandom
On Wed, Sep 29, 2010 at 09:57:53AM -0400, Simon Perreault wrote: > I'm trying to use /dev/srandom, but I can't get even a single byte out > of it. > > $ hexdump -n 1 /dev/srandom > > It just hangs there, sleeping. If I use /dev/urandom instead, it returns > immediately, as expected: > > $ hexdump -n 1 /dev/urandom > 000 0069 > 001 > > I tried on various routers that have been forwarding packets since > forever. I waited a "long time" for the read to succeed. I tried on > OpenBSD 4.3 and 4.6. Am I doing something wrong? Using hexdump(1), apparently - "dd if=/dev/srandom bs=1 count=1 | hexdump" works just fine. You may want to sendbug this one. Joachim -- TFMotD: string2key (8) - map a password into a key http://www.joachimschipper.nl/
Re: How to use /dev/srandom
On 2010-09-29 10:36, Theo de Raadt wrote: > it is hanging because: > > 23208 hexdump CALL read(0,0x81ffc000,0x1) > > It is trying to read too much. A whole buffer, into stdio. > > So it empties the pool it can have, and then has to wait for more. > eventually it does get data, and print 1 char. Thanks! I was using the much slower "add printf()s" debugging method... > I am susprised that hexdump doesn't decide to read less based on the -n > argument. Me too! Thanks a lot for your help, that fixes my issue. Simon -- NAT64/DNS64 open-source --> http://ecdysis.viagenie.ca STUN/TURN server--> http://numb.viagenie.ca vCard 4.0 --> http://www.vcarddav.org
Re: How to use /dev/srandom
it is hanging because: 23208 hexdump CALL read(0,0x81ffc000,0x1) It is trying to read too much. A whole buffer, into stdio. So it empties the pool it can have, and then has to wait for more. eventually it does get data, and print 1 char. I am susprised that hexdump doesn't decide to read less based on the -n argument.
How to use /dev/srandom
Hello, I'm trying to use /dev/srandom, but I can't get even a single byte out of it. To reproduce: $ hexdump -n 1 /dev/srandom It just hangs there, sleeping. If I use /dev/urandom instead, it returns immediately, as expected: $ hexdump -n 1 /dev/urandom 000 0069 001 I tried on various routers that have been forwarding packets since forever. I waited a "long time" for the read to succeed. I tried on OpenBSD 4.3 and 4.6. Am I doing something wrong? Thanks, Simon -- NAT64/DNS64 open-source --> http://ecdysis.viagenie.ca STUN/TURN server--> http://numb.viagenie.ca vCard 4.0 --> http://www.vcarddav.org
