Re: valgrind and openssl
Think back to what tripped this whole discussion. valgrind isn't complaining because the data has been pre-filled, it's complaining because it's never been touched. i.e if it were attacker providable buffer contents then this whole discussion wouldn't have happened. If the attacker can pre-seed uninitialized data in the process, then they can read generated keys directly - and that's far easier than by trying to second guess the RNG. Peter From: dean gaudet [EMAIL PROTECTED] To: openssl-dev@openssl.org Date: 05/21/2008 03:44 PM Subject:Re: valgrind and openssl On Tue, 20 May 2008, Richard Salz wrote: on the other hand it may be a known plaintext attack. Using those words in this context makes it sound that you not only don't understand what is being discussed right here and now, but also that you don't understand the term you just used. Are you sure you understood, e.g., Ted Tso's postings in this thread? Perhaps I'm missing something, but can you show me something that talks about known plaintext attacks in the context of hashing/digests? yes i abused the term. the so-called uninitialized data is actually from the stack right? an attacker generally controls that (i.e. earlier use of the stack probably includes char buf[] which is controllable). i don't know what ordering the entropy is added to the PRNG, but if all the useful entropy goes in first then an attacker might get to control the last 1KiB passed through the SHA1. yes it's unlikely given what we know today that an attacker could manipulate the state down to a sufficiently small number of outputs, but i really don't see the point of letting an attacker have that sort of control. -dean __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED] __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: valgrind and openssl
Peter Waltenberg wrote:
Think back to what tripped this whole discussion.
valgrind isn't complaining because the data has been pre-filled, it's
complaining because it's never been touched.
i.e if it were attacker providable buffer contents then this whole
discussion wouldn't have happened.
If the attacker can pre-seed uninitialized data in the process, then they
can read generated keys directly - and that's far easier than by trying to
second guess the RNG.
Consider this little piece of code:
#include stdio.h
#include stdlib.h
void f(void)
{
char buf[80];
printf(Please enter a line: );
fflush(stdout);
fgets(buf, sizeof buf, stdin);
}
void g(void)
{
char buf[80];
/* Use uninitialized buffer content... */
printf(Buf contains %s\n, buf);
}
int main(void)
{
f();
g();
return 0;
}
On my system, this yields:
[EMAIL PROTECTED]:~/src$ gcc buf.c
[EMAIL PROTECTED]:~/src$ ./a.out
Please enter a line: blark foo
Buf contains blark foo
[EMAIL PROTECTED]:~/src$
:-) The reason this works is that when fgets writes into the stack
memory, this memory is reused in the function g() without being
reinitialized.
So, it is sometimes possible indeed that an attacker will be able to
provide (some of) the content of uninitialized memory, if he gets to
interact with the program in some way.
Regards,
--
Nils Gösche
Don't ask for whom the CTRL-G tolls.
__
OpenSSL Project http://www.openssl.org
Development Mailing List openssl-dev@openssl.org
Automated List Manager [EMAIL PROTECTED]
Re: valgrind and openssl
The PRNG is initialized when you initialize OpenSSL, and I'll concede,
someone COULD use fgets() before that, but it's a really improbable
scenario and in any case - the PRNG being vulnerable is the least of your
worries then
Sure you can stuff known data into there - if you are running in the same
process address space (just read the keys), if you are in the kernel (just
read the keys), if you have the same UID (or root UID) - (just read the
keys), you can even do it via fgets(), no argument, your example is valid -
but can you do it before OpenSSL has been initialized ? - well, maybe some
badly written code COULD do that, but it doesn't matter, if you control the
stack to that extent you can also engineer a buffer overrun, insert your
own code and simply read the generated keys - which is much simpler than
cracking the PRNG.
As MANY others pointed out - simply putting known data in there doesn't
decrease the entropy - worst case it's exactly what it was before. (The
hash function is irreversible in this case - effectively it's compressing
the data). So best case, you do have data there which is machine or
instance dependent and the PNG is even less predictable that it was, worst
case it's as strong as it was before you added the known data.
I'm not claiming this is best practice, but it's not intrinsically bad
either.
Peter
From: Nils Gösche [EMAIL PROTECTED]
To: openssl-dev@openssl.org
Date: 05/21/2008 07:57 PM
Subject:Re: valgrind and openssl
Peter Waltenberg wrote:
Think back to what tripped this whole discussion.
valgrind isn't complaining because the data has been pre-filled, it's
complaining because it's never been touched.
i.e if it were attacker providable buffer contents then this whole
discussion wouldn't have happened.
If the attacker can pre-seed uninitialized data in the process, then
they
can read generated keys directly - and that's far easier than by trying
to
second guess the RNG.
Consider this little piece of code:
#include stdio.h
#include stdlib.h
void f(void)
{
char buf[80];
printf(Please enter a line: );
fflush(stdout);
fgets(buf, sizeof buf, stdin);
}
void g(void)
{
char buf[80];
/* Use uninitialized buffer content... */
printf(Buf contains %s\n, buf);
}
int main(void)
{
f();
g();
return 0;
}
On my system, this yields:
[EMAIL PROTECTED]:~/src$ gcc buf.c
[EMAIL PROTECTED]:~/src$ ./a.out
Please enter a line: blark foo
Buf contains blark foo
[EMAIL PROTECTED]:~/src$
:-) The reason this works is that when fgets writes into the stack
memory, this memory is reused in the function g() without being
reinitialized.
So, it is sometimes possible indeed that an attacker will be able to
provide (some of) the content of uninitialized memory, if he gets to
interact with the program in some way.
Regards,
--
Nils Gösche
Don't ask for whom the CTRL-G tolls.
__
OpenSSL Project http://www.openssl.org
Development Mailing List openssl-dev@openssl.org
Automated List Manager [EMAIL PROTECTED]
__
OpenSSL Project http://www.openssl.org
Development Mailing List openssl-dev@openssl.org
Automated List Manager [EMAIL PROTECTED]
Re: valgrind and openssl
On Tue, May 20, 2008 at 10:43:27PM -0700, dean gaudet wrote: the so-called uninitialized data is actually from the stack right? an attacker generally controls that (i.e. earlier use of the stack probably includes char buf[] which is controllable). i don't know what ordering the entropy is added to the PRNG, but if all the useful entropy goes in first then an attacker might get to control the last 1KiB passed through the SHA1. yes it's unlikely given what we know today that an attacker could manipulate the state down to a sufficiently small number of outputs, but i really don't see the point of letting an attacker have that sort of control. If this is true, then all digital signatures, certificates, that use SHA-1 would have to be discarded. The PRNG will be the least of your problems. Consider that if I were to digitally sign this reply, I am including in this message text I didn't write (namely, the text which you are replying). Or an attacker which gets to control network packets which are sent out via integrity-protected IPSEC connections. Crypto checksums have to be able to deal with this sort of thing, and no, they're not affected. Controlling the last megabyte of data passed through SHA1 wouldn't matter; if you could, then you could induce hash collisions, and SHA-1 would be totally broken as an crypto checksum. - Ted __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
RE: valgrind and openssl
You mean you're not testing *all* of the real code. That's fine, you can't debug everythign at once. if you haven't tested your final production binary then you haven't tested anything at all. You: Two plus two is five. Me: Are you crazy? Two plus two is not five. You: If you don't realize that three plus three is six, then you don't know any math at all. Err, what?! What does testing your final production binary have to do with debugging on code other than your final production binary? Good luck finding people who agree with you. I've been a professional software developer for about 18 years and I've worked on debugging with i've been a professional for longer than you. big whoop. -dean It really doesn't matter how long you've been doing it when you spout nonsense like the above. DS __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
RE: valgrind and openssl
on the other hand it may be a known plaintext attack. what are you guys smoking? -dean This argument has already been refuted in the posts you are replying to. Such an attack would require the algorithm to not meet its specific design security objectives. In other words, you are arguing that this might be a problem if the algorithm is fundamentally broken. Well, duh, using it might be a problem if the algorithm is fundamentally broken. A primary security objective of the PRNG is that mixing in known data can never, ever hurt you. It may not increase the entropy in the PRNG, but it cannot decrease it. This is a specific design object, and you can rely on the PRNG to meet this object. If you don't think the PRNG meets this objective, you should not use it at all because this is one of its primary objectives and the entire design is premised on it. The only issue is that you might not mix in enough unknown data. Mixing in known data can never hurt you. If it can, the PRNG is totally and completely broken. If you have any reason to think it is, please explain why. We will all change the algorithm because this would be a major breakage. You might as well argue that after you sign something with RSA, you should keep the signature secret. After all, there might be some attack where the signature leaks parts of the private key. Sure, there might be. But if you really thought there was any chance of that, you'd abandon the signature algorithm, not work harder to keep the signatures secret. DS __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: valgrind and openssl
on the other hand it may be a known plaintext attack. Using those words in this context makes it sound that you not only don't understand what is being discussed right here and now, but also that you don't understand the term you just used. Are you sure you understood, e.g., Ted Tso's postings in this thread? Perhaps I'm missing something, but can you show me something that talks about known plaintext attacks in the context of hashing/digests? what are you guys smoking? Indeed. /r$ -- STSM, DataPower Chief Programmer WebSphere DataPower SOA Appliances http://www.ibm.com/software/integration/datapower/
Re: valgrind and openssl
On Monday 19 May 2008 15:27:24 dean gaudet wrote: Note that you should always build with no-asm if you're doing this kind of debug analysis. The assembly optimisations are likely to operate at granularities and in ways that valgrind could easily complain about. I don't know that this is the case, but it would certainly make sense to compare before posting a bug report. you know, this is sheer stupidity. you're suggesting that testing the no-asm code is a valid way of testing the assembly code? Whoa, you're getting a tad trigger happy there. I'm not suggesting that testing the no-asm code is a valid way of testing the assembly code, you're putting words in my mouth. I think my last sentence is pretty clear. Debugging and optimising the assembly code is a different ball-game than identifying C-level leaks or other bugs in openssl. That's why I suggested comparing with a no-asm build. If valgrind sees equivalent problems in both, fine. If it sees it only with the assembly optimisations, that's useful information too, right? additionally the suggestion of -DPURIFY as a way of testing the code is also completely broken software engineering practice. No, it's not. This is a PRNG, not a menu widget. You might also want to test with support for symbols and core-dumps, but run without either in production. Heisenburgs can also up when running in a debugger vs not running a debugger, using optimisation flags selectively, [...]. We have a PRNG, the accent is on the R, not the P. This is not valgrind's fault, just a one-off incompatibility with the notion that uninitialised==catastrophe. any special case changes for testing means you're not testing the REAL CODE. With respect, please check the though shalts at the door, thanks. for example if you build -DPURIFY then you also won't get notified of problems with other PRNG seeds which are supposed to be providing random *initialized* data. This is not true. -DPURIFY does *NOT* compile out the use of *initialised* data - that is the whole point. It compiles out any use of *uninitialized* data. (Or at least it's supposed to, Bodo thinks there's another case where an #ifdef would be needed in the corner case where there's a short width parameter - that's beside the point.) not to mention that a system compiled that way is insecure -- so you either have to link your binaries static (to avoid the danger of an insecure shared lib), or set up a chroot for testing. Also not true. in any event YOU'RE NOT TESTING THE REAL CODE. which is to say you're wasting your time if you test under any of these conditions. You haven't got your facts straight. In fact, I'm starting to suspect you've got them deliberately organised in a circle, for the sake of argument. Please make sure you're aware of *precisely* what's going on here, and *precisely* what -DPURIFY changes in openssl. Otherwise your second sentence is theologic, at best. openssl should not be relying on uninitialized data for anything. even if it doesn't matter from the point of view of the PRNG, it should be pretty damn clear it's horrible software engineering practice. More thou shalts. As for your other mail; On Monday 19 May 2008 23:48:14 dean gaudet wrote: On Thu, 15 May 2008, Bodo Moeller wrote: The use of unititialized data in this case is stupid because the entropy of this random data is close to zero. It may be zero, but it may be more, depending on what happened earlier in the program if the same memory locations have been in use before. This may very well include data that would be unpredictable to adversaries -- i.e., entropy; that's the point here. on the other hand it may be a known plaintext attack. For that sentence to be other than BS, it is a minimum requirement that the PRNG be fundamentally broken algorithmically. Please publish, or refrain from muddling the issue. what are you guys smoking? The charred remains of a debian package, it's very noxious. Cheers, Geoff __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: valgrind and openssl
On Tue, 20 May 2008, Richard Salz wrote: on the other hand it may be a known plaintext attack. Using those words in this context makes it sound that you not only don't understand what is being discussed right here and now, but also that you don't understand the term you just used. Are you sure you understood, e.g., Ted Tso's postings in this thread? Perhaps I'm missing something, but can you show me something that talks about known plaintext attacks in the context of hashing/digests? yes i abused the term. the so-called uninitialized data is actually from the stack right? an attacker generally controls that (i.e. earlier use of the stack probably includes char buf[] which is controllable). i don't know what ordering the entropy is added to the PRNG, but if all the useful entropy goes in first then an attacker might get to control the last 1KiB passed through the SHA1. yes it's unlikely given what we know today that an attacker could manipulate the state down to a sufficiently small number of outputs, but i really don't see the point of letting an attacker have that sort of control. -dean __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: valgrind and openssl
It doesn't matter. If you only have one bit of real entropy you are screwed - no matter whether 0 or a 10^15 bits of known data are introduced, and if it's 10^15 bits of data the attacker can't reliably guess, you are definitely better off. And, to put this in perspective, given that the uninitialized memory contents are likely unknowable off the machine - if Debian had screwed up just a little more - and left the uninitialized memory as a source, but taken out the real entropy source, (instead of taking out both), would Debian users be in better or worse shape now ?. Peter From: Thor Lancelot Simon [EMAIL PROTECTED] To: openssl-dev@openssl.org Date: 05/19/2008 05:24 PM Subject:Re: valgrind and openssl On Fri, May 16, 2008 at 11:24:45AM -0400, Geoff Thorpe wrote: On Friday 16 May 2008 00:47:52 Thor Lancelot Simon wrote: On Thu, May 15, 2008 at 11:45:14PM +0200, Bodo Moeller wrote: It may be zero, but it may be more, depending on what happened earlier in the program if the same memory locations have been in use before. This may very well include data that would be unpredictable to adversaries -- i.e., entropy; that's the point here. Unfortunately, it may also very well include data that would be highly predictable to adversaries. If feeding predictable data into a PRNG that was already well seeded with unpredictable data produced a weaker PRNG, then you have found a security bug in the PRNG and I suggest you publish. Yeah, I've heard that a few times. However, consider the pathological case, in which an adversary manages to introduce N-1 bits of known state into your PRNG which has N bits of internal state. Are you comfortable with that? For what value M are you comfortable with N - M bits of the state having been introduced by the adversary? Why? It seems to me that best practice is to not introduce such state if one can avoid it, whether one counts it into an entropy estimate or not. Thor __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED] __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: valgrind and openssl
On Mon, May 19, 2008 at 6:00 AM, Michael Sierchio [EMAIL PROTECTED] wrote: Theodore Tso wrote: ... I'd be comfortable with an adversary knowing the first megabyte of data fed through SHA1, as long as it was followed up by at least 256 bits which the adversary *didn't* know. I'd be comfortable with an adversary knowing the first zetabyte of data fed though SHA1, as long as it was followed up by at least 256 bits which the adversary didn't know. ;-) You are being a few orders of magnitude too optimistic here, though ... ;-) A zettabyte would be 2^78 bits (less if you use the standard decimal version of zetta), but SHA-1 will only handle inputs up to 2^64 -1 bits. Bodo __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: valgrind and openssl
On Mon, May 19, 2008 at 11:47:07AM +0200, Bodo Moeller wrote: You are being a few orders of magnitude too optimistic here, though ... ;-) A zettabyte would be 2^78 bits (less if you use the standard decimal version of zetta), but SHA-1 will only handle inputs up to 2^64 -1 bits. That's true only because size of the message in bits is appended to the end of the message to prevent the obvious extension attacks. (For people who are clueless about how SHA-1 is implemented, the message is padded out with a 1 followed by enough 0 bits so the message is congruent to 448 mod 512 bits, and then a 64-bit message size count, in big-endian format is appended to the message, and the message is processed in 16 byte --- 512 bit --- chunks. Yes that means that if the message is an exact multiple of 16 bytes, and you know the size of the message, the last 16 bytes which is run through the compression algorithm is known to the adversary; it will be a 1, followed by 447 zero bits, followed by the message size as a 64-bit big endian integer. Whoop! Whoop! Danger Will Robinson! Time to run in circles and scream about something you know nothing about!!! It must be a NSA conspiracy!) Seriously, it would be easy to change the padding scheme to accomodate bigger messages, and in fact for a PRNG you don't need to worry about extension attacks, since what you are really depending upon is the crypto hash function's compression function. With Linux's /dev/random, we dispense with message padding completely, since it's not necessary given how we are using the hash's compression function. So it wouldn't be SHA-1, but with some very minor modifications to accomodate a bigger message size count, you could run a zettabyte of known data, followed by 256 bits of known data, and the adversary would still be screwed. (And if they aren't, it's time to replace SHA-1) - Ted __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: valgrind and openssl
On Sun, May 18, 2008 at 10:07:03PM -0400, Theodore Tso wrote: On Sun, May 18, 2008 at 05:24:51PM -0400, Thor Lancelot Simon wrote: So you're comfortable with the adversary knowing, let's say, 511 of the first 512 bits fed through SHA1? *Sigh*. Thor, you clearly have no idea how SHA-1 works. In fact, I'd be comfortable with an adversary knowing the first megabyte of data fed through SHA1, as long as it was followed up by at least 256 bits which the adversary *didn't* know. Thanks for the gratuitous insult. I'd be perfectly happy with the case you'd be happy with, too, but you took my one bit and turned it into 256. What I _wouldn't_ be happy with is a PRNG which has been fed only known data, but enough of it at startup that it agrees to provide output to the user. There are a terrible lot of these around, and pretending that stack contents are random is a great way to accidentally build them. Not feeding in data which you have a pretty darned good idea will be predictable -- potentially as the first bits in at RNG startup -- is to my mind one thing one can should do to avoid the problem. Thor __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: valgrind and openssl
On Mon, May 19, 2008 at 6:30 PM, Thor Lancelot Simon [EMAIL PROTECTED] wrote: On Sun, May 18, 2008 at 10:07:03PM -0400, Theodore Tso wrote: On Sun, May 18, 2008 at 05:24:51PM -0400, Thor Lancelot Simon wrote: So you're comfortable with the adversary knowing, let's say, 511 of the first 512 bits fed through SHA1? *Sigh*. Thor, you clearly have no idea how SHA-1 works. In fact, I'd be comfortable with an adversary knowing the first megabyte of data fed through SHA1, as long as it was followed up by at least 256 bits which the adversary *didn't* know. Thanks for the gratuitous insult. I'd be perfectly happy with the case you'd be happy with, too, but you took my one bit and turned it into 256. What I _wouldn't_ be happy with is a PRNG which has been fed only known data, but enough of it at startup that it agrees to provide output to the user. There are a terrible lot of these around, and pretending that stack contents are random is a great way to accidentally build them. Not feeding in data which you have a pretty darned good idea will be predictable -- potentially as the first bits in at RNG startup -- is to my mind one thing one can should do to avoid the problem. No-one pretends that stacks contents are random. The OpenSSL PRNG tries to keep a tally of how much entropy has been added from external sources. I won't generate any output for key generation and such until it is happy about this amount of entropy. Those stack contents are taken into account with an entropy estimate of 0.0, i.e., not at all. Thus, after feeding those 511 known bits to the OpenSSL PRNG [*], it would still expect just as much additional seeding as before. Your failure scenario has nothing to do with the way this PRNG operates. ]*] Actually the PRNG won't take fractions of bytes, so make that 512 bits, or 504. __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: valgrind and openssl
On Mon, May 19, 2008 at 12:30:42PM -0400, Thor Lancelot Simon wrote: Thanks for the gratuitous insult. I'd be perfectly happy with the case you'd be happy with, too, but you took my one bit and turned it into 256. But your example is NOT what openssl does. I recently had similar issue with Linux's /dev/random, where folks were similarly confused. You see, the issue that SHA-1 takes its input in 512 bits chunks of data. If you are only mixing in 256 bits of randomness, the question is what to do with the other 256 bits. You could waste CPU time zero'ing out those bits, or you can just shrug your shoulders and say, I don't care. The problem is people who don't understand say OMG!!! Ur using unitializated data!, not getting the fact that whole point was to mix in the 256 bits of entropy. The other uninitialized bits might add more information unknown to the adversary (in which case it helps), or it might not (in which case it doesn't matter, because what you're really depending on is the 256 bits of entropy from a good entropy source). Of course, you could use the argument that it's a bad idea because a clueless Debian developer might comment out the call entirely due to some misguided notion that using uninitialized data was somehow a security problem. Until last week, I would have thought that was a silly argument, but apparently there are people that clueless/stupid/ careless out there - Ted __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: valgrind and openssl
On Thu, 15 May 2008, Geoff Thorpe wrote: I forgot to mention something; On Thursday 15 May 2008 12:38:24 John Parker wrote: It is already possible to use openssl and valgrind - just build OpenSSL with -DPURIFY, and it is quite clean. Actually on my system, just -DPURIFY doesn't satisfy valgrind. What I'm asking for is something that both satisfies valgrind and doesn't reduce the keyspace. If you're using an up-to-date version of openssl when you see this (ie. a recent CVS snapshot from our website, even if it's from a stable branch for compatibility reasons), then please post details. -DPURIFY exists to facilitate debuggers that don't like reading uninitialised data, so if that's not the case then please provide details. Note however that there are a variety of gotchas that allow you to create little leaks if you're not careful, and valgrind could well be complaining about those instead. Note that you should always build with no-asm if you're doing this kind of debug analysis. The assembly optimisations are likely to operate at granularities and in ways that valgrind could easily complain about. I don't know that this is the case, but it would certainly make sense to compare before posting a bug report. you know, this is sheer stupidity. you're suggesting that testing the no-asm code is a valid way of testing the assembly code? additionally the suggestion of -DPURIFY as a way of testing the code is also completely broken software engineering practice. any special case changes for testing means you're not testing the REAL CODE. for example if you build -DPURIFY then you also won't get notified of problems with other PRNG seeds which are supposed to be providing random *initialized* data. not to mention that a system compiled that way is insecure -- so you either have to link your binaries static (to avoid the danger of an insecure shared lib), or set up a chroot for testing. in any event YOU'RE NOT TESTING THE REAL CODE. which is to say you're wasting your time if you test under any of these conditions. openssl should not be relying on uninitialized data for anything. even if it doesn't matter from the point of view of the PRNG, it should be pretty damn clear it's horrible software engineering practice. -dean __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: valgrind and openssl
dean gaudet wrote: On Thu, 15 May 2008, Geoff Thorpe wrote: I forgot to mention something; If you're using an up-to-date version of openssl when you see this (ie. a recent CVS snapshot from our website, even if it's from a stable branch for compatibility reasons), then please post details. -DPURIFY exists to facilitate debuggers that don't like reading uninitialised data, so if that's not the case then please provide details. Note however that there are a variety of gotchas that allow you to create little leaks if you're not careful, and valgrind could well be complaining about those instead. Note that you should always build with no-asm if you're doing this kind of debug analysis. The assembly optimisations are likely to operate at granularities and in ways that valgrind could easily complain about. I don't know that this is the case, but it would certainly make sense to compare before posting a bug report. you know, this is sheer stupidity. you're suggesting that testing the no-asm code is a valid way of testing the assembly code? additionally the suggestion of -DPURIFY as a way of testing the code is also completely broken software engineering practice. any special case changes for testing means you're not testing the REAL CODE. Where is the problem with it? When you add debugging print statements for finding an error, you are also not testing the real code, but when you have found the error, you can remove the print statements and are back to the real code. With -DPURIFY you can do the same. for example if you build -DPURIFY then you also won't get notified of problems with other PRNG seeds which are supposed to be providing random *initialized* data. not to mention that a system compiled that way is insecure -- so you either have to link your binaries static (to avoid the danger of an insecure shared lib), or set up a chroot for testing. Why is a system built with -DPURIFY insecure? I presume you don't understand what -DPURIFY does. in any event YOU'RE NOT TESTING THE REAL CODE. which is to say you're wasting your time if you test under any of these conditions. openssl should not be relying on uninitialized data for anything. even if OpenSSL doesn't rely on uninitialized data! it doesn't matter from the point of view of the PRNG, it should be pretty damn clear it's horrible software engineering practice. It's not at all clear that putting uninitialized data into a PRNG is bad software engineering practice, there may be circumstances where it is the last resort (imagine e.g. that the debian developer would have killed /dev/random instead). Ciao, Richard -- Dr. Richard W. Könning Fujitsu Siemens Computers GmbH __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
RE: valgrind and openssl
any special case changes for testing means you're not testing the REAL CODE. You mean you're not testing *all* of the real code. That's fine, you can't debug everythign at once. for example if you build -DPURIFY then you also won't get notified of problems with other PRNG seeds which are supposed to be providing random *initialized* data. not to mention that a system compiled that way is insecure -- so you either have to link your binaries static (to avoid the danger of an insecure shared lib), or set up a chroot for testing. Right, but you know that. So you don't build with -DPURIFY if you care about things that it affects. But sometimes you care about other things. in any event YOU'RE NOT TESTING THE REAL CODE. which is to say you're wasting your time if you test under any of these conditions. You seem to think that code is one monolithic thing that doesn't consist of component parts. In fact, code does consist of component parts, and the code your actually testing may be a different component from the one you change the compilation flags on. openssl should not be relying on uninitialized data for anything. even if it doesn't matter from the point of view of the PRNG, it should be pretty damn clear it's horrible software engineering practice. No, it's not pretty damn clear. The only reason it might be horrible is because it makes the code less predictable. But in this case, predictability is explicitly undesired. Perhaps you can make a coherent argument why it's bad in this particular case, but I doubt it. This is the opposite of the typical case. -dean Good luck finding people who agree with you. I've been a professional software developer for about 18 years and I've worked on debugging with hundreds of other developers. I have *never* met anyone who shared your view. In fact, it strikes me as sheer craziness. It is akin to saying that debuggers should not exist. After all, the release program won't run with a debugger, so how can you debug with one? Clearly every difference between the test environment and the use environment is a trade-off. But being a competent engineer is about making rational trade-offs. I could go into more detail with real-world examples how following your advice above would have turned very simple efforts into Herculean ones, but what you're saying is so obviously absurd, I can't see how it could possibly be worth the effort. DS __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
RE: valgrind and openssl
What I _wouldn't_ be happy with is a PRNG which has been fed only known data, but enough of it at startup that it agrees to provide output to the user. There are a terrible lot of these around, and pretending that stack contents are random is a great way to accidentally build them. Fortunately, OpenSSL does not pretend the stack contents are random. People sometimes accidentally shoot themselves because they handle a gun as if it was unloaded when it's loaded. But it doesn't make sense to ensure the gun is loaded just to ensure you handle it as if it was loaded. However, this is essentially your argument -- don't do something that can only help because if you relied on just that you might be in trouble. Sorry, I reject that argument. Not feeding in data which you have a pretty darned good idea will be predictable -- potentially as the first bits in at RNG startup -- is to my mind one thing one can should do to avoid the problem. Thor You are honestly arguing that the best way to avoid handling a gun as if it were unloaded is to ensure it is loaded before handling it. Isn't that clearly crazy? Why do something that can only hurt you? That you might get in trouble if you rely on it is an argument not to rely on it, not an argument not to do it. DS __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: valgrind and openssl
On Thu, 15 May 2008, Bodo Moeller wrote: On Thu, May 15, 2008 at 11:41 PM, Erik de Castro Lopo [EMAIL PROTECTED] wrote: Goetz Babin-Ebell wrote: But here the use of this uninitialized data is intentional and the programmer are very well aware of what they did. The use of unititialized data in this case is stupid because the entropy of this random data is close to zero. It may be zero, but it may be more, depending on what happened earlier in the program if the same memory locations have been in use before. This may very well include data that would be unpredictable to adversaries -- i.e., entropy; that's the point here. on the other hand it may be a known plaintext attack. what are you guys smoking? -dean __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
RE: valgrind and openssl
On Mon, 19 May 2008, David Schwartz wrote: any special case changes for testing means you're not testing the REAL CODE. You mean you're not testing *all* of the real code. That's fine, you can't debug everythign at once. if you haven't tested your final production binary then you haven't tested anything at all. Good luck finding people who agree with you. I've been a professional software developer for about 18 years and I've worked on debugging with i've been a professional for longer than you. big whoop. -dean __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: valgrind and openssl
The problems occur on Red Hat 5.1 server x86_64. For what it's worth, I don't get errors on (updated :) Ubuntu 7.10. I do get errors even with Bodo's addition to randfile.c. I'd be happy to post the valgrind output if that would be helpful. If this is environment/OS-specific, then it's probably indicative of a libc (or valgrind, or gcc) issue with the fixed-in-the-past versions of those packages on RH5.1-64bit. If compiling and running the same openssl source on a different system doesn't give you problems, that would seem to imply that the problem appears and disappears based on elements that vary rather than elements that are invariant. :-) It's environment dependent. I'm not able to reproduce the valgrind errors even on other 64 bit RH5.1 machines. As far as I'm concerned, openssl valgrinds clean with ./config no-asm -DPURIFY: my goals are achieved. -JP __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: valgrind and openssl
On Mon, May 19, 2008 at 10:48 PM, dean gaudet [EMAIL PROTECTED] wrote: On Thu, 15 May 2008, Bodo Moeller wrote: On Thu, May 15, 2008 at 11:41 PM, Erik de Castro Lopo [EMAIL PROTECTED] wrote: Goetz Babin-Ebell wrote: But here the use of this uninitialized data is intentional and the programmer are very well aware of what they did. The use of unititialized data in this case is stupid because the entropy of this random data is close to zero. It may be zero, but it may be more, depending on what happened earlier in the program if the same memory locations have been in use before. This may very well include data that would be unpredictable to adversaries -- i.e., entropy; that's the point here. on the other hand it may be a known plaintext attack. OK, so I'll seed my random number generator with a bunch of bits you don't know, then you give me however much known plaintext you want, and I'll update the state with that too. Then, I'll start generating random numbers. If you can guess them, you win! Right? Essentially what you're claiming is that you can predict the output of SHA-1 when you know part of the input, but not all of the input. Please explain how! -JP what are you guys smoking? -dean __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED] __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: valgrind and openssl
Hi, If feeding predictable data into a PRNG that was already well seeded with unpredictable data produced a weaker PRNG, then you have found a security bug in the PRNG and I suggest you publish. Yeah, I've heard that a few times. However, consider the pathological case, in which an adversary manages to introduce N-1 bits of known state into your PRNG which has N bits of internal state. Are you comfortable with that? Yes and no. No, because I don't like it, if an attacker even knows a single bit of the internal state of the PRNG. But yes, because it simply doesn't work the way your question makes it seem: If you add known bits to the state, those do _not_ simply replace the state bits but are mixed into it. If you have N state bits, even if you add N^2 or 2^N or whatever bits to the state, you still don't know any single bit of the state. I'll give a simplified example on how this mixing in works, so you can see yourself, why everybody is confident about this: Think of adding bits of known state as really adding N bit numbers via standard addition (ignoring the overflow bit, i.e. addition modulo 2^N). So you have an unknown state and add a known number to it (or any number of known numbers). You still don't know the sum, i.e. the internal state is still perfectly safe. And vice versa, if you do know the resulting state (or even a single bit in it), you can simply do a subtraction and get to know the original state (or the corresponding single bit in it), i.e. if the new state is predictable, so is the original one, i.e. you don't loose any security. On the other hand, if an attacker got to know some bits of the internal state and you add N unknown bits into the state, the result will be unknown again. So the worst case (all bits are known) doesn't do any harm, while the best case (some bits are really unpredictable) is really helpful. HTH, Stefan __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: valgrind and openssl
Thor Lancelot Simon wrote: ... However, consider the pathological case, in which an adversary manages to introduce N-1 bits of known state into your PRNG which has N bits of internal state. ... What you seem not to understand from this discussion is that the internal state is a consequence of input that is processed via a diffusion mechanism, a cryptographic hash such as SHA1 or MD5 or something stronger. These change roughly half the bits of their output when a single input bit is changed. It is not possible to know the state itself without direct inspection of structures whose security is obviously essential for useful random number generation. If your point is that system events are not a good source of entropy, I agree. __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: valgrind and openssl
On Sun, May 18, 2008 at 08:41:36AM -0700, Michael Sierchio wrote: Thor Lancelot Simon wrote: ... However, consider the pathological case, in which an adversary manages to introduce N-1 bits of known state into your PRNG which has N bits of internal state. ... What you seem not to understand from this discussion is that the internal state is a consequence of input that is processed via a diffusion mechanism, a cryptographic hash such as SHA1 or MD5 or something stronger. These change roughly half the bits of their output when a single input bit is changed. So you're comfortable with the adversary knowing, let's say, 511 of the first 512 bits fed through SHA1? Maybe I haven't been clear enough here: I specifically object to introducing easy to know information into the PRNG at startup time, though if one is going to feed it in at other times because it can't hurt and extra iterations of the mixing function are thought to be helpful, I can't really see why not feed in 0x every 100ms from a timer -- it'd do about as much good. Thor __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
RE: valgrind and openssl
Thor Lancelot Simon wrote: So you're comfortable with the adversary knowing, let's say, 511 of the first 512 bits fed through SHA1? I'm comfortable knowing any number of bits fed into or through the SHA1 provided there are also sufficient bits he does not know. The issue of how many bits he does know is a complete and utter red herring. It doesn't matter. All that matters is whether you feed in enough bits he *doesn't* know. Maybe I haven't been clear enough here: I specifically object to introducing easy to know information into the PRNG at startup time, though if one is going to feed it in at other times because it can't hurt and extra iterations of the mixing function are thought to be helpful, I can't really see why not feed in 0x every 100ms from a timer -- it'd do about as much good. It doesn't hurt at startup time. It doesn't hurt at any time. Mixing in data an adversary knows *never* hurts you. What hurts you is *not* mixing in enough information the attacker does not know. You are confusing the issue by focusing on things that are not the problem. DS __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: valgrind and openssl
On Sun, May 18, 2008 at 05:24:51PM -0400, Thor Lancelot Simon wrote: So you're comfortable with the adversary knowing, let's say, 511 of the first 512 bits fed through SHA1? *Sigh*. Thor, you clearly have no idea how SHA-1 works. In fact, I'd be comfortable with an adversary knowing the first megabyte of data fed through SHA1, as long as it was followed up by at least 256 bits which the adversary *didn't* know. Look, SHA-1 works by having a Very Complicated Mixing function that takes a state function, and mixes in the input in a one-way fashion, in chunks of 64 bytes at a time. The initial state looks like this: 67452301 efcdab89 98badcfe 10325476 c3d2e1f0 It doesn't look very random, but that's OK. You have to start *somewhere* --- and it's a public value. If you mix in a megabyte of known data, it is the equivalent of changing the initial state to something else. Effectively, it's another public starting value. As long as follow up the megabyte of known data with 256 bits of unknown data, you could feed another megabyte of known data, and the adversary would have no idea what the internal state of the SHA-1 hash function would look like. If this were not true, SHA-1's mixing function would be so throughly broken that all use of SHA-1 for digital signatures, certificates, etc., would be totally broken. So if you don't trust SHA-1 for use in PRNG's, then you shouldn't trust SHA-1 for *anything*. - Ted __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: valgrind and openssl
Theodore Tso wrote: ... I'd be comfortable with an adversary knowing the first megabyte of data fed through SHA1, as long as it was followed up by at least 256 bits which the adversary *didn't* know. I'd be comfortable with an adversary knowing the first zetabyte of data fed though SHA1, as long as it was followed up by at least 256 bits which the adversary didn't know. ;-) Cheers, Michael __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
RE: valgrind and openssl
Unfortunately, it may also very well include data that would be highly predictable to adversaries. That doesn't matter. I am aware that this is an area without a lot of good theoretical signposts, but I am just not very comfortable feeding arbitrary amounts of possibly-known data into a PRNG. Thor Don't worry about it. One of the specific security objectives of the PRNG is that mixing in known data does not degrade the pool. If you don't trust it not to get worse with known data being mixed in, then you don't trust it at all and you shouldn't use it. If you can't trust your selected algorithms to meet their specific security objectives, then you can't use those algorithms. DS __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: valgrind and openssl
On Fri, May 16, 2008 at 11:24:45AM -0400, Geoff Thorpe wrote: On Friday 16 May 2008 00:47:52 Thor Lancelot Simon wrote: On Thu, May 15, 2008 at 11:45:14PM +0200, Bodo Moeller wrote: It may be zero, but it may be more, depending on what happened earlier in the program if the same memory locations have been in use before. This may very well include data that would be unpredictable to adversaries -- i.e., entropy; that's the point here. Unfortunately, it may also very well include data that would be highly predictable to adversaries. If feeding predictable data into a PRNG that was already well seeded with unpredictable data produced a weaker PRNG, then you have found a security bug in the PRNG and I suggest you publish. Yeah, I've heard that a few times. However, consider the pathological case, in which an adversary manages to introduce N-1 bits of known state into your PRNG which has N bits of internal state. Are you comfortable with that? For what value M are you comfortable with N - M bits of the state having been introduced by the adversary? Why? It seems to me that best practice is to not introduce such state if one can avoid it, whether one counts it into an entropy estimate or not. Thor __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
RE: valgrind and openssl
If feeding predictable data into a PRNG that was already well seeded with unpredictable data produced a weaker PRNG, then you have found a security bug in the PRNG and I suggest you publish. Yeah, I've heard that a few times. However, consider the pathological case, in which an adversary manages to introduce N-1 bits of known state into your PRNG which has N bits of internal state. Are you comfortable with that? Yes. For what value M are you comfortable with N - M bits of the state having been introduced by the adversary? Why? An attacker can feed an infinite amount of known entropy into the pool. The result is that it has as much entropy as it had before he did that. It seems to me that best practice is to not introduce such state if one can avoid it, whether one counts it into an entropy estimate or not. Why? Because it's always possible that the algorithm actually has some unknown defect that might make this bad? What if the algorithm has some unknown defect that doing this fixes? The entropy pool is specifically designed, as a major security objective, such that: 1) An attacker can remove an unlimited amount of entropy from the pool and it is still computationally infeasible to know anything about the next byte that will come out, and 2) An attacker can add an unlimited amount of known entropy to the pool and it is still computationally infeasible to know anything abou the next byte that will come, and 3) An attacker can do both 2 and 3 in any amount in any computation, and the pool is still just as secure as it was before he did so. These are not accidents. These are the specific design criteria for the entropy pool PRNG used and its design specifically reflects these objectives. If you have any reason or evidence to suspect that any of these is not true, please share it. Otherwise, you're just asking us to hope that the hypothetical attack this might somehow make us less vulnerable to is more significant to the hypothetical attack this might somehow make us more vulnerable too. And that would just be your guess. DS __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
RE: valgrind and openssl
Hi, It certainly would, but Valgrind isn't the only analysis tool people might want to use. A runtime flag provides a means of obtaining accurate results with any tool. Unfortunately, for am attacker it also provides a means of (possibly) weakening your program's randomness behind your back, so IMO any way to configure this behaviour at runtime is a potential security issue. IMO it's better to tell people unable to ignore bogus warnings (or make their analysis tool ignore them, e.g. via a suppressions file) to use a special build option, like it's currently done. In fact, I'd generally (i.e. not specifically for OpenSSL) wish developers would invest more of their time into fixing bugs and/or adding features rather than trying to get rid of the last bogus warnings caused by some more or less obscure compiler or analysis tool - that would also (need to) result in a healthier reaction to such warnings by other developers and packagers: just assume that the authors know what they are doing instead of trying to fix even the last warning by whatever inappropriate means. Regards, Stefan P.S.: Of course, I realize it's a problem if you have lots of bogus warnings and can't even find the real problematic ones in between them, but there must be a reasonable compromise somewhere between a lot and zero. __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: valgrind and openssl
On Thu, May 15, 2008 at 11:45:14PM +0200, Bodo Moeller wrote: On Thu, May 15, 2008 at 11:41 PM, Erik de Castro Lopo [EMAIL PROTECTED] wrote: Goetz Babin-Ebell wrote: But here the use of this uninitialized data is intentional and the programmer are very well aware of what they did. The use of unititialized data in this case is stupid because the entropy of this random data is close to zero. It may be zero, but it may be more, depending on what happened earlier in the program if the same memory locations have been in use before. This may very well include data that would be unpredictable to adversaries -- i.e., entropy; that's the point here. Unfortunately, it may also very well include data that would be highly predictable to adversaries. I am aware that this is an area without a lot of good theoretical signposts, but I am just not very comfortable feeding arbitrary amounts of possibly-known data into a PRNG. Thor __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: valgrind and openssl
On Thu, May 15, 2008 at 06:17:03PM -0400, Geoff Thorpe wrote: On Thursday 15 May 2008 17:31:45 Erik de Castro Lopo wrote: Geoff Thorpe wrote: Then tell your linux distribution to use -DPURIFY. Hangon, I've got a better idea. How about the OpenSSL develoeprs fix their library so that the standard version that they ship is valgrind clean. Then the distributions won't need to do anything other than compile it. What, you mean like how the standard version we ship has a good PRNG, so that the distributions don't need to do anything about that either? Funny, that doesn't work so well in the real world. I do have to question whether mixing in predictable data from the stack at RNG initialization time is such a great idea. Thor __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: valgrind and openssl
Bodo Moeller wrote: However, another intentional use of potentially unitialized data is still left as of http://cvs.openssl.org/getfile/openssl/crypto/rand/randfile.c?v=1.47.2.2 : i=fread(buf,1,n,in); if (i = 0) break; /* even if n != i, use the full array */ RAND_add(buf,n,(double)i); Changing this into RAND_add(buf,i,(double)i) should make verification tools happier. Or it could be #ifdef PURIFY RAND_add(buf,i,(double)i); #else RAND_add(buf,n,(double)i); #endif (abusing the PURIFY macro with a more general meaning). Good catch, patch applied :-) Best regards, Lutz __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: valgrind and openssl
On Fri, May 16, 2008 at 6:47 AM, Thor Lancelot Simon [EMAIL PROTECTED] wrote: On Thu, May 15, 2008 at 11:45:14PM +0200, Bodo Moeller wrote: On Thu, May 15, 2008 at 11:41 PM, Erik de Castro Lopo [EMAIL PROTECTED] wrote: Goetz Babin-Ebell wrote: But here the use of this uninitialized data is intentional and the programmer are very well aware of what they did. The use of unititialized data in this case is stupid because the entropy of this random data is close to zero. It may be zero, but it may be more, depending on what happened earlier in the program if the same memory locations have been in use before. This may very well include data that would be unpredictable to adversaries -- i.e., entropy; that's the point here. Unfortunately, it may also very well include data that would be highly predictable to adversaries. Sure. That's not a problem, though. What happens to the PRNG then is not too different from what happens when you use it to output bits (except that with RAND_add(), there is no output that might be seen by the adversary, so seeding with known data should actually be safer than generating output if you're worrying about this kind of things at all). The adversary may know something about what is going on, but the internal state still remains secret; and the internal state's entropy won't be adversely affected more than marginally if at all. (Because of the way the internal state is structured, this stirring achieved even with a fixed input might even be considered a feature to improve the distribution of whatever entropy you already have.) Bodo __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: valgrind and openssl
On Thursday 15 May 2008 16:51:55 John Parker wrote: I'm still seeing a lot of errors from valgrind, even with the latest snapshot. 19 15:12 tar xvfz ../openssl-0.9.8-stable-SNAP-20080515.tar.gz 20 15:12 cd openssl-0.9.8-stable-SNAP-20080515/ 21 15:12 ls 22 15:12 ./config no-asm -DPURIFY 23 15:12 make 24 15:14 valgrind ./apps/openssl genrsa 1024 Please let me know if I'm doing something wrong with this test sequence. The problems occur on Red Hat 5.1 server x86_64. For what it's worth, I don't get errors on (updated :) Ubuntu 7.10. I do get errors even with Bodo's addition to randfile.c. I'd be happy to post the valgrind output if that would be helpful. If this is environment/OS-specific, then it's probably indicative of a libc (or valgrind, or gcc) issue with the fixed-in-the-past versions of those packages on RH5.1-64bit. If compiling and running the same openssl source on a different system doesn't give you problems, that would seem to imply that the problem appears and disappears based on elements that vary rather than elements that are invariant. :-) OTOH, it might just be that the different OSes catch things differently. In any case, I don't have (or want) a RH5.1-64bit installation, so you might as well post some clippings from your output to give us an idea of what it's showing. (If possible, could you please use a snapshot from HEAD for this instead of 0.9.8-stable?) TIA. Cheers, Geoff PS: I wonder whether I should be so dismissive of RH5.1 - as far as I know, it always generated usable keys ... :-) __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: valgrind and openssl
On Friday 16 May 2008 00:47:52 Thor Lancelot Simon wrote: On Thu, May 15, 2008 at 11:45:14PM +0200, Bodo Moeller wrote: It may be zero, but it may be more, depending on what happened earlier in the program if the same memory locations have been in use before. This may very well include data that would be unpredictable to adversaries -- i.e., entropy; that's the point here. Unfortunately, it may also very well include data that would be highly predictable to adversaries. If feeding predictable data into a PRNG that was already well seeded with unpredictable data produced a weaker PRNG, then you have found a security bug in the PRNG and I suggest you publish. (Or put it in a distribution-local patch and wait a couple of years...) After all, the PRNG itself is highly predictable to adversaries - it's open source - so the requirement of the PRNG is for it's stirring mechanism to be essentially mononotic w.r.t. entropy gained from the input. Any unpredictability in the data you feed it should make the PRNG output correspondingly unpredictable - and that should not become more predictable (ie. less unpredictable) because you supplement the existing unpredictable input with predictable input. That's the point - feed whatever you've got into it, and the resulting randomness is (roughly) as good as the total entropy of your input, no matter how sparse that input's entropy was. Cheers, Geoff __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: valgrind and openssl
On Thu, May 15, 2008 at 4:58 PM, John Parker [EMAIL PROTECTED] wrote: In the wake of the issues with Debian, is it possible to modify the source so that it is possible to use valgrind with openssl without reducing the key space? Sure. This might happen with the next release. Are we really relying on uninitialized memory for randomness? Not at all. It's just that OpenSSL in some situations tries to feed possibly uninitialized memory into the random number generator anyway, essentially just for fun and because their *might* be some actual randomness there from whatever happened earlier in the same process. The Debian-internal patch was blatantly overbroad in disabling the essential functionality of the RAND_add() function rather than just avoiding the one case where this function might have been called with uninitialized memory. (That one case is in RAND_load_file(), which would intentionally feed a complete 1024-byte buffer to RAND_add() even if fewer than 1024 bytes had been put into the buffer by fread().) Bodo __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: valgrind and openssl
On May 15, 2008 10:58:07 am John Parker wrote: In the wake of the issues with Debian, is it possible to modify the source so that it is possible to use valgrind with openssl without reducing the key space? It is already possible to use openssl and valgrind - just build OpenSSL with -DPURIFY, and it is quite clean. (we do it all the time here with WvStreams and Pathfinder, and it works like a charm). Have fun. -- Patrick Patterson President and Chief PKI Architect, Carillon Information Security Inc. http://www.carillon.ca __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: valgrind and openssl
Patrick Patterson [EMAIL PROTECTED] writes: On May 15, 2008 10:58:07 am John Parker wrote: In the wake of the issues with Debian, is it possible to modify the source so that it is possible to use valgrind with openssl without reducing the key space? It is already possible to use openssl and valgrind - just build OpenSSL with -DPURIFY, and it is quite clean. Even with -DPURIFY there was (prior to 0.9.8f) some uninitialised memory got used if you didn't have a .rnd file (the stat() in crypto/rand/randfile.c). (I'm not sure whether that change ever got on to 0.9.7. Hmm, I think it didn't. Likely not critical, I guess, though Debian/stable is on 0.9.7. Backporting that (trivial) change seems like something a distribution could uncontroversially do.) __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: valgrind and openssl
It is already possible to use openssl and valgrind - just build OpenSSL with -DPURIFY, and it is quite clean. (we do it all the time here with WvStreams and Pathfinder, and it works like a charm). The problem is that this may reduce the keyspace so that keys are guessable. http://blog.isotoma.com/2008/05/debians_openssl_disaster.html -JP __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: valgrind and openssl
On Thursday 15 May 2008 11:52:08 John Parker wrote: It is already possible to use openssl and valgrind - just build OpenSSL with -DPURIFY, and it is quite clean. (we do it all the time here with WvStreams and Pathfinder, and it works like a charm). The problem is that this may reduce the keyspace so that keys are guessable. No it won't, it removes an entropy source whose quality is known to be unknown, ie. it may add nothing useful, it gets used just in case. Removing it does not reduce the keypsace at all. All you can say is that leaving it there *may* improve the PRNG depending on the user, the environment, the application, and quite probably, the alignment of the planets... The debian patch went further than -DPURIFY, as it removed more than just this unreliable source, it removed all use of reliable sources as well. http://blog.isotoma.com/2008/05/debians_openssl_disaster.html This blog does not suggest that building with -DPURIFY would a problem and nor should it. I think you may have misunderstood the details of this issue. Cheers, Geoff __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: valgrind and openssl
All of this is independent of proper entropy seeding to the PRNG, which is what the debian patch crushed and which in turn led to the high seismic reading in the blogosphere. But it may help explain why I do *not* want us to unilaterally remove the use of uninitialised data in the PRNG. That seems to be motivated by a capitulation to the weight of users (or packagers) who don't know how to read the FAQ. Perhaps what we should do instead is I think we should be less worried how things seem and more worried about the practical consequences. change -DPURIFY to -DNO_UNINIT_DATA or something else which has a clearer intention, so that debug packages (or even base packages that want to be valgrind-friendly) have a straightforward mechanism to apply. Well, a straightforward mechanism that doesn't kill the PRNG outright, I mean (otherwise there is already a highly-publicised patch we could apply...) What I was hoping for was a -DNO_UNINIT_DATA that wouldn't be the default, but wouldn't reduce the keyspace either. Can someone provide a pointer to this highly-publicized patch? I'm afraid I'm dreadfully ignorant of the blogosphere. -JP __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: valgrind and openssl
It is already possible to use openssl and valgrind - just build OpenSSL with -DPURIFY, and it is quite clean. Actually on my system, just -DPURIFY doesn't satisfy valgrind. What I'm asking for is something that both satisfies valgrind and doesn't reduce the keyspace. (we do it all the time here with WvStreams and Pathfinder, and it works like a charm). The problem is that this may reduce the keyspace so that keys are guessable. No it won't, it removes an entropy source whose quality is known to be unknown, ie. it may add nothing useful, it gets used just in case. Removing it does not reduce the keypsace at all. All you can say is that leaving it there *may* improve the PRNG depending on the user, the environment, the application, and quite probably, the alignment of the planets... The debian patch went further than -DPURIFY, as it removed more than just this unreliable source, it removed all use of reliable sources as well. http://blog.isotoma.com/2008/05/debians_openssl_disaster.html This blog does not suggest that building with -DPURIFY would a problem and nor should it. I think you may have misunderstood the details of this issue. I am clearly misunderstanding something. You seem to be saying that -DPURIFY satisfies valgrind but doesn't reduce the keyspace. I'm prepared to take it on faith that -DPURIFY doesn't reduce the keyspace. -JP __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: valgrind and openssl
On May 15, 2008 12:38:24 pm John Parker wrote:
It is already possible to use openssl and valgrind - just build
OpenSSL with -DPURIFY, and it is quite clean.
Actually on my system, just -DPURIFY doesn't satisfy valgrind. What
I'm asking for is something that both satisfies valgrind and doesn't
reduce the keyspace.
What you simply need to do is make a supplemental ignore known issues file
for valgrind - for WvStreams, we use one that has the following in it:
{
more_fun_libcrypto_junk
Memcheck:Value4
fun:BF_encrypt
}
{
more_fun_libcrypto_junk_2
Memcheck:Value4
fun:AES_encrypt
}
{
more_fun_libcrypto_junk_3
Memcheck:Addr4
fun:AES_cbc_encrypt
}
{
more_fun_libcrypto_junk_4
Memcheck:Value4
fun:_x86_AES_encrypt
}
And then we run all of the unit tests through valgrind with the options:
valgrind --tool=memcheck --leak-check=yes --num-callers=10
--suppressions=$(WVSTREAMS_SRC)/wvstreams.supp
Between -DPURIFY and the suppressions, the unit tests come back clean (when we
haven't made any silly mistakes in our own code, of course :)
BTW: I'm not claiming that the above list of suppressions will work 100% for
you - the suppressions above are for things that our code tickles - you may
want to add more of them for those specific areas that your code touches that
ours does not.
Have fun.
--
Patrick Patterson
President and Chief PKI Architect,
Carillon Information Security Inc.
http://www.carillon.ca
__
OpenSSL Project http://www.openssl.org
Development Mailing List openssl-dev@openssl.org
Automated List Manager [EMAIL PROTECTED]
Re: valgrind and openssl
On Thursday 15 May 2008 12:38:24 John Parker wrote: It is already possible to use openssl and valgrind - just build OpenSSL with -DPURIFY, and it is quite clean. Actually on my system, just -DPURIFY doesn't satisfy valgrind. What I'm asking for is something that both satisfies valgrind and doesn't reduce the keyspace. If you're using an up-to-date version of openssl when you see this (ie. a recent CVS snapshot from our website, even if it's from a stable branch for compatibility reasons), then please post details. -DPURIFY exists to facilitate debuggers that don't like reading uninitialised data, so if that's not the case then please provide details. Note however that there are a variety of gotchas that allow you to create little leaks if you're not careful, and valgrind could well be complaining about those instead. This blog does not suggest that building with -DPURIFY would a problem and nor should it. I think you may have misunderstood the details of this issue. I am clearly misunderstanding something. You seem to be saying that -DPURIFY satisfies valgrind but doesn't reduce the keyspace. I'm prepared to take it on faith that -DPURIFY doesn't reduce the keyspace. Well, more generally than some keyspace is the randomness of the PRNG itself. (Your keys are only random if the PRNG's output is random.) But yes, I'm saying that -DPURIFY does not diminish the quality of the PRNG, except *possibly* by some unquantifiable amount that you couldn't safely depend on anyway. As for your other mail; On Thursday 15 May 2008 12:09:46 John Parker wrote: All of this is independent of proper entropy seeding to the PRNG, which is what the debian patch crushed and which in turn led to the high seismic reading in the blogosphere. But it may help explain why I do *not* want us to unilaterally remove the use of uninitialised data in the PRNG. That seems to be motivated by a capitulation to the weight of users (or packagers) who don't know how to read the FAQ. Perhaps what we should do instead is I think we should be less worried how things seem and more worried about the practical consequences. That is more or less what I was doing. I hope that was clear. change -DPURIFY to -DNO_UNINIT_DATA or something else which has a clearer intention, so that debug packages (or even base packages that want to be valgrind-friendly) have a straightforward mechanism to apply. Well, a straightforward mechanism that doesn't kill the PRNG outright, I mean (otherwise there is already a highly-publicised patch we could apply...) What I was hoping for was a -DNO_UNINIT_DATA that wouldn't be the default, but wouldn't reduce the keyspace either. I believe this has been answered. For now, it's called -DPURIFY. Can someone provide a pointer to this highly-publicized patch? I'm afraid I'm dreadfully ignorant of the blogosphere. You started this mail thread, so you go and find it! :-) The patch I was referring to, tongue-in-cheek, is the debian patch that crippled the PRNG. As for the blogosphere, you aren't missing much, I'd recommend that continued ignorance would be far from dreadful - in fact I intend to join you in that respect, once this was-it-debian's-fault-or-openssl's-fault nonsense has died down a bit. Cheers, Geoff __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: valgrind and openssl
John Parker, 2008-05-15: It is already possible to use openssl and valgrind - just build OpenSSL with -DPURIFY, and it is quite clean. Actually on my system, just -DPURIFY doesn't satisfy valgrind. What I'm asking for is something that both satisfies valgrind and doesn't reduce the keyspace. Valgrind can be told to ignore specific errors, using a suppressions file. Never used this with OpenSSL, though. http://valgrind.org/docs/manual/manual-core.html#manual-core.suppress Leandro __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: valgrind and openssl
I forgot to mention something; On Thursday 15 May 2008 12:38:24 John Parker wrote: It is already possible to use openssl and valgrind - just build OpenSSL with -DPURIFY, and it is quite clean. Actually on my system, just -DPURIFY doesn't satisfy valgrind. What I'm asking for is something that both satisfies valgrind and doesn't reduce the keyspace. If you're using an up-to-date version of openssl when you see this (ie. a recent CVS snapshot from our website, even if it's from a stable branch for compatibility reasons), then please post details. -DPURIFY exists to facilitate debuggers that don't like reading uninitialised data, so if that's not the case then please provide details. Note however that there are a variety of gotchas that allow you to create little leaks if you're not careful, and valgrind could well be complaining about those instead. Note that you should always build with no-asm if you're doing this kind of debug analysis. The assembly optimisations are likely to operate at granularities and in ways that valgrind could easily complain about. I don't know that this is the case, but it would certainly make sense to compare before posting a bug report. Cheers, Geoff __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: valgrind and openssl
On Thu, May 15, 2008 at 11:09:46AM -0500, John Parker wrote: change -DPURIFY to -DNO_UNINIT_DATA or something else which has a clearer intention, so that debug packages (or even base packages that want to be valgrind-friendly) have a straightforward mechanism to apply. Well, a straightforward mechanism that doesn't kill the PRNG outright, I mean (otherwise there is already a highly-publicised patch we could apply...) What I was hoping for was a -DNO_UNINIT_DATA that wouldn't be the default, but wouldn't reduce the keyspace either. -DPURIFY *does* do what you want. It doesn't reduce the keyspace. The problem was that what Debian did went far beyond -DPURIFY. The Debian developer in question disabled one call that used uninitialized memory, but then later on, removed another similar call that looked the same, but in fact *was* using initialized data --- said initialized data being real randomness critically necessary for security. - Ted __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: valgrind and openssl
John Parker wrote: change -DPURIFY to -DNO_UNINIT_DATA or something else which has a clearer intention, so that debug packages (or even base packages that want to be valgrind-friendly) have a straightforward mechanism to apply. Well, a straightforward mechanism that doesn't kill the PRNG outright, I mean (otherwise there is already a highly-publicised patch we could apply...) What I was hoping for was a -DNO_UNINIT_DATA that wouldn't be the default, but wouldn't reduce the keyspace either. That is -DPURIFY. Can someone provide a pointer to this highly-publicized patch? I'm afraid I'm dreadfully ignorant of the blogosphere. The Debian patch is the highly publicized patch that kills the PRNG outright. Jeffrey Altman smime.p7s Description: S/MIME Cryptographic Signature
Re: valgrind and openssl
On Thu, May 15, 2008 at 7:53 PM, Theodore Tso [EMAIL PROTECTED] wrote: On Thu, May 15, 2008 at 11:09:46AM -0500, John Parker wrote: What I was hoping for was a -DNO_UNINIT_DATA that wouldn't be the default, but wouldn't reduce the keyspace either. -DPURIFY *does* do what you want. It doesn't reduce the keyspace. The problem was that what Debian did went far beyond -DPURIFY. The Debian developer in question disabled one call that used uninitialized memory, but then later on, removed another similar call that looked the same, but in fact *was* using initialized data --- said initialized data being real randomness critically necessary for security. This similar call would, under certain conditions, use uninitialized data too. I guess Valgrind is more thorough than Purify, because it seems that those using Purify were not shown this as suspicious, and thus -DPURIFY didn't cover this particular case. Of course, totally disabling the offending call in md_rand.c as was done in Debian was blatantly wrong. The correct way would have been to change RAND_load_file() in randfile.c; that's the function thatt might sometimes pass uninitialized data to RAND_add() (intentionally, but without relying on this uninitialized data as a source of randomness). One of the offending RAND_add() calls has already been taken care of about a year ago: http://cvs.openssl.org/filediff?f=openssl/crypto/rand/randfile.cv1=1.47.2.1v2=1.47.2.2 However, another intentional use of potentially unitialized data is still left as of http://cvs.openssl.org/getfile/openssl/crypto/rand/randfile.c?v=1.47.2.2 : i=fread(buf,1,n,in); if (i = 0) break; /* even if n != i, use the full array */ RAND_add(buf,n,(double)i); Changing this into RAND_add(buf,i,(double)i) should make verification tools happier. Or it could be #ifdef PURIFY RAND_add(buf,i,(double)i); #else RAND_add(buf,n,(double)i); #endif (abusing the PURIFY macro with a more general meaning). Bodo __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: valgrind and openssl
On Thu, May 15, 2008 at 12:29 PM, Geoff Thorpe [EMAIL PROTECTED] wrote: I forgot to mention something; On Thursday 15 May 2008 12:38:24 John Parker wrote: It is already possible to use openssl and valgrind - just build OpenSSL with -DPURIFY, and it is quite clean. Actually on my system, just -DPURIFY doesn't satisfy valgrind. What I'm asking for is something that both satisfies valgrind and doesn't reduce the keyspace. If you're using an up-to-date version of openssl when you see this (ie. a recent CVS snapshot from our website, even if it's from a stable branch for compatibility reasons), then please post details. -DPURIFY exists to facilitate debuggers that don't like reading uninitialised data, so if that's not the case then please provide details. Note however that there are a variety of gotchas that allow you to create little leaks if you're not careful, and valgrind could well be complaining about those instead. Note that you should always build with no-asm if you're doing this kind of debug analysis. The assembly optimisations are likely to operate at granularities and in ways that valgrind could easily complain about. I don't know that this is the case, but it would certainly make sense to compare before posting a bug report. I'm still seeing a lot of errors from valgrind, even with the latest snapshot. 19 15:12 tar xvfz ../openssl-0.9.8-stable-SNAP-20080515.tar.gz 20 15:12 cd openssl-0.9.8-stable-SNAP-20080515/ 21 15:12 ls 22 15:12 ./config no-asm -DPURIFY 23 15:12 make 24 15:14 valgrind ./apps/openssl genrsa 1024 Please let me know if I'm doing something wrong with this test sequence. The problems occur on Red Hat 5.1 server x86_64. For what it's worth, I don't get errors on (updated :) Ubuntu 7.10. I do get errors even with Bodo's addition to randfile.c. I'd be happy to post the valgrind output if that would be helpful. -JP __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: valgrind and openssl
Theodore Tso wrote: On Thu, May 15, 2008 at 11:09:46AM -0500, John Parker wrote: change -DPURIFY to -DNO_UNINIT_DATA or something else which has a clearer intention, so that debug packages (or even base packages that want to be valgrind-friendly) have a straightforward mechanism to apply. Well, a straightforward mechanism that doesn't kill the PRNG outright, I mean (otherwise there is already a highly-publicised patch we could apply...) What I was hoping for was a -DNO_UNINIT_DATA that wouldn't be the default, but wouldn't reduce the keyspace either. -DPURIFY *does* do what you want. But thats a compile time option. I would prefer not to have to compile my own version of OpenSSL just to be able to valgrind my program which links against openssl. Erik -- - Erik de Castro Lopo - #!/bin/sh unzip ; strip; touch ; finger ; mount ; gasp ; yes ; more ; umount ; sleep ; __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: valgrind and openssl
Patrick Patterson wrote: On May 15, 2008 10:58:07 am John Parker wrote: In the wake of the issues with Debian, is it possible to modify the source so that it is possible to use valgrind with openssl without reducing the key space? It is already possible to use openssl and valgrind - just build OpenSSL with -DPURIFY, and it is quite clean. A compile time option is not enough. I would like the be able to valgrind my program linked against the standard openssl library I get from my Linux distribution. Erik -- - Erik de Castro Lopo - We reject kings, presidents, and voting. We believe in rough consensus and running code. -- Dave Clark (IETF 1992) __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: valgrind and openssl
On Thursday 15 May 2008 16:56:17 Erik de Castro Lopo wrote: Patrick Patterson wrote: On May 15, 2008 10:58:07 am John Parker wrote: In the wake of the issues with Debian, is it possible to modify the source so that it is possible to use valgrind with openssl without reducing the key space? It is already possible to use openssl and valgrind - just build OpenSSL with -DPURIFY, and it is quite clean. A compile time option is not enough. I would like the be able to valgrind my program linked against the standard openssl library I get from my Linux distribution. Then tell your linux distribution to use -DPURIFY. Or order me an 8-core magic wand from Dell and I'll get right on it. Cheers, Geoff __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: valgrind and openssl
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Erik de Castro Lopo schrieb: | Theodore Tso wrote: | | On Thu, May 15, 2008 at 11:09:46AM -0500, John Parker wrote: | What I was hoping for was a -DNO_UNINIT_DATA that wouldn't be the | default, but wouldn't reduce the keyspace either. | -DPURIFY *does* do what you want. | | But thats a compile time option. I would prefer not to have to compile | my own version of OpenSSL just to be able to valgrind my program which | links against openssl. Then configure your valgrind to ignore this uninitialized use of data. Normally usage of uninitialized data is an oversight done by the programmer that may cause unintended consequences. Because of that valgrind complains about them. But here the use of this uninitialized data is intentional and the programmer are very well aware of what they did. Perhaps it would be a good idea to add a README.valgrind containing a description of what happens and how to configure valgrind to ignore the uninitialized reads ? (Perhaps people that won't read the FAQ will read that) Goetz - -- DMCA: The greed of the few outweighs the freedom of the many -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.4-svn0 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFILKt52iGqZUF3qPYRAtkIAJ47deAtIXtN0DKJGN61CtZyimI3jACfUPDJ ddRbnQn5NF5h0Y6P6pLUHP4= =8wkC -END PGP SIGNATURE- __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: valgrind and openssl
Geoff Thorpe wrote: Then tell your linux distribution to use -DPURIFY. Hangon, I've got a better idea. How about the OpenSSL develoeprs fix their library so that the standard version that they ship is valgrind clean. Then the distributions won't need to do anything other than compile it. Erik -- - Erik de Castro Lopo - The difference between genius and stupidity is that genius has its limits. __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: valgrind and openssl
On Thu, May 15, 2008 at 11:41 PM, Erik de Castro Lopo [EMAIL PROTECTED] wrote: Goetz Babin-Ebell wrote: But here the use of this uninitialized data is intentional and the programmer are very well aware of what they did. The use of unititialized data in this case is stupid because the entropy of this random data is close to zero. It may be zero, but it may be more, depending on what happened earlier in the program if the same memory locations have been in use before. This may very well include data that would be unpredictable to adversaries -- i.e., entropy; that's the point here. __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: valgrind and openssl
Bodo Moeller wrote: It may be zero, but it may be more, depending on what happened earlier in the program if the same memory locations have been in use before. This may very well include data that would be unpredictable to adversaries -- i.e., entropy; that's the point here. Do you know its unpredicatable or are you only guessing? Can a bad guy force it to be predicatable? How much entropy is actually there? Has anyone actually measured it? Erik -- - Erik de Castro Lopo - Using Java as a general purpose application development language is like going big game hunting armed with Nerf weapons. -- Author Unknown __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: valgrind and openssl
On Thu, May 15, 2008 at 4:41 PM, Erik de Castro Lopo [EMAIL PROTECTED] wrote: Goetz Babin-Ebell wrote: But here the use of this uninitialized data is intentional and the programmer are very well aware of what they did. The use of unititialized data in this case is stupid because the entropy of this random data is close to zero. The only sane way to deal with this it to either make it zero or make it truely random. Erik I disagree. If there's a performance cost to making openssl happy with valgrind, I'd rather have there be an option that defaults to optimize security and performance at the expense of debugging capability. Debugging is the infrequent case. Although I disagree, I understand your argument. However, you weaken your position by using the words stupid and sane; they make you seem disrespectful. -JP __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: valgrind and openssl
On Thu, May 15, 2008 at 11:51 PM, Erik de Castro Lopo [EMAIL PROTECTED] wrote: Bodo Moeller wrote: It may be zero, but it may be more, depending on what happened earlier in the program if the same memory locations have been in use before. This may very well include data that would be unpredictable to adversaries -- i.e., entropy; that's the point here. Do you know its unpredicatable or are you only guessing? Can a bad guy force it to be predicatable? How much entropy is actually there? Has anyone actually measured it? All this depends on the specific application. For many, there almost certainly won't be any unpredictable data. For others, in particular long-running interactive software, there certainly will be at least some information that is unpredictable to at least some adversaries. Even if it's just return addresses on the stack, the specific pattern will depend on the program's past, some aspects of which may be unknown to adversaries. We don't care if anyone can force this to be predictable, because we're in no way relying on it to deliver more than zero bits of entropy. We're just hoping there might be some entropy in there sometimes. __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
RE: valgrind and openssl
Geoff Thorpe wrote: Then tell your linux distribution to use -DPURIFY. Hangon, I've got a better idea. How about the OpenSSL develoeprs fix their library so that the standard version that they ship is valgrind clean. Then the distributions won't need to do anything other than compile it. Erik Umm, why? 1) This is an unusual use case. 2) Zeroing memory that doesn't need to be zeroed has a performance cost. 3) It's very easy to recompile a debug build for debugging. IMO, the default build should always be a release build with release optimizations and tradeoffs. A debug build should be the exception. DS __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: valgrind and openssl
Bodo Moeller wrote: We don't care if anyone can force this to be predictable, because we're in no way relying on it to deliver more than zero bits of entropy. So it might end up being zero just by chance right? We're just hoping there might be some entropy in there sometimes. In the practice of engineering, we should try to avoid 'hoping' about anything. Erik -- - Erik de Castro Lopo - Every method you use to prevent or find bugs leaves a residue of subtler bugs against which those methods are ineffectual. -- Bruce Beizer's Pesticide Paradox __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: valgrind and openssl
David Schwartz wrote: Umm, why? 1) This is an unusual use case. This is not an unusual case. I'm a developer and I valgrind my code all the time because fixing problems shown up by valgrind makes my code better. My code is targeting an embedded Linux box and I try to ensure that the system I develop and test on is as close as possible to the embedded target. Using a different version of openssl sort of defeats that aim. 2) Zeroing memory that doesn't need to be zeroed has a performance cost. Yep, I agree, for code that doesn't give valgrind warnings. A couple of years ago I attended a very interesting presentation by Andrew Tridgel of the Samba project. He explained that they avoided zeroing memory wherever possible. However, they used valgrind and had a rigorous policy of fixing all valgrind warnings. Tridge's claim was that doing this maximized performace and guarded against information leakage onto the wire. 3) It's very easy to recompile a debug build for debugging. But if I do that, my test system is no longer the same as my target. IMO, the default build should always be a release build with release optimizations and tradeoffs. A debug build should be the exception. I'm not asking for all uses of malloc to be replaced with calloc, I'm asking for the very small number of areas that produce valgrind warnings to be fixed. Since they are small in number it is unlikely that any performance decrease can even be measureed. Erik -- - Erik de Castro Lopo - The Earth is around 70% water. Fish rule the seas. Humans are over 90% water. It's only a matter of time. __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
RE: valgrind and openssl
David Schwartz wrote: Umm, why? 1) This is an unusual use case. This is not an unusual case. I'm a developer and I valgrind my code all the time because fixing problems shown up by valgrind makes my code better. I didn't say it was an unusual use case for you. It's an unusual use case for OpenSSL. My code is targeting an embedded Linux box and I try to ensure that the system I develop and test on is as close as possible to the embedded target. Using a different version of openssl sort of defeats that aim. Then you have to make a tradeoff. You can either debug in a release build and suffer the added complexity or you can release a debug build and suffer the performance loss. But others don't have to make that tradeoff, and there's no reason your problem should force them to. 2) Zeroing memory that doesn't need to be zeroed has a performance cost. Yep, I agree, for code that doesn't give valgrind warnings. A couple of years ago I attended a very interesting presentation by Andrew Tridgel of the Samba project. He explained that they avoided zeroing memory wherever possible. However, they used valgrind and had a rigorous policy of fixing all valgrind warnings. Tridge's claim was that doing this maximized performace and guarded against information leakage onto the wire. He's absolutely right. Here, Tridge's argument works the other way. We *want* as much information as possible to leak into the PRNG. 3) It's very easy to recompile a debug build for debugging. But if I do that, my test system is no longer the same as my target. Then you are welcome to ship a debug build with your target. That's your tradeoff to make. The vast majority of people don't have such a tradeoff, so it's kind of silly that they should suffer its costs just because you have to. IMO, the default build should always be a release build with release optimizations and tradeoffs. A debug build should be the exception. I'm not asking for all uses of malloc to be replaced with calloc, I'm asking for the very small number of areas that produce valgrind warnings to be fixed. Since they are small in number it is unlikely that any performance decrease can even be measureed. Then you are welcome to compile with -DPURIFY even in your release build. That does exactly what you ask for. I've already explained why that's not the default -- your case is unusual and so it should *not* be the default. A single extra flag is an extremely minimal cost. DS __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
RE: valgrind and openssl
Would a runtime flag for don't seed with uninitialized memory, rather than (or in addition to) -DPURIFY, satisfy everybody? John __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: valgrind and openssl
In the practice of engineering, we should try to avoid 'hoping' about anything. Don't know much about cryptography, do you? /r$ -- STSM, DataPower Chief Programmer WebSphere DataPower SOA Appliances http://www.ibm.com/software/integration/datapower/
RE: valgrind and openssl
Would a runtime flag for don't seed with uninitialized memory, rather than (or in addition to) -DPURIFY, satisfy everybody? Everybody? It seems to me that only one or two people who don't really understand what's going on are complaining. OpenSSL should stay as it is. A contributed valgrind suppressions file would be useful. /r$ -- STSM, DataPower Chief Programmer WebSphere DataPower SOA Appliances http://www.ibm.com/software/integration/datapower/
RE: valgrind and openssl
Would a runtime flag for don't seed with uninitialized memory, rather than (or in addition to) -DPURIFY, satisfy everybody? John I don't think it's necessary, since compiling with '-DPURIFY' is so ridiculously easy, but I have no objection to it. An evironment variable would probably be the easiest way to do it. DS __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
RE: valgrind and openssl
Everybody? It seems to me that only one or two people who don't really understand what's going on are complaining. Wanting to get accurate runtime analysis results with a release build is not an unreasonable request. OpenSSL should stay as it is. A contributed valgrind suppressions file would be useful. It certainly would, but Valgrind isn't the only analysis tool people might want to use. A runtime flag provides a means of obtaining accurate results with any tool. John __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: valgrind and openssl
On Thursday 15 May 2008 17:31:45 Erik de Castro Lopo wrote: Geoff Thorpe wrote: Then tell your linux distribution to use -DPURIFY. Hangon, I've got a better idea. How about the OpenSSL develoeprs fix their library so that the standard version that they ship is valgrind clean. Then the distributions won't need to do anything other than compile it. What, you mean like how the standard version we ship has a good PRNG, so that the distributions don't need to do anything about that either? Funny, that doesn't work so well in the real world. Distributions always do something before compiling packages, as debian has so succinctly and spectacularly demonstrated. They pick target architecture settings for the distribution that are independent of the build host (eg. cross-compilation for non-x86 hosts, etc) and another other configuration options they think useful/necessary. Eg. static/dynamic, PIC or not, symbols or not, installation paths, optional features (and dependencies), documentation, [...]. Sometimes they even throw in patches, which is where they diverge dangerously from what we provide. If it is the distribution's preference to have openssl unnecessarily memset and/or unnecessarily restrict its seeding to pander to an unmodified and unconfigured valgrind, be that in the standard package or any other debug package, then that is their choice. We provide a (supported) method for doing this, it's called -DPURIFY. It doesn't require *any* source-code patching. ahem. Valgrind is a great tool and no doubt many non-noobs put it to good use on a daily basis. It helps find non-deterministic behaviour. That it finds non-deterministic behaviour in our PRNG should not be cause for basement-dwellers the world over to rise up against common-sense. We have a FAQ in case you get confused, and valgrind can also be taught to work around such cases as this. And again, you can build openssl to side-step the false-positives for a very small overhead. Just what level of base, ill-informed, and incompetent debugging help do we need to cater to? And to what non-technical extents are we prepared to go for it? Cheers, Geoff __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: valgrind and openssl
On Fri, May 16, 2008 at 12:39 AM, David Schwartz [EMAIL PROTECTED] wrote: 2) Zeroing memory that doesn't need to be zeroed has a performance cost. This particular argument doesn't actually apply here. We wouldn't have to zeroize any memory, we just wouldn't feed those bytes that are not known to have been initialized into RAND_add(). The cost of RAND_add() is a lot higher than that of memset(), so we'd even gain some performance. But we'd lose the randomness that is available when the bytes not currently known to have been initialized have in fact been previously initialized in a way not known by an attacker. __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
