Re: OpenSSL and DH parameters
On 10/24/13 1:59 PM, Dave Thompson wrote: (For EC, the specified curve must also be acceptable to client(s) per ClientHello extension, which encourages using the callback or choosing a popular curve like P-256.) So, my understanding is that if the tmp_ecdh is set to a curve which is not supported by the client, then OpenSSL ought to just skip the elliptic curve cipher suites and pick the next acceptable cipher suite supported by both the client and server. Is this not the case? I was puzzled by this message: http://www.metzdowd.com/pipermail/cryptography/2013-October/018330.html It suggests that if the client offers P-256 and P-384, but the server's tmp_ecdh is set to P-521, then the server will pick an elliptic curve cipher suite anyway, try to force P-521 on the client, and the handshake will fail. Hence his assertion With TLS, no EC is better than crippled EC. This seems very wrong to me! If the client and server can't agree on a curve, shouldn't the server just pick a non-elliptic-curve cipher suite that both it and the client support instead? It seems like offering EC should never be worse than not offering EC! The following draft also seems to suggest the same thing, that if client and server both support an elliptic curve suite, they will pick it, and then discover that they don't have any curves in common, and give up, rather than picking a non-EC suite: http://datatracker.ietf.org/doc/draft-gutmann-tls-eccsuites/ Is this true? And why? It doesn't seem like it should work that way. --Patrick __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager majord...@openssl.org
Re: not fork-safe if pids wrap
On 8/22/13 12:46 PM, Nico Williams wrote: The parent might be multi-threaded, leading to the risk that a thread in the parent and the child will obtain the same PRNG outputs until the parent thread that fork()ed completes the re-seeding. That's a good point; I hadn't thought of that. Also, it's not a requirement that pthread_atfork() require -lpthread. It's entirely possible (and on Solaris 10 and up, for example, it is in fact so) that pthread_atfork() is in libc. That actually makes much more sense, since pthread_atfork() really has nothing to do with threads. But at least on Linux, pthread_atfork() is part of -lpthread. If you are going to exec() anyways you should have used vfork(), or better! you should have used posix_spawn() or equivalent. On Linux, posix_spawn still ends up calling the atfork handlers, if (and only if) you specify any file_actions. I was actually in an unfortunate situation recently where the atfork handlers of a closed-source OpenGL library were causing crashes, so to run an external process with output redirection, I had to posix_spawn /bin/sh (with no file_actions), and then give /bin/sh a shell command to perform the output redirections and then exec the program I really wanted to run. Ugly! --Patrick __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager majord...@openssl.org
Re: not fork-safe if pids wrap
On 8/21/13 8:55 AM, Nico Williams wrote: OpenSSL should use pthread_atfork() and mix in more /dev/urandom into its pool in the child-side of the fork(), Only a child-side handler is needed, FYI, unless there's locks to acquire and release, in which case you also need a pre-fork and parent-side handlers, or unless fork() is just a good excuse to add entropy to the pool on the parent side anyways :) Yeah, it seems like a good excuse. Actually, it probably makes more sense to only add entropy on the parent side, since the parent is likely to live longer, and there's a good chance the child is just going to exec() anyway, in which case adding entropy to it will have been for nothing. The downside with using pthread_atfork() is that it requires pthreads. Since you pointed out there are still non-threaded libcs (at least for static linking), that would be an issue. Most other libraries I've seen handle this by saving the pid in a static variable, and then comparing the current pid to it. This has the advantage of not needing pthreads, and also of only adding the entropy to the child if it is actually needed (i. e. it doesn't exec after fork). For example, GnuTLS: https://gitorious.org/gnutls/gnutls/blobs/master/lib/nettle/rnd.c#line429 libevent: https://github.com/libevent/libevent/blob/master/arc4random.c#L413 and libottery: https://github.com/nmathewson/libottery/blob/master/src/ottery.c#L523 The only thing that bothers me about doing the pid check is that in theory it could still fail, although it's a really unlikely case. Imagine a parent process that forks a child process. The child doesn't generate any random numbers, so the reseed doesn't happen in the child. The parent dies, and then the child forks a grandchild. In an incredibly rare and unlucky case, the grandchild could have the same pid as the original parent, and then the grandchild wouldn't detect it had forked. --Patrick __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager majord...@openssl.org
not fork-safe if pids wrap (was Re: DLL hell)
On 8/15/13 11:51 PM, Patrick Pelletier wrote: On Aug 15, 2013, at 10:38 PM, Nico Williams wrote: Hmm, I've only read the article linked from there: http://android-developers.blogspot.com/2013/08/some-securerandom-thoughts.html Yeah, that's the only place I've seen it, and then the Google+ thread I linked to is essentially the comment area for that post. We (meaning those of us commenting in the thread) haven't gotten any official answer from Google, but Nikolay Elenkov has been very helpful in reconstructing what seems to be happening. We've exchanged a few more posts this evening, and it appears that what's happening is that OpenSSL is correctly self-seeding when system_server starts, but then system_server forks (without execing) to start multiple processes, and these processes are producing the same random sequence. It's not yet entirely clear why, since the OpenSSL source code looks like it's trying to be fork-safe, but it appears that somehow in practice it's not succeeding. s/system_server/zygote/ So, it appears that the problem is that since OpenSSL merely mixes in the pid to the existing random state, once the pids wrap, you will have had two processes that have generated the exact same random sequence, since they started with the same state (before the fork) and mixed in the same thing (the pid) after the fork, resulting in the same output. (This is in contrast to the approach of comparing the old and new pids, and doing a full reseed from /dev/urandom if they differ, which is what is done by Nick Mathewson's preliminary but already excellent-looking libottery.) Nikolay Elenkov wrote a proof-of-concept that shows the pid-wrapping bug on Android, and then I took it one step further and wrote a proof-of-concept using OpenSSL in C, demonstrating that this is an underlying OpenSSL bug: https://gist.github.com/ppelleti/6290984 An easy way to work around this, if you don't mind linking against pthreads, is to do this at the start of your application, after initializing OpenSSL: typedef void (*voidfunc) (void); if (ENGINE_get_default_RAND () == NULL) pthread_atfork (NULL, (voidfunc) RAND_poll, (voidfunc) RAND_poll); But, of course, this ought to eventually be fixed in OpenSSL itself. (By using the pid-comparison trick that libottery uses, rather than just mixing in the pid.) I'm happy to submit a patch, if we think there's a good chance it would be considered? --Patrick __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager majord...@openssl.org
Re: not fork-safe if pids wrap (was Re: DLL hell)
On 8/21/13 12:19 AM, Patrick Pelletier wrote: Nikolay Elenkov wrote a proof-of-concept that shows the pid-wrapping bug on Android, and then I took it one step further and wrote a proof-of-concept using OpenSSL in C, demonstrating that this is an underlying OpenSSL bug: https://gist.github.com/ppelleti/6290984 Hmmm... so I'm able to reproduce the bug with my little program when using the version of OpenSSL that ships with Ubuntu 12.04 (OpenSSL 1.0.1 14 Mar 2012). But I just built an OpenSSL off the tip of master in github (OpenSSL 1.1.0-dev xx XXX ), and my test program doesn't produce any duplicate random numbers when linked against that OpenSSL. So, this would suggest the bug has already been fixed, but I'm not sure how, since md_rand.c hasn't been changed since 2011, and it's still doing the same pid-mixing trick. Anybody else have any observations or thoughts on this? --Patrick __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager majord...@openssl.org
Re: DLL hell
On Aug 15, 2013, at 10:38 PM, Nico Williams wrote: Hmm, I've only read the article linked from there: http://android-developers.blogspot.com/2013/08/some-securerandom-thoughts.html Yeah, that's the only place I've seen it, and then the Google+ thread I linked to is essentially the comment area for that post. We (meaning those of us commenting in the thread) haven't gotten any official answer from Google, but Nikolay Elenkov has been very helpful in reconstructing what seems to be happening. We've exchanged a few more posts this evening, and it appears that what's happening is that OpenSSL is correctly self-seeding when system_server starts, but then system_server forks (without execing) to start multiple processes, and these processes are producing the same random sequence. It's not yet entirely clear why, since the OpenSSL source code looks like it's trying to be fork-safe, but it appears that somehow in practice it's not succeeding. I'll take a look. For now it seems there should be no need to set any thread-related callbacks, no? Or if they are needed, we should make them no-ops on OSes with thread libraries. Threading callbacks are currently needed for any application that isn't single threaded. (The situation is still the same as what's covered in the 2002 O'Reilly book, Network Security with OpenSSL. It has some sample code for setting them up on Windows and pthreads.) I totally agree with you that I'd love to see support for at least pthreads and Windows threads baked into OpenSSL, much like libevent does, for example. But I suspect that convincing the OpenSSL developers will be an uphill battle. (Just to be clear about my role: I'm an OpenSSL user, an occasional contributor to the OpenSSL wiki, and also one of several admins on the wiki, simply because I was interested and volunteered. But I'm not an OpenSSL developer, nor do I have a direct line to any of the developers. Nor do I speak for anyone but myself. So this is just me throwing opinions and speculation from the peanut gallery.) I think that a crypto library should have no worse initialization/finalization/thread safety/fork safety semantics than libpkcs11 in Solaris/Illumos, which: a) thread-safely ref-counts C_Initialize()/C_Finalize() calls, b) leaves locking around objects to the app, c) re-initializes (and loses all objects) on the child-side of fork(), d) no thread/lock callback setters. (It's necessary to finalize all objects that refer to sessions for crypto coprocessors, TPMs, tokens, as well as any other stateful objects on the child side of fork(), either that or establish new sessions. It shouldn't be necessary to finalize key objects, say, but hey, it's what PKCS#11 requires.) Better yet: implied initialization (using pthread_once()). I'm in total agreement with you on that, although even libraries that I consider pretty good in that regard, like libevent and GnuTLS, don't quite achieve that. A lot of people seem to love static linking. There are problems with dynamic linking, to be sure, but static linking with dynamic dlopen() is insane, and anyways, static linking unfortunately means having to keep track of total dependencies at the program link-edit stage, which is also basically insane (though at least that could be fixed). Agree. And yeah, I hadn't been thinking about static libc. OS X, at least, has the (mostly) nice property that libc (called libSystem) can't be static. Oh. Is there any reason not to blow that away, or at least build-time select which to use? I'm in agreement with you; I just don't think you're going to get the OpenSSL folks on board. They'll probably say something like we want to be totally agnostic to threading library without acknowledging that pthreads and Windows threads cover the vast majority of modern mainstream operating systems. Great. I was hoping that the response wouldn't be something like no way, we need these callback setting functions for XYZ reasons or, worse, no way. Unfortunately, I think the response will be that. (The OpenSSL folks just haven't weighed in on this thread yet.) That's why I was floating the idea of writing an unofficial companion library that would smooth over these rough spots and provide a batteries included approach to people who want it, without having to convince the OpenSSL project to change the core library, which I think would be an uphill battle at best. --Patrick __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager majord...@openssl.org
Re: DLL hell
On 8/15/13 10:24 AM, Nico Williams wrote: . Recent developments, like Android's failure to properly initialize OpenSSL's PRNG make me think it's time to table (in the British sense) the issue once more. Can you point to any article or post which explains exactly what the OpenSSL half of the Android issue was? (I understand the Harmony SecureRandom issue, but that's a separate thing.) OpenSSL is supposed to call RAND_poll on the first call to RAND_bytes, and RAND_poll knows how to seed from /dev/urandom on systems that have it, which should include Android. Neither I nor others speculating on Google+ could figure out why this wasn't the case, and why explicit seeding would have been necessary: https://plus.google.com/+AndroidDevelopers/posts/YxWzeNQMJS2 There should be no need for run-time, global initialization of OpenSSL (or any sub-system of it) by applications. Moreover, any existing functions for such initialization should become no-ops or do their work just once, and in a thread-safe way. As much configuration of threading and other system libraries should be done at build-time as possible; as little should be done at run-time as possible. Even for special-purpose OSes. Figuring out the right sequence of initialization functions to call, even when just one application is using OpenSSL, has not been entirely clear. In particular, see my rambling talk-page discourse on the OpenSSL Wiki about what is and isn't necessary in order to get ENGINEs initialized, and how it depends upon some build-time #defines in a non-obvious way: http://wiki.opensslfoundation.com/index.php/Talk:Libcrypto_API OpenSSL requires too much explicit and global run-time initialization of it by applications in ways that preclude (or at least used to and still might cause problems for) multiple distinct callers of OpenSSL in the same process. And yet the layered software systems we've built over the past decade result in just that happening. Thank you! I'm glad I'm not the only one who feels this is a big problem. It's something I've expressed concern about in the past, albeit in a parenthesized paragraph about halfway through a long, far-reaching rant: http://lists.randombit.net/pipermail/cryptography/2012-October/003388.html Typical examples include: applications that use TLS and Kerberos, PAM applications (each module might need to use OpenSSL, and so might the application), systems running without nscd (the name service switch modules may need OpenSSL, and so might the app), and so on. Some examples close to my heart: * using the OpenSSL Ruby Gem, while also using another Ruby Gem that depends on OpenSSL indirectly * using OpenSSL directly while also using libevent's optional integration with OpenSSL Huh? Which is it? Must apps call CRYPTO_THREADID_set_callback() even though [o]n Windows and many Unix systems, OpenSSL automatically uses the multi-threaded versions of the standard libraries? Why? One would think that the multi-threaded versions of the standard libraries on such OSes would provide all that OpenSSL needs. I don't get it. I think what it means is that We link against thread-safe versions of the standard library (which implies there are thread-unsafe versions of the standard library, which I think might be true on Windows, but I'm pretty sure there isn't any thread-unsafe version of the standard library on Linux or OS X) so that, for example, errno doesn't get clobbered by multiple threads. But we still don't call any threading functions to make OpenSSL itself threadsafe, so you'll have to provide this boilerplate yourself in every OpenSSL application you write. I could also rant about OpenSSL's homemade thread local storage (TLS, in the other sense of the acronym) implementation, rather than using the OS's API for thread local storage. This can result in memory leaks when a thread terminates, unless you have complete control over the lifecycle of every thread and can call a cleanup function before the thread terminates. (And, as you've noted, in a modern application with many layers of libraries, you don't have such control.) Finally, what can I do to help? Definitely to the extent that you can figure out what the actual behavior is (although that itself is not easy and usually requires experimentation and studying the source code in-depth), document your findings on the OpenSSL wiki: http://wiki.opensslfoundation.com/ (Though hopefully any such clarifications can be propagated to the more official documentation as well.) And I certainly hope that the initialization and thread-safety issues can be addressed in libcrypto itself, but if they can't or won't, then I think we (meaning those interested) should write a small add-on library to go along with OpenSSL, that would include standard implementations of the thread safety callbacks, and perhaps we could convince users of OpenSSL to accept it as a de-facto
Re: OpenSSL wikibook
On Feb 2, 2013, at 2:46 AM, Matt Caswell wrote: I have previously submitted a largish patch for documentation around the OpenSSL EC library. Unfortunately there seems little interest in it, and it has been hanging around in RT for some while: https://rt.openssl.org/Ticket/Display.html?id=2799 (username guest, password guest) This documentation is in the POD format used by OpenSSL so isn't ideal for a direct conversion onto the wiki. However it seems a shame for it to be inaccessible so I've published it onto your wikibooks site anyway. I agree on both counts. Ideally, the actual API documentation should be maintained by the OpenSSL project, so it can be version-controlled along with the source, and the documentation can be installed as man pages. Ultimately, my vision for the OpenSSL wikibook is that it focus more on what would you use these functions for, and how would you use them together rather than these are the arguments and return values. Something that explains a bit more of the big picture and real-world usage. I see it more as a replacement for the out-of-date O'Reilly book, rather than a replacement for the OpenSSL manpages. However, at this point, I am very happy to have content of any sort in the wikibook, and I thank you for your contribution! I've reorganized the table of contents a bit, so that all of your pages are directly accessible from the table of contents. (Again, this man-page style format isn't quite what I'm looking for in the long term, but I'm quite happy to have it for now!) Thanks so much for contributing, --Patrick
OpenSSL wikibook
Since the quality of OpenSSL documentation, and the ease of contributing to it, has been a subject of discussion on both the openssl-users list and the cryptography list in the past few months, and since the only commercial book on OpenSSL is over a decade old now, I thought it would be worthwhile to start an OpenSSL wikibook: https://en.wikibooks.org/wiki/OpenSSL All I have in place right now is a skeleton of a table of contents, but I'm hoping that OpenSSL users will contribute to the book in their areas of expertise, or as they learn new things that they wish had been documented. --Patrick
