Re: DLL hell
FYI, in a few weeks I'll have some time to actually implement and submit patches. I'll attempt to identify useful points for automatic self-initialization (any hints as to commonly used first calls, not counting the callback setters, would be welcomed). I'll also have to spend sometime with the build system to autodetect correct build options. The rest should be simple enough. I do plan to stay away from RNG and fork-safety issues, except for re-seeding or perturbing the PRNG on the child-side of fork(). I'm not sure how to test, except for, perhaps, building two sets of shared objects with different SONAMEs so as to be able to load two instances of the same libraries (and their callers) in one process. Some ideas as to what to test for would be nice. I was thinking of instrumenting OpenSSL entry points with calls to a utility that uses dladdr() and stack walkers to ensure that each {caller, OpenSSL instance} pair are always paired up correctly and no caller inappropriately calls a different OpenSSL instance. Would such a test need to be in-tree and build-time configurable? But this seems more like a test of the run-time linker loader than a test of OpenSSL. Ideas on how to test are welcomed. That's the plan. If any of you see anything wrong with it, please save me time and effort by letting me know. Nico -- __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager majord...@openssl.org
not fork-safe if pids wrap (was Re: DLL hell)
On 8/15/13 11:51 PM, Patrick Pelletier wrote: On Aug 15, 2013, at 10:38 PM, Nico Williams wrote: Hmm, I've only read the article linked from there: http://android-developers.blogspot.com/2013/08/some-securerandom-thoughts.html Yeah, that's the only place I've seen it, and then the Google+ thread I linked to is essentially the comment area for that post. We (meaning those of us commenting in the thread) haven't gotten any official answer from Google, but Nikolay Elenkov has been very helpful in reconstructing what seems to be happening. We've exchanged a few more posts this evening, and it appears that what's happening is that OpenSSL is correctly self-seeding when system_server starts, but then system_server forks (without execing) to start multiple processes, and these processes are producing the same random sequence. It's not yet entirely clear why, since the OpenSSL source code looks like it's trying to be fork-safe, but it appears that somehow in practice it's not succeeding. s/system_server/zygote/ So, it appears that the problem is that since OpenSSL merely mixes in the pid to the existing random state, once the pids wrap, you will have had two processes that have generated the exact same random sequence, since they started with the same state (before the fork) and mixed in the same thing (the pid) after the fork, resulting in the same output. (This is in contrast to the approach of comparing the old and new pids, and doing a full reseed from /dev/urandom if they differ, which is what is done by Nick Mathewson's preliminary but already excellent-looking libottery.) Nikolay Elenkov wrote a proof-of-concept that shows the pid-wrapping bug on Android, and then I took it one step further and wrote a proof-of-concept using OpenSSL in C, demonstrating that this is an underlying OpenSSL bug: https://gist.github.com/ppelleti/6290984 An easy way to work around this, if you don't mind linking against pthreads, is to do this at the start of your application, after initializing OpenSSL: typedef void (*voidfunc) (void); if (ENGINE_get_default_RAND () == NULL) pthread_atfork (NULL, (voidfunc) RAND_poll, (voidfunc) RAND_poll); But, of course, this ought to eventually be fixed in OpenSSL itself. (By using the pid-comparison trick that libottery uses, rather than just mixing in the pid.) I'm happy to submit a patch, if we think there's a good chance it would be considered? --Patrick __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager majord...@openssl.org
Re: not fork-safe if pids wrap (was Re: DLL hell)
On 8/21/13 12:19 AM, Patrick Pelletier wrote: Nikolay Elenkov wrote a proof-of-concept that shows the pid-wrapping bug on Android, and then I took it one step further and wrote a proof-of-concept using OpenSSL in C, demonstrating that this is an underlying OpenSSL bug: https://gist.github.com/ppelleti/6290984 Hmmm... so I'm able to reproduce the bug with my little program when using the version of OpenSSL that ships with Ubuntu 12.04 (OpenSSL 1.0.1 14 Mar 2012). But I just built an OpenSSL off the tip of master in github (OpenSSL 1.1.0-dev xx XXX ), and my test program doesn't produce any duplicate random numbers when linked against that OpenSSL. So, this would suggest the bug has already been fixed, but I'm not sure how, since md_rand.c hasn't been changed since 2011, and it's still doing the same pid-mixing trick. Anybody else have any observations or thoughts on this? --Patrick __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager majord...@openssl.org
Re: not fork-safe if pids wrap (was Re: DLL hell)
On 21 August 2013 03:19, Patrick Pelletier c...@funwithsoftware.org wrote: On 8/15/13 11:51 PM, Patrick Pelletier wrote: On Aug 15, 2013, at 10:38 PM, Nico Williams wrote: Hmm, I've only read the article linked from there: http://android-developers.**blogspot.com/2013/08/some-** securerandom-thoughts.htmlhttp://android-developers.blogspot.com/2013/08/some-securerandom-thoughts.html Yeah, that's the only place I've seen it, and then the Google+ thread I linked to is essentially the comment area for that post. We (meaning those of us commenting in the thread) haven't gotten any official answer from Google, but Nikolay Elenkov has been very helpful in reconstructing what seems to be happening. We've exchanged a few more posts this evening, and it appears that what's happening is that OpenSSL is correctly self-seeding when system_server starts, but then system_server forks (without execing) to start multiple processes, and these processes are producing the same random sequence. It's not yet entirely clear why, since the OpenSSL source code looks like it's trying to be fork-safe, but it appears that somehow in practice it's not succeeding. s/system_server/zygote/ So, it appears that the problem is that since OpenSSL merely mixes in the pid to the existing random state, once the pids wrap, you will have had two processes that have generated the exact same random sequence, since they started with the same state (before the fork) and mixed in the same thing (the pid) after the fork, resulting in the same output. (This is in contrast to the approach of comparing the old and new pids, and doing a full reseed from /dev/urandom if they differ, which is what is done by Nick Mathewson's preliminary but already excellent-looking libottery.) Nikolay Elenkov wrote a proof-of-concept that shows the pid-wrapping bug on Android, and then I took it one step further and wrote a proof-of-concept using OpenSSL in C, demonstrating that this is an underlying OpenSSL bug: https://gist.github.com/**ppelleti/6290984https://gist.github.com/ppelleti/6290984 An easy way to work around this, if you don't mind linking against pthreads, is to do this at the start of your application, after initializing OpenSSL: typedef void (*voidfunc) (void); if (ENGINE_get_default_RAND () == NULL) pthread_atfork (NULL, (voidfunc) RAND_poll, (voidfunc) RAND_poll); But, of course, this ought to eventually be fixed in OpenSSL itself. (By using the pid-comparison trick that libottery uses, rather than just mixing in the pid.) I'm happy to submit a patch, if we think there's a good chance it would be considered? Something needs to be done, but won't this re-introduce the problem of /dev/random starvation, leading to more use of /dev/urandom (on platforms where this is a problem)? Mixing in the time seems like a safer solution that should also fix the problem. Possibly only when the PID changes. --Patrick __**__**__ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager majord...@openssl.org
Re: not fork-safe if pids wrap (was Re: DLL hell)
On Wed, Aug 21, 2013 at 2:19 AM, Patrick Pelletier c...@funwithsoftware.org wrote: An easy way to work around this, if you don't mind linking against pthreads, is to do this at the start of your application, after initializing OpenSSL: typedef void (*voidfunc) (void); if (ENGINE_get_default_RAND () == NULL) pthread_atfork (NULL, (voidfunc) RAND_poll, (voidfunc) RAND_poll); This is a pretty standard thing to do, and Solaris' libpkcs11 does it (not to add entropy but to re-initialize, since PKCS#11 requires all session and object handles to no longer be usable on the child-side of fork()). But, of course, this ought to eventually be fixed in OpenSSL itself. (By using the pid-comparison trick that libottery uses, rather than just mixing in the pid.) I'm happy to submit a patch, if we think there's a good chance it would be considered? OpenSSL should use pthread_atfork() and mix in more /dev/urandom into its pool in the child-side of the fork(), Only a child-side handler is needed, FYI, unless there's locks to acquire and release, in which case you also need a pre-fork and parent-side handlers, or unless fork() is just a good excuse to add entropy to the pool on the parent side anyways :) Nico -- __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager majord...@openssl.org
Re: not fork-safe if pids wrap (was Re: DLL hell)
On Wed, Aug 21, 2013 at 5:41 AM, Ben Laurie b...@links.org wrote: Something needs to be done, but won't this re-introduce the problem of /dev/random starvation, leading to more use of /dev/urandom (on platforms where this is a problem)? Mixing in the time seems like a safer solution that should also fix the problem. Possibly only when the PID changes. Stirring in time and PID seems like just a fail-safe. Some bytes from /dev/urandom should also be added -- it won't hang once seeded (or ever on Linux, but hopefully a simple service can be added by users to seed urandom from random). Provided one read from /dev/random has been done I think perturbing the pool with time + PID + urandom should suffice. Nico -- __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager majord...@openssl.org
Re: DLL hell
On Aug 15, 2013, at 10:38 PM, Nico Williams wrote: Hmm, I've only read the article linked from there: http://android-developers.blogspot.com/2013/08/some-securerandom-thoughts.html Yeah, that's the only place I've seen it, and then the Google+ thread I linked to is essentially the comment area for that post. We (meaning those of us commenting in the thread) haven't gotten any official answer from Google, but Nikolay Elenkov has been very helpful in reconstructing what seems to be happening. We've exchanged a few more posts this evening, and it appears that what's happening is that OpenSSL is correctly self-seeding when system_server starts, but then system_server forks (without execing) to start multiple processes, and these processes are producing the same random sequence. It's not yet entirely clear why, since the OpenSSL source code looks like it's trying to be fork-safe, but it appears that somehow in practice it's not succeeding. I'll take a look. For now it seems there should be no need to set any thread-related callbacks, no? Or if they are needed, we should make them no-ops on OSes with thread libraries. Threading callbacks are currently needed for any application that isn't single threaded. (The situation is still the same as what's covered in the 2002 O'Reilly book, Network Security with OpenSSL. It has some sample code for setting them up on Windows and pthreads.) I totally agree with you that I'd love to see support for at least pthreads and Windows threads baked into OpenSSL, much like libevent does, for example. But I suspect that convincing the OpenSSL developers will be an uphill battle. (Just to be clear about my role: I'm an OpenSSL user, an occasional contributor to the OpenSSL wiki, and also one of several admins on the wiki, simply because I was interested and volunteered. But I'm not an OpenSSL developer, nor do I have a direct line to any of the developers. Nor do I speak for anyone but myself. So this is just me throwing opinions and speculation from the peanut gallery.) I think that a crypto library should have no worse initialization/finalization/thread safety/fork safety semantics than libpkcs11 in Solaris/Illumos, which: a) thread-safely ref-counts C_Initialize()/C_Finalize() calls, b) leaves locking around objects to the app, c) re-initializes (and loses all objects) on the child-side of fork(), d) no thread/lock callback setters. (It's necessary to finalize all objects that refer to sessions for crypto coprocessors, TPMs, tokens, as well as any other stateful objects on the child side of fork(), either that or establish new sessions. It shouldn't be necessary to finalize key objects, say, but hey, it's what PKCS#11 requires.) Better yet: implied initialization (using pthread_once()). I'm in total agreement with you on that, although even libraries that I consider pretty good in that regard, like libevent and GnuTLS, don't quite achieve that. A lot of people seem to love static linking. There are problems with dynamic linking, to be sure, but static linking with dynamic dlopen() is insane, and anyways, static linking unfortunately means having to keep track of total dependencies at the program link-edit stage, which is also basically insane (though at least that could be fixed). Agree. And yeah, I hadn't been thinking about static libc. OS X, at least, has the (mostly) nice property that libc (called libSystem) can't be static. Oh. Is there any reason not to blow that away, or at least build-time select which to use? I'm in agreement with you; I just don't think you're going to get the OpenSSL folks on board. They'll probably say something like we want to be totally agnostic to threading library without acknowledging that pthreads and Windows threads cover the vast majority of modern mainstream operating systems. Great. I was hoping that the response wouldn't be something like no way, we need these callback setting functions for XYZ reasons or, worse, no way. Unfortunately, I think the response will be that. (The OpenSSL folks just haven't weighed in on this thread yet.) That's why I was floating the idea of writing an unofficial companion library that would smooth over these rough spots and provide a batteries included approach to people who want it, without having to convince the OpenSSL project to change the core library, which I think would be an uphill battle at best. --Patrick __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager majord...@openssl.org
Re: DLL hell
On 8/15/2013 10:24 AM, Nico Williams wrote: Hi, I'm sorry if this has all been discussed extensively before. A brief search for DLL hell in the archives turns up disappointingly (and surprisingly) little. I do see a thread with messages from my erstwhile colleagues at Sun/Oracle, so I know it's been discussed, e.g., here: http://www.mail-archive.com/openssl-dev@openssl.org/msg27453.html . Recent developments, like Android's failure to properly initialize OpenSSL's PRNG make me think it's time to table (in the British sense) the issue once more. To summarize the rest of this long post (please forgive me): I think a lot of the init logic heralds from the original SSLeay days. There seems to be intent that initialization is supposed to happen in main() in the application and libraries shouldn't be calling initialization routines in OpenSSL. However, I agree that all the init logic for the library should be handled automatically and transparently in a thread-safe manner. -- Thomas Hruska Shining Light Productions Home of BMP2AVI and Win32 OpenSSL. http://www.slproweb.com/ __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager majord...@openssl.org
Re: DLL hell
On Fri, Aug 16, 2013 at 07:17:22AM -0700, Thomas J. Hruska wrote: I think a lot of the init logic heralds from the original SSLeay days. There seems to be intent that initialization is supposed to happen in main() in the application and libraries shouldn't be calling initialization routines in OpenSSL. This is a big problem, when main() has no knowledge of OpenSSL, rather OpenSSL is used indirectly via an intermediate library, that may even be dynamically loaded (e.g. Java dynamically loading GSSAPI, with Heimdal's GSS library using OpenSSL). Now it is certainly not appropriate for other libraries to call OpenSSL one-time initialization functions. The result is a mess. However, I agree that all the init logic for the library should be handled automatically and transparently in a thread-safe manner. Therefore, it is probably time to consider moving the OpenSSL library initialization code into OpenSSL itself, with the set of ciphers and digests to initialize by default as well as the thread locking mechanism chosen at compile time. -- Viktor. __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager majord...@openssl.org
Re: DLL hell
On Thu, Aug 15, 2013 at 11:51:05PM -0700, Patrick Pelletier wrote: Oh. Is there any reason not to blow that away, or at least build-time select which to use? I'm in agreement with you; I just don't think you're going to get the OpenSSL folks on board. They'll probably say something like we want to be totally agnostic to threading library without acknowledging that pthreads and Windows threads cover the vast majority of modern mainstream operating systems. Ah... I need OpenSSL developers to consider this. Would that mean re-posting to the openssl-dev list? Great. I was hoping that the response wouldn't be something like no way, we need these callback setting functions for XYZ reasons or, worse, no way. Unfortunately, I think the response will be that. (The OpenSSL folks just haven't weighed in on this thread yet.) That's why I was I'm ever an optimist and I fail to see any reason to not make initialization automatic and safe on all major platforms, keeping the old callback setters as no-ops and as fallbacks in cases where build-time configuration specifically requires that those setters not be no-ops. The alternative has to be don't *EVER* use OpenSSL from a library, or always link with and initialize OpenSSL in every program that might -no matter how indirectly- use an OpenSSL-using library, and *clearly* that can't be what the OpenSSL devs want, or if it is, then it's clearly way too late. floating the idea of writing an unofficial companion library that would smooth over these rough spots and provide a batteries included approach to people who want it, without having to convince the OpenSSL project to change the core library, which I think would be an uphill battle at best. That can't really work unless *every* OpenSSL-using library used it, or unless we specifically go for using symbol interposition (which means dynamic linking, FYI, so it'd not work for statically-linked builds). I'd like to get authoritative answers to my questions before considering alternatives. Nico -- __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager majord...@openssl.org
Re: DLL hell
On Fri, Aug 16, 2013 at 02:44:23PM +, Viktor Dukhovni wrote: On Fri, Aug 16, 2013 at 07:17:22AM -0700, Thomas J. Hruska wrote: I think a lot of the init logic heralds from the original SSLeay days. There seems to be intent that initialization is supposed to happen in main() in the application and libraries shouldn't be calling initialization routines in OpenSSL. This is a big problem, when main() has no knowledge of OpenSSL, rather OpenSSL is used indirectly via an intermediate library, that may even be dynamically loaded (e.g. Java dynamically loading GSSAPI, with Heimdal's GSS library using OpenSSL). Right! Now it is certainly not appropriate for other libraries to call OpenSSL one-time initialization functions. The result is a mess. Exactly. Therefore, it is probably time to consider moving the OpenSSL library initialization code into OpenSSL itself, with the set of ciphers and digests to initialize by default as well as the thread locking mechanism chosen at compile time. But would patches for this be welcomed? Nico -- __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager majord...@openssl.org
Re: DLL hell
On 8/15/13 10:24 AM, Nico Williams wrote: . Recent developments, like Android's failure to properly initialize OpenSSL's PRNG make me think it's time to table (in the British sense) the issue once more. Can you point to any article or post which explains exactly what the OpenSSL half of the Android issue was? (I understand the Harmony SecureRandom issue, but that's a separate thing.) OpenSSL is supposed to call RAND_poll on the first call to RAND_bytes, and RAND_poll knows how to seed from /dev/urandom on systems that have it, which should include Android. Neither I nor others speculating on Google+ could figure out why this wasn't the case, and why explicit seeding would have been necessary: https://plus.google.com/+AndroidDevelopers/posts/YxWzeNQMJS2 There should be no need for run-time, global initialization of OpenSSL (or any sub-system of it) by applications. Moreover, any existing functions for such initialization should become no-ops or do their work just once, and in a thread-safe way. As much configuration of threading and other system libraries should be done at build-time as possible; as little should be done at run-time as possible. Even for special-purpose OSes. Figuring out the right sequence of initialization functions to call, even when just one application is using OpenSSL, has not been entirely clear. In particular, see my rambling talk-page discourse on the OpenSSL Wiki about what is and isn't necessary in order to get ENGINEs initialized, and how it depends upon some build-time #defines in a non-obvious way: http://wiki.opensslfoundation.com/index.php/Talk:Libcrypto_API OpenSSL requires too much explicit and global run-time initialization of it by applications in ways that preclude (or at least used to and still might cause problems for) multiple distinct callers of OpenSSL in the same process. And yet the layered software systems we've built over the past decade result in just that happening. Thank you! I'm glad I'm not the only one who feels this is a big problem. It's something I've expressed concern about in the past, albeit in a parenthesized paragraph about halfway through a long, far-reaching rant: http://lists.randombit.net/pipermail/cryptography/2012-October/003388.html Typical examples include: applications that use TLS and Kerberos, PAM applications (each module might need to use OpenSSL, and so might the application), systems running without nscd (the name service switch modules may need OpenSSL, and so might the app), and so on. Some examples close to my heart: * using the OpenSSL Ruby Gem, while also using another Ruby Gem that depends on OpenSSL indirectly * using OpenSSL directly while also using libevent's optional integration with OpenSSL Huh? Which is it? Must apps call CRYPTO_THREADID_set_callback() even though [o]n Windows and many Unix systems, OpenSSL automatically uses the multi-threaded versions of the standard libraries? Why? One would think that the multi-threaded versions of the standard libraries on such OSes would provide all that OpenSSL needs. I don't get it. I think what it means is that We link against thread-safe versions of the standard library (which implies there are thread-unsafe versions of the standard library, which I think might be true on Windows, but I'm pretty sure there isn't any thread-unsafe version of the standard library on Linux or OS X) so that, for example, errno doesn't get clobbered by multiple threads. But we still don't call any threading functions to make OpenSSL itself threadsafe, so you'll have to provide this boilerplate yourself in every OpenSSL application you write. I could also rant about OpenSSL's homemade thread local storage (TLS, in the other sense of the acronym) implementation, rather than using the OS's API for thread local storage. This can result in memory leaks when a thread terminates, unless you have complete control over the lifecycle of every thread and can call a cleanup function before the thread terminates. (And, as you've noted, in a modern application with many layers of libraries, you don't have such control.) Finally, what can I do to help? Definitely to the extent that you can figure out what the actual behavior is (although that itself is not easy and usually requires experimentation and studying the source code in-depth), document your findings on the OpenSSL wiki: http://wiki.opensslfoundation.com/ (Though hopefully any such clarifications can be propagated to the more official documentation as well.) And I certainly hope that the initialization and thread-safety issues can be addressed in libcrypto itself, but if they can't or won't, then I think we (meaning those interested) should write a small add-on library to go along with OpenSSL, that would include standard implementations of the thread safety callbacks, and perhaps we could convince users of OpenSSL to accept it as a de-facto
Re: DLL hell
On Thu, Aug 15, 2013 at 10:58 PM, Patrick Pelletier c...@funwithsoftware.org wrote: On 8/15/13 10:24 AM, Nico Williams wrote: . Recent developments, like Android's failure to properly initialize OpenSSL's PRNG make me think it's time to table (in the British sense) the issue once more. Can you point to any article or post which explains exactly what the OpenSSL half of the Android issue was? (I understand the Harmony SecureRandom issue, but that's a separate thing.) OpenSSL is supposed to call RAND_poll on the first call to RAND_bytes, and RAND_poll knows how to seed from /dev/urandom on systems that have it, which should include Android. Neither I nor others speculating on Google+ could figure out why this wasn't the case, and why explicit seeding would have been necessary: https://plus.google.com/+AndroidDevelopers/posts/YxWzeNQMJS2 Hmm, I've only read the article linked from there: http://android-developers.blogspot.com/2013/08/some-securerandom-thoughts.html Not enough info there :( I don't really feel like finding the relevant OpenJDK JCA code, nor the Android derivative of it. It'd be easier to ask them for more details. If they were wrong about OpenSSL in this respect and the problem was truly specific to Android then I apologize for spreading a falsehood. Figuring out the right sequence of initialization functions to call, even when just one application is using OpenSSL, has not been entirely clear. In particular, see my rambling talk-page discourse on the OpenSSL Wiki about what is and isn't necessary in order to get ENGINEs initialized, and how it depends upon some build-time #defines in a non-obvious way: I'll take a look. For now it seems there should be no need to set any thread-related callbacks, no? Or if they are needed, we should make them no-ops on OSes with thread libraries. [...] Thank you! I'm glad I'm not the only one who feels this is a big problem. It's something I've expressed concern about in the past, albeit in a parenthesized paragraph about halfway through a long, far-reaching rant: http://lists.randombit.net/pipermail/cryptography/2012-October/003388.html I've ranted myself about this privately many a time. I... just never got involved. I felt this [possibly incorrectly attributed to OpenSSL] event was the straw that broke the camel's back for me. I think that a crypto library should have no worse initialization/finalization/thread safety/fork safety semantics than libpkcs11 in Solaris/Illumos, which: a) thread-safely ref-counts C_Initialize()/C_Finalize() calls, b) leaves locking around objects to the app, c) re-initializes (and loses all objects) on the child-side of fork(), d) no thread/lock callback setters. (It's necessary to finalize all objects that refer to sessions for crypto coprocessors, TPMs, tokens, as well as any other stateful objects on the child side of fork(), either that or establish new sessions. It shouldn't be necessary to finalize key objects, say, but hey, it's what PKCS#11 requires.) Better yet: implied initialization (using pthread_once()). * using the OpenSSL Ruby Gem, while also using another Ruby Gem that depends on OpenSSL indirectly * using OpenSSL directly while also using libevent's optional integration with OpenSSL I'm sure there's many more :( Huh? Which is it? Must apps call CRYPTO_THREADID_set_callback() even though [o]n Windows and many Unix systems, OpenSSL automatically uses the multi-threaded versions of the standard libraries? Why? One would think that the multi-threaded versions of the standard libraries on such OSes would provide all that OpenSSL needs. I don't get it. I think what it means is that We link against thread-safe versions of the standard library (which implies there are thread-unsafe versions of the standard library, which I think might be true on Windows, but I'm pretty sure there isn't any thread-unsafe version of the standard library on Linux or OS X) so that, for example, errno doesn't get clobbered by multiple threads. But we still don't call any threading functions to make OpenSSL itself threadsafe, so you'll have to provide this boilerplate yourself in every OpenSSL application you write. Well, *static* libc.a's tend to not be very thread-safe. Solaris 10 and Linux still today both have a number of process models: - dynamically linked, linked with a threading library - dynamically linked, NOT linked with a threading library - linked with static libc, linked with a threading library - linked with static libc, NOT linked with a threading library and interesting things happen if in the not linked with a thread library cases if an object loaded with dlopen() does link with a thread library (even more interesting if the program is statically linked with libc). The dlopen() upgrade to threaded model case happens when you use the name service switch without an nscd daemon or PAM or... A lot of people seem to love static linking.