openssl 0.9.8n issue with no-tlsext
Hello, after updating from openssl 0.9.8l to openssl 0.9.8n, I'm unable to connect to a TLS enabled SMTP server: ./openssl s_client -connect smtp.scriptroom.net:25 -starttls smtp -debug 28141:error:14092073:SSL routines:SSL3_GET_SERVER_HELLO:bad packet length:s3_clnt.c:878: openssl is compiled with the no-tlsext option. no-tlsext was added back in 2009 as openssl 0.9.8j had trouble connecting to a Centos 3 based server. (http://marc.info/?l=openssl-devm=123192990505188) openssl-0.9.8m is also affected. Any idea what might be going on? Thanks in advance, Thomas __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager majord...@openssl.org
RE: OpenSSL 1.0.0 released - VMS
Hello, I am happy that 1.0.0 is released. Thank you all for the hard work and time spent for the community. I was really hoping and looking for a VMS ready 1.0.0 release. Some of us have sent many patches, suggestions - unfortunately, not all of those changes have got through to the released code. Therefore, I sadly need to rapport the following build failures on OpenVMS (IA64) system. (see at the bottom) ... after all, I wonder has been tested this 1.0.0 release at all? Obviously, not on the OpenVMS platform - and this should be a clear responsibility of the project member who is responsible for the VMS code health. ...all patches has arrived. I understand if there is no time for merging in into some beta release... but please - this is an official release of one of the most important security related software... and it is delivered untested. I am starting to speculate that there might be some personal interest in not providing functional OpenVMS code, because I can not imagine what could stop a professional and responsible code reviewer to merge into the code a patch submitted by developers that is obviously needed to achieve proper functionality. I have written earlier also, but here it is again: I would be glad to provide any help, including coding, even donating a box running OpenVMS for the OpenSSL community, whatever else - in order to have OpenSSL code build on OpenVMS as well. Please, ensure us, all developers that this project is worth spending time and effort... and our patches will land into the code soon or later improving the stability and the functionality for everybody's benefit. ...otherwise seeing that our work is for nothing... it is not an uplifting feeling :( 1. Compiling The cversion.c File. (LIBRARY,LIB) #include buildinf.h .^ %CC-F-NOINCLFILEF, Cannot find file buildinf.h specified in #include directive. at line number 62 in file USRDSK:[ZAY.WORK.OPENSSL-100.CRYPTO]CVERSION.C;1 2. - Compiling The UI Library Files. (LIBRARY,LIB) ui_err.c ui_lib.c int UI_method_set_prompt_constructor(UI_METHOD *method, char *(*prompt_constructor)(UI* ui, const char* object_desc, const char* object_name)); ^ %CC-W-LONGEXTERN, The external identifier name exceeds 31 characters; truncated to UI_METHOD_SET_PROMPT _CONSTRUCTO. at line number 313 in file USRDSK:[ZAY.WORK.OPENSSL-100.INCLUDE.OPENSSL]UI.H;2 char* (*UI_method_get_prompt_constructor(UI_METHOD *method))(UI*, const char*, const char*); ^ %CC-W-LONGEXTERN, The external identifier name exceeds 31 characters; truncated to UI_METHOD_GET_PROMPT _CONSTRUCTO. at line number 319 in file USRDSK:[ZAY.WORK.OPENSSL-100.INCLUDE.OPENSSL]UI.H;2 %LIBRAR-W-COMCOD, compilation warnings in module UI_LIB file USRDSK:[ZAY.WORK.OPENSSL-100.IA64.OBJ.CRYPT O]UI_LIB.OBJ;1 3. --- %ILINK-W-COMPWARN, compilation warnings module: UI_LIB file: USRDSK:[ZAY.WORK.OPENSSL-100.IA64.EXE.CRYPTO]LIBCRYPTO.OLB;1 %ILINK-W-NUDFSYMS, 2 undefined symbols: %ILINK-I-UDFSYM,SSLEAY %ILINK-I-UDFSYM,SSLEAY_VERSION %ILINK-W-USEUNDEF, undefined symbol SSLEAY_VERSION referenced section: $CODE$ offset: %X00011740 slot: 2 module: SPEED file: USRDSK:[ZAY.WORK.OPENSSL-100.IA64.OBJ.APPS]SPEED.OBJ;1 %ILINK-W-USEUNDEF, undefined symbol SSLEAY_VERSION referenced section: $CODE$ offset: %X000117A0 slot: 2 module: SPEED file: USRDSK:[ZAY.WORK.OPENSSL-100.IA64.OBJ.APPS]SPEED.OBJ;1 %ILINK-W-USEUNDEF, undefined symbol SSLEAY_VERSION referenced section: $CODE$ offset: %X00011A50 slot: 2 module: SPEED file: USRDSK:[ZAY.WORK.OPENSSL-100.IA64.OBJ.APPS]SPEED.OBJ;1 %ILINK-W-USEUNDEF, undefined symbol SSLEAY referenced section: $CODE$ offset: %X03F0 slot: 2 module: VERSION file: USRDSK:[ZAY.WORK.OPENSSL-100.IA64.OBJ.APPS]VERSION.OBJ;1 %ILINK-W-USEUNDEF, undefined symbol SSLEAY_VERSION referenced section: $CODE$ offset: %X0410 slot: 2 module: VERSION file: USRDSK:[ZAY.WORK.OPENSSL-100.IA64.OBJ.APPS]VERSION.OBJ;1 %ILINK-W-USEUNDEF, undefined symbol SSLEAY_VERSION referenced section: $CODE$ offset: %X0460 slot: 2 module: VERSION file: USRDSK:[ZAY.WORK.OPENSSL-100.IA64.OBJ.APPS]VERSION.OBJ;1 Regards, Z -Original Message- From: OpenSSL [mailto:open...@openssl.org] Sent: den 29 mars 2010 16:52 To: openssl-dev@openssl.org; openssl-us...@openssl.org; openssl-annou...@openssl.org Subject: OpenSSL 1.0.0 released -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 OpenSSL version 1.0.0 released == __ OpenSSL Project
Re: OpenSSL 1.0.0 released - VMS
On Tue, Mar 30, 2010, Arpadffy Zoltan wrote: Hello, I am happy that 1.0.0 is released. Thank you all for the hard work and time spent for the community. I was really hoping and looking for a VMS ready 1.0.0 release. Some of us have sent many patches, suggestions - unfortunately, not all of those changes have got through to the released code. Have any of these patches been sent to the request tracker? Steve. -- Dr Stephen N. Henson. OpenSSL project core developer. Commercial tech support now available see: http://www.openssl.org __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager majord...@openssl.org
Re: openssl 0.9.8n issue with no-tlsext
On Tue, Mar 30, 2010 at 7:35 AM, Thomas Jarosch thomas.jaro...@intra2net.com wrote: 28141:error:14092073:SSL routines:SSL3_GET_SERVER_HELLO:bad packet length:s3_clnt.c:878: openssl is compiled with the no-tlsext option. no-tlsext was added back in 2009 as openssl 0.9.8j had trouble connecting to a Centos 3 based server. (http://marc.info/?l=openssl-devm=123192990505188) openssl-0.9.8m is also affected. Any idea what might be going on? A tcpdump would be very helpful. It might be that the reneg extension is sent even with no-tlsext, although I haven't checked the code. (But if the server is TLS intolerant, then it's really time to fix the server.) AGL -- Adam Langley a...@imperialviolet.org http://www.imperialviolet.org __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager majord...@openssl.org
Re: openssl 0.9.8n issue with no-tlsext
On Mar 30, 2010, at 3:04 PM, Adam Langley wrote: On Tue, Mar 30, 2010 at 7:35 AM, Thomas Jarosch thomas.jaro...@intra2net.com wrote: 28141:error:14092073:SSL routines:SSL3_GET_SERVER_HELLO:bad packet length:s3_clnt.c:878: openssl is compiled with the no-tlsext option. no-tlsext was added back in 2009 as openssl 0.9.8j had trouble connecting to a Centos 3 based server. (http://marc.info/?l=openssl-devm=123192990505188) openssl-0.9.8m is also affected. Any idea what might be going on? A tcpdump would be very helpful. Or just add the -msg option to the command line. I can see what is happening, though: as of RFC 5746, disabling TLS extensions isn't really tenable because you can't do secure renegotiation without those, and can't even quite do a secure *initial* negotiation because that requires at least sending the pseudo-ciphersuite number 0x00 0xFF -- which to current servers is equivalent to a TLS extension. So client-side OpenSSL is buggy if compiled with no-tlsext (in 0.9.8m and 0.9.8n) because it sends that pseudo-ciphersuite number without being able to handle the TLS extension then expected in the server's response. So the no-tlsext build shouldn't be sending the pseudo- ciphersuite number. However, then you'd soon have problems connecting to some updated servers, as these may start to *demand* confirmation that clients are updated to support RFC 5746. So the fix won't help you in the long run. Bodo __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager majord...@openssl.org
Re: openssl 0.9.8n issue with no-tlsext
Hello, On Tuesday, 30. March 2010 15:51:31 Bodo Moeller wrote: So client-side OpenSSL is buggy if compiled with no-tlsext (in 0.9.8m and 0.9.8n) because it sends that pseudo-ciphersuite number without being able to handle the TLS extension then expected in the server's response. So the no-tlsext build shouldn't be sending the pseudo- ciphersuite number. However, then you'd soon have problems connecting to some updated servers, as these may start to *demand* confirmation that clients are updated to support RFC 5746. So the fix won't help you in the long run. Thanks for your explanation. I could remove the no-tlsext option as the servers under our control haven been upgraded from Centos 3 to Centos 5. I'm just thinking what might happen if f.e. a TLS enabled postfix connects to an old Centos 3 based server to deliver emails. Guess that would fail like in 2009, wouldn't it? Cheers, Thomas __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager majord...@openssl.org
Re: openssl 0.9.8n issue with no-tlsext
On Tuesday, 30. March 2010 16:01:54 Thomas Jarosch wrote: I'm just thinking what might happen if f.e. a TLS enabled postfix connects to an old Centos 3 based server to deliver emails. Guess that would fail like in 2009, wouldn't it? Just rechecked the issue from 2009 (http://marc.info/?l=openssl-devm=123192990505188) and it was related to SSLv3 and not to TLS. So this could work out fine IMHO. Cheers, Thomas __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager majord...@openssl.org
[openssl.org #2211] Segfault running 'openssl cms -decrypt', version 1.0.0
Hello, I think I've found a bug in openssl. When I run the following command to decrypt a file in CMS format (DER encoded), openssl crashes with a segmentation fault: openssl cms -decrypt -binary -inform der -in inputfile -recip certs/enc.crt-nopass -out outputfile The inputfile is attached (and doesn't contain any secrets), the certificate/private key is available on request. When I run the same command in 'valgrind', it does complete and I get a successfully decrypted file. I have also tried using other versions of openssl (openssl-0.9.8n, openssl-1.0.0-beta4, openssl-1.0.0-beta5) and they all fail in the same way. The backtrace is as follows: [New Thread 0xb7d5f6b0 (LWP 14207)] Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0xb7d5f6b0 (LWP 14207)] 0x08129638 in asn1_template_clear (pval=0xb3468004, tt=0x8202cc0) at tasn_new.c:315 315 *pval = NULL; (gdb) bt #0 0x08129638 in asn1_template_clear (pval=0xb3468004, tt=0x8202cc0) at tasn_new.c:315 #1 0x0812953e in ASN1_template_new (pval=0xb3468004, tt=0x8202cc0) at tasn_new.c:272 #2 0x0812939b in asn1_item_ex_combine_new (pval=0xb367fff0, it=0x8202cfc, combine=0) at tasn_new.c:201 #3 0x0812905d in ASN1_item_ex_new (pval=0xb367fff0, it=0x8202cfc) at tasn_new.c:85 #4 0x0812b41d in ASN1_item_ex_d2i (pval=0xb367fff0, in=0xbfffd484, len=871, it=0x8202cfc, tag=0, aclass=128, opt=1 '\001', ctx=0xbfffd730) at tasn_dec.c:401 #5 0x0812bca4 in asn1_template_noexp_d2i (val=0xb367fff0, in=0xbfffd544, len=2536, tt=0x82035d4, opt=1 '\001', ctx=0xbfffd730) at tasn_dec.c:733 #6 0x0812b983 in asn1_template_ex_d2i (val=0xb367fff0, in=0xbfffd544, inlen=2536, tt=0x82035d4, opt=1 '\001', ctx=0xbfffd730) at tasn_dec.c:608 #7 0x0812b55a in ASN1_item_ex_d2i (pval=0xb3656ffc, in=0xbfffd5e4, len=2536, it=0x8203638, tag=16, aclass=0, opt=0 '\0', ctx=0xbfffd730) at tasn_dec.c:449 #8 0x0812bd08 in asn1_template_noexp_d2i (val=0xb3656ffc, in=0xbfffd640, len=2541, tt=0x82039b4, opt=0 '\0', ctx=0xbfffd730) at tasn_dec.c:747 #9 0x0812b8a7 in asn1_template_ex_d2i (val=0xb3656ffc, in=0xbfffd6a4, inlen=2543, tt=0x82039b4, opt=0 '\0', ctx=0xbfffd730) at tasn_dec.c:576 #10 0x0812b55a in ASN1_item_ex_d2i (pval=0xbfffd72c, in=0xbfffd768, len=2543, it=0x8203aa8, tag=16, aclass=0, opt=0 '\0', ctx=0xbfffd730) at tasn_dec.c:449 #11 0x0812ad9d in ASN1_item_d2i (pval=0xbfffd72c, in=0xbfffd768, len=2556, it=0x8203aa8) at tasn_dec.c:136 #12 0x08121a29 in ASN1_item_d2i_bio (it=0x8203aa8, in=0xb3591fc0, x=0x0) at a_d2i_fp.c:116 #13 0x081649a4 in d2i_CMS_bio (bp=0xb3591fc0, cms=0x0) at cms_io.c:82 #14 0x08087eb0 in cms_main (argc=11, argv=0xbfffde58) at cms.c:793 #15 0x0804a7ad in do_cmd (prog=0xb3d59fa0, argc=11, argv=0xbfffde58) at openssl.c:413 #16 0x0804a47d in main (Argc=11, Argv=0xbfffde58) at openssl.c:312 openssl was configured with: ./config enable-cms -d (to obtain a proper backtrace). The output of 'make report' without the '-d' flag is: OpenSSL self-test report: OpenSSL version: 1.0.0 Last change: Add missing function EVP_CIPHER_CTX_copy(). This copi... Options: enable-cms no-gmp no-jpake no-krb5 no-md2 no-rc5 no-rfc3779 no-shared no-store no-zlib no-zlib-dynamic static-engine OS (uname): Linux sdb-test 2.6.26-2-686 #1 SMP Wed Feb 10 08:59:21 UTC 2010 i686 GNU/Linux OS (config): i686-whatever-linux2 Target (default): linux-elf Target: linux-elf Compiler: Using built-in specs. Target: i486-linux-gnu Configured with: ../src/configure -v --with-pkgversion='Debian 4.3.2-1.1' --with-bugurl=file:///usr/share/doc/gcc-4.3/README.Bugs --enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr --enable-shared --with-system-zlib --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --enable-nls --with-gxx-include-dir=/usr/include/c++/4.3 --program-suffix=-4.3 --enable-clocale=gnu --enable-libstdcxx-debug --enable-objc-gc --enable-mpfr --enable-targets=all --enable-cld --enable-checking=release --build=i486-linux-gnu --host=i486-linux-gnu --target=i486-linux-gnu Thread model: posix gcc version 4.3.2 (Debian 4.3.2-1.1) Test passed. If I can do anything to aid in debugging this, let me know. Best Regards, Ronald. inputfile Description: Binary data
Failed to compile OpenSSL 0.9.8n with compression disabled
doing ./config no-comp ; make on OpenSSL 0.9.8n I get this:
gcc -I../crypto -I.. -I../include -DDSO_DLFCN -DHAVE_DLFCN_H -m64
-DL_ENDIAN -DTERMIO -O3 -Wall -DMD32_REG_T=int -DOPENSSL_BN_ASM_MONT
-DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DAES_ASM -c -o s23_clnt.o
s23_clnt.c
s23_clnt.c: In function 'ssl23_client_hello':
s23_clnt.c:375: error: 'j' undeclared (first use in this function)
s23_clnt.c:375: error: (Each undeclared identifier is reported only once
s23_clnt.c:375: error: for each function it appears in.)
s23_clnt.c:381: error: 'comp' undeclared (first use in this function)
make[1]: *** [s23_clnt.o] Error 1
Looking closer, the variables j and comp are in a preprocessor block:
#ifndef OPENSSL_NO_COMP
int j;
SSL_COMP *comp;
#endif
at line 209 of s23_clnt.c
Comparing to a 0.9.8l (which works), it looks like the #ifndef is new, as
0.9.8l had these declared without the preprocessor wrappings:
static int ssl23_client_hello(SSL *s)
{
unsigned char *buf;
unsigned char *p,*d;
int i,j,ch_len;
unsigned long Time,l;
int ssl2_compat;
int version = 0, version_major, version_minor;
SSL_COMP *comp;
int ret;
Any reason not to remove the preprocessor #ifndef from 0.9.8n?
Thanks,
Jeff
Re: Failed to compile OpenSSL 0.9.8n with compression disabled
On Tue, Mar 30, 2010, Jeff Davey wrote:
doing ./config no-comp ; make on OpenSSL 0.9.8n I get this:
gcc -I../crypto -I.. -I../include -DDSO_DLFCN -DHAVE_DLFCN_H -m64
-DL_ENDIAN -DTERMIO -O3 -Wall -DMD32_REG_T=int -DOPENSSL_BN_ASM_MONT
-DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DAES_ASM -c -o s23_clnt.o
s23_clnt.c
s23_clnt.c: In function 'ssl23_client_hello':
s23_clnt.c:375: error: 'j' undeclared (first use in this function)
s23_clnt.c:375: error: (Each undeclared identifier is reported only once
s23_clnt.c:375: error: for each function it appears in.)
s23_clnt.c:381: error: 'comp' undeclared (first use in this function)
make[1]: *** [s23_clnt.o] Error 1
Looking closer, the variables j and comp are in a preprocessor block:
#ifndef OPENSSL_NO_COMP
int j;
SSL_COMP *comp;
#endif
at line 209 of s23_clnt.c
Comparing to a 0.9.8l (which works), it looks like the #ifndef is new, as
0.9.8l had these declared without the preprocessor wrappings:
static int ssl23_client_hello(SSL *s)
{
unsigned char *buf;
unsigned char *p,*d;
int i,j,ch_len;
unsigned long Time,l;
int ssl2_compat;
int version = 0, version_major, version_minor;
SSL_COMP *comp;
int ret;
Any reason not to remove the preprocessor #ifndef from 0.9.8n?
Oops, part of the update got left out in 0.9.8. This is a patch:
http://cvs.openssl.org/chngview?cn=19518
Steve.
--
Dr Stephen N. Henson. OpenSSL project core developer.
Commercial tech support now available see: http://www.openssl.org
__
OpenSSL Project http://www.openssl.org
Development Mailing List openssl-dev@openssl.org
Automated List Manager majord...@openssl.org
RE: libcrypto safe for library use?
Mark Phalan wrote: In this case, I presume 'pkinit' only supports one threading model (or one set of compatible threading models). So it can set the callbacks. It can set the callbacks but it can't set them in a way which is safe from races. It can set them before it performs any OpenSSL operations. How is that not safe from races? Then set default callbacks in your code that calls OpenSSL. OpenSSL can't do it, because it has no idea what threading models your code uses. Perhaps I don't understand exactly what you mean by threading model. Are you referring to 1:1, n:1 etc? That's part of it. It's also what threading library you are using. I think it would be reasonable for OpenSSL as delivered with an OS to use the native threading model of the OS. Is there something I'm missing? If you only support one threading model, there's no problem. Just set callbacks that are safe for that threading model before you use OpenSSL. If the horse has left the stable and the code supports more than one threading model, then the problem is provable insolvable. There is simply no way for OpenSSL to know what kind of locks are adequate. If your code supports only one threading model, then you can tell OpenSSL this by setting the callbacks. What I could do is say that when delivered as part of OpenSolaris OpenSSL assumes the OS's threading model. If an application wants to change that they the application should set its own callbacks. If that's what you want, it's trivial to implement. Simply set callbacks that are safe for the OS' threading model before you call any OpenSSL functions. Just to be totally clear. All I'd like to see is a compile-time option (it can be disabled by default) to set a default set of locking callbacks native to the OS it's compiling on. I'm not suggesting that the ability to set locking callbacks be removed. If you are able to craft a default set of locking callbacks that are safe for your application, then do so and set them. I don't see what assistance you need from OpenSSL. If that is your case, your solution is trivial -- set those callbacks. If you know they're safe, set them. Since libldap_r knows what threading model the application is using, it can do this. OpenSSL doesn't know, so it can't do this. It can do this but only by potentially introducing a race. What is the race exactly? If they're already set, setting them again to precisely the same callbacks is harmless. And if the problem is an application setting them to different callbacks, OpenSSL can't help you -- if an application changes the callbacks from the default to unsafe callbacks, you're still hosed. Existing code that uses OpenSSL will still change those callbacks from the defaults. New applications that don't use OpenSSL at all won't be affected by the change. New libraries could just as easily be coded to always set known safe callbacks. There is only a race if some application either reverts to no callbacks or sets unsafe callbacks. Both of those operations will still be fatal with the suggested change. DS __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager majord...@openssl.org
RE: libcrypto safe for library use?
Mark Phalan wrote: On 29 Mar 2010, at 20:20, David Schwartz dav...@webmaster.com wrote: Mark Phalan wrote: I think this fix is actually a bad one. I'm still not clear why you think that. Because it doesn't solve the problem case -- where one library user sets callbacks that another library user isn't happy with. It won't help existing code, because it will already set callbacks (or it's hopelessly broken). It might help new code, but if we're going to help new code, we should do it a better way. Even without the proposed change libraries should at least check before clobbering the existing callbacks. Checking before setting will be racy but still better than blindly replacing callbacks. I don't get what is this race you see. Are you talking about libraries that unset the OpenSSL callbacks while another library running in the same application might be using OpenSSL? That would still be fatal, and can never be made to work anyway. The proposal doesn't make anything worse it 1) removes the race if libraries set the callbacks properly How is that? I don't see how it removes the race. There are two cases for existing code: 1) They all set callbacks that are 100% compatible -- in which case there never was any race to fix. 2) Some of them set callbacks that are not 100% compatible with the callbacks others set -- in this case, there's still a race. and 2) fixes MT issues if they don't. If they don't set locking callbacks where such is clearly documented to be required, they're so hopelessly broken there's no point in trying to fix them in OpenSSL. What other requirements do they ignore? Of course there is always the problem of what happens on dlclose() of the lib if the callbacks are set... You can't shutdown a library while another library in that same application is or might be using it. It will never be possible to make that work. Again, that would be such hopelessly broken behavior that there's no point in trying to fix it for existing libraries/applications. A sensible way to fix it for new applications/libraries might make sense. The issue is that existing code may set the locking callbacks badly and the horse has already left the stable (we can't redesign them). We can't easily get all the libraries which might end up eventually loading OpenSSL to change their APIs so that locking callbacks can be set but I think it wouldn't be too difficult to get them to do a check for existing callbacks before doing the set. Libraries which don't set the callbacks immediately become MT safe (as far as OpenSSL calls are concerned). For new implementations, then best solution is an OpenSSL wrapper that can arbitrate when OpenSSL needs to be loaded and unloaded and who is using it. This could certainly be built into OpenSSL. But I don't think it would help existing applications unless they at least make some kind of 'add OpenSSL user' and 'remove OpenSSL user' kind of call. I don't see how this helps in that case -- the existing code will continue to set the locking callbacks badly, overriding the sane default. Indeed. Until those are changed they will be broken both now and after the proposed change. The proposal offers a sane way out. If we're talking about fixing it for new code, I think either an OpenSSL wrapper library, OpenSSL wrapper support in the application, or an 'add OpenSSL user'/'remove OpenSSL user' function in OpenSSL is the right way to go. DS __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager majord...@openssl.org
Re: libcrypto safe for library use?
David Schwartz wrote: Mark Phalan wrote: In this case, I presume 'pkinit' only supports one threading model (or one set of compatible threading models). So it can set the callbacks. It can set the callbacks but it can't set them in a way which is safe from races. It can set them before it performs any OpenSSL operations. How is that not safe from races? Still don't get it. Lets use Linux as an example. You have a top level executable whose only runtime linked libraries are libc, libdl, libpthread. This library from 3 independent threads decides it needs to dynamically load via the libdl API call dlopen() the following libraries: libldap_r (this in turn has a runtime library dependency on libcrypto libssl amongst other DSOs). libcurl (this in turn has a runtime library dependency on libcrypto libssl amongst other DSOs). libstunnel (this in turn has a runtime library dependency on libcrypto libssl amongst other DSOs). There are less contrived cases; Perl and Python built into executables that in turn link with OpenSSL as well. The top level application doesn't even know OpenSSL exists. It doesn't care that is exists, since the API contract is has relates to some other functionality. The dynamic linker will de-duplicate the DSO being loaded, so when libcrypto is loaded for the 2nd time it will realize it is already loaded and reference count it up one and hand the same DSO address to each of the 3 callers to share. Since that is the point of DSOs. How does each of the 3 threads arbitrate in a thread-safe manner the initialization of the OpenSSL related libraries. How does any one of them know it is the first user ? Once each of the 3 users has finished using their copy of OpenSSL they can request the dynamic linker unload it dlclose(). So it is also necessary for OpenSSL to know when the last user has closed the library and it needs to perform memory freeing and cleanup because there are no users left. How does any one of them know it is the last user ? I think these cases should have equal merit, if you are going to fix the load case you must also fix the unload case. Now you can re look at the problem and simplify it on the basis what can be do if we know its a single thread. Then from that vantage point work out what small areas of the process need to be protected. If you ask me the dynamic linker should provide support for this issue itself, the ability to embed a control block in the executable that provides callbacks with the dynamic linker providing serialization of running those callback and exposing the most basic locking primitives (mutex) for use only from inside those callbacks. Windows provides this infrastructure already, _DllMain symbol and the fact that all platform provide access to the Windows API and the mutex system calls are available. Maybe open source systems do as well but we are just not aware of the hooking mechanism in this list. The issue is we don't want the thread-aware code to manage this in the OpenSSL DSOs since that will mess up the single-thread (thread unaware) users of OpenSSL. This also means we don't want a runtime linker dependancy on libpthread just for this purpose (libcrypto currently doesn't have one, libssl does). Since the thread locking primitives needs to be provided by the platform and this is a linker caused issue, it is the dynamic link that should provide the mechanism to clean up this mess. This effectively becomes imposed baggage by the dynamic linker due to _SOME_ users being multi-threaded users (as well as allowing the linker freedom to load dependancies on a whim). Perhaps I don't understand exactly what you mean by threading model. Are you referring to 1:1, n:1 etc? That's part of it. It's also what threading library you are using. You can still have a default theading model and still support all known threading models via reconfiguration before first use. There is no conflict here please stop implying there is one. What I could do is say that when delivered as part of OpenSolaris OpenSSL assumes the OS's threading model. If an application wants to change that they the application should set its own callbacks. If that's what you want, it's trivial to implement. Simply set callbacks that are safe for the OS' threading model before you call any OpenSSL functions. This is exactly where the issue lies. We know we need to do this (please stop repeating back to us that this is what we need to do). The issue is we can not do this in a thread-safe manner since: * The dynamic linker provides no arbitration (this is who I really blame) * OpenSSL the library provides no arbitration (due to it having a single thread / thread unaware use case). Just to be totally clear. All I'd like to see is a compile-time option (it can be disabled by default) to set a default set of locking callbacks native to the OS it's compiling on. I'm not suggesting
Re: libcrypto safe for library use?
Darryl Miles wrote: David Schwartz wrote: Mark Phalan wrote: In this case, I presume 'pkinit' only supports one threading model (or one set of compatible threading models). So it can set the callbacks. It can set the callbacks but it can't set them in a way which is safe from races. It can set them before it performs any OpenSSL operations. How is that not safe from races? Still don't get it. Lets use Linux as an example. This is actually one of the easiest to solve. Of course the bigger problem is that solutions here are all platform-dependent, and maintenance will be a pain. ELF shared libraries support .init and .fini sections to contain code that should be executed just after load and just before unload. Assuming you had a default set of callbacks in the library, it would be simple to set them here. Likewise any other shared library that manipulates callbacks can do its own cleanup safely in their own .fini section. The other feature you need, to make this reasonable, is support for weak external symbols. I don't know of any compiler support for these but certainly they're supported at the assembler and linker level. The default callbacks should reference pthread functions as weak externs, and just no-op if the externs are unresolved at runtime. As an aside, it's probably still unsafe to assume any particular thread model, on any OS. Third party thread libraries are still out there that run cross-platform (e.g. GNU Pth). You have a top level executable whose only runtime linked libraries are libc, libdl, libpthread. This library from 3 independent threads decides it needs to dynamically load via the libdl API call dlopen() the following libraries: libldap_r (this in turn has a runtime library dependency on libcrypto libssl amongst other DSOs). libcurl (this in turn has a runtime library dependency on libcrypto libssl amongst other DSOs). libstunnel (this in turn has a runtime library dependency on libcrypto libssl amongst other DSOs). There are less contrived cases; Perl and Python built into executables that in turn link with OpenSSL as well. The top level application doesn't even know OpenSSL exists. It doesn't care that is exists, since the API contract is has relates to some other functionality. The dynamic linker will de-duplicate the DSO being loaded, so when libcrypto is loaded for the 2nd time it will realize it is already loaded and reference count it up one and hand the same DSO address to each of the 3 callers to share. Since that is the point of DSOs. How does each of the 3 threads arbitrate in a thread-safe manner the initialization of the OpenSSL related libraries. How does any one of them know it is the first user ? Once each of the 3 users has finished using their copy of OpenSSL they can request the dynamic linker unload it dlclose(). So it is also necessary for OpenSSL to know when the last user has closed the library and it needs to perform memory freeing and cleanup because there are no users left. How does any one of them know it is the last user ? I think these cases should have equal merit, if you are going to fix the load case you must also fix the unload case. Now you can re look at the problem and simplify it on the basis what can be do if we know its a single thread. Then from that vantage point work out what small areas of the process need to be protected. If you ask me the dynamic linker should provide support for this issue itself, the ability to embed a control block in the executable that provides callbacks with the dynamic linker providing serialization of running those callback and exposing the most basic locking primitives (mutex) for use only from inside those callbacks. Windows provides this infrastructure already, _DllMain symbol and the fact that all platform provide access to the Windows API and the mutex system calls are available. Maybe open source systems do as well but we are just not aware of the hooking mechanism in this list. The issue is we don't want the thread-aware code to manage this in the OpenSSL DSOs since that will mess up the single-thread (thread unaware) users of OpenSSL. This also means we don't want a runtime linker dependancy on libpthread just for this purpose (libcrypto currently doesn't have one, libssl does). Since the thread locking primitives needs to be provided by the platform and this is a linker caused issue, it is the dynamic link that should provide the mechanism to clean up this mess. This effectively becomes imposed baggage by the dynamic linker due to _SOME_ users being multi-threaded users (as well as allowing the linker freedom to load dependancies on a whim). Perhaps I don't understand exactly what you mean by threading model. Are you referring to 1:1, n:1 etc? That's part of it. It's also what threading library you are using. You can still have a default theading model and still support all known threading models via reconfiguration before first use.
Re: libcrypto safe for library use?
Howard Chu wrote: Darryl Miles wrote: Still don't get it. Lets use Linux as an example. This is actually one of the easiest to solve. Of course the bigger problem is that solutions here are all platform-dependent, and maintenance will be a pain. ELF shared libraries support .init and .fini sections to contain code that should be executed just after load and just before unload. Assuming you had a default set of callbacks in the library, it would be simple to set them here. Likewise any other shared library that manipulates callbacks can do its own cleanup safely in their own .fini section. The other feature you need, to make this reasonable, is support for weak external symbols. I don't know of any compiler support for these but certainly they're supported at the assembler and linker level. The default callbacks should reference pthread functions as weak externs, and just no-op if the externs are unresolved at runtime. Since a dynamic linker requires the use of threading primitives (mutexes and such) internally (if itself supports multi-threaded usage) it is highly likely it has access to an implementation to advise from the .init and .fini. So the next matter would be for the .init and .fini functions to provide a reference back to the dynamic linker as an argument when those callbacks get invoked. I shall call this the module handle. From this module handle the dynamic linker needs to provide API accessible implementations for: * Symbol lookup (maybe this is already extern and implicitly available to all DSOs) * Mutex API (init, lock, unlock, trylock, end, sizeof()) * Reference counting increment/decrement API (init, increment, decrement, end, sizeof()) - To be honest this can be written on top of a mutex, everything can be built on top of a working mutex, but exposing an API for something obviously useful here might stop wheel reinvention! [FWIW - The sizeof() relates to the amount of space required for opaque storage, in the interests of future proofing and the fact these are not performance implementations more space than is required should be reserved and 64bit should be reserving double the amount 32bit does] Just to reiterate the major point of this reply ... THE DYNAMIC LINKER NEEDS TO PROVIDE API What I think I need to clarify is that from the module's point of view it is the dynamic linker which is providing the implementations, not libpthread. The fact these function pointers may directly access an already mapped copy of similar libpthread function/symbols in some platforms should be treated as a coincidence. Infact I would probably advocate not doing that and having libdl at least use a delegation pattern internally to allow a shim layer to iron out minor differences between APIs (the mutex API is exposes and the mutex API is consumes). I'm trying to remove the notion of a symbol out of it, since the module (OpenSSL) can not assume any particular mutex implementation such as POSIX Threads. It doesn't matter what the implementation is it just needs an API provided by the dynamic linker. This means OpenSSL doesn't directly access POSIX Thread definitions at compile time for this purpose, it references some definitions supplied by dlfcn.h. If you start using weak symbols, then you end up down the route of making OpenSSL responsible for knowing about and at runtime trying to detect which mutex implementation exists and is available. This is the whole matter I'm trying to clearly push back to the dynamic linker. This is not OpenSSL's problem! Darryl __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager majord...@openssl.org
Re: libcrypto safe for library use?
Howard Chu wrote: ELF shared libraries support .init and .fini sections to contain code that should be executed just after load and just before unload. Assuming you had a default set of callbacks in the library, it would be simple to set them here. Likewise any other shared library that manipulates callbacks can do its own cleanup safely in their own .fini section. We could be in luck on this point, in that if the dynamic linker already arbitrates the execution of .init/.fini sections by enforcing a single-threaded model then the code within the section does not need to be thread aware. Now the next issue remains. Adding a new runtime dependency to libcrypto.so on the platforms native threading implementation. This is a issue since it is currently clean and some voices want to keep it so (seemly even for platforms where multi-thread library support is the norm and the most popular usage pattern on that platform). It does seem that glibc can keep libpthread out of the mandatory dependencies, I can only guess this is because it implements system calls internally that can be built on top of to implement its own locking requirements. Just the kind of thing it can do but other libraries don't want to do. So again back to the packaging idea could a libcrypto_r be created, which is just a stub DSO. * It has a mandatory dependency on libcrypto. * It may have a mandatory dependency on libpthread (or whichever platform threading library is in use) or it may detect and load at runtime. * It re-exports all the libcrypto symbols (so the application doesn't need to link against libcrypto directly at build link time). * It adds the necessary .init/.fini stuff. * It can even encapsulates some libcrypto API calls as necessary to provide a change in policy. Maybe this is the solution you were alluding to Howard ? As an aside, it's probably still unsafe to assume any particular thread model, on any OS. Third party thread libraries are still out there that run cross-platform (e.g. GNU Pth). On this kind of subject libcrypto_r would need to provide a documented way to override the assumed default. However how to do this without relying on using either of them is another headache. I'm thinking environment variables, configuration files, file existence tests and other such ways to configure a DSO at runtime. This is only an issue if the different threading libraries are used in incompatible ways and/or if the expected default implementation is not available on the platform. For example there can many implementations of spinlocks within a single application that all co-exist, fewer mutex implementations and fewer still; ways to manage the creation of threads. Darryl __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager majord...@openssl.org
RE: libcrypto safe for library use?
Darryl Miles wrote: How does each of the 3 threads arbitrate in a thread-safe manner the initialization of the OpenSSL related libraries. How does any one of them know it is the first user ? For existing code, there is no fix. They will set incompatible callbacks and they will break. I don't see anything OpenSSL can do to fix this other to implement default callbacks and to claim attempts to change the callbacks succeeded while silently failing them. Once each of the 3 users has finished using their copy of OpenSSL they can request the dynamic linker unload it dlclose(). So it is also necessary for OpenSSL to know when the last user has closed the library and it needs to perform memory freeing and cleanup because there are no users left. How does any one of them know it is the last user ? This is a great example of a case where OpenSSL should be wrapped. Again, I don't think you can fix existing code. It's already going to do the wrong thing, for example, shutting down OpenSSL. (Unless you want to make OpenSSL shutdown functions silently do nothing but claim success.) You can still have a default theading model and still support all known threading models via reconfiguration before first use. There is no conflict here please stop implying there is one. That's what we have now. The default threading model is single-threaded. Others are supported by reconfiguration before first use. Setting the callback(s) is unsafe. Since another thread might be using them. Do you not get that ? There is no locking around the setting of them so therefore they are thread-unsafe, the method to set locking is not re-entrant. I harp back to my request for re-entrant thread-safe OpenSSL APIs for setting up the threading callbacks. I agree, but I don't think there's anything we can do to help existing code. Existing code already does the wrong thing. I suggested a register OpenSSL user and deregister OpenSSL user function which permits the user to specify the threading model it's using and trigger a reference count. (Overlapping callers with different threading models would result in an error.) I don't think you can avoid a dependency on the system threading library though, but I don't see why that would be an issue. Many single-threaded programs wind up requiring the threading library on many platforms anyway as it may contain functions like 'clock_gettime' or 'sched_yield'. (Does anyone know of a platform where this is a problem?) DS __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager majord...@openssl.org
Re: OpenSSL 1.0.0 released - VMS
From: Dr. Stephen Henson st...@openssl.org Have any of these patches been sent to the request tracker? A similar question was raised back around 12-NOV-2009: Can you (and others in this thread) please submit bug fix patches to the request tracker (r...@openssl.org) so they don't get overlooked?? which got this still-applicable (whining) response: From my point of view, overlooked is less of a problem than rejected. And some of the stuff which gets added appears to have been tested by no one. For example, I offered revised VMS builders which (among other things): Obviated the annoying copying of C source files into the test directory (symlinks being unsupported on many VMS systems and/or file systems). Added an SSL_ prefix to the object library names so that a victim might have some chance of identifying the things amid the clutter in SYS$SHARE: A bunch of other stuff which I've forgotten. I also tried building the stuff before I submitted my suggestions. Some of these proposed changes involve functional changes which would affect the victims (object library and shared image file name changes, for example), so deserve some discussion. I see very little such discussion here. Suggestions get submitted. Some get adopted. Some get rejected. I find out what happened when I see the next betaN kit. It's not very satisfying, and I've largely stopped caring. (This may or may not be seen as much of a loss.) Since then, I've tried to engage in some discussion of some changes I'd like to see, but only Arpadffy Zoltan and I seem to be interested, and, even when we agree, we have no control over what gets done when. It seems pointeless to invest much time submitting extensive patches of which only small parts are accepted. I'd prefer to have the argument first, but that seems almost never to happen. For example, at least six months ago, I suggested a change to the names of the shareable images on VMS to correspond more closely with the naming scheme HP uses in its OpenSSL kit. Mr. Zoltan also reminded us that HP ships 32-bit and 64-bit binaries, and suggested making their generation easier. The 1.0.0 release seemed to me to be an ideal time to make those changes. Repeated attempts to discuss these changes failed to elicit a response. Apparently, no one tests any release, beta or final, on VMS until it's too late. I'm still testing, but I may have VMS builders working for 1.0.0 which minimize pointless file copying and source tree mutilation. I believe that they will even work on newer VMS systems (V8.3) where symlinks are supported (on ODS5 file systems), and the source kit is unwrapped so that the symlinks are extracted as symlinks. There are, of course, some remaining mysteries: 1. What's the purpose of this in apps?: md4.c - ../crypto/md4/md4.c It's the only symlink in there, and it's not obvious to me why. 2. There are no symlinks in include/openssl in the source kit for these header files: jpake.h md2.h rc5.h store.h Conscious decision or oversight? 3. In the new crypto/whrlpool code, wp_block.c causes problems on the VAX, where there is no 64-bit integer type to use for u64. There seems to be code in there which pretends that the size of u64 is open to discussion (sizeof(u64)), but it seems (to me) to be pretty clear that other things will break if it's not 64 bits. I assume that this stuff can be disabled on the VAX, but I haven't yet looked for a standard method for doing that. (Assuming that no one has any nice 32-bit code to insert to obviate that.) I'll publish what I have after I get through a VAX build. Perhaps Wednesday, but no bets. Steven M. Schweda s...@antinode-info 382 South Warwick Street(+1) 651-699-9818 Saint Paul MN 55105-2547 __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager majord...@openssl.org
