I sent this to the openssl-users forum, but it seems I should be reporting bug here.
After switching from OpenSSL 0.9.7e to OpenSSL 0.9.7g we started experiencing strange problems with certificate verification on Linux systems and crashes on Solaris systems. We tracked this down to the fact the CA name of one of our trusted CAs contains an underscore character which results in the name being a T61STRING. All of the other trusted CAs have names which are PRINTABLESTRINGS. Whether a given certificate could be verified against a store containing the trusted CA certificates or not seemed to be dependent on the order that the trusted CA certificates were originally added to the certificate store. After debugging the process we found that final order of the certificates in the trusted store after sorting was dependent on the order that they were added. After further investigation we found that a new name comparison routine 'asn1_string_memcmp' in x509_cmp.c added in version 0.9.7f which is used for comparing name strings of different types always assumes that a longer string is greater than a shorter. Such an assumption plays havoc with the binary search that is used to locate a given trusted certificate in the certificate store. On our Solaris 8 systems the inconsistent behavior of the comparison has exposed a bug in the Solaris C library qsort routine which runs off the beginning of the list of elements to sort when it gets an unexpected result from the comparison callback routine. We tried a fix to the 'asn1_string_memcmp' routine to always compare the strings up to the shortest length, and then only if the strings are equal to that point declaring the longer string greater than the shorter. This fixed all the issues that we were seeing including the crashes on Solaris. This problem appears to exist in all the OpenSSL releases from OpenSSL 0.9.7f though OpenSSL 0.9.8a. I'm still not convinced that all of the other assumptions made in the X509 name comparison routines are valid for subsequent use of the sorted list in a binary search. We are not seeing any more problems though. Ken Bowler Chief Architect Signiant Corporation. ______________________________________________________________________ OpenSSL Project http://www.openssl.org Development Mailing List [email protected] Automated List Manager [EMAIL PROTECTED]
