Andy Polyakov <[EMAIL PROTECTED]> said:
> Well, it went totally off topic, but anyway...
>> rsa_oaep_test also breaks when built on SPARC Solaris 2.6 using Sun's SC5.0.
>> The first decryption fails, then the program dumps core when BN_bin2bn() calls
>> malloc() to load key2->n. It looks like the malloc arena is getting trashed.
>
> From my earlier experience core dumps in Solaris 2.x malloc are usually
> insinuated by things like double-freeing of same portion, freeing of
> never allocated chunk and similar earlier in the code. I used to link
> with /usr/lib/libbsdmalloc.a (which is resistant to such things) if
> fast-acting cure is desired.
> > I'm using "-xdepend -xO5 -xarch=v8 -xstrconst";
>
> Can you confirm that test passes if you manually link rsa_oaep_test with
> -lbsdmalloc?
Yes, linking with -lbsdmalloc rsa_oaep_test stops dumping core but all
3 decryptions fail. Linking without -lbsdmalloc the first decryption
fails then it dumps core in a call to malloc().
I'm investigating further...
______________________________________________________________________
OpenSSL Project http://www.openssl.org
Development Mailing List [EMAIL PROTECTED]
Automated List Manager [EMAIL PROTECTED]
