On Tue, Sep 26, 2017, Blumenthal, Uri - 0553 - MITLL wrote: > Working on pkcs11 engine, I discovered a bug in crypto/rsa/rsa_pmeth.c in > pkey_rsa_encrypt() and pkey_rsa_decrypt(). > > They cause a crash when called with out==NULL. Normally it should not happen > ??? but when an engine is called, and it cannot process the padding ??? it > reverts to the original OpenSSL-provided pkey_rsa_encrypt() or > pkey_rsa_decrypt() (as appropriate). OpenSSL pkeyutl makes two calls when the > key is not directly available (aka not presented in a disk file), and the > first call with out==NULL crashes when RSA_private_decrypt() or > RSA_public_encrypt() tries to copy the result to out. >
The original RSA pkey method has the flag EVP_PKEY_FLAG_AUTOARGLEN set which handles the NULL output automatically so it is not handled in pkey_rsa_*(). The ENGINE should either set this flag itself too or deal with NULL arguments manually if that is not appropriate. Steve. -- Dr Stephen N. Henson. OpenSSL project core developer. Commercial tech support now available see: http://www.openssl.org -- openssl-dev mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev
