I think there's some confusion here... OpenSSL's pkeyutl does indeed call something with out==NULL, but it's not calling RSA_private_decrypt() or RSA_public_encrypt() directly, it's calling the EVP_PKEY functions. In *those* functions, there is a check to see if the output argument is NULL and to return the appropriate size in that case. There is no promise that the lower level functions called by the EVP_PKEY interface does the same.
As a matter of fact, the manual page for RSA_private_decrypt and RSA_public_encrypt says this: ... RSA_public_encrypt() encrypts the flen bytes at from (usually a session key) using the public key rsa and stores the ciphertext in to. to must point to RSA_size(rsa) bytes of memory. ... RSA_private_decrypt() decrypts the flen bytes at from using the private key rsa and stores the plaintext in to. to must point to a memory section large enough to hold the decrypted data (which is smaller than RSA_size(rsa)). padding is the padding mode that was used to encrypt the data. ... That should make it clear that a NULL output buffer isn't acceptable at that level. Cheers, Richard In message <a8bf4fbb-3e80-4126-9826-e9a64c46f...@ll.mit.edu> on Tue, 26 Sep 2017 20:29:11 +0000, "Blumenthal, Uri - 0553 - MITLL" <u...@ll.mit.edu> said: uri> Working on pkcs11 engine, I discovered a bug in crypto/rsa/rsa_pmeth.c in pkey_rsa_encrypt() and uri> pkey_rsa_decrypt(). uri> uri> They cause a crash when called with out==NULL. Normally it should not happen – but when an uri> engine is called, and it cannot process the padding – it reverts to the original OpenSSL-provided uri> pkey_rsa_encrypt() or pkey_rsa_decrypt() (as appropriate). OpenSSL pkeyutl makes two calls when uri> the key is not directly available (aka not presented in a disk file), and the first call with out==NULL uri> crashes when RSA_private_decrypt() or RSA_public_encrypt() tries to copy the result to out. uri> uri> The fix should be adding something like uri> uri> if (out == NULL) { uri> uri> int klen = RSA_size(ctx->pkey->pkey.rsa); uri> uri> *outlen = klen; uri> uri> return 1; uri> uri> } uri> uri> right before the call to RSA_public_encrypt(). uri> uri> P.S. It’s more critical in pkey_rsa_encrypt(), because it’s more likely that the engine would handle uri> the decryption operation completely by itself. uri> uri> -- uri> uri> Regards, uri> uri> Uri Blumenthal uri> -- openssl-dev mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev