Hi, This is regarding an issue reported here in this link - https://github.com/openssl/openssl/issues/8872 - This is blocking my development progress. I'm seeing this issue - capi_rsa_priv_enc() - capi engine: function not supported, when I tried to upgrade my application from using OpenSSL version 1.0.2r to 1.1.1b. This is causing TLS handshake to fail. In my case, I'm creating a self signed certificate that is used by Server application loading CAPI engine. Certificate is created by calling CertCreateSelfSignCertificate(--) - wincrypt.h - using szOID_RSA_SHA256RSA signature algorithm and "Microsoft Enhanced RSA and AES Cryptographic Provider".
This failure doesn't look like it's because of TLS1.3 as turning off TLS1.3 while configure with "no-tls1_3" also caused the same problem. Here is the call stack that is causing the reported problem when SSL_accept() is called. capi.dll!capi_rsa_priv_enc(int flen, const unsigned char * from, unsigned char * to, rsa_st * rsa, int padding) libcrypto-1_1.dll!RSA_private_encrypt(int flen, const unsigned char * from, unsigned char * to, rsa_st * rsa, int padding) libcrypto-1_1.dll!pkey_rsa_sign(evp_pkey_ctx_st * ctx, unsigned char * sig, unsigned int * siglen, const unsigned char * tbs, unsigned int tbslen) libcrypto-1_1.dll!EVP_PKEY_sign(evp_pkey_ctx_st * ctx, unsigned char * sig, unsigned int * siglen, const unsigned char * tbs, unsigned int tbslen) libcrypto-1_1.dll!EVP_DigestSignFinal(evp_md_ctx_st * ctx, unsigned char * sigret, unsigned int * siglen) libcrypto-1_1.dll!EVP_DigestSign(evp_md_ctx_st * ctx, unsigned char * sigret, unsigned int * siglen, const unsigned char * tbs, unsigned int tbslen) libssl-1_1.dll!tls_construct_cert_verify(ssl_st * s, wpacket_st * pkt) libssl-1_1.dll!write_state_machine(ssl_st * s) libssl-1_1.dll!state_machine(ssl_st * s, int server) libssl-1_1.dll!ossl_statem_accept(ssl_st * s) libssl-1_1.dll!SSL_do_handshake(ssl_st * s) libssl-1_1.dll!SSL_accept(ssl_st * s) Is there a solution for this? Or Do I need to switch to some other Signature algorithm like ECDSA? Can I use this or anything else instead of RSA? If I should switch to ECDSA, should I use "szOID_ECDSA_SHA256" (wincrypt.h) as signature algorithm, use "PROV_EC_ECDSA_SIG" while calling CryptAcquireContext() and call ENGINE_set_default() with ENGINE_METHOD_EC to support ECDSA using capi.dll? Thank you in advance. -- Regards, Sandeep
