> From: openssl-users [mailto:openssl-users-boun...@openssl.org] On Behalf Of
> sandeep m.v
> Sent: Monday, July 15, 2019 11:56
> I'm seeing this issue - capi_rsa_priv_enc() - capi engine: function not
> supported,
> when I tried to upgrade my application from using OpenSSL version 1.0.2r to
> 1.1.1b.
> This is causing TLS handshake to fail.
> In my case, I'm creating a self signed certificate that is used by Server
> application
> loading CAPI engine.
> Certificate is created by calling CertCreateSelfSignCertificate(--) -
> wincrypt.h -
> using szOID_RSA_SHA256RSA signature algorithm and "Microsoft Enhanced RSA and
> AES Cryptographic Provider".
It's been some years since I worked with OpenSSL CAPI support, and in
particular I haven't done anything with the CAPI engine for 1.1.
For 1.0.2, though, I ended up forking the OpenSSL CAPI engine support and
enhancing it in a number of places. I changed capi_load_privkey and the
functions it calls (capi_find_key and capi_get_pkey) to silently determine if
the provider type in the context was wrong, and if so correct it.
CAPI is a fairly horrible API (CNG is somewhat better), and in particular is
very fragile when there's a mismatch between the provider type in the CAPI
context and the provider type specified for the key. It may be that your CAPI
context specifies a provider other than the Enhanced RSA and AES one.
The CAPI engine for 1.0.2 (at least the version I forked from) also had a
shortcoming which Steve Henson had suggested a fix for, but which wasn't in the
code. That's down in capi_get_key where it calls CryptGetUserKey. If
CryptGetUserKey fails with NTE_NO_KEY (you have to call GetLastError to check),
then try again with the other key type, by XORing keyspec with 3:
if (!CryptGetUserKey(key->hprov, keyspec, &key->key))
{
success = FALSE;
errcode = GetLastError();
if (errcode == NTE_NO_KEY)
{
keyspec ^= 3;
success = CryptGetUserKey(key->hprov, keyspec, &key->key);
}
...
}
This handles the case where you have an RSA key marked as a key-exchange key
and you need a signing key, or vice versa.
All of that may be fixed already in the 1.1 CAPI engine, or be irrelevant to
your problem, of course. But those are potential issues when using RSA keys
with CAPI.
--
Michael Wojcik
Distinguished Engineer, Micro Focus
