X509_get_pubkey() in OpenSSL 3.0?

[Jason Schultz]

I thought I should start a new thread since this question was buried in my 
"FIPS" thread and I dont' think it has anything to do with FIPS and OpenSSL 
providers. I'm hitting another problem that I think is related to the migration 
to OpenSSL 3.0, as this code works with OpenSSL 1.1.1 (and 1.0.2 before it). 
When looking at the documentation pages for 1.1.1 vs 3.0, I'm not seeing any 
differences between the OpenSSL APIs I'm calling in the 2 different release 
levels.

Here is the sequence, I'm basically setting up my certificate and private key, 
both in PEM format, for the server, then I need to extract some information 
from them:

    ctx = SSL_CTX_new_ex(non_fips_libctx, NULL, TLS_method());
    SSL_CTX_use_PrivateKey_file(ctx,<keyfile>,SSL_FILETYPE_PEM);
    SSL_CTX_use_certificate_file(ctx,<certfile>,SSL_FILETYPE_PEM);
    SSL_CTX_check_private_key(ctx);
    fp = fopen(<certfile>, "r");
    mycert = PEM_read_X509(fp, NULL, 0, NULL);
    pkey = X509_get_pubkey(mycert);

All functions return good statuses or non-NULL pointers until the last one, 
X509_get_pubkey() returns NULL. I have tried this with RSA and Elliptic Curve 
cert/key pairs. I have tried with pairs created with OpenSSL 1.1.1 as well as 
3.0 via the command line.

This seems like a pretty straightforward operation, but it appears that 
key->pkey is NULL in the code below:

EVP_PKEY *X509_PUBKEY_get0(const X509_PUBKEY *key)
{
    if (key == NULL) {
        ERR_raise(ERR_LIB_X509, ERR_R_PASSED_NULL_PARAMETER);
        return NULL;
    }

    if (key->pkey == NULL) {
        /* We failed to decode the key when we loaded it, or it was never set */
        ERR_raise(ERR_LIB_EVP, EVP_R_DECODE_ERROR);
        return NULL;
    }

    return key->pkey;
}

I got to this code from the error information I dumped from OpenSSL:

00079FF8647F0000:error:03000072:digital envelope routines:(unknown 
function):decode error:crypto/x509/x_pubkey.c:444:

I can paste certificates if anyone thinks it will help, but I've tried several 
types of cert/key pairs, and using the command line to do "openssl x509 
-pubkey..." displays the public key just fine with the certificate file in 
question.

Here is an example of the openssl command line to create one of the cert/key 
pairs that hits this error:

openssl req -new -x509 -nodes -newkey ec:<(openssl ecparam -name secp384r1) 
-keyout threecert.key -out threecert.crt -days 365

Aside: While I was waiting for an answer on this, I started working on removing 
some of the deprecated functions in my code, for example, PEM_read_DHparams(). 
I ended up finding the DECODER functions and plan on doing something like:


Thanks,

Jason