So here's what I'm trying to do in a unit test, wiring in the method
replacement seems to work (i.e. my priv decrypt callback is called at the right
place) but the encrypt --> decrypt doesn't work in a unit test so it certainly
won't work there or anywhere else.
So I load the certificate and get key handles using capi, brief snippets:
result = CryptAcquireCertificatePrivateKey(
m_pccert_context,
CRYPT_ACQUIRE_COMPARE_KEY_FLAG,
NULL,
&m_crypto_provider,
&key_spec,
NULL);
...
// Get Private Key
result = CryptGetUserKey(
m_crypto_provider,
AT_KEYEXCHANGE,
&m_cert_key_pair);
...
// Get the public key information for the certificate.
result = CryptImportPublicKeyInfo(
m_crypto_provider,
X509_ASN_ENCODING,
&m_pccert_context->pCertInfo->SubjectPublicKeyInfo,
&m_cert_public_key);
That all works since I can then (pub) encrypt and (priv) decrypt a sample
message in capi, I do some other sanity tests and get to my interop test, I
load the x509 certificate from the m_pccert_context, i.e.
BIO * input = BIO_new_mem_buf_fn (
(void*) m_pccert_context->pbCertEncoded,
(UINT32) m_pccert_context->cbCertEncoded);
m_x509_cert = d2i_X509_bio_fn(input, NULL);
and then pull the public key to encrypt:
EVP_PKEY * my_pkey = X509_get_pubkey_fn(m_x509_cert);
my_rsa_key->n = BN_dup_fn(my_pkey->pkey.rsa->n);
my_rsa_key->e = BN_dup_fn(my_pkey->pkey.rsa->e);
When I extract and print the modulus (i.e. the n) from both my_rsa_key and the
m_cert_public_key (exported as a PUBLICKEYBLOB), they're the same (but
reversed), so I proceed to encrypt my test message with openssl:
enc_mess_len = RSA_public_encrypt(
strlen(test_mess),
(unsigned char *) test_mess,
(unsigned char *) enc_mess,
my_rsa_key,
RSA_PKCS1_OAEP_PADDING);
And decrypt with capi:
if(!CryptDecrypt(
m_cert_key_pair, //__in HCRYPTKEY hKey,
NULL, //__in HCRYPTHASH hHash,
TRUE, //__in BOOL Final,
CRYPT_OAEP, //__in DWORD dwFlags,
(BYTE*) enc_mess, //__inout BYTE *pbData,
&decrypted_len //__inout DWORD *pdwDataLen,
))
{
// Figure out what went wrong.
DWORD last_error = GetLastError();
return -1;
}
And it always fails with NTE_BAD_DATA. (I've eliminated some function wrappers
but this is basically it.) I've tried different padding schemes, e.g.
CRYPT_OAEP <---> RSA_PKCS1_OAEP_PADDING
?? the default <-----> RSA_PKCS1_PADDING
I've tried CRYPT_DECRYPT_RSA_NO_PADDING_CHECK (and get an NTE_BAD_FLAGS error)
thanks ms, I've tried reversing the encrypted buffer, all to no avail.
Am I missing something here? Thanks in advance .... N
---
Nou Dadoun
ndad...@teradici.com
604-628-1215
______________________________________________________________________
OpenSSL Project http://www.openssl.org
User Support Mailing List openssl-users@openssl.org
Automated List Manager majord...@openssl.org
