Sorry I knew I'd forget something, I've put the my_rsa_key declaration and initialization in the right place marked /**** here ****/ ... N
--- Nou Dadoun ndad...@teradici.com 604-628-1215 -----Original Message----- From: owner-openssl-us...@openssl.org [mailto:owner-openssl-us...@openssl.org] On Behalf Of Nou Dadoun Sent: March 9, 2012 12:01 PM To: openssl-users@openssl.org Subject: Encrypting with openssl and decrypting with capi cryptdecrypt So here's what I'm trying to do in a unit test, wiring in the method replacement seems to work (i.e. my priv decrypt callback is called at the right place) but the encrypt --> decrypt doesn't work in a unit test so it certainly won't work there or anywhere else. So I load the certificate and get key handles using capi, brief snippets: result = CryptAcquireCertificatePrivateKey( m_pccert_context, CRYPT_ACQUIRE_COMPARE_KEY_FLAG, NULL, &m_crypto_provider, &key_spec, NULL); ... // Get Private Key result = CryptGetUserKey( m_crypto_provider, AT_KEYEXCHANGE, &m_cert_key_pair); ... // Get the public key information for the certificate. result = CryptImportPublicKeyInfo( m_crypto_provider, X509_ASN_ENCODING, &m_pccert_context->pCertInfo->SubjectPublicKeyInfo, &m_cert_public_key); That all works since I can then (pub) encrypt and (priv) decrypt a sample message in capi, I do some other sanity tests and get to my interop test, I load the x509 certificate from the m_pccert_context, i.e. BIO * input = BIO_new_mem_buf_fn ( (void*) m_pccert_context->pbCertEncoded, (UINT32) m_pccert_context->cbCertEncoded); m_x509_cert = d2i_X509_bio_fn(input, NULL); and then pull the public key to encrypt: /**** here ****/ RSA* my_rsa_key = RSA_generate_key(1024, RSA_F4, NULL, NULL; EVP_PKEY * my_pkey = X509_get_pubkey_fn(m_x509_cert); my_rsa_key->n = BN_dup_fn(my_pkey->pkey.rsa->n); my_rsa_key->e = BN_dup_fn(my_pkey->pkey.rsa->e); When I extract and print the modulus (i.e. the n) from both my_rsa_key and the m_cert_public_key (exported as a PUBLICKEYBLOB), they're the same (but reversed), so I proceed to encrypt my test message with openssl: enc_mess_len = RSA_public_encrypt( strlen(test_mess), (unsigned char *) test_mess, (unsigned char *) enc_mess, my_rsa_key, RSA_PKCS1_OAEP_PADDING); And decrypt with capi: if(!CryptDecrypt( m_cert_key_pair, //__in HCRYPTKEY hKey, NULL, //__in HCRYPTHASH hHash, TRUE, //__in BOOL Final, CRYPT_OAEP, //__in DWORD dwFlags, (BYTE*) enc_mess, //__inout BYTE *pbData, &decrypted_len //__inout DWORD *pdwDataLen, )) { // Figure out what went wrong. DWORD last_error = GetLastError(); return -1; } And it always fails with NTE_BAD_DATA. (I've eliminated some function wrappers but this is basically it.) I've tried different padding schemes, e.g. CRYPT_OAEP <---> RSA_PKCS1_OAEP_PADDING ?? the default <-----> RSA_PKCS1_PADDING I've tried CRYPT_DECRYPT_RSA_NO_PADDING_CHECK (and get an NTE_BAD_FLAGS error) thanks ms, I've tried reversing the encrypted buffer, all to no avail. Am I missing something here? Thanks in advance .... N --- Nou Dadoun ndad...@teradici.com 604-628-1215 ______________________________________________________________________ OpenSSL Project http://www.openssl.org User Support Mailing List openssl-users@openssl.org Automated List Manager majord...@openssl.org ______________________________________________________________________ OpenSSL Project http://www.openssl.org User Support Mailing List openssl-users@openssl.org Automated List Manager majord...@openssl.org
