Turns out this was a problem with the data from the windows app, and not with my code (the CAPI developer fixed it). I was assuming that the CAPI code was correct. Doh!
In case anyone else ever has to do something similar, keep in mind that the CAPI values are little-endian and have to be byte reversed before calling BN_bin2bn(). Scott G. Kelly wrote: > I'm trying unsuccessfully to complete a DH exchange with a CAPI > application. The CAPI app is doing a hex dump of its DH values (p, g, > pub_key and priv_key), so I've imported them into some test code to try > to isolate the problem. The p/g values are the oakley 1024 group. > > I've defined uint8_t arrays containing the dumped CAPI KEYBLOB values. I > have CAPI-compatibility structures defined for the KEYBLOB's, and am > typecasting them to the uint8_t arrays: > > ClientPrivKeyBlobPtr = (MicrosoftDHPrivKeyBlobType *) ArrayPtr; > > I'm loading the p, g, and priv_key values into a DH structure using > calls like this: > > DH->p = BN_bin2bn( ClientPrivKeyBlobPtr->prime, 128, NULL ); > DH->g = BN_bin2bn( ClientPrivKeyBlbbPtr->generator, 128, NULL ); > DH->priv_key = BN_bin2bn( ClientPrivKeyBlobPtr->secret, 128, NULL ); > > I've also eye-balled hex dumps of the values at the prime, geneator, and > secret pointers to make sure the CAPI-compatible structures are properly > defined, aligned, etc. > > When I call DH_generate_key( DH ), the pub_key value does not match the > expected value, so I assume I'm doing something wrong here. > > To verify that the basic approach is sound, I loaded just the prime and > generator, called DH_generate_key( DH ), dumped out the resulting > pub_key/priv_key, and then attempted the same test (import as arrays, > load with BN_bin2bn(), etc) with these values, and it worked fine. > > I read on MSDN that the generator, secret, and pubkey are little-endian, > so I've tried reversing these before loading them with BN_bin2bn(), but > alas, that fails as well. > > Am I missing something obvious here? > > Thanks, > > Scott > ______________________________________________________________________ > OpenSSL Project http://www.openssl.org > Development Mailing List openssl-dev@openssl.org > Automated List Manager majord...@openssl.org ______________________________________________________________________ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager majord...@openssl.org
