Hi, I've looked at the archives and didn't see any apropos discussions so I
thought I'd go straight to the list:
We use the openssl & crypto libraries in several places in our product both in
fips and non-fips modes.
We have a set of unit tests to exercise various portions of our code to ensure
that our implementations using these tools work the way that they're supposed
to.
I've run into an unusual problem, we have a set of crypto (encrypt/decrypt)
tests and a separate set of certificate tests (e.g. retrieve a certificate and
its CA and do a certificate verification).
If I run the certificate tests first and then the crypto tests, all the tests
pass and everything works great.
If I run the crypto tests first and then the certificate test, the verification
fails due to a "signature" failure. This implies to me that the crypto tests
are leaving something in the openssl/crypto machinery in a funky state which
breaks the subsequent certificate signature computation.
A couple of questions:
What can I do to completely clean the openssl/crypto state to ensure that this
doesn't happen? I've added:
CRYPTO_cleanup_all_ex_data();
ENGINE_cleanup();
But this appears to be inadequate (I suspect necessary but insufficient), any
suggestions?
(I've reordered the tests so that they're passing now but I'd like to avoid
this hack if I can.)
Second question, I added a verification callback routine, e.g.
static int verify_callback(int ok, X509_STORE_CTX *stor)
{
if(!ok)
{
printf("verify_callback Certificate Verification Error: %s\n",
X509_verify_cert_error_string(stor->error));
}
else
{
printf("verify_callback Certificate Verification Success\n");
}
return ok;
}
I've put a breakpoint in it and noticed that when verifying a certificate, this
callback is called twice,
In the successful order above (cert then crypto tests) both calls have ok == 1,
In the unsuccessful order (crypto then cert), the first call is 1 and the
second is 0 with a "signature error" reported.
Why is it called twice and what's the difference? (I suspect the second is
signature checking and the first is everything else but I'm curious).
Thanks in advance ... N
---
Nou Dadoun
ndad...@teradici.com
604-628-1215
