hi all, I've been working on several issues involving PKCS#11 engine
we ship with OpenSSL as part of Solaris and hit a problem of memory leakage
when the engine was used. The problem can't be attributed strictly to
OpenSSL or to the engine. The problem is how digests work:
OpenSSL: ctx init, dgst init/update/final, ctx cleanup
PKCS#11: dgst init/update/final
where some memory is allocated in dgst init and freed in ctx cleanup
(not dgst final) in OpenSSL, but allocated in dgst init and freed in dgst
final in PKCS#11 tokens. So, if you don't call xyzFinal() it's fine wrt
memory usage in OpenSSL but it's a problem in PKCS#11 app.
it's usually not a problem because when there is digest init there
is also digest final some time after that. However, not for HMAC
computation. Probably due to the way how TLS PRF function is processed, ipad
and opad helper digest contexts are not finalized in HMAC_Final(). This fact
is used in tls1_P_hash() where HMAC_Init_ex()'s are called in reinit mode,
reusing already used i/o_ctx's. This is not a problem for OpenSSL since all
ctx's are cleaned in HMAC_CTX_cleanup() in the very end but it's a problem
on PKCS#11 side because there is no EVP_MD_CTX_cleanup() equivalent. Yes,
cleanup()-like function is called on the engine side but the only way would
be to call PKCS#11's Final()-like function again. So, Final()-like function
would be called twice in most cases which is not expected; the session
doesn't exist after the first C_DigestFinal().
when I manually call EVP_DigestFinal_ex()'s for i/o_ctx's in the
tls1_P_hash() loop for the last chunk it gets rid of those memory leaks.
This is more a hack than anything else because it doesn't solve the problem
when HMAC functions are directly used by an application (that uses the
engine). Fixing HMAC_Final() with finalizing both pad contexts and then
rewriting the loop using EVP_MD_CTX_copy_ex()'s calls on temp ctx's doesn't
help either because due too excessive copy operations every SSL handshake
becomes several tens of percent slower. And the solution itself is not nice
either.
what would be the best solution on OpenSSL side? Another approach
could be that several EVP_DigestXXX()'s could be used instead of
HMAC_Final() (or, eh, grouped into HMAC_Final2) and then HMAC_Final() could
be changed but I would like to fix it the same way as in OpenSSL - if you
decide to fix it of course. Having separate patches is too painful.
thanks, Jan.
--
Jan Pechanec
______________________________________________________________________
OpenSSL Project http://www.openssl.org
Development Mailing List [email protected]
Automated List Manager [EMAIL PROTECTED]
