I am developing an ENGINE for OpenSSL 1.0.0a and 0.9.8o. Among other things
this engine implements digest methods like SHA1.
While testing this engine I discovered memory leaks when using
PEM_write_bio_PKCS8PrivateKey().
I traced the leak back to the inner and outer SHA1 digest contexts used for
HMAC in PKCS5_PBKDF2_HMAC().
The leak is related to the md_data in the EVP_MD_CTX. My engine saves it's own
data in ctx->md_data which includes allocation of additional pointers.
That data gets initialized by my ENGINE implementation of the init() callback
and destroyed by the cleanup() callback.
One thing I have noticed about the ENGINE callback design is that the init()
callback cannot check the md_data to see if it was previously already
initialized.
OpenSSL doesn't initialize the memory to 0 when md_data is allocated. So my
init() callback has no choice but to assume that the md_data is uninitialized.
So if the init() callback is called more than once without calling the
cleanup() callback my ENGINE implementation will introduce a memory leak.
This is where the source of the problem lies.
In the code below the "while(tkeylen)" loop may result in multiple passes of
the HMAC code. On each pass it does HMAC_Init_ex(),HMAC_Update(),HMAC_Final().
HMAC_CTX_cleanup() is only called once after exiting the while loop.
Each time HMAC_Init_ex() is called my ENGINE SHA1 init() callback is called and
it initializes the md_data for the inner and outer digest contexts.
But HMAC_Final() never calls EVP_DigestFinal_ex() or EVP_MD_CTX_cleanup() on
these inner and outer digest contexts.
Because these inner/outer digest context are not finalized or cleaned up, each
pass through the loop results in them being re-initialized by my init()
callback and memory leaks introduced.
If I modify HMAC_Final() to call EVP_MD_CTX_cleanup(&ctx->i_ctx) and
EVP_MD_CTX_cleanup(&ctx->o_ctx), then my memory leaks go away.
So either there is a bug in OpenSSL or I am misunderstanding the ENGINE digest
interfaces.
int PKCS5_PBKDF2_HMAC(const char *pass, int passlen,
const unsigned char *salt, int saltlen, int iter,
const EVP_MD *digest,
int keylen, unsigned char *out)
{
unsigned char digtmp[EVP_MAX_MD_SIZE], *p, itmp[4];
int cplen, j, k, tkeylen, mdlen;
unsigned long i = 1;
HMAC_CTX hctx;
mdlen = EVP_MD_size(digest);
if (mdlen < 0)
return 0;
HMAC_CTX_init(&hctx);
p = out;
tkeylen = keylen;
if(!pass)
passlen = 0;
else if(passlen == -1)
passlen = strlen(pass);
while(tkeylen)
{
if(tkeylen > mdlen)
cplen = mdlen;
else
cplen = tkeylen;
/* We are unlikely to ever use more than 256 blocks (5120 bits!)
* but just in case...
*/
itmp[0] = (unsigned char)((i >> 24) & 0xff);
itmp[1] = (unsigned char)((i >> 16) & 0xff);
itmp[2] = (unsigned char)((i >> 8) & 0xff);
itmp[3] = (unsigned char)(i & 0xff);
HMAC_Init_ex(&hctx, pass, passlen, digest, NULL);
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< source of memory leaks is from 2nd and later
passes here
HMAC_Update(&hctx, salt, saltlen);
HMAC_Update(&hctx, itmp, 4);
HMAC_Final(&hctx, digtmp, NULL);
memcpy(p, digtmp, cplen);
for(j = 1; j < iter; j++)
{
HMAC(digest, pass, passlen,
digtmp, mdlen, digtmp, NULL);
for(k = 0; k < cplen; k++)
p[k] ^= digtmp[k];
}
tkeylen-= cplen;
i++;
p+= cplen;
}
HMAC_CTX_cleanup(&hctx);
#ifdef DEBUG_PKCS5V2
fprintf(stderr, "Password:\n");
h__dump (pass, passlen);
fprintf(stderr, "Salt:\n");
h__dump (salt, saltlen);
fprintf(stderr, "Iteration count %d\n", iter);
fprintf(stderr, "Key:\n");
h__dump (out, keylen);
#endif
return 1;
}
int HMAC_Final(HMAC_CTX *ctx, unsigned char *md, unsigned int *len)
{
int j;
unsigned int i;
unsigned char buf[EVP_MAX_MD_SIZE];
j=EVP_MD_block_size(ctx->md);
if (!EVP_DigestFinal_ex(&ctx->md_ctx,buf,&i))
goto err;
if (!EVP_MD_CTX_copy_ex(&ctx->md_ctx,&ctx->o_ctx))
goto err;
if (!EVP_DigestUpdate(&ctx->md_ctx,buf,i))
goto err;
if (!EVP_DigestFinal_ex(&ctx->md_ctx,md,len))
goto err;
>>>>>>>>>>>>> THE INNER (ctx->i_ctx ) AND OUTER (ctx->o_ctx ) DIGEST CONTEXTS
>>>>>>>>>>>>> ARE NOT CLEANED UP OR FINALIZED HERE.
return 1;
err:
return 0;
}
--
Robert Dugal Senior Software Developer
Certicom Corp. A Subsidiary of Research In Motion
[email protected]
direct 905.501.3848
fax 905.507.4230
www.certicom.com
---------------------------------------------------------------------
This transmission (including any attachments) may contain confidential
information, privileged material (including material protected by the
solicitor-client or other applicable privileges), or constitute non-public
information. Any use of this information by anyone other than the intended
recipient is prohibited. If you have received this transmission in error,
please immediately reply to the sender and delete this information from your
system. Use, dissemination, distribution, or reproduction of this transmission
by unintended recipients is not authorized and may be unlawful.
______________________________________________________________________
OpenSSL Project http://www.openssl.org
Development Mailing List [email protected]
Automated List Manager [email protected]
