Doesn't work. Causes a segfault in EVP_DigestUpdate() because the ctx->update
pointer is NULL.
This is because without type->ctx_size being nonzero the update method is not
set in EVP_DigestInit_ex().
See this code in EVP_DigestInit_ex()
if (ctx->digest != type)
{
if (ctx->digest && ctx->digest->ctx_size)
OPENSSL_free(ctx->md_data);
ctx->digest=type;
if (!(ctx->flags & EVP_MD_CTX_FLAG_NO_INIT) && type->ctx_size)
<<<<<<<<<<<<<<<<<<<<<<<<< type->ctx_size is 0
{
ctx->update = type->update;
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<doesn't get set
ctx->md_data=OPENSSL_malloc(type->ctx_size);
if (ctx->md_data == NULL)
{
EVPerr(EVP_F_EVP_DIGESTINIT_EX,
ERR_R_MALLOC_FAILURE);
return 0;
}
}
}
--
Robert Dugal Senior Software Developer
Certicom Corp. A Subsidiary of Research In Motion
[email protected]
direct 905.501.3848
fax 905.507.4230
www.certicom.com
-----Original Message-----
From: [email protected] [mailto:[email protected]] On
Behalf Of Dr. Stephen Henson
Sent: Thursday, November 25, 2010 11:45 AM
To: [email protected]
Subject: Re: memory leak in HMAC_Final() when using my ENGINE
On Thu, Nov 25, 2010, Robert Dugal wrote:
> I am developing an ENGINE for OpenSSL 1.0.0a and 0.9.8o. Among other things
> this engine implements digest methods like SHA1.
>
> While testing this engine I discovered memory leaks when using
> PEM_write_bio_PKCS8PrivateKey().
> I traced the leak back to the inner and outer SHA1 digest contexts used for
> HMAC in PKCS5_PBKDF2_HMAC().
> The leak is related to the md_data in the EVP_MD_CTX. My engine saves it's
> own data in ctx->md_data which includes allocation of additional pointers.
> That data gets initialized by my ENGINE implementation of the init() callback
> and destroyed by the cleanup() callback.
> One thing I have noticed about the ENGINE callback design is that the init()
> callback cannot check the md_data to see if it was previously already
> initialized.
> OpenSSL doesn't initialize the memory to 0 when md_data is allocated. So my
> init() callback has no choice but to assume that the md_data is uninitialized.
> So if the init() callback is called more than once without calling the
> cleanup() callback my ENGINE implementation will introduce a memory leak.
> This is where the source of the problem lies.
>
> In the code below the "while(tkeylen)" loop may result in multiple passes of
> the HMAC code. On each pass it does HMAC_Init_ex(),HMAC_Update(),HMAC_Final().
> HMAC_CTX_cleanup() is only called once after exiting the while loop.
>
> Each time HMAC_Init_ex() is called my ENGINE SHA1 init() callback is called
> and it initializes the md_data for the inner and outer digest contexts.
> But HMAC_Final() never calls EVP_DigestFinal_ex() or EVP_MD_CTX_cleanup() on
> these inner and outer digest contexts.
> Because these inner/outer digest context are not finalized or cleaned up,
> each pass through the loop results in them being re-initialized by my init()
> callback and memory leaks introduced.
>
> If I modify HMAC_Final() to call EVP_MD_CTX_cleanup(&ctx->i_ctx) and
> EVP_MD_CTX_cleanup(&ctx->o_ctx), then my memory leaks go away.
> So either there is a bug in OpenSSL or I am misunderstanding the ENGINE
> digest interfaces.
>
The auto allocation of md_data is a convenience measure which reduces code
duplication in some cases (for example software based implementations). The
number of cleanup calls is kept to a minimum to reduce memory fragmentation by
allowing contexts to be reused.
In some cases (such as yours) the default behaviour isn't appropriate. The
amount of memory auto allocated is determined by the ctx_size field of the
EVP_MD structure. Have you tried setting this to zero? That should shut off
the auto allocation and freeing and leave the implementation to handle it in
its init() and cleanup() callbacks.
Steve.
--
Dr Stephen N. Henson. OpenSSL project core developer.
Commercial tech support now available see: http://www.openssl.org
______________________________________________________________________
OpenSSL Project http://www.openssl.org
Development Mailing List [email protected]
Automated List Manager [email protected]
---------------------------------------------------------------------
This transmission (including any attachments) may contain confidential
information, privileged material (including material protected by the
solicitor-client or other applicable privileges), or constitute non-public
information. Any use of this information by anyone other than the intended
recipient is prohibited. If you have received this transmission in error,
please immediately reply to the sender and delete this information from your
system. Use, dissemination, distribution, or reproduction of this transmission
by unintended recipients is not authorized and may be unlawful.
______________________________________________________________________
OpenSSL Project http://www.openssl.org
Development Mailing List [email protected]
Automated List Manager [email protected]
