RE: ENGINE reference leak using EVP_Digest*

[Erik Tkal]

In addition, I find that EVP_DigestInit_ex ends up bumping the reference count 
*twice*...

In eng_table.c:

       /* Try to initialise the ENGINE? */
       if((ret->funct_ref > 0) || !(table_flags & ENGINE_TABLE_FLAG_NOINIT))
              initres = engine_unlocked_init(ret);
       else
              initres = 0;
       if(initres)
              {
              /* Update 'funct' */
              if((fnd->funct != ret) && engine_unlocked_init(ret))
                     {
                     /* If there was a previous default we release it. */
                     if(fnd->funct)
                           engine_unlocked_finish(fnd->funct, 0);

To clarify the version, it's 1.0.1 built as FIPS-capable, but OpenSSL FIPS is 
not enabled during this usage.

....................................
Erik Tkal
Juniper OAC/UAC/Pulse Development


Sent: Tuesday, March 27, 2012 10:45 AM
To: openssl-dev@openssl.org
Subject: RE: ENGINE reference leak using EVP_Digest*

BTW, this is with OpenSSL 1.0.1; I did not try an older version.


Sent: Tuesday, March 27, 2012 10:20 AM
To: openssl-dev@openssl.org<mailto:openssl-dev@openssl.org>
Subject: ENGINE reference leak using EVP_Digest*


I have some code that uses the following sequence of calls (e.g.):



EVP_MD_CTX ctx;

EVP_MD_CTX_init(&ctx);



EVP_DigestInit_ex(&ctx, EVP_sha1(), NULL);

EVP_DigestUpdate(&ctx, pData, nSize);

EVP_DigestFinal_ex(&ctx, pOut, NULL);



EVP_DigestInit_ex(&ctx, EVP_sha1(), NULL);

EVP_DigestUpdate(&ctx, pData, nSize);

EVP_DigestFinal_ex(&ctx, pOut, NULL);



EVP_DigestInit_ex(&ctx, EVP_sha1(), NULL);

EVP_DigestUpdate(&ctx, pData, nSize);

EVP_DigestFinal_ex(&ctx, pOut, NULL);



EVP_MD_CTX_cleanup(&ctx);



The docs say that "After calling EVP_DigestFinal_ex() no additional calls to 
EVP_DigestUpdate() can be made, but EVP_DigestInit_ex() can be called to 
initialize a new digest operation."



In my case, there is an ENGINE registered to perform the SHA1 operations.



What happens, though is that EVP_DigestInit_ex() increments the ENGINE 
reference, but EVP_DigestFinal_ex() does not decrement it.  Only the 
EVP_MD_CTX_cleanup() call is decrementing the reference count, and thus my 
ENGINE never ends up being actually freed on the ENGINE_finish() call.



Is this a known issue?  Do I really need to cleanup/reinit my CTX around each 
hash sequence?



....................................

Erik Tkal

Juniper OAC/UAC/Pulse Development