RE: ENGINE reference leak using FIPS-capable OpenSSL

[Erik Tkal]

Any takers?  Should I be able to build a FIPS-capable OpenSSL and have some of 
the implementation be provided via an ENGINE (e.g. let's say I have a hardware 
module to perform AES) but some by the OpenSSL FIPS canister?  Or is it truly 
all or nothing?

  Thanks.
....................................
Erik Tkal
Juniper OAC/UAC/Pulse Development


From: owner-openssl-...@openssl.org [mailto:owner-openssl-...@openssl.org] On 
Behalf Of Erik Tkal
Sent: Monday, April 16, 2012 10:02 AM
To: openssl-dev@openssl.org
Subject: ENGINE reference leak using FIPS-capable OpenSSL

I've been investigating a memory leak in using a FIPS-capable OpenSSL in 
non-FIPS mode.

For example, the following code does not seem to be correct in evp_enc.c:

int EVP_CipherInit_ex(EVP_CIPHER_CTX *ctx, const EVP_CIPHER *cipher, ENGINE 
*impl,
...
#ifndef OPENSSL_NO_ENGINE
              if(impl)
                     {
                     if (!ENGINE_init(impl))
                           {
                           EVPerr(EVP_F_EVP_CIPHERINIT_EX, 
EVP_R_INITIALIZATION_ERROR);
                           return 0;
                           }
                     }
              else
                     /* Ask if an ENGINE is reserved for this job */
                     impl = ENGINE_get_cipher_engine(cipher->nid);
              if(impl)
                     {
                     /* There's an ENGINE for this job ... (apparently) */
                     const EVP_CIPHER *c = ENGINE_get_cipher(impl, cipher->nid);
                     if(!c)
                           {
                           /* One positive side-effect of US's export
                           * control history, is that we should at least
                           * be able to avoid using US mispellings of
                           * "initialisation"? */
                           EVPerr(EVP_F_EVP_CIPHERINIT_EX, 
EVP_R_INITIALIZATION_ERROR);
                           return 0;
                           }
                     /* We'll use the ENGINE's private cipher definition */
                     cipher = c;
                     /* Store the ENGINE functional reference so we know
                     * 'cipher' came from an ENGINE and we need to release
                     * it when done. */
                     ctx->engine = impl;
                     }
              else
                     ctx->engine = NULL;
#endif

#ifdef OPENSSL_FIPS
              return FIPS_cipherinit(ctx, cipher, key, iv, enc);
#else

So the code goes through all the motions of honoring the engine that exists and 
incrementing reference counts, etc, but then blindly calls FIPS_cipherinit(), 
which ends up removing the engine pointer from the context.  I see in some 
cases this behaviour, yet in others the call is wrapped with a test for FIPS 
mode.  E.g.:

int EVP_DigestInit_ex(EVP_MD_CTX *ctx, const EVP_MD *type, ENGINE *impl)
...
#ifdef OPENSSL_FIPS
       if (FIPS_mode())
              {
              if (FIPS_digestinit(ctx, type))
                     return 1;
              OPENSSL_free(ctx->md_data);
              ctx->md_data = NULL;
              return 0;
              }
#endif
       return ctx->digest->init(ctx);
       }

Note that the Update call however has:

int EVP_DigestUpdate(EVP_MD_CTX *ctx, const void *data, size_t count)
       {
#ifdef OPENSSL_FIPS
       return FIPS_digestupdate(ctx, data, count);
#else
       return ctx->update(ctx,data,count);
#endif
       }

Should *all* calls be protected with a test for FIPS_mode()?  Or is it 
documented that a FIPS-capable OpenSSL is not compatible with the usage of 
engines, even in non-FIPS mode?


....................................
Erik Tkal
Juniper OAC/UAC/Pulse Development