[openssl.org #2418] BUG: 0.9.8m Can't decrypt a PKCS7 using ECB Cipher generated from 0.9.7g.

[Randall Geyer via RT]

The issue is encountered in the following function in 
src\openssl\openssl-0.9.8m\crypto\evp\evp_lib.c:
 
int EVP_CIPHER_get_asn1_iv(EVP_CIPHER_CTX *c, ASN1_TYPE *type)
       {
       int i=0,l;
 
       if (type != NULL) 
              {
              l=EVP_CIPHER_CTX_iv_length(c);
              OPENSSL_assert(l <= sizeof c->iv);
              i=ASN1_TYPE_get_octetstring(type,c->oiv,l);
              if (i != l)
                     return(-1);
              else if (i > 0)
                     memcpy(c->iv,c->oiv,l);
              }
       return(i);
       }

When 9.7g decrypts a 9.7g encryption: In EVP_CIPHER_get_ans1_iv(), l and i are 
both 8, all is well, returns 8.
 
When 9.7g decrypts a 9.8m encryption: In EVP_CIPHER_get_ans1_iv(), it is 
basically a no-op because the parameter ANS1_TYPE *type is NULL, returns zero.
 
When 9.8m decrypts a 9.8m encryption: In EVP_CIPHER_get_ans1_iv(), it is 
basically a no-op because the parameter ANS1_TYPE *type is NULL, returns zero.
 
When 9.8m decrypts a 9.7g encryption: In EVP_CIPHER_get_ans1_iv(), l is 0 and i 
is 8, returns -1 and we are in failure mode.
 
I modified the code to avoid comparing IV length when the cipher is ECB - since 
there is supposed to be no IV for an ECB.
 
int EVP_CIPHER_get_asn1_iv(EVP_CIPHER_CTX *c, ASN1_TYPE *type)
       {
       int i=0,l;
 
       if (type != NULL && EVP_CIPHER_CTX_mode(c) != EVP_CIPHER_ECB_MODE) 
              {
              l=EVP_CIPHER_CTX_iv_length(c);
              OPENSSL_assert(l <= sizeof c->iv);
              i=ASN1_TYPE_get_octetstring(type,c->oiv,l);
              if (i != l)
                     return(-1);
              else if (i > 0)
                     memcpy(c->iv,c->oiv,l);
              }
       return(i);
       }
 
The issue stems from what appears to have been a bug in 
openssl-0.9.7g\crypto\evp\evp_locl.h, because it has an iv_len parameter, which 
was ultimately removed in 0.9.8m, and iv_len is now hardcoded to = 0.
 
in ...\openssl-0.9.7g\crypto\evp\evp_locl.h BLOCK_CIPHER_def_ecb is defined as:
#define BLOCK_CIPHER_def_ecb(cname, kstruct, nid, block_size, key_len, \
        iv_len, flags, init_key, cleanup, set_asn1, \
        get_asn1, ctrl) \
BLOCK_CIPHER_def1(cname, ecb, ecb, ECB, kstruct, nid, block_size, key_len, \
    iv_len, flags, init_key, cleanup, set_asn1, get_asn1, ctrl)
 
in ...\openssl-0.9.8m\crypto\evp\evp_locl.h BLOCK_CIPHER_def_ecb no longer has 
iv_len variable as a parameter, but hardcodes 0 as the parameter:
#define BLOCK_CIPHER_def_ecb(cname, kstruct, nid, block_size, key_len, \
                             flags, init_key, cleanup, set_asn1, \
                             get_asn1, ctrl) \
BLOCK_CIPHER_def1(cname, ecb, ecb, ECB, kstruct, nid, block_size, key_len, \
                  0, flags, init_key, cleanup, set_asn1, get_asn1, ctrl)
 
I was only using these 2 releases, so I am not sure how many other releases are 
affected. 

 

The issue is encountered in the following function in src\openssl\openssl-0.9.8m\crypto\evp\evp_lib.c:

 

int EVP_CIPHER_get_asn1_iv(EVP_CIPHER_CTX *c, ASN1_TYPE *type)
       {
       int i=0,l;

 

       if (type != NULL)
              {
              l=EVP_CIPHER_CTX_iv_length(c);
              OPENSSL_assert(l <= sizeof c->iv);
              i=ASN1_TYPE_get_octetstring(type,c->oiv,l);
              if (i != l)
                     return(-1);
              else if (i > 0)
                     memcpy(c->iv,c->oiv,l);
              }
       return(i);
       }

When 9.7g decrypts a 9.7g encryption: In EVP_CIPHER_get_ans1_iv(), l and i are both 8, all is well, returns 8.

 

When 9.7g decrypts a 9.8m encryption: In EVP_CIPHER_get_ans1_iv(), it is basically a no-op because the parameter ANS1_TYPE *type is NULL, returns zero.

 

When 9.8m decrypts a 9.8m encryption: In EVP_CIPHER_get_ans1_iv(), it is basically a no-op because the parameter ANS1_TYPE *type is NULL, returns zero.

 

When 9.8m decrypts a 9.7g encryption: In EVP_CIPHER_get_ans1_iv(), l is 0 and i is 8, returns -1 and we are in failure mode.

 

I modified the code to avoid comparing IV length when the cipher is ECB - since there is supposed to be no IV for an ECB.

 

int EVP_CIPHER_get_asn1_iv(EVP_CIPHER_CTX *c, ASN1_TYPE *type)
       {
       int i=0,l;

 

       if (type != NULL && EVP_CIPHER_CTX_mode(c) != EVP_CIPHER_ECB_MODE)
              {
              l=EVP_CIPHER_CTX_iv_length(c);
              OPENSSL_assert(l <= sizeof c->iv);
              i=ASN1_TYPE_get_octetstring(type,c->oiv,l);
              if (i != l)
                     return(-1);
              else if (i > 0)
                     memcpy(c->iv,c->oiv,l);
              }
       return(i);
       }

 

The issue stems from what appears to have been a bug in openssl-0.9.7g\crypto\evp\evp_locl.h, because it has an iv_len parameter, which was ultimately removed in 0.9.8m, and iv_len is now hardcoded to = 0.

 

in ...\openssl-0.9.7g\crypto\evp\evp_locl.h BLOCK_CIPHER_def_ecb is defined as:
#define BLOCK_CIPHER_def_ecb(cname, kstruct, nid, block_size, key_len, \
        iv_len, flags, init_key, cleanup, set_asn1, \
        get_asn1, ctrl) \
BLOCK_CIPHER_def1(cname, ecb, ecb, ECB, kstruct, nid, block_size, key_len, \
    iv_len, flags, init_key, cleanup, set_asn1, get_asn1, ctrl)

 

in ...\openssl-0.9.8m\crypto\evp\evp_locl.h BLOCK_CIPHER_def_ecb no longer has iv_len variable as a parameter, but hardcodes 0 as the parameter:
#define BLOCK_CIPHER_def_ecb(cname, kstruct, nid, block_size, key_len, \
                             flags, init_key, cleanup, set_asn1, \
                             get_asn1, ctrl) \
BLOCK_CIPHER_def1(cname, ecb, ecb, ECB, kstruct, nid, block_size, key_len, \
                  0, flags, init_key, cleanup, set_asn1, get_asn1, ctrl)

 

I was only using these 2 releases, so I am not sure how many other releases are affected.