Program crashes in EVP_CipherUpdate while doing T-DES Decryption

[Ambarish Mitra]

Hi all,

Platform: Redhat Linux Advanced Server 4.0
Compiler: g++ 3.3.2     (C++ compiler used)
Linkage: openssl libraries statically linked with the application program.
Version: OpenSSL 0.9.7a Feb 19 2003 (I tried with a recent version of
openssl, but same result).

I am trying out the EVP API functions to encrypt and decrypt a string using
Triple DES in CBC mode. The program is actually a shared object, which is
loaded by another EXE. The program gets a base64 string, which I first do a
base64 decode and then apply the T-des decryption EVP.

The encryption works fine, whereas the decryption program crashes at
EVP_CipherUpdate with the following stack trace:

#0  0x00000000 in ?? ()
#1  0xb4f9df57 in EVP_EncryptUpdate () from /home/ambarish/appssl.so
#2  0xb4f9e047 in EVP_EncryptUpdate () from /home/ambarish/appssl.so
#3  0xb4f9e18d in EVP_DecryptUpdate () from
/home/ambarish/PFG/impersonation/authzplugin/appssl.so
#4  0xb4f5d4ec in tdesDecrypt (in=0x8fac420
"æÛ¥YK©Ô¶#íoBUK¾ýÓEAÖ®\016©Ïßæ\223\217ð³á\026Å\223\226é<\036ñ=", len=40,
    output=0x8fac450 "ì\nf", outlen=0xb6650ecc) at tdes.cpp:141
#5  0xb4f57b0e in DecrFn (pContext=0x8f7b8c8, pFnBlock=0x8fac408,
pInfo=0x8fac3f0) at appssl.cpp:259

It is not multi-threaded at this point (I have to provide MT support later),
so I have not used the Crypto locking functions in the code.


The code snippet is given below. The encryption program is exactly the same
except:
EVP_CipherInit_ex(&ctx, EVP_des_ede3_cbc(), NULL, key, iv, ENCRYPT);  //
This works fine.


...

        unsigned char *binText;                 // The binary text after
base64 decode
        unsigned int binTextLen = 0;            // Spaces allocated for
cipher text
        unsigned int cipherTextLen = 0;         // Cipher Text Length after
base64 decoding
        unsigned char *plainText;               // The plantext after
decryption
        unsigned int plainTextLen = 0;          // The plaintext length

// Base64 Decode the cookie, and then  Decrpyt the Cookie

        binTextLen = (((strlen(input) + 3)/4) * 3);
        binText = new unsigned char [binTextLen];

        b64Decode(binText, dnipcookie, &cipherTextLen); // We base64 decode
input. o/p will be held in binText

        plainText = new unsigned char [cipherTextLen];
        tdesDecrypt(binText, cipherTextLen, plainText, &plainTextLen);
// Tdes Decrypt. The output will be held in plainText.

        plainText[plainTextLen] = '\0';

        delete[] plainText;
        delete[] binText;
...

void b64Decode(unsigned char *ret, const char *buf, unsigned int *len)
{
        int tlen;
        int buflen = strlen(buf);

        tlen = EVP_DecodeBlock(ret, (unsigned char *)buf, buflen);

        *len = tlen;
}


int tdesDecrypt(unsigned char *in, unsigned int len, unsigned char *output,
unsigned int *outlen)
{
        unsigned char initVector[8] =
{0xfe,0xdc,0xba,0x98,0x76,0x54,0x32,0x10};        //The initialization
vector will be derived from this
        unsigned char iv[EVP_MAX_IV_LENGTH], key[EVP_MAX_KEY_LENGTH];

        int outbuflen=0;
        int outbuflen2=0;

        EVP_CIPHER_CTX ctx;

        unsigned char salt[8] = {0x53,0x41,0x4C,0x54,0x73,0x61,0x6C,0x74};
        unsigned char keyInput[24] =
{0x3F,0x6F,0x6B,0x69,0x20,0x5E,0x5F,0x45,0x65,0x54,0x5D,0x56,0x63,0x68,0x6E,
0x6F,0x14,0x32,0x2C,0x41,0x3F,0xD3,0x9B,0xA3};

        memcpy(iv,initVector,sizeof(iv));

        int keysize = EVP_BytesToKey(EVP_des_ede3_cbc(), EVP_md5(), salt,
keyInput, sizeof(keyInput), 1, key, iv);

        EVP_CIPHER_CTX_init(&ctx);
        EVP_CipherInit_ex(&ctx, EVP_des_ede3_cbc(), NULL, key, iv, DECRYPT);

        EVP_CipherUpdate(&ctx, output, &outbuflen, in,len);                     
////////////
Crashes here. /////////////

        EVP_CipherFinal_ex(&ctx, &output[outbuflen], &outbuflen2);

        EVP_CIPHER_CTX_cleanup(&ctx);

        output[outbuflen + outbuflen2] = '\0';
        *outlen = outbuflen + outbuflen2;

        return 0;
}


DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the 
property of Persistent Systems Ltd. It is intended only for the use of the 
individual or entity to which it is addressed. If you are not the intended 
recipient, you are not authorized to read, retain, copy, print, distribute or 
use this message. If you have received this communication in error, please 
notify the sender and delete all copies of this message. Persistent Systems 
Ltd. does not accept any liability for virus infected mails.
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    openssl-users@openssl.org
Automated List Manager                           [EMAIL PROTECTED]