openssl generated RSA keys in Crypto++

[John Windle]

I'm trying to use RSA keys generated by openssl in Crypto++ encryption 
and decryption routines. I realise that openssl outputs the keys in PEM 
format and Crypto++ uses DER format.

I thought PEM format "consists of the DER format base64 encoded with 
additional header and footer lines.", Openssl manual 
(http://www.openssl.org/docs/apps/rsa.html), so I thought that if I just 
got rid of the header and footer lines I'd have a DER Base64 encoded 
file that I could feed into the Crypto++ routines as the keys.

When I try this and try encoding data, using the public key, it appears 
to produce an encoded message, ie it all completes successfully. But 
when I try and decode the message, private key, I get a "BER decode 
error" exception.

The routines I have work fine so long as the keys are generated with the 
crypto++ functions.

If anyone out there can offer advice or point me in the right direction 
it would be much appreciated.

The openssl command lines I'm using are:
        openssl genrsa -out privkey.asc 2048
        openssl rsa -in privkey.asc -pubout -out pubkey.asc
I seperated my code into 3 programs to help with debugging. The crypto 
programs that comprise my tests are (stolen right out of the test 
program :-):

Key generation:
---------------
void GenerateRSAKey(unsigned int keyLength,
                    const char *privFilename,
                    const char *pubFilename,
                    const char *seed)
{
        RandomPool randPool;
        randPool.Put((byte *)seed, strlen(seed));
        RSAES_OAEP_SHA_Decryptor priv(randPool, keyLength);
        Base64Encoder privFile(new FileSink(privFilename));
        priv.DEREncode(privFile);
        privFile.MessageEnd();
        RSAES_OAEP_SHA_Encryptor pub(priv);
        Base64Encoder pubFile(new FileSink(pubFilename));
        pub.DEREncode(pubFile);
        pubFile.MessageEnd();
}
int main(int argc, char** argv)
{
    GenerateRSAKey( 2048, "privkey.asc", "pubkey.asc", "A random string 
or 345%");
}

Encrypt the message:
--------------------
string RSAEncryptString(const char *pubFilename,
                        const char *seed,
                        const char *message)
{
        FileSource pubFile(pubFilename, true, new Base64Decoder);
        RSAES_OAEP_SHA_Encryptor pub(pubFile);
        RandomPool randPool;
        randPool.Put((byte *)seed, strlen(seed));
	string result;
	StringSource(message, true, new PK_EncryptorFilter(randPool, pub, new 
HexEncoder(new StringSink(result))));
	Base64Encoder(new StringSink(result))));
	return result;
}

int main(int argc, char** argv)
{
    string encStr;
    FILE *op;
    try {
        encStr =  RSAEncryptString("pubkey.asc", "dribbleSeed23", "A 
few good men, 175 to be exact\n");
        op = fopen("rsaMsg.txt", "w");
        fprintf(op, "%s\n", encStr.c_str());
        fclose(op);
    }
    catch (std::exception const& e)
    {
        // This clause will also catch CryptoPP::Exception as well as
        // all derived exception classes
        cout << "std::exception caught:" << endl
              << e.what() << endl;

    }
}
Decrypt the message:
--------------------
string RSADecryptString(const char *privFilename,
                        const char *ciphertext)
{
        FileSource privFile(privFilename, true, new Base64Decoder);
        RSAES_OAEP_SHA_Decryptor priv(privFile);
        RandomPool randomPool;
	string result;
	StringSource(ciphertext,
	             true,
		     StringSink(result))));
	             new HexDecoder(new PK_DecryptorFilter(randomPool, priv, 
new StringSink(result))));
	return result;
}

int main(int argc, char** argv)
{
    string decrStr;
    FILE *ip;
    char ipStr[3000];
    ip = fopen("rsaMsg.txt", "r");
    fgets(ipStr, 3000, ip);
    // get rid of the \n at the end of the line
    ipStr[strlen(ipStr) - 1] = '\0';
    try {
        decrStr = RSADecryptString("privkey.asc", ipStr);
        printf("%s\n", decrStr.c_str());
        fclose(ip);
    }
    catch (std::exception const& e)
    {
        // This clause will also catch CryptoPP::Exception as well as
        // all derived exception classes, except for SpecificException
        // which has already been caught above.
        cout << "std::exception caught:" << endl
              << e.what() << endl;
    }
}