FIPS version of RSA_generate_key_ex puts FIPS library in a bad state

[Perrow, Graeme]

If I attempt to create an RSA key pair with a size of >4096 bits using the FIPS 
library (FIPS 2.0.5, OpenSSL 1.0.1h), I get an error ("data too large for 
modulus"), but doing so seems to put the FIPS library into a bad state. 
Subsequent calls return failure and the error stack indicates that the FIPS 
self-test has failed.

Source for a short C program is attached. If I attempt to use a small key size 
(say 500 bits), generating the key fails but subsequent actions are OK. A key 
size between 1024 and 4096 works as expected. If I try to use a key size of 
4097, generating the key fails but then subsequent FIPS calls also fail.

I am seeing this on both Windows 7 and Linux.

Graeme Perrow


#include <stdio.h>

#include <openssl/crypto.h>
#include <openssl/err.h>
#include <openssl/rsa.h>
#include <openssl/evp.h>

static void print_err( const char *ctx )
/**************************************/
{
    int err_code        = ERR_get_error();
    int lib             = ERR_GET_LIB( err_code );
    int func            = ERR_GET_FUNC( err_code );
    int reason          = ERR_GET_REASON( err_code );

    fprintf( stderr, "Error from %s:\n\tcode 
%d\n\tlib=%d/%s\n\tfunc=%d/%s\n\treason=%d/%s\n",
             ctx, err_code, lib, ERR_lib_error_string(err_code),
             func, ERR_func_error_string(err_code),
             reason, ERR_reason_error_string(err_code) );
}

int main( int, char ** )
/**********************/
{
    ERR_load_crypto_strings();
    if( !FIPS_mode_set(1) ) {
        print_err( "FIPS_mode_set" );
        return 1;
    }
    printf( "FIPS initialization succeeded\n" );

    // Create 4097-bit key
    BIGNUM *exponent = BN_new();
    RSA *rsa_key = RSA_new();
    BN_set_word( exponent, RSA_F4 );
    int rc = RSA_generate_key_ex( rsa_key, 4097, exponent, NULL );
    if( rc == 0 ) {
        print_err( "RSA_generate_key_ex" );
    } else {
        printf( "RSA key generation succeeded\n" );
    }

    // Now try something else - some AES encryption
    EVP_CIPHER_CTX e_ctx;

    EVP_CIPHER_CTX_init( &e_ctx );
    if( EVP_EncryptInit_ex( &e_ctx, EVP_aes_128_cbc(), NULL, NULL, NULL ) == 0 
) {
        print_err( "EVP_EncryptInit_ex" );
        return 1;
    }
    printf( "EVP_EncryptInit_ex succeeded\n" );
    
    return 0;
}