[openssl.org #3022] Binary curves broken in FIPS mode

[Matt Caswell via RT]

Hello

When using OpenSSL-1.0.1e-fips a call to PEM_write_bio_PrivateKey
silently fails and produces a corrupt pem file when using an
EVP_PKEY_EC key and a binary curve. The same function works fine when
not using a FIPS capable OpenSSL. I suspect the same problem will
affect any ASN.1 routines that attempt to write the private key.

Please see attached:
* A test case c file that demonstrates the problem
* Two example corrupt pem files
* A patch to rectify the problem

The patch has been tested against OpenSSL-1.0.1e and corrects the following:
* Modifies eckey_param2type in ec_ameth.c to check for a 0 return from
i2d_ECParameters to prevent a silent failure
* Modifies the checks to see if the functions should have been called
in EC_GROUP_get_trinomial_basis and EC_GROUP_get_pentanomial_basis
within ec_asn1.c, so that they work in FIPS mode

Please can someone apply this patch, as binary curves are currently
broken in FIPS mode.

Thanks

Matt


#include <openssl/evp.h>
#include <openssl/ec.h>
#include <openssl/obj_mac.h>
#include <openssl/err.h>

EVP_PKEY *generate_key(int curve);



int main(int arc, char *argv[])
{
	EVP_PKEY *key;
	int i;

	/* Initialise the library */
	ERR_load_crypto_strings();

	/* Generate an Elliptic Curve key (for ECDSA) */
	for(i=0; i<2; i++)
	{
		key = generate_key(i==0?NID_sect233k1:NID_sect131r1);

		if(key == NULL)
		{
			printf("Error generating key\n");
			goto err;
		}


		BIO *bp;
		char filename[80];
		sprintf(filename, "out%d.pem", i);
		if(NULL == (bp = BIO_new_file(filename, "wb"))) goto err;
		if (1 != PEM_write_bio_PrivateKey(bp, key, NULL, NULL, 0, 0, NULL)) goto err;
		BIO_free(bp);
		EVP_PKEY_free(key);
	}


	printf("Finished\n");
	goto finish;
	/* Clean up */
err:
	ERR_print_errors_fp(stderr);
finish:
	ERR_free_strings();
}


EVP_PKEY *generate_key(int curve)
{
	EVP_PKEY_CTX *pctx = NULL, *kctx = NULL;
	EVP_PKEY *params = NULL, *key = NULL;


	/* Create the context for generating the parameters */
	if(!(pctx = EVP_PKEY_CTX_new_id(EVP_PKEY_EC, NULL))) goto err;
	if(!EVP_PKEY_paramgen_init(pctx)) goto err;

	/*if(!EVP_PKEY_CTX_set_ec_paramgen_curve_nid(pctx, NID_sect233k1)) goto err;*/
	if(!EVP_PKEY_CTX_set_ec_paramgen_curve_nid(pctx, NID_sect131r1)) goto err;

	/* Generate parameters */
	if (!EVP_PKEY_paramgen(pctx, &params)) goto err;

	/* Create context for the key generation */
	if(!(kctx = EVP_PKEY_CTX_new(params, NULL))) goto err; 

	/* Generate the key */

	if(!EVP_PKEY_keygen_init(kctx)) goto err;

	if (!EVP_PKEY_keygen(kctx, &key)) goto err;

	goto finish;
err:
	ERR_print_errors_fp(stderr);
finish:
	if(pctx) EVP_PKEY_CTX_free(pctx);
	if(params) EVP_PKEY_free(params);
	if(kctx) EVP_PKEY_CTX_free(kctx);

	return key;
}

out0-corrupt.pem
Description: Binary data

out1-corrupt.pem
Description: Binary data

ec2patch.patch
Description: Binary data