There appears to be an issue with the way OpenSSL handles X9.31 padding in
RSA signatures. According to the X9.31 standard (section 4.2, step 4):
RR shall be raised to the power d modulo n. The signature S is either
the result or its complement to n, whichever is smaller.
S = min { RR^d mod n, n - (RR^d mod n) }
In their terminology:
S is the signature (they use a capital sigma, but I'm using S for the email)
RR is the hash of the message surrounded by their padding characters, header,
and trailer.
d is the private exponent
n is the
public modulus
The issue is the 'whichever is smaller' part. In crypto/rsa/rsa_eay.c, at the
end of RSA_eay_private_encrypt():
494 if (padding == RSA_X931_PADDING)
495 {
496 BN_sub(f, rsa->n, ret);
497 if (BN_cmp(ret, f))
498 res = f;
499 else
500 res = ret;
501 }
Line 497 does does not check if 'f' is *smaller* than 'ret', it checks if it is
*different* from ret.
I believe the code should be:
494 if (padding == RSA_X931_PADDING)
495 {
496 BN_sub(f, rsa->n, ret);
497 if (BN_cmp(ret, f) > 0)
498 res = f;
499 else
500 res = ret;
501 }
Patch attached.
diff --git a/crypto/rsa/rsa_eay.c b/crypto/rsa/rsa_eay.c
index 16f000f..64c23f7 100644
--- a/crypto/rsa/rsa_eay.c
+++ b/crypto/rsa/rsa_eay.c
@@ -494,7 +494,7 @@ static int RSA_eay_private_encrypt(int flen, const unsigned
char *from,
if (padding == RSA_X931_PADDING)
{
BN_sub(f, rsa->n, ret);
- if (BN_cmp(ret, f))
+ if (BN_cmp(ret, f) > 0)
res = f;
else
res = ret;
