[TLS] TLS 1.2 - is it allowed to strip the leading zero byte(s) in RSA signature in ServerKeyExchange?

[M K Saravanan]

Hi,

I recently encountered the below issue:

TLS1.2
ECDHE_RSA
server certificate: 2048-bit RSA (= 256 bytes)
ServerKeyExchange hash/sign algorithm: rsa_pkcs1_sha1

The server was sending the ServerKeyExchange with 255 byte as length for
the RSA signature (i.e. the leading zero was stripped) instead of 256 like
this:

====================
Handshake Protocol: Server Key Exchange
    Handshake Type: Server Key Exchange (12)
    Length: 328
    EC Diffie-Hellman Server Params
        Curve Type: named_curve (0x03)
        Named Curve: secp256r1 (0x0017)
        Pubkey Length: 65
        Pubkey: 042206562efea8bd47bf014a9e650c42f27078643c553671…
        Signature Algorithm: rsa_pkcs1_sha1 (0x0201)
        Signature Length: 255
        Signature: d1bf915eca2ec0bcdda6f90a398fe5378d2028a22574d213…
====================

Is this allowed?  i.e. stripping the leading zero of the RSA signature and
marking the length as 255?   It is not clear to me from the RFC5246 whether
it is allowed or not.

(client was failing to verify the signature due to this).

with regards,
Saravanan

_______________________________________________
TLS mailing list
TLS@ietf.org
https://www.ietf.org/mailman/listinfo/tls