Re: [JDK-8146293] - Proposal to fix RSASSA-PSS AlgorithmChecker constraints for TLS 1.2

[Sean Mullan]

Hi Chris,

Comments inline ...
On 2/10/17 4:41 PM, Christopher Fox wrote:
We have been looking into supporting RSASSA-PSS signature algorithms
within the chain of an end-entity certificate used for TLS 1.2. The EE
certificate itself is not signed with RSASSA-PSS.

As mentioned in JDK-8146293
<https://bugs.openjdk.java.net/browse/JDK-8146293>, we run into the
exception: java.security.cert.CertificateException: Certificates does
not conform to algorithm constraints

Upon closer inspection we believe there are 2 workarounds for this issue:

1)
Update 
sun.security.provider.certpath.AlgorithmChecker#check(java.security.cert.Certificate,
java.util.Collection<java.lang.String>) to call getSigAlgName from the
provided certificate (var1), instead of the
converted sun.security.x509.X509CertImpl (var3).

    Looking at the code in question:

        public void check(Certificate var1, Collection<String> var2) throws 
CertPathValidatorException {
            if(var1 instanceof X509Certificate && this.constraints != null) {
                X509CertImpl var3 = null;

                try {
                    var3 = X509CertImpl.toImpl((X509Certificate)var1);
                } catch (CertificateException var15) {
                    throw new CertPathValidatorException(var15);
                }

                PublicKey var4 = var3.getPublicKey();
                String var5 = var3.getSigAlgName();
                AlgorithmId var6 = null;

                try {
                    var6 = (AlgorithmId)var3.get("x509.algorithm");
                } catch (CertificateException var14) {
                    throw new CertPathValidatorException(var14);
                }

                AlgorithmParameters var7 = var6.getParameters();
                if(!this.constraints.permits(SIGNATURE_PRIMITIVE_SET, var5, 
var7)) {
                    throw new CertPathValidatorException("Algorithm constraints 
check failed: " + var5, (Throwable)null, (CertPath)null, -1, 
BasicReason.ALGORITHM_CONSTRAINED);
                } else {
                ....

    The problem is that the sun.security.x509.X509CertImpl cannot
    convert the RSASSA-PSS algorithm OID to its friendly name when
    var3.getSigAlgName() is called:

    NOTE: In this case var1 is a instance
    of org.bouncycastle.jce.provider.X509CertificateObject

    In our tests, making this change results in a successful TLS
    connection without further changes:

    - Stringvar5 = var3.getSigAlgName();
    + Stringvar5 = ((X509Certificate)var1).getSigAlgName();

We have just fixed this in JDK 9: 
http://hg.openjdk.java.net/jdk9/dev/jdk/rev/d911fe42d2da

2) Update sun.security.x509.AlgorithmId to properly map the RSASSA-PSS
algorithm OID to its friendly name. We have not experimented with this
option, but believe it would have the same outcome, but with more code
to change.

I think that's a more involved changes that will be addressed by 
JDK-8146293.

Thanks,
Sean