[JDK-8146293] - Proposal to fix RSASSA-PSS AlgorithmChecker constraints for TLS 1.2

[Christopher Fox]

Hello,


We have been looking into supporting RSASSA-PSS signature algorithms within the 
chain of an end-entity certificate used for TLS 1.2. The EE certificate itself 
is not signed with RSASSA-PSS.


As mentioned in JDK-8146293<https://bugs.openjdk.java.net/browse/JDK-8146293>, 
we run into the exception: java.security.cert.CertificateException: 
Certificates does not conform to algorithm constraints


Upon closer inspection we believe there are 2 workarounds for this issue:


1) Update 
sun.security.provider.certpath.AlgorithmChecker#check(java.security.cert.Certificate,
 java.util.Collection<java.lang.String>) to call getSigAlgName from the 
provided certificate (var1), instead of the converted 
sun.security.x509.X509CertImpl (var3).


Looking at the code in question:


    public void check(Certificate var1, Collection<String> var2) throws 
CertPathValidatorException {
        if(var1 instanceof X509Certificate && this.constraints != null) {
            X509CertImpl var3 = null;

            try {
                var3 = X509CertImpl.toImpl((X509Certificate)var1);
            } catch (CertificateException var15) {
                throw new CertPathValidatorException(var15);
            }

            PublicKey var4 = var3.getPublicKey();
            String var5 = var3.getSigAlgName();
            AlgorithmId var6 = null;

            try {
                var6 = (AlgorithmId)var3.get("x509.algorithm");
            } catch (CertificateException var14) {
                throw new CertPathValidatorException(var14);
            }

            AlgorithmParameters var7 = var6.getParameters();
            if(!this.constraints.permits(SIGNATURE_PRIMITIVE_SET, var5, var7)) {
                throw new CertPathValidatorException("Algorithm constraints 
check failed: " + var5, (Throwable)null, (CertPath)null, -1, 
BasicReason.ALGORITHM_CONSTRAINED);
            } else {
            ....

The problem is that the sun.security.x509.X509CertImpl cannot convert the 
RSASSA-PSS algorithm OID to its friendly name when var3.getSigAlgName() is 
called:

[cid:6a0141b3-a283-46ca-9db8-115cafc77a07]
NOTE: In this case var1 is a instance of 
org.bouncycastle.jce.provider.X509CertificateObject

In our tests, making this change results in a successful TLS connection without 
further changes:

- String var5 = var3.getSigAlgName();
+ String var5 = ((X509Certificate)var1).getSigAlgName();


2) Update sun.security.x509.AlgorithmId to properly map the RSASSA-PSS 
algorithm OID to its friendly name. We have not experimented with this option, 
but believe it would have the same outcome, but with more code to change.


Any thoughts from the community on which approach would be accepted into the 
JDK, or alternative suggestions not mentioned here, are appreciated.


Thanks,

Chris Fox

Senior Software Engineer @ MobileIron