On Tue, Jul 27, 2010 at 10:09 AM, Pat <lync...@gmail.com> wrote: > Hello, > > Can anyone explain what is going wrong with the following scenario? > > Using NSPR 4.8, NSS 3.12.6, JSS 4.3.1 with JDK 1.6_21 on Windows XP > Professional SP 3. FIPS mode is enabled. > > I'm trying to open an LDAP connection to an LDAP server (Apache > Directory Server) running locally on the same system. Both SSL > contexts (server and client) are configured to use the KeyManager > (PKCS11 KeyStore), TrustManager and SecureRandom obtained from the > SunPKCS11-NSS provider. > > I have enabled the debug logging for the TLS handshake and I can see > that both sides are using the same certificate for identification and > that this certificate as well as the appropriate CA certificates are > being found in the NSS database. > > When executing the code with FIPS mode disabled, the handshake is > successful. However, with FIPS mode enabled, the following stacktrace > is produced: > > 2010-07-27 08:51:02,154;48156;ERROR;ds.DsServiceImplLiveTest; > (main);Client: > javax.net.ssl.SSLException: java.security.ProviderException: Could not > generate premaster secret > at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java: > 190) > at > com.sun.net.ssl.internal.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java: > 1623) [...omitted...] > Caused by: java.security.ProviderException: Could not generate > premaster secret > at > sun.security.pkcs11.P11TlsRsaPremasterSecretGenerator.engineGenerateKey(P11TlsRsaPremasterSecretGenerator.java: > 87) > at javax.crypto.KeyGenerator.generateKey(DashoA13*..) > at > com.sun.net.ssl.internal.ssl.RSAClientKeyExchange.<init>(RSAClientKeyExchange.java: > 91) > at > com.sun.net.ssl.internal.ssl.ClientHandshaker.serverHelloDone(ClientHandshaker.java: > 673) > at > com.sun.net.ssl.internal.ssl.ClientHandshaker.processMessage(ClientHandshaker.java: > 230) > at > com.sun.net.ssl.internal.ssl.Handshaker.processLoop(Handshaker.java: > 529) [...omitted...] > Caused by: sun.security.pkcs11.wrapper.PKCS11Exception: > CKR_ATTRIBUTE_VALUE_INVALID > at sun.security.pkcs11.wrapper.PKCS11.C_GenerateKey(Native Method) > at > sun.security.pkcs11.P11TlsRsaPremasterSecretGenerator.engineGenerateKey(P11TlsRsaPremasterSecretGenerator.java: > 81) [...omitted...]
The stack trace of the exception shows that the C_GenerateKey call used to generate the premaster secret failed with CKR_ATTRIBUTE_VALUE_INVALID. Since there is Sun Java code involved, I won't be able to fully debug this. I'll give you some hints on how I would debug this, and hopefully you can track this down. 1. Search in the NSS source tree for "CKR_ATTRIBUTE_VALUE_INVALID" to identify the places where CKR_ATTRIBUTE_VALUE_INVALID may be returned: http://mxr.mozilla.org/security/ident?i=CKR_ATTRIBUTE_VALUE_INVALID&filter= Focus only on the files in security/nss/lib/softoken, as the SunPKCS11 provider is only using the "softoken" of NSS. 2. Since you get this error when the NSS softoken is in FIPS mode, security/nss/lib/softoken/fipstokn.c is worth a look. In FIPS mode, the PKCS #11 C_Foo function is mapped to the FC_Foo function in security/nss/lib/softoken/fipstokn.c. So C_GenerateKey becomes FC_GenerateKey: http://bonsai.mozilla.org/cvsblame.cgi?file=mozilla/security/nss/lib/softoken/fipstokn.c&rev=1.28&mark=1292,1300-1307#1291 NSC_GenerateKey is the common implementation of C_GenerateKey between FIPS and non-FIPS mode. So you see clearly that FC_GenerateKey does some special checks for FIPS mode, and then invoke the common code. One of the special check is that the caller must specify that the new key be sensitive (CKA_SENSITIVE), which means the new key cannot be exported in unencrypted form. If this check fails, FC_GenerateKey returns CKR_ATTRIBUTE_VALUE_INVALID. Given the info you provided, this is most likely the place where the NSS softoken failed with CKR_ATTRIBUTE_VALUE_INVALID in FIPS mode. If so, I'm afraid that the fix is to modify sun.security.pkcs11 to be able to generate and use RSA premaster secrets with the CKA_SENSITIVE attribute set. If this is not the failure spot, then you can use the method I outlined above to track this down. Wan-Teh -- dev-tech-crypto mailing list dev-tech-crypto@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-tech-crypto
