Just a note to say I haven't forgotten this. I hope to look at this this week unless someone beats me to it.
Mark On 16/10/2019 17:55, George Stanchev wrote: > > On 15/10/2019 22:15, George Stanchev wrote: >> Hi, >> >> I would need some help with tracking an issue with TC 8.5.47 (windows x64, >> java: azul 1.8.0_222) configured with [1] and tcnative-1.dll. When a simple >> client tries to connect to the server, the server hangs on SSL handshake >> until either the client times out on read or the server times out (if I set >> the HttpsURLConnection#setConnectTimeout(0) and ...#setReadTimeout(0)). I >> have enabled the client side SSL trace and everything goes well until ECDH >> key exchange (for brevity I have enabled only one TLS suite >> "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA"). I can provide the debug logs if >> requested. The cacerts we use is whatever comes with Azul's Java distro >> which has ~150 entries + the custom cert for testing. The issue comes from >> how the connector deals with trusted certs because if I reduce the cacerts >> to contain only the test certificate, the request is being served without an >> issue. Also if I remove the tcnative-1.dll from TC there is no issue either. >> >> Perhaps I am missing something. Any help is appreciated. > > <Mark wrote> > This sounds like it is repeatable and that you have a system you can test > with. On that basis here are a few things to try: > > 1. Take a 3 thread dumps ~5s apart of the Tomcat process when TLS handshake > is hanging. > > 2. Try a binary search to try and determine if the issue is the number of > certificates in the trust store or is caused by a specific certificate. > > It sounds like there might be an issue converting one or more of the trusted > certs in the trust store to a format OpenSSL can work with. > > </Mark wrote> > > So the thread dumps didn't prove to be very useful - at least I couldn't see > anything. I attached one of them (dump3.txt). But the trial and error in the > binary searched proved it is not any certificate but a magic number of > certificates in cacerts - 125 hang and 124 work. I don't know if it is a > memory issue or buffer size issue - I tried removing and adding random certs > around the 124/125 boundary and which one is out and in doesn't make a > difference. I realize that a cert is rather large blob so it still could be > memory issue with the size of the cert doesn't matter that much. In > 8.5.47-src\java\org\apache\tomcat\util\net\openssl\OpenSSLContext.java#init(kms, > tms, sr) we pass the DER-encoded certs to OpenSSL: > > // Pass along the DER encoded certificates of the accepted > client > // certificate issuers, so that their subjects can be > presented > // by the server during the handshake to allow the client > choosing > // an acceptable certificate > for (X509Certificate caCert : > x509TrustManager.getAcceptedIssuers()) { > SSLContext.addClientCACertificateRaw(ctx, > caCert.getEncoded()); > if (log.isDebugEnabled()) > log.debug(sm.getString("openssl.addedClientCaCert", > caCert.toString())); > } > > And this is not where it hangs. I stepped through the code through the > handshaker but still haven't been able to figure out the hang point as I am > not familiar with the details of that portion of TC code . I've attached two > cacerts that you can plug in as truststore to try out the issue. > > George > > > >> >> George >> >> [1] >> >> <Connector >> port="8443" SSLEnabled="true" maxHttpHeaderSize="8192" >> maxThreads="150" acceptCount="100" enableLookups="false" >> disableUploadTimeout="true" >> scheme="https" secure="true" clientAuth="true" sslProtocol="TLS" >> sslEnabledProtocols="+TLSv1 +TLSv1.1 +TLSv1.2" >> protocol="org.apache.coyote.http11.Http11NioProtocol" >> keystoreType="JKS" >> keystoreFile="${server.conf.dir}/serena.keystore" keystorePass="changeit" >> keyAlias="jboss" URIEncoding="UTF-8" useServerCipherSuitesOrder="true" >> ciphers="TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384, >> TLS_CHACHA20_POLY1305_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, >> TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, >> TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, >> TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, >> TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, >> TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256, >> TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, >> TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, >> TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, >> TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, >> TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256, >> TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, >> TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, >> TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, >> TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, >> TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, >> TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, >> TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, >> TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, >> TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_128_GCM_SHA256, >> TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_128_CBC_SHA256, >> TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA, >> TLS_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_3DES_EDE_CBC_SHA, >> TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, >> TLS_ECDHE_ECDSA_WITH_AES_128_SHA_GCM_SHA256, >> TLS_ECDHE_ECDSA_WITH_AES_128_SHA_CBC_SHA256, >> TLS_ECDHE_ECDSA_WITH_AES_128_CCM, TLS_ECDHE_ECDSA_WITH_AES_128_CCM_8, >> TLS_ECDHE_ECDSA_WITH_AES_256_GCM_CCM_8, >> TLS_ECDHE_ECDSA_WITH_AES_256_CCM, TLS_ECDHE_ECDSA_WITH_AES_256_SHA, >> TLS_ECDHE_ECDSA_WITH_AES_256_CCM_8, >> TLS_DHE_DSS_WITH_AES_256_CBC_SHA256, >> TLS_RSA_WITH_CAMELLIA_128_CBC_SHA256, >> TLS_RSA_WITH_CAMELLIA_256_CBC_SHA256, >> TLS_DHE_DSS_WITH_CAMELLIA_128_CBC_SHA256, >> TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA256, >> TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA256, >> TLS_ECDHE_ECDSA_WITH_CAMELLIA_128_CBC_SHA256, >> TLS_ECDHE_ECDSA_WITH_CAMELLIA_256_CBC_SHA384, >> TLS_ECDHE_RSA_WITH_CAMELLIA_128_CBC_SHA256, >> TLS_ECDHE_RSA_WITH_CAMELLIA_256_CBC_SHA384" /> >> >> > > B‹KKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKCB•È[œÝXœØÜšX™KK[XZ[ > ˆ\Ù\œË][œÝXœØÜšX™PÛXØ] > ˜\XÚK›Ü™ÃB‘›ÜˆY][Û˜[ÛÛ[X[™ËK[XZ[ > ˆ\Ù\œËZ[ÛXØ] > ˜\XÚK›Ü™ÃBƒ > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org