Hi all, More analysis and now getting to a weird point:
Chain of events: ——————————— Start tomcat with SSL and JMX enabled… Reload SSL config with JMX (this time for only one host) > curl -u user:pass > "https://docker2:8443/manager/jmxproxy?invoke=Catalina:type=ProtocolHandler,port=8443&op=reloadSslHostConfig&ps=docker2“ OK - Operation reloadSslHostConfig without return value Now query the same certificate with JMX > curl -u user:pass > 'https://docker2:8443/manager/jmxproxy?qry=Catalina:type=SSLHostConfig,ThreadPool="https-jsse-nio2-8443",name="docker2";' OK - Number of results: 1 … I see that an *empty* Default certificate is added to SSLHostConfigs at SSLHostConfig.java:217 private void registerDefaultCertificate() { if (defaultCertificate == null) { // <<< ??? defaultCertificate = new SSLHostConfigCertificate( this, SSLHostConfigCertificate.Type.UNDEFINED); // << MEEK certificates.add(defaultCertificate); } } WAIT: it’s called from here: // ------------------------------- OpenSSL specific configuration properties // TODO: These certificate setters can be removed once it is no longer // necessary to support the old configuration attributes (Tomcat 10?). public String getCertificateChainFile() { registerDefaultCertificate(); return defaultCertificate.getCertificateChainFile(); } And the callstack: SSLHostConfig.registerDefaultCertificate() line: 217 SSLHostConfig.getCertificateChainFile() line: 733 NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method] NativeMethodAccessorImpl.invoke(Object, Object[]) line: 62 DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 43 Method.invoke(Object, Object...) line: 566 BaseModelMBean.getAttribute(String) line: 176 DefaultMBeanServerInterceptor.getAttribute(ObjectName, String) line: 641 JmxMBeanServer.getAttribute(ObjectName, String) line: 678 Now Reload SSL config with JMX > curl -u user:pass > "https://docker2:8443/manager/jmxproxy?invoke=Catalina:type=ProtocolHandler,port=8443&op=reloadSslHostConfig&ps=docker2“ javax.management.RuntimeOperationsException… Apparently there are two certs in the config now for this host. The *empty* Default certificate is loaded and SSLUtilBase.java:301 fails as there is no certificate file in the Keystore. Mark can you confirm that this is a bug? Thanks and have a nice Sunday Peter > Am 14.12.2019 um 18:29 schrieb logo <l...@kreuser.name>: > > Hi all, > > I have a problem with the reloading of my ssl configs after an update of > certs from LetsEncrypt or my internal CA. > > Tomcat is 8.5.50, JDK is 11.0.5+10-post-Debian-2 > > I have used basically Christopher Schultz’s tool from > https://github.com/ChristopherSchultz/apache-tomcat-stuff/tree/master/bin > <https://github.com/ChristopherSchultz/apache-tomcat-stuff/tree/master/bin>. > > > ▶ curl -u user:pass > "https://host:8443/manager/jmxproxy?invoke=Catalina:type=ProtocolHandler,port=8443&op=reloadSslHostConfigs > > <https://host:8443/manager/jmxproxy?invoke=Catalina:type=ProtocolHandler,port=8443&op=reloadSslHostConfigs>" > Error - javax.management.RuntimeOperationsException: Exception invoking > method reloadSslHostConfigs > javax.management.RuntimeOperationsException: Exception invoking method > reloadSslHostConfigs > at > org.apache.tomcat.util.modeler.BaseModelMBean.invoke(BaseModelMBean.java:295) > at > java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:809) > at > java.management/com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801) > at > org.apache.catalina.manager.JMXProxyServlet.invokeOperationInternal(JMXProxyServlet.java:273) > at > org.apache.catalina.manager.JMXProxyServlet.invokeOperation(JMXProxyServlet.java:207) > at > org.apache.catalina.manager.JMXProxyServlet.doGet(JMXProxyServlet.java:116) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:634) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:741) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) > at > org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) > at > org.apache.catalina.filters.HttpHeaderSecurityFilter.doFilter(HttpHeaderSecurityFilter.java:126) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) > at > org.apache.catalina.filters.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:109) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) > at > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199) > at > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) > at > org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:668) > at > org.apache.catalina.valves.RequestFilterValve.process(RequestFilterValve.java:348) > at > org.apache.catalina.valves.RemoteAddrValve.invoke(RemoteAddrValve.java:52) > at > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139) > at > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) > at > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) > at > org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:678) > at > org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:747) > at > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) > at > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) > at > org.apache.coyote.http2.StreamProcessor.service(StreamProcessor.java:324) > at > org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) > at > org.apache.coyote.http2.StreamProcessor.process(StreamProcessor.java:69) > at org.apache.coyote.http2.StreamRunnable.run(StreamRunnable.java:35) > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > at > org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) > at java.base/java.lang.Thread.run(Thread.java:834) > Caused by: java.lang.IllegalArgumentException: > java.lang.IllegalArgumentException > at > org.apache.tomcat.util.net.AbstractEndpoint.addSslHostConfig(AbstractEndpoint.java:252) > at > org.apache.tomcat.util.net.AbstractEndpoint.reloadSslHostConfig(AbstractEndpoint.java:311) > at > org.apache.tomcat.util.net.AbstractEndpoint.reloadSslHostConfigs(AbstractEndpoint.java:320) > at > org.apache.coyote.http11.AbstractHttp11Protocol.reloadSslHostConfigs(AbstractHttp11Protocol.java:517) > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.base/java.lang.reflect.Method.invoke(Method.java:566) > at > org.apache.tomcat.util.modeler.BaseModelMBean.invoke(BaseModelMBean.java:287) > ... 38 more > Caused by: java.lang.IllegalArgumentException > at > org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:100) > at > org.apache.tomcat.util.net.AbstractEndpoint.addSslHostConfig(AbstractEndpoint.java:250) > ... 46 more > Caused by: java.io.IOException > at > org.apache.tomcat.util.net.SSLUtilBase.getKeyManagers(SSLUtilBase.java:301) > at > org.apache.tomcat.util.net.SSLUtilBase.createSSLContext(SSLUtilBase.java:246) > at > org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:98) > ... 47 more > > > It’s working after a restart of tomcat, then it may be successful a couple of > times, but then something is happening (Access of the website?) and boom the > exception occurs. > I am able to force this error, if I call a imx query with curl first. > > curl -u user:pass > "https://host:8443/manager/jmxproxy?qry=Catalina:type=ProtocolHandler,port=8443 > > <https://host:8443/manager/jmxproxy?qry=Catalina:type=ProtocolHandler,port=8443>“ > > > > I have tried to debug the problem and tracked it down to an empty KeyStore on > the „defaultSSLHostConfigName" due to an empty certificateKeystore and with a > default non-existing certificateKeystoreFile "/home/tomcat/.keystore“ and an > null certificateFile . I see some of my configured attributes - but not the > cert file. From there on I need help. I have no clue where to look next. > Maybe Mark or Chris may lead me to the spot where all this is reset. Chris, > is your script working on 8.5? > > Code is: AbstractEndpoint.java:252 > > public void reloadSslHostConfig(String hostName) { > SSLHostConfig sslHostConfig = sslHostConfigs.get(hostName); > if (sslHostConfig == null) { > throw new IllegalArgumentException( > sm.getString("endpoint.unknownSslHostName", hostName)); > } > addSslHostConfig(sslHostConfig, true); // << sslHostConfig has no > Cert anymore, yet it is not null > } > > > And in SSLUtilBase.java:301 > > KeyStore ks = certificate.getCertificateKeystore(); // << none of the > possible locations are filled anymore > ... > if (ks == null) { > if (certificate.getCertificateFile() == null) { > throw new IOException(sm.getString("jsse.noCertFile")); > } > ... > > > The site is still accessible in the browser even though the JMX requests > fail???? > > My SSL config is: > > <Connector port="8443" > protocol="org.apache.coyote.http11.Http11Nio2Protocol" > > sslImplementationName="org.apache.tomcat.util.net.jsse.JSSEImplementation" > allowTrace="false" > maxThreads="150" > SSLEnabled="true" > compression="off" > scheme="https" > server="Apache Tomcat" > secure="true" > defaultSSLHostConfigName="host" > > <UpgradeProtocol className="org.apache.coyote.http2.Http2Protocol" > compression="on" /> > > <SSLHostConfig > hostName="host" > honorCipherOrder="true" > certificateVerification="none" > > certificateRevocationListFile="${catalina.base}/conf/ssl/ca-bundle-client.crl" > truststoreFile="${catalina.base}/conf/ssl/cacerts.jks" > truststorePassword="changeit" > protocols="TLSv1.2+TLSv1.3" > > ciphers="HIGH:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:!DSS"> > <Certificate > certificateKeystoreFile="${catalina.base}/conf/ssl/docker.p12" > certificateKeystorePassword="changeit" > certificateKeyAlias="docker" > type="RSA" /> > </SSLHostConfig> > … > > I have a couple more of them (self signed CA and LetsEncrypt). But in the > reloadSslHostConfigs they are correctly initialized. > > Any idea? Maybe it’s just a misunderstanding? Or a bug? > > Peter >
smime.p7s
Description: S/MIME cryptographic signature
