Hi all, I have a problem with the reloading of my ssl configs after an update of certs from LetsEncrypt or my internal CA.
Tomcat is 8.5.50, JDK is 11.0.5+10-post-Debian-2 I have used basically Christopher Schultz’s tool from https://github.com/ChristopherSchultz/apache-tomcat-stuff/tree/master/bin. ▶ curl -u user:pass "https://host:8443/manager/jmxproxy?invoke=Catalina:type=ProtocolHandler,port=8443&op=reloadSslHostConfigs"; Error - javax.management.RuntimeOperationsException: Exception invoking method reloadSslHostConfigs javax.management.RuntimeOperationsException: Exception invoking method reloadSslHostConfigs at org.apache.tomcat.util.modeler.BaseModelMBean.invoke(BaseModelMBean.java:295) at java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:809) at java.management/com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801) at org.apache.catalina.manager.JMXProxyServlet.invokeOperationInternal(JMXProxyServlet.java:273) at org.apache.catalina.manager.JMXProxyServlet.invokeOperation(JMXProxyServlet.java:207) at org.apache.catalina.manager.JMXProxyServlet.doGet(JMXProxyServlet.java:116) at javax.servlet.http.HttpServlet.service(HttpServlet.java:634) at javax.servlet.http.HttpServlet.service(HttpServlet.java:741) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.catalina.filters.HttpHeaderSecurityFilter.doFilter(HttpHeaderSecurityFilter.java:126) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.catalina.filters.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:109) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:668) at org.apache.catalina.valves.RequestFilterValve.process(RequestFilterValve.java:348) at org.apache.catalina.valves.RemoteAddrValve.invoke(RemoteAddrValve.java:52) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:678) at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:747) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) at org.apache.coyote.http2.StreamProcessor.service(StreamProcessor.java:324) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) at org.apache.coyote.http2.StreamProcessor.process(StreamProcessor.java:69) at org.apache.coyote.http2.StreamRunnable.run(StreamRunnable.java:35) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: java.lang.IllegalArgumentException: java.lang.IllegalArgumentException at org.apache.tomcat.util.net.AbstractEndpoint.addSslHostConfig(AbstractEndpoint.java:252) at org.apache.tomcat.util.net.AbstractEndpoint.reloadSslHostConfig(AbstractEndpoint.java:311) at org.apache.tomcat.util.net.AbstractEndpoint.reloadSslHostConfigs(AbstractEndpoint.java:320) at org.apache.coyote.http11.AbstractHttp11Protocol.reloadSslHostConfigs(AbstractHttp11Protocol.java:517) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.apache.tomcat.util.modeler.BaseModelMBean.invoke(BaseModelMBean.java:287) ... 38 more Caused by: java.lang.IllegalArgumentException at org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:100) at org.apache.tomcat.util.net.AbstractEndpoint.addSslHostConfig(AbstractEndpoint.java:250) ... 46 more Caused by: java.io.IOException at org.apache.tomcat.util.net.SSLUtilBase.getKeyManagers(SSLUtilBase.java:301) at org.apache.tomcat.util.net.SSLUtilBase.createSSLContext(SSLUtilBase.java:246) at org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:98) ... 47 more It’s working after a restart of tomcat, then it may be successful a couple of times, but then something is happening (Access of the website?) and boom the exception occurs. I am able to force this error, if I call a imx query with curl first. curl -u user:pass "https://host:8443/manager/jmxproxy?qry=Catalina:type=ProtocolHandler,port=8443“ I have tried to debug the problem and tracked it down to an empty KeyStore on the „defaultSSLHostConfigName" due to an empty certificateKeystore and with a default non-existing certificateKeystoreFile "/home/tomcat/.keystore“ and an null certificateFile . I see some of my configured attributes - but not the cert file. From there on I need help. I have no clue where to look next. Maybe Mark or Chris may lead me to the spot where all this is reset. Chris, is your script working on 8.5? Code is: AbstractEndpoint.java:252 public void reloadSslHostConfig(String hostName) { SSLHostConfig sslHostConfig = sslHostConfigs.get(hostName); if (sslHostConfig == null) { throw new IllegalArgumentException( sm.getString("endpoint.unknownSslHostName", hostName)); } addSslHostConfig(sslHostConfig, true); // << sslHostConfig has no Cert anymore, yet it is not null } And in SSLUtilBase.java:301 KeyStore ks = certificate.getCertificateKeystore(); // << none of the possible locations are filled anymore ... if (ks == null) { if (certificate.getCertificateFile() == null) { throw new IOException(sm.getString("jsse.noCertFile")); } ... The site is still accessible in the browser even though the JMX requests fail???? My SSL config is: <Connector port="8443" protocol="org.apache.coyote.http11.Http11Nio2Protocol" sslImplementationName="org.apache.tomcat.util.net.jsse.JSSEImplementation" allowTrace="false" maxThreads="150" SSLEnabled="true" compression="off" scheme="https" server="Apache Tomcat" secure="true" defaultSSLHostConfigName="host" > <UpgradeProtocol className="org.apache.coyote.http2.Http2Protocol" compression="on" /> <SSLHostConfig hostName="host" honorCipherOrder="true" certificateVerification="none" certificateRevocationListFile="${catalina.base}/conf/ssl/ca-bundle-client.crl" truststoreFile="${catalina.base}/conf/ssl/cacerts.jks" truststorePassword="changeit" protocols="TLSv1.2+TLSv1.3" ciphers="HIGH:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:!DSS"> <Certificate certificateKeystoreFile="${catalina.base}/conf/ssl/docker.p12" certificateKeystorePassword="changeit" certificateKeyAlias="docker" type="RSA" /> </SSLHostConfig> … I have a couple more of them (self signed CA and LetsEncrypt). But in the reloadSslHostConfigs they are correctly initialized. Any idea? Maybe it’s just a misunderstanding? Or a bug? Peter
smime.p7s
Description: S/MIME cryptographic signature
