On Wed, Aug 5, 2020, 18:46 James H. H. Lampert <jam...@touchtonecorp.com>
wrote:
> Ladies and Gentlemen:
>
> I've now proceeded to the "real" server, with the Tomcat portion of the
> procedure refined to give me plenty of "undo" capability. And it turns
> out I need it.
>
> It seems that with the unwanted update to 7.0.57 that happened on
> launching the test spot instances, the Let's Encrypt certs worked just
> fine.
>
> But applying the procedure to the *real* development instance (7.0.40)
> blew up in my face, failing to open the connectors. Here is an excerpt
> from catalina.out, showing the stacktraces.
>
> > 05-Aug-2020 23:00:52.038 WARNING [main]
> org.apache.catalina.startup.SetAllPropertiesRule.begin
> [SetAllPropertiesRule]{Server/Service/Connector} Setting property
> 'bufferSize' to '1024' did not find a matching property.
> > 05-Aug-2020 23:00:52.085 WARNING [main]
> org.apache.catalina.startup.SetAllPropertiesRule.begin
> [SetAllPropertiesRule]{Server/Service/Connector} Setting property
> 'bufferSize' to '1024' did not find a matching property.
> > 05-Aug-2020 23:00:52.189 INFO [main]
> org.apache.catalina.startup.VersionLoggerListener.log Server version:
> Apache Tomcat/8.5.40
> > 05-Aug-2020 23:00:52.189 INFO [main]
> org.apache.catalina.startup.VersionLoggerListener.log Server built:
> May 2 2019 18:02:51 UTC
> > 05-Aug-2020 23:00:52.194 INFO [main]
> org.apache.catalina.startup.VersionLoggerListener.log Server number:
> 8.5.40.0
> > 05-Aug-2020 23:00:52.194 INFO [main]
> org.apache.catalina.startup.VersionLoggerListener.log OS Name:
> Linux
> > 05-Aug-2020 23:00:52.194 INFO [main]
> org.apache.catalina.startup.VersionLoggerListener.log OS Version:
> 4.14.121-85.96.amzn1.x86_64
> > 05-Aug-2020 23:00:52.194 INFO [main]
> org.apache.catalina.startup.VersionLoggerListener.log Architecture:
> amd64
> > 05-Aug-2020 23:00:52.195 INFO [main]
> org.apache.catalina.startup.VersionLoggerListener.log Java Home:
> /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.201.b09-0.43.amzn1.x86_64/jre
> > 05-Aug-2020 23:00:52.195 INFO [main]
> org.apache.catalina.startup.VersionLoggerListener.log JVM Version:
> 1.8.0_201-b09
> > 05-Aug-2020 23:00:52.195 INFO [main]
> org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor:
> Oracle Corporation
> > 05-Aug-2020 23:00:52.195 INFO [main]
> org.apache.catalina.startup.VersionLoggerListener.log CATALINA_BASE:
> /usr/share/tomcat8
> > 05-Aug-2020 23:00:52.196 INFO [main]
> org.apache.catalina.startup.VersionLoggerListener.log CATALINA_HOME:
> /usr/share/tomcat8
> > 05-Aug-2020 23:00:52.196 INFO [main]
> org.apache.catalina.startup.VersionLoggerListener.log Command line
> argument: -Dcatalina.base=/usr/share/tomcat8
> > 05-Aug-2020 23:00:52.196 INFO [main]
> org.apache.catalina.startup.VersionLoggerListener.log Command line
> argument: -Dcatalina.home=/usr/share/tomcat8
> > 05-Aug-2020 23:00:52.197 INFO [main]
> org.apache.catalina.startup.VersionLoggerListener.log Command line
> argument: -Djava.endorsed.dirs=
> > 05-Aug-2020 23:00:52.197 INFO [main]
> org.apache.catalina.startup.VersionLoggerListener.log Command line
> argument: -Djava.io.tmpdir=/var/cache/tomcat8/temp
> > 05-Aug-2020 23:00:52.197 INFO [main]
> org.apache.catalina.startup.VersionLoggerListener.log Command line
> argument:
> -Djava.util.logging.config.file=/usr/share/tomcat8/conf/logging.properties
> > 05-Aug-2020 23:00:52.197 INFO [main]
> org.apache.catalina.startup.VersionLoggerListener.log Command line
> argument: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
> > 05-Aug-2020 23:00:52.198 INFO [main]
> org.apache.catalina.core.AprLifecycleListener.lifecycleEvent The APR based
> Apache Tomcat Native library which allows optimal performance in production
> environments was not found on the java.library.path:
> [/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib]
> > 05-Aug-2020 23:00:52.422 INFO [main]
> org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler
> ["https-jsse-nio-8443"]
> > 05-Aug-2020 23:00:52.848 SEVERE [main]
> org.apache.catalina.core.StandardService.initInternal Failed to initialize
> connector [Connector[HTTP/1.1-8443]]
> > org.apache.catalina.LifecycleException: Failed to initialize component
> [Connector[HTTP/1.1-8443]]
> > at
> org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:112)
> > at
> org.apache.catalina.core.StandardService.initInternal(StandardService.java:552)
> > at
> org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:107)
> > at
> org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:875)
> > at
> org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:107)
> > at org.apache.catalina.startup.Catalina.load(Catalina.java:639)
> > at org.apache.catalina.startup.Catalina.load(Catalina.java:662)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:498)
> > at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:309)
> > at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:492)
> > Caused by: org.apache.catalina.LifecycleException: Protocol handler
> initialization failed
> > at
> org.apache.catalina.connector.Connector.initInternal(Connector.java:995)
> > at
> org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:107)
> > ... 12 more
> > Caused by: java.lang.IllegalArgumentException: Cannot store
> non-PrivateKeys
> > at org.apache.tomcat.util.net
> .AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:100)
> > at org.apache.tomcat.util.net
> .AbstractJsseEndpoint.initialiseSsl(AbstractJsseEndpoint.java:72)
> > at org.apache.tomcat.util.net
> .NioEndpoint.bind(NioEndpoint.java:244)
> > at org.apache.tomcat.util.net
> .AbstractEndpoint.init(AbstractEndpoint.java:1105)
> > at org.apache.tomcat.util.net
> .AbstractJsseEndpoint.init(AbstractJsseEndpoint.java:224)
> > at
> org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:581)
> > at
> org.apache.coyote.http11.AbstractHttp11Protocol.init(AbstractHttp11Protocol.java:68)
> > at
> org.apache.catalina.connector.Connector.initInternal(Connector.java:993)
> > ... 13 more
> > Caused by: java.security.KeyStoreException: Cannot store non-PrivateKeys
> > at
> sun.security.provider.JavaKeyStore.engineSetKeyEntry(JavaKeyStore.java:261)
> > at
> sun.security.provider.JavaKeyStore$JKS.engineSetKeyEntry(JavaKeyStore.java:56)
> > at
> sun.security.provider.KeyStoreDelegator.engineSetKeyEntry(KeyStoreDelegator.java:117)
> > at
> sun.security.provider.JavaKeyStore$DualFormatJKS.engineSetKeyEntry(JavaKeyStore.java:70)
> > at java.security.KeyStore.setKeyEntry(KeyStore.java:1140)
> > at org.apache.tomcat.util.net
> .SSLUtilBase.getKeyManagers(SSLUtilBase.java:313)
> > at org.apache.tomcat.util.net
> .SSLUtilBase.createSSLContext(SSLUtilBase.java:239)
> > at org.apache.tomcat.util.net
> .AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:98)
> > ... 20 more
> >
> > 05-Aug-2020 23:00:52.857 INFO [main]
> org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler
> ["https-jsse-nio-7443"]
> > 05-Aug-2020 23:00:52.861 SEVERE [main]
> org.apache.catalina.core.StandardService.initInternal Failed to initialize
> connector [Connector[HTTP/1.1-7443]]
> > org.apache.catalina.LifecycleException: Failed to initialize component
> [Connector[HTTP/1.1-7443]]
> > at
> org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:112)
> > at
> org.apache.catalina.core.StandardService.initInternal(StandardService.java:552)
> > at
> org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:107)
> > at
> org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:875)
> > at
> org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:107)
> > at org.apache.catalina.startup.Catalina.load(Catalina.java:639)
> > at org.apache.catalina.startup.Catalina.load(Catalina.java:662)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:498)
> > at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:309)
> > at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:492)
> > Caused by: org.apache.catalina.LifecycleException: Protocol handler
> initialization failed
> > at
> org.apache.catalina.connector.Connector.initInternal(Connector.java:995)
> > at
> org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:107)
> > ... 12 more
> > Caused by: java.lang.IllegalArgumentException: Cannot store
> non-PrivateKeys
> > at org.apache.tomcat.util.net
> .AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:100)
> > at org.apache.tomcat.util.net
> .AbstractJsseEndpoint.initialiseSsl(AbstractJsseEndpoint.java:72)
> > at org.apache.tomcat.util.net
> .NioEndpoint.bind(NioEndpoint.java:244)
> > at org.apache.tomcat.util.net
> .AbstractEndpoint.init(AbstractEndpoint.java:1105)
> > at org.apache.tomcat.util.net
> .AbstractJsseEndpoint.init(AbstractJsseEndpoint.java:224)
> > at
> org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:581)
> > at
> org.apache.coyote.http11.AbstractHttp11Protocol.init(AbstractHttp11Protocol.java:68)
> > at
> org.apache.catalina.connector.Connector.initInternal(Connector.java:993)
> > ... 13 more
>
> Caused by: java.security.KeyStoreException:
Cannot store non-PrivateKeys
>
If you pasted the full stack trace, then here we have the last "caused by",
showing one issue
> at
> sun.security.provider.JavaKeyStore.engineSetKeyEntry(JavaKeyStore.java:261)
> > at
> sun.security.provider.JavaKeyStore$JKS.engineSetKeyEntry(JavaKeyStore.java:56)
> > at
> sun.security.provider.KeyStoreDelegator.engineSetKeyEntry(KeyStoreDelegator.java:117)
> > at
> sun.security.provider.JavaKeyStore$DualFormatJKS.engineSetKeyEntry(JavaKeyStore.java:70)
> > at java.security.KeyStore.setKeyEntry(KeyStore.java:1140)
> > at org.apache.tomcat.util.net
> .SSLUtilBase.getKeyManagers(SSLUtilBase.java:313)
> > at org.apache.tomcat.util.net
> .SSLUtilBase.createSSLContext(SSLUtilBase.java:239)
> > at org.apache.tomcat.util.net
> .AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:98)
> > ... 20 more
> >
>
>
> I suspect that I will need to bring the Tomcat up to 8.5.57 here, too,
> before it will work. Presumably, that means tripping the same process
> that made the mess on the test spot instance.
>
> It seems that when the unwanted update happened,
>
> 1. /etc/tomcat8/server.xml was left at least relatively undisturbed: our
> connectors were undamaged.
>
> 2. /etc/tomcat8/tomcat-users.xml was also left undisturbed.
>
> 3. /var/lib/tomcat8/webapps/manager/WEB-INF/web.xml was also left at
> least relatively undisturbed: because of the size of our WAR files, we
> increase the max-file-size and max-request-size from 50MB to 500M, and
> that was as we left it.
>
> 4. /var/lib/tomcat8/webapps/manager/META-INF/context.xml, however, was
> reset to the "factory" state, with the RemoteAddrValve active.
>
> 5. The default ROOT context overlaid our ROOT context, leaving
> /var/lib/tomcat8/webapps/ROOT filled with both our files and the default
> ones. Our ROOT.war, on the other hand, was left intact, and if I stop
> Tomcat, remove the ROOT context directory, and then restart Tomcat, it
> does unpack our ROOT.war correctly.
>
> 6. The /var/lib/tomcat8/webapps/examples context directory, which we
> always remove, was reinstalled.
>
> Can anybody make sense of why some things changed, while others were
> left alone?
>
