Hello users, I am new to Tomcat and to this mailing list. Looked far and wide for a solution to my problem, but couldn't find anything effective. I found other folks asking about similar issues. I then looked through the source and think I got a solution that I'd like to share as a patch.
The problem is this: Trying to use Apache Tomcat with an OpenSSL Engine that has proprietary private ECC key format fails. The private key file is not PEM, and only this specific OpenSSL Engine can load such a private ECC key. When the server.xml configuration includes reference to a proprietary format private ECC key, in a Service/Connector/SSLHostConfig/Certificate/certificateKeyFile, the run-time fails to initialize a new SSL context. As a result, TLS doesn't get established, connection fails. I have tried Tomcat7, 9 and 10. To illustrate the configuration in server.xml, it includes elements like these: <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="MySslEngine" /> <Service name="Catalina"> <!-- Define an SSL Coyote HTTP/1.1 Connector on port 8443 --> <Connector protocol="org.apache.coyote.http11.Http11NioProtocol" port="8443" maxThreads="200" scheme="https" SSLEnabled="true"> <SSLHostConfig caCertificateFile="my-keys/ca.pem" certificateVerification="require" certificateVerificationDepth="10"> <Certificate certificateKeyFile="my-keys/server.key" certificateFile="my-keys/server.pem" certificateChainFile="my-keys/server-chain.pem" type="EC" /> </SSLHostConfig> </Connector> <!-- ... --> </Service> The logs may include lines like these: 05-Mar-2021 14:37:07.175 INFO [main] org.apache.tomcat.util.net.openssl.OpenSSLUtil.getKeyManagers The certificate [/opt/my-keys/server.pem] or its private key [/opt/my-keys/server.key] could not be processed using a JSSE key manager and will be given directly to OpenSSL 05-Mar-2021 14:37:07.176 WARNING [main] org.apache.tomcat.util.net.openssl.OpenSSLContext.init Error initializing SSL context java.lang.Exception: Unable to load certificate key /opt/my-keys/server.key (error:0909006C:PEM routines:get_name:no start line) at org.apache.tomcat.jni.SSLContext.setCertificate(Native Method) at org.apache.tomcat.util.net.openssl.OpenSSLContext.addCertificate(OpenSSLContext.java:379) at org.apache.tomcat.util.net.openssl.OpenSSLContext.init(OpenSSLContext.java:250) at org.apache.tomcat.util.net.SSLUtilBase.createSSLContext(SSLUtilBase.java:246) at org.apache.tomcat.util.net.AprEndpoint.createSSLContext(AprEndpoint.java:401) at org.apache.tomcat.util.net.AprEndpoint.bind(AprEndpoint.java:367) at org.apache.tomcat.util.net.AbstractEndpoint.bindWithCleanup(AbstractEndpoint.java:1164) at org.apache.tomcat.util.net.AbstractEndpoint.init(AbstractEndpoint.java:1177) at org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:574) at org.apache.coyote.http11.AbstractHttp11Protocol.init(AbstractHttp11Protocol.java:82) at org.apache.catalina.connector.Connector.initInternal(Connector.java:1052) at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136) at org.apache.catalina.core.StandardService.initInternal(StandardService.java:558) at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136) at org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:1045) at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136) at org.apache.catalina.startup.Catalina.load(Catalina.java:747) at org.apache.catalina.startup.Catalina.load(Catalina.java:769) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:302) at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:472) My understanding of the root cause is that Tomcat doesn't support a proprietary format of the private ECC key. It insists that the key be in PEM format, in a file or in a keystore. What I needed was support for the "engine" key format. Similar to the feature of the "openssl digest" command in the following invocation: openssl dgst \ -sign my-keys/server.key \ *-keyform ENGINE* \ -engine MySslEngine \ -out signature.bin \ my-input When the key has the form "engine", the key is loaded using the ENGINE_load_private_key API ( https://www.openssl.org/docs/man1.1.0/man3/ENGINE_load_private_key.html). I have come up with a small change to the Tomcat Native library that resolves the problem for me. It is not as general as the "engine" key form in the openssl command line. The change below simply attempts to load the private key through the ENGINE_load_private_key if load_pem_key fails. Please consider the change as a patch to the Tomcat Native library: --- tomcat-native-1.2.26-src/native/include/ssl_private.h 2020-12-10 09:09:19.000000000 -0800 +++ tomcat-native-1.2.26-src.changed/native/include/ssl_private.h 2021-03-08 15:35:56.516968836 -0800 @@ -51,6 +51,7 @@ */ #ifndef OPENSSL_NO_ENGINE #include <openssl/engine.h> +extern ENGINE *tcn_ssl_engine; #endif #ifndef RAND_MAX --- tomcat-native-1.2.26-src/native/src/sslcontext.c 2020-12-10 09:09:19.000000000 -0800 +++ tomcat-native-1.2.26-src.changed/native/src/sslcontext.c 2021-03-08 15:38:18.264651853 -0800 @@ -1034,7 +1034,13 @@ } } else { - if ((c->keys[idx] = load_pem_key(c, key_file)) == NULL) { + if ((c->keys[idx] = load_pem_key(c, key_file)) == NULL +#ifndef OPENSSL_NO_ENGINE + && tcn_ssl_engine && + (c->keys[idx] = ENGINE_load_private_key(tcn_ssl_engine, key_file, + NULL, NULL)) == NULL +#endif + ) { ERR_error_string(SSL_ERR_get(), err); tcn_Throw(e, "Unable to load certificate key %s (%s)", key_file, err); Thanks, Edin Hodzic