Hi Josef and Kotaro, Thanks for identifying this scenario. I am away from the office for a bit but will try to review Kotaro’s changes in the linked PR. The regression is within Jetty’s code, and requires a new API to be invoked. NiFi does not have an existing method to configure a specific key to use within the keystore, and thus has always encouraged the use of a keystore with a single certificate and key (PrivateKeyEntry).
However, I will note that the initial scenario described by Josef seems to use a wildcard certificate, and this is explicitly mentioned in the documentation as not supported and discouraged [1]. > Wildcard certificates (i.e. two nodes node1.nifi.apache.org and > node2.nifi.apache.org being assigned the same certificate with a CN or SAN > entry of *.nifi.apache.org) are not officially supported and not recommended. > There are numerous disadvantages to using wildcard certificates, and a > cluster working with wildcard certificates has occurred in previous versions > out of lucky accidents, not intentional support. Wildcard SAN entries are > acceptable if each cert maintains an additional unique SAN entry and CN entry. I understand the challenges around automating key and certificate management and regenerating/expiring certificates appropriately. The TLS Toolkit exists to assist with this process, and there are ongoing improvements being made. However, fully supporting wildcard certificates would require substantial refactoring in the core framework and is not planned for any immediate attention. [1] https://nifi.apache.org/docs/nifi-docs/html/toolkit-guide.html#wildcard_certificates Andy LoPresto [email protected] [email protected] He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Aug 19, 2020, at 11:13 AM, Kotaro Terada <[email protected]> wrote: > > Hi Josef and teams, > > I encountered the same problem, and I have created a patch to fix it [1]. > > I guess the only way to fix the problem is to apply the patch and rebuild > NiFi, since the current implementation unfortunately doesn't seem to support > keystores with multiple certificates. Could someone please give support to > review the PR and proceed to fix it? > > [1] https://issues.apache.org/jira/browse/NIFI-7730 > <https://issues.apache.org/jira/browse/NIFI-7730> > > Thanks, > Kotaro > > > On Thu, Aug 20, 2020 at 12:51 AM <[email protected] > <mailto:[email protected]>> wrote: > Hi guys > > > > As we are waiting for some fixed bugs in NiFI 1.12.0, we upgraded today from > 1.11.4 to the newest version on one of our secured test single VM instances. > However, NiFi crashed during startup, error message below. It tells us that > KeyStores with multiple certificates are not supported. As you know we have > to use two keystores (keystore & truststore): > > Keystore with PrivateKey and Signed Cert -> only one Cert, the one belongs to > the PrivateKey (picture far below) > Truststore Keystore with CA Certs -> Multiple CA certs as we have imported > the cacerts from linux > > > I see two potential issues now, but I didn’t found the time to execute > further tests. > > > > We don’t have multiple certs in the keystore with the privateKey as you can > see in the picture far below, but of course we have SAN (Subject Alternative > Names) as we have ton’s of NiFi instances running and it’s more than annoying > to configure/generate a keypair for each instance. So the workaround was to > insert all our NiFi instances as SAN and that way we were able to use one > single keystore for all our NiFi instances (some of them are even clustered, > some not). However my assumption is that the mentioned workaround potentially > breaks now NiFi, this was working until NiFi 1.11.4. We know from security > perspective the workaround is/was not ideal, but we don’t have the manpower > to generate manually that many certs every 1-2 years when the certs are > expiring and it’s anyway completely separated from public networks. > > > > In the truststore we have multiple certs, but that’s very common that you use > eg. Linux cacerts as template for that. > > > > So to sum up, we can’t start NiFi anymore after the upgrade – any thoughts > how to fix the issue with the keystores? Or shall I open a bugticket on Jira? > > > > Cheers Josef > > > > 2020-08-19 16:23:43,334 INFO [main] o.e.jetty.server.handler.ContextHandler > Started > o.e.j.w.WebAppContext@2a4f5433{nifi-error,/,file:///opt/nifi-1.12.0/work/jetty/nifi-web-error-1.12.0.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.12.0.nar-unpacked/NAR-INF/bundled-dependencies/nifi-web-error-1.12.0.war} > > 2020-08-19 16:23:43,346 INFO [main] o.e.jetty.util.ssl.SslContextFactory > x509=X509@5d22a04d(1,h=[nifi-01.root.net <http://nifi-01.root.net/>, > nifi-02.root.net <http://nifi-02.root.net/>, nifi-03.root.net > <http://nifi-03.root.net/>, nifi-04.root.net <http://nifi-04.root.net/>, > nifi-05.root.net <http://nifi-05.root.net/>, nifi-06.root.net > <http://nifi-06.root.net/>, nifi-07.root.net <http://nifi-07.root.net/>, > nifi-08.root.net <http://nifi-08.root.net/>, nifi-09.root.net > <http://nifi-09.root.net/>, nifi-10.root.net <http://nifi-10.root.net/>, > nifi-11.root.net <http://nifi-11.root.net/>, nifi-12.root.net > <http://nifi-12.root.net/>, nifi-13.root.net <http://nifi-13.root.net/>, > nifi-14.root.net <http://nifi-14.root.net/>, nifi-15.root.net > <http://nifi-15.root.net/>, nifi-16.root.net <http://nifi-16.root.net/>, > nifi-17.root.net <http://nifi-17.root.net/>, nifi-18.root.net > <http://nifi-18.root.net/>, nifi-19.root.net <http://nifi-19.root.net/>, > nifi-20.root.net <http://nifi-20.root.net/>, nifi-21.root.net > <http://nifi-21.root.net/>, nifi-22.root.net <http://nifi-22.root.net/>, > nifi-23.root.net <http://nifi-23.root.net/>, nifi-24.root.net > <http://nifi-24.root.net/>, nifi-94.root.net <http://nifi-94.root.net/>, > nifi-95.root.net <http://nifi-95.root.net/>, nifi-96.root.net > <http://nifi-96.root.net/>, nifi-97.root.net <http://nifi-97.root.net/>, > nifi-98.root.net <http://nifi-98.root.net/>, nifi-99.root.net > <http://nifi-99.root.net/>, nifi-01.root.net <http://nifi-01.root.net/>, > nifi-02.root.net <http://nifi-02.root.net/>, nifi-03.root.net > <http://nifi-03.root.net/>, nifi-04.root.net <http://nifi-04.root.net/>, > nifi-05.root.net <http://nifi-05.root.net/>, nifi-06.root.net > <http://nifi-06.root.net/>, nifi-07.root.net <http://nifi-07.root.net/>, > nifi-08.root.net <http://nifi-08.root.net/>, nifi-09.root.net > <http://nifi-09.root.net/>, nifi-lan-01.root.net > <http://nifi-lan-01.root.net/>, nifi-lan-02.root.net > <http://nifi-lan-02.root.net/>, nifi-lan-03.root.net > <http://nifi-lan-03.root.net/>, nifi-lan-04.root.net > <http://nifi-lan-04.root.net/>, nifi-lan-05.root.net > <http://nifi-lan-05.root.net/>, nifi-lan-06.root.net > <http://nifi-lan-06.root.net/>, nifi-lan-07.root.net > <http://nifi-lan-07.root.net/>, nifi-lan-08.root.net > <http://nifi-lan-08.root.net/>, nifi-lan-09.root.net > <http://nifi-lan-09.root.net/>, nifi-lan-10.root.net > <http://nifi-lan-10.root.net/>, nifi-lan-11.root.net > <http://nifi-lan-11.root.net/>, nifi-lan-12.root.net > <http://nifi-lan-12.root.net/>, nifi-lan-13.root.net > <http://nifi-lan-13.root.net/>, nifi-lan-14.root.net > <http://nifi-lan-14.root.net/>, nifi-lan-15.root.net > <http://nifi-lan-15.root.net/>, nifi-lan-16.root.net > <http://nifi-lan-16.root.net/>, nifi-lan-17.root.net > <http://nifi-lan-17.root.net/>, nifi-lan-18.root.net > <http://nifi-lan-18.root.net/>, nifi-lan-19.root.net > <http://nifi-lan-19.root.net/>, nifi-lan-20.root.net > <http://nifi-lan-20.root.net/>, nifi-lan-21.root.net > <http://nifi-lan-21.root.net/>, nifi-lan-22.root.net > <http://nifi-lan-22.root.net/>, nifi-lan-23.root.net > <http://nifi-lan-23.root.net/>, nifi-lan-24.root.net > <http://nifi-lan-24.root.net/>, nifi-lan-94.root.net > <http://nifi-lan-94.root.net/>, nifi-lan-95.root.net > <http://nifi-lan-95.root.net/>, nifi-lan-96.root.net > <http://nifi-lan-96.root.net/>, nifi-lan-97.root.net > <http://nifi-lan-97.root.net/>, nifi-lan-98.root.net > <http://nifi-lan-98.root.net/>, nifi-lan-99.root.net > <http://nifi-lan-99.root.net/>, nifi-lan-01.root.net > <http://nifi-lan-01.root.net/>, nifi-lan-02.root.net > <http://nifi-lan-02.root.net/>, nifi-lan-03.root.net > <http://nifi-lan-03.root.net/>, nifi-lan-04.root.net > <http://nifi-lan-04.root.net/>, nifi-lan-05.root.net > <http://nifi-lan-05.root.net/>, nifi-lan-06.root.net > <http://nifi-lan-06.root.net/>, nifi-lan-07.root.net > <http://nifi-lan-07.root.net/>, nifi-lan-08.root.net > <http://nifi-lan-08.root.net/>, nifi-lan-09.root.net > <http://nifi-lan-09.root.net/>],w=[]) for > SslContextFactory@4e32e1f9[provider=null,keyStore=file:///etc/nifi/certs/nifi_wildcard_keystore.jks,trustStore=file:///etc/nifi/certs/sc-truststore.jks] > > 2020-08-19 16:23:43,348 WARN [main] org.apache.nifi.web.server.JettyServer > Failed to start web server... shutting down. > > java.lang.IllegalStateException: KeyStores with multiple certificates are not > supported on the base class org.eclipse.jetty.util.ssl.SslContextFactory. > (Use org.eclipse.jetty.util.ssl.SslContextFactory$Server or > org.eclipse.jetty.util.ssl.SslContextFactory$Client instead) > > at > org.eclipse.jetty.util.ssl.SslContextFactory.newSniX509ExtendedKeyManager(SslContextFactory.java:1275) > > at > org.eclipse.jetty.util.ssl.SslContextFactory.getKeyManagers(SslContextFactory.java:1256) > > at > org.eclipse.jetty.util.ssl.SslContextFactory.load(SslContextFactory.java:374) > > at > org.eclipse.jetty.util.ssl.SslContextFactory.doStart(SslContextFactory.java:245) > > at > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) > > at > org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) > > at > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) > > at > org.eclipse.jetty.server.SslConnectionFactory.doStart(SslConnectionFactory.java:92) > > at > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) > > at > org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) > > at > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) > > at > org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:320) > > at > org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:81) > > at > org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:231) > > at > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) > > at org.eclipse.jetty.server.Server.doStart(Server.java:385) > > at > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) > > at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:1058) > > at org.apache.nifi.NiFi.<init>(NiFi.java:158) > > at org.apache.nifi.NiFi.<init>(NiFi.java:72) > > at org.apache.nifi.NiFi.main(NiFi.java:301) > > 2020-08-19 16:23:43,348 INFO [Thread-1] org.apache.nifi.NiFi Initiating > shutdown of Jetty web server... > > > > <image001.png> >
