Hi Andy & Kotaro Thank you for your comments. So this means we can’t upgrade to nifi 1.12.0 :-( (except if we change the certs, which is no option at the moment).
@Andy: I’m aware of the wildcard certificate notes from the documentation. We don’t have a wildcard certificate with a ‘*’ sign in it. We are using SAN with multiple explicit nodenames like nifi1.domain.com, nifi2.domain.com. Does this causes the same issues as you mentioned with a real wildcard or would that be fine? This isn’t clear for me when reading the documentation. We can’t use the NiFiToolkit as we have to use our own Corporate CA which is at the moment not automatically provisionable, so the CSRs need to be done manually and it would be a huge work to create and maintain the certificates. Cheers Josef From: Andy LoPresto <alopre...@apache.org> Reply to: "users@nifi.apache.org" <users@nifi.apache.org> Date: Thursday, 20 August 2020 at 01:06 To: "users@nifi.apache.org" <users@nifi.apache.org> Subject: Re: NiFi 1.12.0 - KeyStores with multiple certificates are not supported Hi Josef and Kotaro, Thanks for identifying this scenario. I am away from the office for a bit but will try to review Kotaro’s changes in the linked PR. The regression is within Jetty’s code, and requires a new API to be invoked. NiFi does not have an existing method to configure a specific key to use within the keystore, and thus has always encouraged the use of a keystore with a single certificate and key (PrivateKeyEntry). However, I will note that the initial scenario described by Josef seems to use a wildcard certificate, and this is explicitly mentioned in the documentation as not supported and discouraged [1]. Wildcard certificates (i.e. two nodes node1.nifi.apache.org<http://node1.nifi.apache.org> and node2.nifi.apache.org<http://node2.nifi.apache.org> being assigned the same certificate with a CN or SAN entry of *.nifi.apache.org<http://nifi.apache.org>) are not officially supported and not recommended. There are numerous disadvantages to using wildcard certificates, and a cluster working with wildcard certificates has occurred in previous versions out of lucky accidents, not intentional support. Wildcard SAN entries are acceptable if each cert maintains an additional unique SAN entry and CN entry. I understand the challenges around automating key and certificate management and regenerating/expiring certificates appropriately. The TLS Toolkit exists to assist with this process, and there are ongoing improvements being made. However, fully supporting wildcard certificates would require substantial refactoring in the core framework and is not planned for any immediate attention. [1] https://nifi.apache.org/docs/nifi-docs/html/toolkit-guide.html#wildcard_certificates Andy LoPresto alopre...@apache.org<mailto:alopre...@apache.org> alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 On Aug 19, 2020, at 11:13 AM, Kotaro Terada <kota...@apache.org<mailto:kota...@apache.org>> wrote: Hi Josef and teams, I encountered the same problem, and I have created a patch to fix it [1]. I guess the only way to fix the problem is to apply the patch and rebuild NiFi, since the current implementation unfortunately doesn't seem to support keystores with multiple certificates. Could someone please give support to review the PR and proceed to fix it? [1] https://issues.apache.org/jira/browse/NIFI-7730 Thanks, Kotaro On Thu, Aug 20, 2020 at 12:51 AM <josef.zahn...@swisscom.com<mailto:josef.zahn...@swisscom.com>> wrote: Hi guys As we are waiting for some fixed bugs in NiFI 1.12.0, we upgraded today from 1.11.4 to the newest version on one of our secured test single VM instances. However, NiFi crashed during startup, error message below. It tells us that KeyStores with multiple certificates are not supported. As you know we have to use two keystores (keystore & truststore): 1. Keystore with PrivateKey and Signed Cert -> only one Cert, the one belongs to the PrivateKey (picture far below) 2. Truststore Keystore with CA Certs -> Multiple CA certs as we have imported the cacerts from linux I see two potential issues now, but I didn’t found the time to execute further tests. We don’t have multiple certs in the keystore with the privateKey as you can see in the picture far below, but of course we have SAN (Subject Alternative Names) as we have ton’s of NiFi instances running and it’s more than annoying to configure/generate a keypair for each instance. So the workaround was to insert all our NiFi instances as SAN and that way we were able to use one single keystore for all our NiFi instances (some of them are even clustered, some not). However my assumption is that the mentioned workaround potentially breaks now NiFi, this was working until NiFi 1.11.4. We know from security perspective the workaround is/was not ideal, but we don’t have the manpower to generate manually that many certs every 1-2 years when the certs are expiring and it’s anyway completely separated from public networks. In the truststore we have multiple certs, but that’s very common that you use eg. Linux cacerts as template for that. So to sum up, we can’t start NiFi anymore after the upgrade – any thoughts how to fix the issue with the keystores? Or shall I open a bugticket on Jira? Cheers Josef 2020-08-19 16:23:43,334 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@2a4f5433{nifi-error,/,file:///opt/nifi-1.12.0/work/jetty/nifi-web-error-1.12.0.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.12.0.nar-unpacked/NAR-INF/bundled-dependencies/nifi-web-error-1.12.0.war} 2020-08-19 16:23:43,346 INFO [main] o.e.jetty.util.ssl.SslContextFactory x509=X509@5d22a04d(1,h=[nifi-01.root.net<http://nifi-01.root.net/>, nifi-02.root.net<http://nifi-02.root.net/>, nifi-03.root.net<http://nifi-03.root.net/>, nifi-04.root.net<http://nifi-04.root.net/>, nifi-05.root.net<http://nifi-05.root.net/>, nifi-06.root.net<http://nifi-06.root.net/>, nifi-07.root.net<http://nifi-07.root.net/>, nifi-08.root.net<http://nifi-08.root.net/>, nifi-09.root.net<http://nifi-09.root.net/>, nifi-10.root.net<http://nifi-10.root.net/>, nifi-11.root.net<http://nifi-11.root.net/>, nifi-12.root.net<http://nifi-12.root.net/>, nifi-13.root.net<http://nifi-13.root.net/>, nifi-14.root.net<http://nifi-14.root.net/>, nifi-15.root.net<http://nifi-15.root.net/>, nifi-16.root.net<http://nifi-16.root.net/>, nifi-17.root.net<http://nifi-17.root.net/>, nifi-18.root.net<http://nifi-18.root.net/>, nifi-19.root.net<http://nifi-19.root.net/>, nifi-20.root.net<http://nifi-20.root.net/>, nifi-21.root.net<http://nifi-21.root.net/>, nifi-22.root.net<http://nifi-22.root.net/>, nifi-23.root.net<http://nifi-23.root.net/>, nifi-24.root.net<http://nifi-24.root.net/>, nifi-94.root.net<http://nifi-94.root.net/>, nifi-95.root.net<http://nifi-95.root.net/>, nifi-96.root.net<http://nifi-96.root.net/>, nifi-97.root.net<http://nifi-97.root.net/>, nifi-98.root.net<http://nifi-98.root.net/>, nifi-99.root.net<http://nifi-99.root.net/>, nifi-01.root.net<http://nifi-01.root.net/>, nifi-02.root.net<http://nifi-02.root.net/>, nifi-03.root.net<http://nifi-03.root.net/>, nifi-04.root.net<http://nifi-04.root.net/>, nifi-05.root.net<http://nifi-05.root.net/>, nifi-06.root.net<http://nifi-06.root.net/>, nifi-07.root.net<http://nifi-07.root.net/>, nifi-08.root.net<http://nifi-08.root.net/>, nifi-09.root.net<http://nifi-09.root.net/>, nifi-lan-01.root.net<http://nifi-lan-01.root.net/>, nifi-lan-02.root.net<http://nifi-lan-02.root.net/>, nifi-lan-03.root.net<http://nifi-lan-03.root.net/>, nifi-lan-04.root.net<http://nifi-lan-04.root.net/>, nifi-lan-05.root.net<http://nifi-lan-05.root.net/>, nifi-lan-06.root.net<http://nifi-lan-06.root.net/>, nifi-lan-07.root.net<http://nifi-lan-07.root.net/>, nifi-lan-08.root.net<http://nifi-lan-08.root.net/>, nifi-lan-09.root.net<http://nifi-lan-09.root.net/>, nifi-lan-10.root.net<http://nifi-lan-10.root.net/>, nifi-lan-11.root.net<http://nifi-lan-11.root.net/>, nifi-lan-12.root.net<http://nifi-lan-12.root.net/>, nifi-lan-13.root.net<http://nifi-lan-13.root.net/>, nifi-lan-14.root.net<http://nifi-lan-14.root.net/>, nifi-lan-15.root.net<http://nifi-lan-15.root.net/>, nifi-lan-16.root.net<http://nifi-lan-16.root.net/>, nifi-lan-17.root.net<http://nifi-lan-17.root.net/>, nifi-lan-18.root.net<http://nifi-lan-18.root.net/>, nifi-lan-19.root.net<http://nifi-lan-19.root.net/>, nifi-lan-20.root.net<http://nifi-lan-20.root.net/>, nifi-lan-21.root.net<http://nifi-lan-21.root.net/>, nifi-lan-22.root.net<http://nifi-lan-22.root.net/>, nifi-lan-23.root.net<http://nifi-lan-23.root.net/>, nifi-lan-24.root.net<http://nifi-lan-24.root.net/>, nifi-lan-94.root.net<http://nifi-lan-94.root.net/>, nifi-lan-95.root.net<http://nifi-lan-95.root.net/>, nifi-lan-96.root.net<http://nifi-lan-96.root.net/>, nifi-lan-97.root.net<http://nifi-lan-97.root.net/>, nifi-lan-98.root.net<http://nifi-lan-98.root.net/>, nifi-lan-99.root.net<http://nifi-lan-99.root.net/>, nifi-lan-01.root.net<http://nifi-lan-01.root.net/>, nifi-lan-02.root.net<http://nifi-lan-02.root.net/>, nifi-lan-03.root.net<http://nifi-lan-03.root.net/>, nifi-lan-04.root.net<http://nifi-lan-04.root.net/>, nifi-lan-05.root.net<http://nifi-lan-05.root.net/>, nifi-lan-06.root.net<http://nifi-lan-06.root.net/>, nifi-lan-07.root.net<http://nifi-lan-07.root.net/>, nifi-lan-08.root.net<http://nifi-lan-08.root.net/>, nifi-lan-09.root.net<http://nifi-lan-09.root.net/>],w=[]) for SslContextFactory@4e32e1f9[provider=null,keyStore=file:///etc/nifi/certs/nifi_wildcard_keystore.jks,trustStore=file:///etc/nifi/certs/sc-truststore.jks<file:///etc/nifi/certs/nifi_wildcard_keystore.jks,trustStore=file:/etc/nifi/certs/sc-truststore.jks>] 2020-08-19 16:23:43,348 WARN [main] org.apache.nifi.web.server.JettyServer Failed to start web server... shutting down. java.lang.IllegalStateException: KeyStores with multiple certificates are not supported on the base class org.eclipse.jetty.util.ssl.SslContextFactory. (Use org.eclipse.jetty.util.ssl.SslContextFactory$Server or org.eclipse.jetty.util.ssl.SslContextFactory$Client instead) at org.eclipse.jetty.util.ssl.SslContextFactory.newSniX509ExtendedKeyManager(SslContextFactory.java:1275) at org.eclipse.jetty.util.ssl.SslContextFactory.getKeyManagers(SslContextFactory.java:1256) at org.eclipse.jetty.util.ssl.SslContextFactory.load(SslContextFactory.java:374) at org.eclipse.jetty.util.ssl.SslContextFactory.doStart(SslContextFactory.java:245) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) at org.eclipse.jetty.server.SslConnectionFactory.doStart(SslConnectionFactory.java:92) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:320) at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:81) at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:231) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.server.Server.doStart(Server.java:385) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:1058) at org.apache.nifi.NiFi.<init>(NiFi.java:158) at org.apache.nifi.NiFi.<init>(NiFi.java:72) at org.apache.nifi.NiFi.main(NiFi.java:301) 2020-08-19 16:23:43,348 INFO [Thread-1] org.apache.nifi.NiFi Initiating shutdown of Jetty web server... <image001.png>
smime.p7s
Description: S/MIME Cryptographic Signature
