The nifi.security.keyPasswd was not filled in, so it looked like this (which is the default configuration):
nifi.security.keyPasswd= On Wed, Mar 4, 2020 at 11:36 AM Endre Kovacs <andrewsmit...@protonmail.com.invalid> wrote: > Hi Nathan, > > There is already a ticket about this: > https://issues.apache.org/jira/browse/NIFI-7219 > > Best regards, > Endre > > Sent with ProtonMail Secure Email. > > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ > On Wednesday, March 4, 2020 5:25 PM, Nathan Gough <thena...@gmail.com> > wrote: > > > I've opened https://issues.apache.org/jira/browse/NIFI-7223 to track and > > I'm working on a fix for this. > > > > Nathan > > > > On Tue, Mar 3, 2020 at 6:17 PM Nathan Gough thena...@gmail.com wrote: > > > > > Hi Joe, > > > Just to confirm here - was the nifi.security.keyPasswd not defined at > all > > > in your nifi.properties? Did you have to add the property and give it > the > > > correct value? Or was it in the nifi.properties file but blank? Or > were the > > > keyPasswd and keystorePasswd different values? > > > Thanks, > > > Nathan > > > On Tue, Mar 3, 2020 at 3:38 PM Joe Gresock jgres...@gmail.com wrote: > > > > > > > Yep, setting the nifi.security.keyPasswd to the same as > > > > nifi.security.keystorePasswd fixed it. Thanks for the insight, Endre! > > > > On Tue, Mar 3, 2020 at 2:01 PM Joe Witt joe.w...@gmail.com wrote: > > > > > > > > > relevant change I believe is here: > > > > > > > > > https://github.com/apache/nifi/commit/46d3b6b0dc28f04da124be7685f82bec52e88775 > > > > > > > > > and > > > > > is from https://issues.apache.org/jira/browse/NIFI-6927 > > > > > It looks to me like this was fixing an improper naming/usage issue > > > > > that > > > > > has been present but if so we probably should have addressed not > in this > > > > > bug fix line. Will defer to Troy/Andy for more context and next > steps > > > > > On Tue, Mar 3, 2020 at 5:53 AM Joe Witt joe.w...@gmail.com wrote: > > > > > > > > > > > If accurate....We need to look into whether this was a mistake > and > > > > > > fix it > > > > > > > > > > > if so. And we need to reflect this in the migration guide > > > > > > On Tue, Mar 3, 2020 at 4:40 AM Ryan Ward ryan.wa...@gmail.com > > > > > > wrote: > > > > > > > > > > > > Endre - thanks that was it > > > > > > > On Tue, Mar 3, 2020 at 6:50 AM Endre Kovacs > > > > > > > andrewsmit...@protonmail.com.invalid wrote: > > > > > > > > > > > > > > > Hi, > > > > > > > > One additional thing: > > > > > > > > we encountered something strange as well: > > > > > > > > on 1.11.2 clustered, kerberized: request replication worked > well. > > > > > > > > on 1.11.3 clustered, kerberized: request replication did not > work, > > > > > > > > unless > > > > > > > > you specify, and set > > > > > > > > nifi.security.keyPasswd > > > > > > > > to the very same password as the > > > > > > > > nifi.security.keystorePasswd > > > > > > > > For us this resolved the issue. > > > > > > > > Best regards, > > > > > > > > Endre > > > > > > > > Sent with ProtonMail Secure Email. > > > > > > > > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ > > > > > > > > On Tuesday, March 3, 2020 12:40 PM, Ryan Ward < > > > > > > > > ryan.wa...@gmail.com> > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi Joe - Did you resolve your issue? If so I am wondering > what > > > > > > > > > the > > > > > > > > > fix > > > > > > > > > > > > > was as I'm seeing the same error on my cluster. > > > > > > > > > > > > > > > > > On Thu, Feb 27, 2020 at 3:13 AM Endre Kovacs < > > > > > > > > > andrewsmit...@protonmail.com.invalid> wrote: > > > > > > > > > > > > > > > > > > > Hi Joe, > > > > > > > > > > > > > > > > > > > > 1. Have you tried connecting/debugging with openssl? > From one > > > > > > > > > > pod > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > > > the other: > > > > > > > > > > > > > > > > > > (openssl s_client -debug -CAfile > > > > > > > > > > > > > > > > > > > > > > > > > > ca-bundle-signing-node-certificates.crt -cert > my-client-cert.crt > > > > > > > > -connect > > > > > > > > nifi-3.nifi-headless.lizardspock.svc.cluster.local:6007) > > > > > > > > > > > > > > > > > > 2. certs can also be verified by: > > > > > > > > > > openssl verify -verbose -CAfile ca-bundle.crt > > > > > > > > > > my-client-cert.crt > > > > > > > > > > > > > > > > > > > > > > > > > 3. Can you check if no intermediary CAs are missing > from the > > > > > > > > > > nodes > > > > > > > > > > > > > > > > > > > > > > > truststore? > > > > > > > > > > > > > > > > > > 4. This exception is coming from inter-node > communication > > > > > > > > > > (replication > > > > > > > > > > of request from one node to the other). This means > that it is > > > > > > > > > > unrelated > > > > > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > external user's authentication by client certificate. The > question > > > > > > > > is: > > > > > > > > > > > > is > > > > > > > > > > > > > > > your inter node communication secured by the trusted root CA > (that > > > > > > > > you > > > > > > > > > > > > are > > > > > > > > > > > > > > > sure that the CA cert is present in the trust store) or is it > > > > > > > > secured > > > > > > > > by > > > > > > > > > > > > > selfsigned CA (which's CA may be lacking from your > truststore)? > > > > > > > > > > > > > > > > > > 5. `nifi.security.needClientAuth` is not part of NiFi > > > > > > > > > > properties > > > > > > > > > > any > > > > > > > > > > > > > > > > > > > > > > > more. If SSL is turned on, and no > > > > > > > > `nifi.security.user.login.identity.provider` is set, then > client > > > > > > > > cert > > > > > > > > > > > > based > > > > > > > > > > > > > > > auth is the default. But supplying this property have no > > > > > > > > detrimental > > > > > > > > > > > > effect > > > > > > > > > > > > > > > anyhow. > > > > > > > > > > > > > > > > > > Best regards, > > > > > > > > > > Endre > > > > > > > > > > > > > > > > > > > > Sent with ProtonMail Secure Email. > > > > > > > > > > > > > > > > > > > > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ > > > > > > > > > > On Wednesday, February 26, 2020 6:22 PM, Joe Gresock > > > > > > > > > > > > > > > > > > > > > > > > > > jgres...@gmail.com wrote: > > > > > > > > > > > > > > > > > > > Were there any changes with how the trust store is > used in > > > > > > > > > > > 1.11.3? I > > > > > > > > > > > > > > had a > > > > > > > > > > > > > > > > > > > 1.11.0 deployment working with the following settings, > but > > > > > > > > > > > when I > > > > > > > > > > > > > deployed > > > > > > > > > > > > > > > > > > > 1.11.3, the cluster can't seem to replicate requests > to itself: > > > > > > > > > > > nifi.remote.input.host=<redacted> > > > > > > > > > > > nifi.remote.input.secure=true > > > > > > > > > > > nifi.remote.input.socket.port=32440 > > > > > > > > > > > nifi.remote.input.http.enabled=true > > > > > > > > > > > nifi.cluster.protocol.is.secure=true > > > > > > > > > > > nifi.cluster.is.node=true > > > > > > > > > nifi.cluster.node.address=nifi-3.nifi-headless.lizardspock.svc.cluster.local > > > > > > > > > > > > > > > nifi.cluster.node.protocol.port=6007 > > > > > > > > > nifi.web.https.host=nifi-3.nifi-headless.lizardspock.svc.cluster.local > > > > > > > > > > > > > > > nifi.web.https.port=8443 > > > > > > > > > > > nifi.security.keystore=./conf/keystore.jks > > > > > > > > > > > nifi.security.keystoreType=jks > > > > > > > > > > > nifi.security.keystorePasswd=<password> > > > > > > > > > > > nifi.security.keyPasswd= > > > > > > > > > > > nifi.security.truststore=./conf/truststore.jks > > > > > > > > > > > nifi.security.truststoreType=jks > > > > > > > > > > > nifi.security.truststorePasswd=<password> > > > > > > > > > > > nifi.security.needClientAuth=true > > > > > > > > > > > A trusted client cert that worked against the old > cluster is > > > > > > > > > > > getting > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > same trust error (PKIX path building failed). I've > verified > > > > > > > > > > > that > > > > > > > > > > > the > > > > > > > > > > > > > > > > client cert was issued by an issuer that is definitely > in the > > > > > > > > > > > ./conf/truststore.jks as a trustedCertEntry. > > > > > > > > > > > 2020-02-26 17:11:09,573 WARN [Replicate Request > Thread-7] > > > > > > > > > > > o.a.n.c.c.h.r.ThreadPoolRequestReplicator > > > > > > > > > > > javax.net.ssl.SSLHandshakeException: > > > > > > > > > > > sun.security.validator.ValidatorException: PKIX path > building > > > > > > > > > > > failed: > > > > > > > > > > > > > > > > > > > > sun.security.provider.certpath.SunCertPathBuilderException: > > > > > > > > > > > unable > > > > > > > > > > > > to > > > > > > > > > > > > > > > find > > > > > > > > > > > > > > > > > > > valid certification path to r > > > > > > > > > > > equested target > > > > > > > > > > > at > sun.security.ssl.Alerts.getSSLException(Alerts.java:192) > > > > > > > > > > > at > > > > > > > > > > > > sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946) > > > > > > > > > > > > > > > > at > sun.security.ssl.Handshaker.fatalSE(Handshaker.java:316) > > > > > > > > > > > at > sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310) > > > > > > > > > > > at > > > > > > > > > sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639) > > > > > > > > > > > > > > > at > > > > > > > > > sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223) > > > > > > > > > > > > > > > at > > > > > > > > > > > > sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037) > > > > > > > > > > > > > > > > at > > > > > > > > > > > > sun.security.ssl.Handshaker.process_record(Handshaker.java:965) > > > > > > > > > > > > > > > > at > > > > > > > > sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1064) > > > > > > > > > > > > > > > at > > > > > > > > > sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367) > > > > > > > > > > > > > > > at > > > > > > > > > sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395) > > > > > > > > > > > > > > > at > > > > > > > > > sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379) > > > > > > > > > > > > > > > at > > > > > > > > > okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:302) > > > > > > > > > > > > > > > at > > > > > > > > > okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:270) > > > > > > > > > > > > > > > at > > > > > > > > > okhttp3.internal.connection.RealConnection.connect(RealConnection.java:162) > > > > > > > > > > > > > > > at > > > > > > > > > okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:257) > > > > > > > > > > > > > > > at > > > > > > > > > okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:135) > > > > > > > > > > > > > > > at > > > > > > > > > okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:114) > > > > > > > > > > > > > > > at > > > > > > > > > okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42) > > > > > > > > > > > > > > > at > > > > > > > > > okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) > > > > > > > > > > > > > > > at > > > > > > > > > okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) > > > > > > > > > > > > > > > at > > > > > > > > > okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93) > > > > > > > > > > > > > > > at > > > > > > > > > okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) > > > > > > > > > > > > > > > at > > > > > > > > > okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) > > > > > > > > > > > > > > > at > > > > > > > > > okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) > > > > > > > > > > > > > > > at > > > > > > > > > okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) > > > > > > > > > > > > > > > at > > > > > > > > > okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:126) > > > > > > > > > > > > > > > at > > > > > > > > > okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) > > > > > > > > > > > > > > > at > > > > > > > > > okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) > > > > > > > > > > > > > > > at > > > > > > > > > > okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:200) > > > > > > > > > > > > > > > > at okhttp3.RealCall.execute(RealCall.java:77) > > > > > > > > > > > at > > > > > > > > > org.apache.nifi.cluster.coordination.http.replication.okhttp.OkHttpReplicationClient.replicate(OkHttpReplicationClient.java:143) > > > > > > > > > > > > > > > at > > > > > > > > > org.apache.nifi.cluster.coordination.http.replication.okhttp.OkHttpReplicationClient.replicate(OkHttpReplicationClient.java:137) > > > > > > > > > > > > > > > at > > > > > > > > > org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator.replicateRequest(ThreadPoolRequestReplicator.java:647) > > > > > > > > > > > > > > > at > > > > > > > > > org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator$NodeHttpRequest.run(ThreadPoolRequestReplicator.java:839) > > > > > > > > > > > > > > > at > > > > > > > > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > > > > > > > > > > > > > > > at > java.util.concurrent.FutureTask.run(FutureTask.java:266) > > > > > > > > > > > at > > > > > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > > > > > > > > > > > > > > at > > > > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > > > > > > > > > > > > > > at java.lang.Thread.run(Thread.java:748) > > > > > > > > > > > Caused by: sun.security.validator.ValidatorException: > PKIX path > > > > > > > > > > > building > > > > > > > > > > > > > > > > > > > > failed: > > > > > > > > > > > > sun.security.provider.certpath.SunCertPathBuilderException: > > > > > > > > > > > > > > unable > > > > > > > > > > > > > > > > > > > to find valid certification path to requested target > > > > > > > > > > > at > > > > > > > > > > > sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397) > > > > > > > > > > > > > > > > at > > > > > > > > > sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302) > > > > > > > > > > > > > > > at > > > > > > > > > > > > sun.security.validator.Validator.validate(Validator.java:262) > > > > > > > > > > > > > > > > at > > > > > > > > > sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:330) > > > > > > > > > > > > > > > at > > > > > > > > > sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:237) > > > > > > > > > > > > > > > at > > > > > > > > > sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:132) > > > > > > > > > > > > > > > at > > > > > > > > > sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1621) > > > > > > > > > > > > > > > ... 35 common frames omitted > > > > > > > > > > > Caused by: > > > > > > > > > > > > sun.security.provider.certpath.SunCertPathBuilderException: > > > > > > > > > > > > > > > > > > > unable to find valid certification path to requested > target > > > > > > > > > > > at > > > > > > > > > sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) > > > > > > > > > > > > > > > at > > > > > > > > > sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) > > > > > > > > > > > > > > > at > > > > > > > > java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280) > > > > > > > > > > > > > > > at > > > > > > > > > > > sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392) > > > > > > > > > > > > > > > > ... 41 common frames omitted > > > > > > > > > > > Thanks, > > > > > > > > > > > Joe > > > > > > > > -- > > > > Be on your guard; stand firm in the faith; be courageous; be strong. > Do > > > > everything in love. -1 Corinthians 16:13-14 > > > -- Be on your guard; stand firm in the faith; be courageous; be strong. Do everything in love. -*1 Corinthians 16:13-14*
