I've opened https://issues.apache.org/jira/browse/NIFI-7223 to track and I'm working on a fix for this.
Nathan On Tue, Mar 3, 2020 at 6:17 PM Nathan Gough <thena...@gmail.com> wrote: > Hi Joe, > > Just to confirm here - was the nifi.security.keyPasswd not defined at all > in your nifi.properties? Did you have to add the property and give it the > correct value? Or was it in the nifi.properties file but blank? Or were the > keyPasswd and keystorePasswd different values? > > Thanks, > Nathan > > On Tue, Mar 3, 2020 at 3:38 PM Joe Gresock <jgres...@gmail.com> wrote: > >> Yep, setting the nifi.security.keyPasswd to the same as >> nifi.security.keystorePasswd fixed it. Thanks for the insight, Endre! >> >> On Tue, Mar 3, 2020 at 2:01 PM Joe Witt <joe.w...@gmail.com> wrote: >> >> > relevant change I believe is here: >> > >> > >> https://github.com/apache/nifi/commit/46d3b6b0dc28f04da124be7685f82bec52e88775 >> > and >> > is from https://issues.apache.org/jira/browse/NIFI-6927 >> > >> > It *looks* to me like this was fixing an improper naming/usage issue >> that >> > has been present but if so we probably should have addressed not in this >> > bug fix line. Will defer to Troy/Andy for more context and next steps >> > >> > On Tue, Mar 3, 2020 at 5:53 AM Joe Witt <joe.w...@gmail.com> wrote: >> > >> > > If accurate....We need to look into whether this was a mistake and >> fix it >> > > if so. And we need to reflect this in the migration guide >> > > >> > > On Tue, Mar 3, 2020 at 4:40 AM Ryan Ward <ryan.wa...@gmail.com> >> wrote: >> > > >> > >> Endre - thanks that was it >> > >> >> > >> On Tue, Mar 3, 2020 at 6:50 AM Endre Kovacs >> > >> <andrewsmit...@protonmail.com.invalid> wrote: >> > >> >> > >> > Hi, >> > >> > >> > >> > One additional thing: >> > >> > >> > >> > we encountered something strange as well: >> > >> > >> > >> > on 1.11.2 clustered, kerberized: request replication worked well. >> > >> > >> > >> > on 1.11.3 clustered, kerberized: request replication did not work, >> > >> unless >> > >> > you specify, and set >> > >> > nifi.security.keyPasswd >> > >> > >> > >> > to the very same password as the >> > >> > >> > >> > nifi.security.keystorePasswd >> > >> > >> > >> > For us this resolved the issue. >> > >> > >> > >> > Best regards, >> > >> > Endre >> > >> > >> > >> > Sent with [ProtonMail](https://protonmail.com) Secure Email. >> > >> > >> > >> > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ >> > >> > On Tuesday, March 3, 2020 12:40 PM, Ryan Ward < >> ryan.wa...@gmail.com> >> > >> > wrote: >> > >> > >> > >> > > Hi Joe - Did you resolve your issue? If so I am wondering what >> the >> > fix >> > >> > was as I'm seeing the same error on my cluster. >> > >> > > >> > >> > > On Thu, Feb 27, 2020 at 3:13 AM Endre Kovacs < >> > >> > andrewsmit...@protonmail.com.invalid> wrote: >> > >> > > >> > >> > >> Hi Joe, >> > >> > >> >> > >> > >> 1. Have you tried connecting/debugging with openssl? From one >> pod >> > to >> > >> > the other: >> > >> > >> (openssl s_client -debug -CAfile >> > >> > ca-bundle-signing-node-certificates.crt -cert my-client-cert.crt >> > >> -connect >> > >> > nifi-3.nifi-headless.lizardspock.svc.cluster.local:6007) >> > >> > >> >> > >> > >> 2. certs can also be verified by: >> > >> > >> openssl verify -verbose -CAfile ca-bundle.crt >> my-client-cert.crt >> > >> > >> >> > >> > >> 3. Can you check if no intermediary CAs are missing from the >> nodes >> > >> > truststore? >> > >> > >> >> > >> > >> 4. This exception is coming from inter-node communication >> > >> (replication >> > >> > of request from one node to the other). This means that it is >> > unrelated >> > >> to >> > >> > external user's authentication by client certificate. The question >> is: >> > >> is >> > >> > your inter node communication secured by the trusted root CA (that >> you >> > >> are >> > >> > sure that the CA cert is present in the trust store) or is it >> secured >> > by >> > >> > selfsigned CA (which's CA may be lacking from your truststore)? >> > >> > >> >> > >> > >> 5. `nifi.security.needClientAuth` is not part of NiFi >> properties >> > any >> > >> > more. If SSL is turned on, and no >> > >> > `nifi.security.user.login.identity.provider` is set, then client >> cert >> > >> based >> > >> > auth is the default. But supplying this property have no >> detrimental >> > >> effect >> > >> > anyhow. >> > >> > >> >> > >> > >> Best regards, >> > >> > >> Endre >> > >> > >> >> > >> > >> Sent with ProtonMail Secure Email. >> > >> > >> >> > >> > >> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ >> > >> > >> On Wednesday, February 26, 2020 6:22 PM, Joe Gresock >> > >> > jgres...@gmail.com wrote: >> > >> > >> >> > >> > >>> Were there any changes with how the trust store is used in >> > 1.11.3? I >> > >> > had a >> > >> > >>> 1.11.0 deployment working with the following settings, but >> when I >> > >> > deployed >> > >> > >>> 1.11.3, the cluster can't seem to replicate requests to itself: >> > >> > >>> nifi.remote.input.host=<redacted> >> > >> > >>> nifi.remote.input.secure=true >> > >> > >>> nifi.remote.input.socket.port=32440 >> > >> > >>> nifi.remote.input.http.enabled=true >> > >> > >>> nifi.cluster.protocol.is.secure=true >> > >> > >>> nifi.cluster.is.node=true >> > >> > >>> >> > >> > >> > >> >> > >> nifi.cluster.node.address=nifi-3.nifi-headless.lizardspock.svc.cluster.local >> > >> > >>> nifi.cluster.node.protocol.port=6007 >> > >> > >>> >> > >> >> nifi.web.https.host=nifi-3.nifi-headless.lizardspock.svc.cluster.local >> > >> > >>> nifi.web.https.port=8443 >> > >> > >>> nifi.security.keystore=./conf/keystore.jks >> > >> > >>> nifi.security.keystoreType=jks >> > >> > >>> nifi.security.keystorePasswd=<password> >> > >> > >>> nifi.security.keyPasswd= >> > >> > >>> nifi.security.truststore=./conf/truststore.jks >> > >> > >>> nifi.security.truststoreType=jks >> > >> > >>> nifi.security.truststorePasswd=<password> >> > >> > >>> nifi.security.needClientAuth=true >> > >> > >>> A trusted client cert that worked against the old cluster is >> > getting >> > >> > the >> > >> > >>> same trust error (PKIX path building failed). I've verified >> that >> > the >> > >> > >>> client cert was issued by an issuer that is definitely in the >> > >> > >>> ./conf/truststore.jks as a trustedCertEntry. >> > >> > >>> 2020-02-26 17:11:09,573 WARN [Replicate Request Thread-7] >> > >> > >>> o.a.n.c.c.h.r.ThreadPoolRequestReplicator >> > >> > >>> javax.net.ssl.SSLHandshakeException: >> > >> > >>> sun.security.validator.ValidatorException: PKIX path building >> > >> failed: >> > >> > >>> sun.security.provider.certpath.SunCertPathBuilderException: >> unable >> > >> to >> > >> > find >> > >> > >>> valid certification path to r >> > >> > >>> equested target >> > >> > >>> at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) >> > >> > >>> at >> sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946) >> > >> > >>> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:316) >> > >> > >>> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310) >> > >> > >>> at >> > >> > >>> >> > >> > >> > >> >> > >> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639) >> > >> > >>> at >> > >> > >>> >> > >> > >> > >> >> > >> sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223) >> > >> > >>> at >> sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037) >> > >> > >>> at >> sun.security.ssl.Handshaker.process_record(Handshaker.java:965) >> > >> > >>> at >> > >> > >>> >> sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1064) >> > >> > >>> at >> > >> > >>> >> > >> > >> > >> >> > >> sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367) >> > >> > >>> at >> > >> > >>> >> > >> >> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395) >> > >> > >>> at >> > >> > >>> >> > >> >> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379) >> > >> > >>> at >> > >> > >>> >> > >> > >> > >> >> > >> okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:302) >> > >> > >>> at >> > >> > >>> >> > >> > >> > >> >> > >> okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:270) >> > >> > >>> at >> > >> > >>> >> > >> > >> > >> >> > >> okhttp3.internal.connection.RealConnection.connect(RealConnection.java:162) >> > >> > >>> at >> > >> > >>> >> > >> > >> > >> >> > >> okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:257) >> > >> > >>> at >> > >> > >>> >> > >> > >> > >> >> > >> okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:135) >> > >> > >>> at >> > >> > >>> >> > >> > >> > >> >> > >> okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:114) >> > >> > >>> at >> > >> > >>> >> > >> > >> > >> >> > >> okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42) >> > >> > >>> at >> > >> > >>> >> > >> > >> > >> >> > >> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) >> > >> > >>> at >> > >> > >>> >> > >> > >> > >> >> > >> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) >> > >> > >>> at >> > >> > >>> >> > >> > >> > >> >> > >> okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93) >> > >> > >>> at >> > >> > >>> >> > >> > >> > >> >> > >> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) >> > >> > >>> at >> > >> > >>> >> > >> > >> > >> >> > >> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) >> > >> > >>> at >> > >> > >>> >> > >> > >> > >> >> > >> okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) >> > >> > >>> at >> > >> > >>> >> > >> > >> > >> >> > >> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) >> > >> > >>> at >> > >> > >>> >> > >> > >> > >> >> > >> okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:126) >> > >> > >>> at >> > >> > >>> >> > >> > >> > >> >> > >> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) >> > >> > >>> at >> > >> > >>> >> > >> > >> > >> >> > >> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) >> > >> > >>> at >> > >> > >>> >> > okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:200) >> > >> > >>> at okhttp3.RealCall.execute(RealCall.java:77) >> > >> > >>> at >> > >> > >>> >> > >> > >> > >> >> > >> org.apache.nifi.cluster.coordination.http.replication.okhttp.OkHttpReplicationClient.replicate(OkHttpReplicationClient.java:143) >> > >> > >>> at >> > >> > >>> >> > >> > >> > >> >> > >> org.apache.nifi.cluster.coordination.http.replication.okhttp.OkHttpReplicationClient.replicate(OkHttpReplicationClient.java:137) >> > >> > >>> at >> > >> > >>> >> > >> > >> > >> >> > >> org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator.replicateRequest(ThreadPoolRequestReplicator.java:647) >> > >> > >>> at >> > >> > >>> >> > >> > >> > >> >> > >> org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator$NodeHttpRequest.run(ThreadPoolRequestReplicator.java:839) >> > >> > >>> at >> > >> > >>> >> > >> >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >> > >> > >>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >> > >> > >>> at >> > >> > >>> >> > >> > >> > >> >> > >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >> > >> > >>> at >> > >> > >>> >> > >> > >> > >> >> > >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >> > >> > >>> at java.lang.Thread.run(Thread.java:748) >> > >> > >>> Caused by: sun.security.validator.ValidatorException: PKIX path >> > >> > building >> > >> > >>> failed: >> > sun.security.provider.certpath.SunCertPathBuilderException: >> > >> > unable >> > >> > >>> to find valid certification path to requested target >> > >> > >>> at >> > >> > >>> >> > sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397) >> > >> > >>> at >> > >> > >>> >> > >> > >> > >> >> > >> sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302) >> > >> > >>> at >> sun.security.validator.Validator.validate(Validator.java:262) >> > >> > >>> at >> > >> > >>> >> > >> > >> > >> >> > >> sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:330) >> > >> > >>> at >> > >> > >>> >> > >> > >> > >> >> > >> sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:237) >> > >> > >>> at >> > >> > >>> >> > >> > >> > >> >> > >> sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:132) >> > >> > >>> at >> > >> > >>> >> > >> > >> > >> >> > >> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1621) >> > >> > >>> ... 35 common frames omitted >> > >> > >>> Caused by: >> > >> sun.security.provider.certpath.SunCertPathBuilderException: >> > >> > >>> unable to find valid certification path to requested target >> > >> > >>> at >> > >> > >>> >> > >> > >> > >> >> > >> sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) >> > >> > >>> at >> > >> > >>> >> > >> > >> > >> >> > >> sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) >> > >> > >>> at >> > >> > >>> >> java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280) >> > >> > >>> at >> > >> > >>> >> > sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392) >> > >> > >>> ... 41 common frames omitted >> > >> > >>> Thanks, >> > >> > >>> Joe >> > >> >> > > >> > >> >> >> -- >> Be on your guard; stand firm in the faith; be courageous; be strong. Do >> everything in love. -*1 Corinthians 16:13-14* >> >
