Ok, I added all the server certs and my administrator's client cert to the trust store, and they all still got PKIX path building failed. So I redeployed nifi 1.11.1, and now it works again.
Joe On Wed, Feb 26, 2020 at 6:21 PM Joe Gresock <jgres...@gmail.com> wrote: > Yes, on Kubernetes. > > FWIW, I do see changes to > nifi-commons/nifi-security-utils/src/main/java/org/apache/nifi/security/util/SslContextFactory.java > involving a new function createTrustSslContextWithTrustManagers(), among > other related changes. > > I'm going to try directly adding the client certs to my trust store to see > if that works (instead of just adding the client certs' issuer). > > On Wed, Feb 26, 2020 at 6:08 PM Joe Witt <joe.w...@gmail.com> wrote: > >> on kubernetes is a key detail here too... >> >> On Wed, Feb 26, 2020 at 10:01 AM Joe Gresock <jgres...@gmail.com> wrote: >> >> > Java 8 >> > >> > On Wed, Feb 26, 2020 at 5:59 PM Nathan Gough <thena...@gmail.com> >> wrote: >> > >> > > Hi Joe, >> > > >> > > I just set up a secure cluster with NiFi 1.11.3 and am not seeing any >> > > issues like you describe. >> > > >> > > Are you running Java 8 or Java 11? >> > > >> > > Nathan >> > > >> > > On Wed, Feb 26, 2020 at 12:22 PM Joe Gresock <jgres...@gmail.com> >> wrote: >> > > >> > > > Were there any changes with how the trust store is used in 1.11.3? >> I >> > > had a >> > > > 1.11.0 deployment working with the following settings, but when I >> > > deployed >> > > > 1.11.3, the cluster can't seem to replicate requests to itself: >> > > > >> > > > nifi.remote.input.host=<redacted> >> > > > nifi.remote.input.secure=true >> > > > nifi.remote.input.socket.port=32440 >> > > > nifi.remote.input.http.enabled=true >> > > > >> > > > nifi.cluster.protocol.is.secure=true >> > > > nifi.cluster.is.node=true >> > > > >> > > > >> > > >> > >> nifi.cluster.node.address=nifi-3.nifi-headless.lizardspock.svc.cluster.local >> > > > nifi.cluster.node.protocol.port=6007 >> > > > >> > > > >> nifi.web.https.host=nifi-3.nifi-headless.lizardspock.svc.cluster.local >> > > > nifi.web.https.port=8443 >> > > > >> > > > nifi.security.keystore=./conf/keystore.jks >> > > > nifi.security.keystoreType=jks >> > > > nifi.security.keystorePasswd=<password> >> > > > nifi.security.keyPasswd= >> > > > nifi.security.truststore=./conf/truststore.jks >> > > > nifi.security.truststoreType=jks >> > > > nifi.security.truststorePasswd=<password> >> > > > nifi.security.needClientAuth=true >> > > > >> > > > A trusted client cert that worked against the old cluster is getting >> > the >> > > > same trust error (PKIX path building failed). I've verified that >> the >> > > > client cert was issued by an issuer that is definitely in the >> > > > ./conf/truststore.jks as a trustedCertEntry. >> > > > >> > > > 2020-02-26 17:11:09,573 WARN [Replicate Request Thread-7] >> > > > o.a.n.c.c.h.r.ThreadPoolRequestReplicator >> > > > javax.net.ssl.SSLHandshakeException: >> > > > sun.security.validator.ValidatorException: PKIX path building >> failed: >> > > > sun.security.provider.certpath.SunCertPathBuilderException: unable >> to >> > > find >> > > > valid certification path to r >> > > > equested target >> > > > at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) >> > > > at >> > sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946) >> > > > at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:316) >> > > > at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310) >> > > > at >> > > > >> > > > >> > > >> > >> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639) >> > > > at >> > > > >> > > >> > >> sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223) >> > > > at >> > sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037) >> > > > at >> > > sun.security.ssl.Handshaker.process_record(Handshaker.java:965) >> > > > at >> > > > sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1064) >> > > > at >> > > > >> > > > >> > > >> > >> sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367) >> > > > at >> > > > >> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395) >> > > > at >> > > > >> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379) >> > > > at >> > > > >> > > > >> > > >> > >> okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:302) >> > > > at >> > > > >> > > > >> > > >> > >> okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:270) >> > > > at >> > > > >> > > >> > >> okhttp3.internal.connection.RealConnection.connect(RealConnection.java:162) >> > > > at >> > > > >> > > > >> > > >> > >> okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:257) >> > > > at >> > > > >> > > > >> > > >> > >> okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:135) >> > > > at >> > > > >> > > > >> > > >> > >> okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:114) >> > > > at >> > > > >> > > > >> > > >> > >> okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42) >> > > > at >> > > > >> > > > >> > > >> > >> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) >> > > > at >> > > > >> > > > >> > > >> > >> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) >> > > > at >> > > > >> > > >> > >> okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93) >> > > > at >> > > > >> > > > >> > > >> > >> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) >> > > > at >> > > > >> > > > >> > > >> > >> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) >> > > > at >> > > > >> > > > >> > > >> > >> okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) >> > > > at >> > > > >> > > > >> > > >> > >> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) >> > > > at >> > > > >> > > > >> > > >> > >> okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:126) >> > > > at >> > > > >> > > > >> > > >> > >> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) >> > > > at >> > > > >> > > > >> > > >> > >> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) >> > > > at >> > > > okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:200) >> > > > at okhttp3.RealCall.execute(RealCall.java:77) >> > > > at >> > > > >> > > > >> > > >> > >> org.apache.nifi.cluster.coordination.http.replication.okhttp.OkHttpReplicationClient.replicate(OkHttpReplicationClient.java:143) >> > > > at >> > > > >> > > > >> > > >> > >> org.apache.nifi.cluster.coordination.http.replication.okhttp.OkHttpReplicationClient.replicate(OkHttpReplicationClient.java:137) >> > > > at >> > > > >> > > > >> > > >> > >> org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator.replicateRequest(ThreadPoolRequestReplicator.java:647) >> > > > at >> > > > >> > > > >> > > >> > >> org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator$NodeHttpRequest.run(ThreadPoolRequestReplicator.java:839) >> > > > at >> > > > >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >> > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) >> > > > at >> > > > >> > > > >> > > >> > >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >> > > > at >> > > > >> > > > >> > > >> > >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >> > > > at java.lang.Thread.run(Thread.java:748) >> > > > Caused by: sun.security.validator.ValidatorException: PKIX path >> > building >> > > > failed: sun.security.provider.certpath.SunCertPathBuilderException: >> > > unable >> > > > to find valid certification path to requested target >> > > > at >> > > > sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397) >> > > > at >> > > > >> > > >> > >> sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302) >> > > > at >> > sun.security.validator.Validator.validate(Validator.java:262) >> > > > at >> > > > >> > > > >> > > >> > >> sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:330) >> > > > at >> > > > >> > > > >> > > >> > >> sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:237) >> > > > at >> > > > >> > > > >> > > >> > >> sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:132) >> > > > at >> > > > >> > > > >> > > >> > >> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1621) >> > > > ... 35 common frames omitted >> > > > Caused by: >> sun.security.provider.certpath.SunCertPathBuilderException: >> > > > unable to find valid certification path to requested target >> > > > at >> > > > >> > > > >> > > >> > >> sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) >> > > > at >> > > > >> > > > >> > > >> > >> sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) >> > > > at >> > > > java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280) >> > > > at >> > > > sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392) >> > > > ... 41 common frames omitted >> > > > >> > > > Thanks, >> > > > Joe >> > > > >> > > >> > >> >
