Hi Joe,
1. Have you tried connecting/debugging with openssl? From one pod to the other:
(openssl s_client -debug -CAfile ca-bundle-signing-node-certificates.crt
-cert my-client-cert.crt -connect
nifi-3.nifi-headless.lizardspock.svc.cluster.local:6007)
2. certs can also be verified by:
openssl verify -verbose -CAfile ca-bundle.crt my-client-cert.crt
3. Can you check if no intermediary CAs are missing from the nodes truststore?
4. This exception is coming from inter-node communication (replication of
request from one node to the other). This means that it is unrelated to
external user's authentication by client certificate. The question is: is your
inter node communication secured by the trusted root CA (that you are sure that
the CA cert is present in the trust store) or is it secured by selfsigned CA
(which's CA may be lacking from your truststore)?
5. `nifi.security.needClientAuth` is not part of NiFi properties any more. If
SSL is turned on, and no `nifi.security.user.login.identity.provider` is set,
then client cert based auth is the default. But supplying this property have no
detrimental effect anyhow.
Best regards,
Endre
Sent with ProtonMail Secure Email.
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Wednesday, February 26, 2020 6:22 PM, Joe Gresock jgres...@gmail.com
wrote:
> Were there any changes with how the trust store is used in 1.11.3? I had a
> 1.11.0 deployment working with the following settings, but when I deployed
> 1.11.3, the cluster can't seem to replicate requests to itself:
> nifi.remote.input.host=<redacted>
> nifi.remote.input.secure=true
> nifi.remote.input.socket.port=32440
> nifi.remote.input.http.enabled=true
> nifi.cluster.protocol.is.secure=true
> nifi.cluster.is.node=true
> nifi.cluster.node.address=nifi-3.nifi-headless.lizardspock.svc.cluster.local
> nifi.cluster.node.protocol.port=6007
> nifi.web.https.host=nifi-3.nifi-headless.lizardspock.svc.cluster.local
> nifi.web.https.port=8443
> nifi.security.keystore=./conf/keystore.jks
> nifi.security.keystoreType=jks
> nifi.security.keystorePasswd=<password>
> nifi.security.keyPasswd=
> nifi.security.truststore=./conf/truststore.jks
> nifi.security.truststoreType=jks
> nifi.security.truststorePasswd=<password>
> nifi.security.needClientAuth=true
> A trusted client cert that worked against the old cluster is getting the
> same trust error (PKIX path building failed). I've verified that the
> client cert was issued by an issuer that is definitely in the
> ./conf/truststore.jks as a trustedCertEntry.
> 2020-02-26 17:11:09,573 WARN [Replicate Request Thread-7]
> o.a.n.c.c.h.r.ThreadPoolRequestReplicator
> javax.net.ssl.SSLHandshakeException:
> sun.security.validator.ValidatorException: PKIX path building failed:
> sun.security.provider.certpath.SunCertPathBuilderException: unable to find
> valid certification path to r
> equested target
> at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
> at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946)
> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:316)
> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310)
> at
> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639)
> at
> sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223)
> at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037)
> at sun.security.ssl.Handshaker.process_record(Handshaker.java:965)
> at
> sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1064)
> at
> sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367)
> at
> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395)
> at
> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379)
> at
> okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:302)
> at
> okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:270)
> at
> okhttp3.internal.connection.RealConnection.connect(RealConnection.java:162)
> at
> okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:257)
> at
> okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:135)
> at
> okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:114)
> at
> okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42)
> at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
> at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
> at
> okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
> at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
> at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
> at
> okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
> at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
> at
> okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:126)
> at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
> at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
> at
> okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:200)
> at okhttp3.RealCall.execute(RealCall.java:77)
> at
> org.apache.nifi.cluster.coordination.http.replication.okhttp.OkHttpReplicationClient.replicate(OkHttpReplicationClient.java:143)
> at
> org.apache.nifi.cluster.coordination.http.replication.okhttp.OkHttpReplicationClient.replicate(OkHttpReplicationClient.java:137)
> at
> org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator.replicateRequest(ThreadPoolRequestReplicator.java:647)
> at
> org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator$NodeHttpRequest.run(ThreadPoolRequestReplicator.java:839)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: sun.security.validator.ValidatorException: PKIX path building
> failed: sun.security.provider.certpath.SunCertPathBuilderException: unable
> to find valid certification path to requested target
> at
> sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397)
> at
> sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302)
> at sun.security.validator.Validator.validate(Validator.java:262)
> at
> sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:330)
> at
> sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:237)
> at
> sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:132)
> at
> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1621)
> ... 35 common frames omitted
> Caused by: sun.security.provider.certpath.SunCertPathBuilderException:
> unable to find valid certification path to requested target
> at
> sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
> at
> sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
> at
> java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
> at
> sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392)
> ... 41 common frames omitted
> Thanks,
> Joe
