Re: 1.11.3 trust store error
Hi Joe, Just to confirm here - was the nifi.security.keyPasswd not defined at all in your nifi.properties? Did you have to add the property and give it the correct value? Or was it in the nifi.properties file but blank? Or were the keyPasswd and keystorePasswd different values? Thanks, Nathan On Tue, Mar 3, 2020 at 3:38 PM Joe Gresock wrote: > Yep, setting the nifi.security.keyPasswd to the same as > nifi.security.keystorePasswd fixed it. Thanks for the insight, Endre! > > On Tue, Mar 3, 2020 at 2:01 PM Joe Witt wrote: > > > relevant change I believe is here: > > > > > https://github.com/apache/nifi/commit/46d3b6b0dc28f04da124be7685f82bec52e88775 > > and > > is from https://issues.apache.org/jira/browse/NIFI-6927 > > > > It *looks* to me like this was fixing an improper naming/usage issue that > > has been present but if so we probably should have addressed not in this > > bug fix line. Will defer to Troy/Andy for more context and next steps > > > > On Tue, Mar 3, 2020 at 5:53 AM Joe Witt wrote: > > > > > If accurateWe need to look into whether this was a mistake and fix > it > > > if so. And we need to reflect this in the migration guide > > > > > > On Tue, Mar 3, 2020 at 4:40 AM Ryan Ward wrote: > > > > > >> Endre - thanks that was it > > >> > > >> On Tue, Mar 3, 2020 at 6:50 AM Endre Kovacs > > >> wrote: > > >> > > >> > Hi, > > >> > > > >> > One additional thing: > > >> > > > >> > we encountered something strange as well: > > >> > > > >> > on 1.11.2 clustered, kerberized: request replication worked well. > > >> > > > >> > on 1.11.3 clustered, kerberized: request replication did not work, > > >> unless > > >> > you specify, and set > > >> > nifi.security.keyPasswd > > >> > > > >> > to the very same password as the > > >> > > > >> > nifi.security.keystorePasswd > > >> > > > >> > For us this resolved the issue. > > >> > > > >> > Best regards, > > >> > Endre > > >> > > > >> > Sent with [ProtonMail](https://protonmail.com) Secure Email. > > >> > > > >> > ‐‐‐ Original Message ‐‐‐ > > >> > On Tuesday, March 3, 2020 12:40 PM, Ryan Ward > > > >> > wrote: > > >> > > > >> > > Hi Joe - Did you resolve your issue? If so I am wondering what the > > fix > > >> > was as I'm seeing the same error on my cluster. > > >> > > > > >> > > On Thu, Feb 27, 2020 at 3:13 AM Endre Kovacs < > > >> > andrewsmit...@protonmail.com.invalid> wrote: > > >> > > > > >> > >> Hi Joe, > > >> > >> > > >> > >> 1. Have you tried connecting/debugging with openssl? From one > pod > > to > > >> > the other: > > >> > >> (openssl s_client -debug -CAfile > > >> > ca-bundle-signing-node-certificates.crt -cert my-client-cert.crt > > >> -connect > > >> > nifi-3.nifi-headless.lizardspock.svc.cluster.local:6007) > > >> > >> > > >> > >> 2. certs can also be verified by: > > >> > >> openssl verify -verbose -CAfile ca-bundle.crt my-client-cert.crt > > >> > >> > > >> > >> 3. Can you check if no intermediary CAs are missing from the > nodes > > >> > truststore? > > >> > >> > > >> > >> 4. This exception is coming from inter-node communication > > >> (replication > > >> > of request from one node to the other). This means that it is > > unrelated > > >> to > > >> > external user's authentication by client certificate. The question > is: > > >> is > > >> > your inter node communication secured by the trusted root CA (that > you > > >> are > > >> > sure that the CA cert is present in the trust store) or is it > secured > > by > > >> > selfsigned CA (which's CA may be lacking from your truststore)? > > >> > >> > > >> > >> 5. `nifi.security.needClientAuth` is not part of NiFi properties > > any > > >> > more. If SSL is turned on, and no > > >> > `nifi.security.user.login.identity.provider` is set, then client > cert > > >> based > > >> > auth is the default. But supplying this property have no detrimental > > >> effect > > >> > anyhow. > > >> > >> > > >> > >> Best regards, > > >> > >> Endre > > >> > >> > > >> > >> Sent with ProtonMail Secure Email. > > >> > >> > > >> > >> ‐‐‐ Original Message ‐‐‐ > > >> > >> On Wednesday, February 26, 2020 6:22 PM, Joe Gresock > > >> > jgres...@gmail.com wrote: > > >> > >> > > >> > >>> Were there any changes with how the trust store is used in > > 1.11.3? I > > >> > had a > > >> > >>> 1.11.0 deployment working with the following settings, but when > I > > >> > deployed > > >> > >>> 1.11.3, the cluster can't seem to replicate requests to itself: > > >> > >>> nifi.remote.input.host= > > >> > >>> nifi.remote.input.secure=true > > >> > >>> nifi.remote.input.socket.port=32440 > > >> > >>> nifi.remote.input.http.enabled=true > > >> > >>> nifi.cluster.protocol.is.secure=true > > >> > >>> nifi.cluster.is.node=true > > >> > >>> > > >> > > > >> > > > nifi.cluster.node.address=nifi-3.nifi-headless.lizardspock.svc.cluster.local > > >> > >>> nifi.cluster.node.protocol.port=6007 > > >> > >>> > > >> nifi.web.https.host=nifi-3.nifi-headless.lizardspock.svc.cluster.local > > >> > >>>
Re: 1.11.3 trust store error
Yep, setting the nifi.security.keyPasswd to the same as nifi.security.keystorePasswd fixed it. Thanks for the insight, Endre! On Tue, Mar 3, 2020 at 2:01 PM Joe Witt wrote: > relevant change I believe is here: > > https://github.com/apache/nifi/commit/46d3b6b0dc28f04da124be7685f82bec52e88775 > and > is from https://issues.apache.org/jira/browse/NIFI-6927 > > It *looks* to me like this was fixing an improper naming/usage issue that > has been present but if so we probably should have addressed not in this > bug fix line. Will defer to Troy/Andy for more context and next steps > > On Tue, Mar 3, 2020 at 5:53 AM Joe Witt wrote: > > > If accurateWe need to look into whether this was a mistake and fix it > > if so. And we need to reflect this in the migration guide > > > > On Tue, Mar 3, 2020 at 4:40 AM Ryan Ward wrote: > > > >> Endre - thanks that was it > >> > >> On Tue, Mar 3, 2020 at 6:50 AM Endre Kovacs > >> wrote: > >> > >> > Hi, > >> > > >> > One additional thing: > >> > > >> > we encountered something strange as well: > >> > > >> > on 1.11.2 clustered, kerberized: request replication worked well. > >> > > >> > on 1.11.3 clustered, kerberized: request replication did not work, > >> unless > >> > you specify, and set > >> > nifi.security.keyPasswd > >> > > >> > to the very same password as the > >> > > >> > nifi.security.keystorePasswd > >> > > >> > For us this resolved the issue. > >> > > >> > Best regards, > >> > Endre > >> > > >> > Sent with [ProtonMail](https://protonmail.com) Secure Email. > >> > > >> > ‐‐‐ Original Message ‐‐‐ > >> > On Tuesday, March 3, 2020 12:40 PM, Ryan Ward > >> > wrote: > >> > > >> > > Hi Joe - Did you resolve your issue? If so I am wondering what the > fix > >> > was as I'm seeing the same error on my cluster. > >> > > > >> > > On Thu, Feb 27, 2020 at 3:13 AM Endre Kovacs < > >> > andrewsmit...@protonmail.com.invalid> wrote: > >> > > > >> > >> Hi Joe, > >> > >> > >> > >> 1. Have you tried connecting/debugging with openssl? From one pod > to > >> > the other: > >> > >> (openssl s_client -debug -CAfile > >> > ca-bundle-signing-node-certificates.crt -cert my-client-cert.crt > >> -connect > >> > nifi-3.nifi-headless.lizardspock.svc.cluster.local:6007) > >> > >> > >> > >> 2. certs can also be verified by: > >> > >> openssl verify -verbose -CAfile ca-bundle.crt my-client-cert.crt > >> > >> > >> > >> 3. Can you check if no intermediary CAs are missing from the nodes > >> > truststore? > >> > >> > >> > >> 4. This exception is coming from inter-node communication > >> (replication > >> > of request from one node to the other). This means that it is > unrelated > >> to > >> > external user's authentication by client certificate. The question is: > >> is > >> > your inter node communication secured by the trusted root CA (that you > >> are > >> > sure that the CA cert is present in the trust store) or is it secured > by > >> > selfsigned CA (which's CA may be lacking from your truststore)? > >> > >> > >> > >> 5. `nifi.security.needClientAuth` is not part of NiFi properties > any > >> > more. If SSL is turned on, and no > >> > `nifi.security.user.login.identity.provider` is set, then client cert > >> based > >> > auth is the default. But supplying this property have no detrimental > >> effect > >> > anyhow. > >> > >> > >> > >> Best regards, > >> > >> Endre > >> > >> > >> > >> Sent with ProtonMail Secure Email. > >> > >> > >> > >> ‐‐‐ Original Message ‐‐‐ > >> > >> On Wednesday, February 26, 2020 6:22 PM, Joe Gresock > >> > jgres...@gmail.com wrote: > >> > >> > >> > >>> Were there any changes with how the trust store is used in > 1.11.3? I > >> > had a > >> > >>> 1.11.0 deployment working with the following settings, but when I > >> > deployed > >> > >>> 1.11.3, the cluster can't seem to replicate requests to itself: > >> > >>> nifi.remote.input.host= > >> > >>> nifi.remote.input.secure=true > >> > >>> nifi.remote.input.socket.port=32440 > >> > >>> nifi.remote.input.http.enabled=true > >> > >>> nifi.cluster.protocol.is.secure=true > >> > >>> nifi.cluster.is.node=true > >> > >>> > >> > > >> > nifi.cluster.node.address=nifi-3.nifi-headless.lizardspock.svc.cluster.local > >> > >>> nifi.cluster.node.protocol.port=6007 > >> > >>> > >> nifi.web.https.host=nifi-3.nifi-headless.lizardspock.svc.cluster.local > >> > >>> nifi.web.https.port=8443 > >> > >>> nifi.security.keystore=./conf/keystore.jks > >> > >>> nifi.security.keystoreType=jks > >> > >>> nifi.security.keystorePasswd= > >> > >>> nifi.security.keyPasswd= > >> > >>> nifi.security.truststore=./conf/truststore.jks > >> > >>> nifi.security.truststoreType=jks > >> > >>> nifi.security.truststorePasswd= > >> > >>> nifi.security.needClientAuth=true > >> > >>> A trusted client cert that worked against the old cluster is > getting > >> > the > >> > >>> same trust error (PKIX path building failed). I've verified that > the > >> > >>> client cert was issued by an issuer that is definitely in the >
Re: 1.11.3 trust store error
relevant change I believe is here: https://github.com/apache/nifi/commit/46d3b6b0dc28f04da124be7685f82bec52e88775 and is from https://issues.apache.org/jira/browse/NIFI-6927 It *looks* to me like this was fixing an improper naming/usage issue that has been present but if so we probably should have addressed not in this bug fix line. Will defer to Troy/Andy for more context and next steps On Tue, Mar 3, 2020 at 5:53 AM Joe Witt wrote: > If accurateWe need to look into whether this was a mistake and fix it > if so. And we need to reflect this in the migration guide > > On Tue, Mar 3, 2020 at 4:40 AM Ryan Ward wrote: > >> Endre - thanks that was it >> >> On Tue, Mar 3, 2020 at 6:50 AM Endre Kovacs >> wrote: >> >> > Hi, >> > >> > One additional thing: >> > >> > we encountered something strange as well: >> > >> > on 1.11.2 clustered, kerberized: request replication worked well. >> > >> > on 1.11.3 clustered, kerberized: request replication did not work, >> unless >> > you specify, and set >> > nifi.security.keyPasswd >> > >> > to the very same password as the >> > >> > nifi.security.keystorePasswd >> > >> > For us this resolved the issue. >> > >> > Best regards, >> > Endre >> > >> > Sent with [ProtonMail](https://protonmail.com) Secure Email. >> > >> > ‐‐‐ Original Message ‐‐‐ >> > On Tuesday, March 3, 2020 12:40 PM, Ryan Ward >> > wrote: >> > >> > > Hi Joe - Did you resolve your issue? If so I am wondering what the fix >> > was as I'm seeing the same error on my cluster. >> > > >> > > On Thu, Feb 27, 2020 at 3:13 AM Endre Kovacs < >> > andrewsmit...@protonmail.com.invalid> wrote: >> > > >> > >> Hi Joe, >> > >> >> > >> 1. Have you tried connecting/debugging with openssl? From one pod to >> > the other: >> > >> (openssl s_client -debug -CAfile >> > ca-bundle-signing-node-certificates.crt -cert my-client-cert.crt >> -connect >> > nifi-3.nifi-headless.lizardspock.svc.cluster.local:6007) >> > >> >> > >> 2. certs can also be verified by: >> > >> openssl verify -verbose -CAfile ca-bundle.crt my-client-cert.crt >> > >> >> > >> 3. Can you check if no intermediary CAs are missing from the nodes >> > truststore? >> > >> >> > >> 4. This exception is coming from inter-node communication >> (replication >> > of request from one node to the other). This means that it is unrelated >> to >> > external user's authentication by client certificate. The question is: >> is >> > your inter node communication secured by the trusted root CA (that you >> are >> > sure that the CA cert is present in the trust store) or is it secured by >> > selfsigned CA (which's CA may be lacking from your truststore)? >> > >> >> > >> 5. `nifi.security.needClientAuth` is not part of NiFi properties any >> > more. If SSL is turned on, and no >> > `nifi.security.user.login.identity.provider` is set, then client cert >> based >> > auth is the default. But supplying this property have no detrimental >> effect >> > anyhow. >> > >> >> > >> Best regards, >> > >> Endre >> > >> >> > >> Sent with ProtonMail Secure Email. >> > >> >> > >> ‐‐‐ Original Message ‐‐‐ >> > >> On Wednesday, February 26, 2020 6:22 PM, Joe Gresock >> > jgres...@gmail.com wrote: >> > >> >> > >>> Were there any changes with how the trust store is used in 1.11.3? I >> > had a >> > >>> 1.11.0 deployment working with the following settings, but when I >> > deployed >> > >>> 1.11.3, the cluster can't seem to replicate requests to itself: >> > >>> nifi.remote.input.host= >> > >>> nifi.remote.input.secure=true >> > >>> nifi.remote.input.socket.port=32440 >> > >>> nifi.remote.input.http.enabled=true >> > >>> nifi.cluster.protocol.is.secure=true >> > >>> nifi.cluster.is.node=true >> > >>> >> > >> nifi.cluster.node.address=nifi-3.nifi-headless.lizardspock.svc.cluster.local >> > >>> nifi.cluster.node.protocol.port=6007 >> > >>> >> nifi.web.https.host=nifi-3.nifi-headless.lizardspock.svc.cluster.local >> > >>> nifi.web.https.port=8443 >> > >>> nifi.security.keystore=./conf/keystore.jks >> > >>> nifi.security.keystoreType=jks >> > >>> nifi.security.keystorePasswd= >> > >>> nifi.security.keyPasswd= >> > >>> nifi.security.truststore=./conf/truststore.jks >> > >>> nifi.security.truststoreType=jks >> > >>> nifi.security.truststorePasswd= >> > >>> nifi.security.needClientAuth=true >> > >>> A trusted client cert that worked against the old cluster is getting >> > the >> > >>> same trust error (PKIX path building failed). I've verified that the >> > >>> client cert was issued by an issuer that is definitely in the >> > >>> ./conf/truststore.jks as a trustedCertEntry. >> > >>> 2020-02-26 17:11:09,573 WARN [Replicate Request Thread-7] >> > >>> o.a.n.c.c.h.r.ThreadPoolRequestReplicator >> > >>> javax.net.ssl.SSLHandshakeException: >> > >>> sun.security.validator.ValidatorException: PKIX path building >> failed: >> > >>> sun.security.provider.certpath.SunCertPathBuilderException: unable >> to >> > find >> > >>> valid certification path to r >> > >>> equested
Re: 1.11.3 trust store error
If accurateWe need to look into whether this was a mistake and fix it if so. And we need to reflect this in the migration guide On Tue, Mar 3, 2020 at 4:40 AM Ryan Ward wrote: > Endre - thanks that was it > > On Tue, Mar 3, 2020 at 6:50 AM Endre Kovacs > wrote: > > > Hi, > > > > One additional thing: > > > > we encountered something strange as well: > > > > on 1.11.2 clustered, kerberized: request replication worked well. > > > > on 1.11.3 clustered, kerberized: request replication did not work, unless > > you specify, and set > > nifi.security.keyPasswd > > > > to the very same password as the > > > > nifi.security.keystorePasswd > > > > For us this resolved the issue. > > > > Best regards, > > Endre > > > > Sent with [ProtonMail](https://protonmail.com) Secure Email. > > > > ‐‐‐ Original Message ‐‐‐ > > On Tuesday, March 3, 2020 12:40 PM, Ryan Ward > > wrote: > > > > > Hi Joe - Did you resolve your issue? If so I am wondering what the fix > > was as I'm seeing the same error on my cluster. > > > > > > On Thu, Feb 27, 2020 at 3:13 AM Endre Kovacs < > > andrewsmit...@protonmail.com.invalid> wrote: > > > > > >> Hi Joe, > > >> > > >> 1. Have you tried connecting/debugging with openssl? From one pod to > > the other: > > >> (openssl s_client -debug -CAfile > > ca-bundle-signing-node-certificates.crt -cert my-client-cert.crt -connect > > nifi-3.nifi-headless.lizardspock.svc.cluster.local:6007) > > >> > > >> 2. certs can also be verified by: > > >> openssl verify -verbose -CAfile ca-bundle.crt my-client-cert.crt > > >> > > >> 3. Can you check if no intermediary CAs are missing from the nodes > > truststore? > > >> > > >> 4. This exception is coming from inter-node communication > (replication > > of request from one node to the other). This means that it is unrelated > to > > external user's authentication by client certificate. The question is: is > > your inter node communication secured by the trusted root CA (that you > are > > sure that the CA cert is present in the trust store) or is it secured by > > selfsigned CA (which's CA may be lacking from your truststore)? > > >> > > >> 5. `nifi.security.needClientAuth` is not part of NiFi properties any > > more. If SSL is turned on, and no > > `nifi.security.user.login.identity.provider` is set, then client cert > based > > auth is the default. But supplying this property have no detrimental > effect > > anyhow. > > >> > > >> Best regards, > > >> Endre > > >> > > >> Sent with ProtonMail Secure Email. > > >> > > >> ‐‐‐ Original Message ‐‐‐ > > >> On Wednesday, February 26, 2020 6:22 PM, Joe Gresock > > jgres...@gmail.com wrote: > > >> > > >>> Were there any changes with how the trust store is used in 1.11.3? I > > had a > > >>> 1.11.0 deployment working with the following settings, but when I > > deployed > > >>> 1.11.3, the cluster can't seem to replicate requests to itself: > > >>> nifi.remote.input.host= > > >>> nifi.remote.input.secure=true > > >>> nifi.remote.input.socket.port=32440 > > >>> nifi.remote.input.http.enabled=true > > >>> nifi.cluster.protocol.is.secure=true > > >>> nifi.cluster.is.node=true > > >>> > > > nifi.cluster.node.address=nifi-3.nifi-headless.lizardspock.svc.cluster.local > > >>> nifi.cluster.node.protocol.port=6007 > > >>> > nifi.web.https.host=nifi-3.nifi-headless.lizardspock.svc.cluster.local > > >>> nifi.web.https.port=8443 > > >>> nifi.security.keystore=./conf/keystore.jks > > >>> nifi.security.keystoreType=jks > > >>> nifi.security.keystorePasswd= > > >>> nifi.security.keyPasswd= > > >>> nifi.security.truststore=./conf/truststore.jks > > >>> nifi.security.truststoreType=jks > > >>> nifi.security.truststorePasswd= > > >>> nifi.security.needClientAuth=true > > >>> A trusted client cert that worked against the old cluster is getting > > the > > >>> same trust error (PKIX path building failed). I've verified that the > > >>> client cert was issued by an issuer that is definitely in the > > >>> ./conf/truststore.jks as a trustedCertEntry. > > >>> 2020-02-26 17:11:09,573 WARN [Replicate Request Thread-7] > > >>> o.a.n.c.c.h.r.ThreadPoolRequestReplicator > > >>> javax.net.ssl.SSLHandshakeException: > > >>> sun.security.validator.ValidatorException: PKIX path building failed: > > >>> sun.security.provider.certpath.SunCertPathBuilderException: unable to > > find > > >>> valid certification path to r > > >>> equested target > > >>> at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) > > >>> at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946) > > >>> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:316) > > >>> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310) > > >>> at > > >>> > > > sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639) > > >>> at > > >>> > > > sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223) > > >>> at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037) > > >>> at
Re: 1.11.3 trust store error
Endre - thanks that was it On Tue, Mar 3, 2020 at 6:50 AM Endre Kovacs wrote: > Hi, > > One additional thing: > > we encountered something strange as well: > > on 1.11.2 clustered, kerberized: request replication worked well. > > on 1.11.3 clustered, kerberized: request replication did not work, unless > you specify, and set > nifi.security.keyPasswd > > to the very same password as the > > nifi.security.keystorePasswd > > For us this resolved the issue. > > Best regards, > Endre > > Sent with [ProtonMail](https://protonmail.com) Secure Email. > > ‐‐‐ Original Message ‐‐‐ > On Tuesday, March 3, 2020 12:40 PM, Ryan Ward > wrote: > > > Hi Joe - Did you resolve your issue? If so I am wondering what the fix > was as I'm seeing the same error on my cluster. > > > > On Thu, Feb 27, 2020 at 3:13 AM Endre Kovacs < > andrewsmit...@protonmail.com.invalid> wrote: > > > >> Hi Joe, > >> > >> 1. Have you tried connecting/debugging with openssl? From one pod to > the other: > >> (openssl s_client -debug -CAfile > ca-bundle-signing-node-certificates.crt -cert my-client-cert.crt -connect > nifi-3.nifi-headless.lizardspock.svc.cluster.local:6007) > >> > >> 2. certs can also be verified by: > >> openssl verify -verbose -CAfile ca-bundle.crt my-client-cert.crt > >> > >> 3. Can you check if no intermediary CAs are missing from the nodes > truststore? > >> > >> 4. This exception is coming from inter-node communication (replication > of request from one node to the other). This means that it is unrelated to > external user's authentication by client certificate. The question is: is > your inter node communication secured by the trusted root CA (that you are > sure that the CA cert is present in the trust store) or is it secured by > selfsigned CA (which's CA may be lacking from your truststore)? > >> > >> 5. `nifi.security.needClientAuth` is not part of NiFi properties any > more. If SSL is turned on, and no > `nifi.security.user.login.identity.provider` is set, then client cert based > auth is the default. But supplying this property have no detrimental effect > anyhow. > >> > >> Best regards, > >> Endre > >> > >> Sent with ProtonMail Secure Email. > >> > >> ‐‐‐ Original Message ‐‐‐ > >> On Wednesday, February 26, 2020 6:22 PM, Joe Gresock > jgres...@gmail.com wrote: > >> > >>> Were there any changes with how the trust store is used in 1.11.3? I > had a > >>> 1.11.0 deployment working with the following settings, but when I > deployed > >>> 1.11.3, the cluster can't seem to replicate requests to itself: > >>> nifi.remote.input.host= > >>> nifi.remote.input.secure=true > >>> nifi.remote.input.socket.port=32440 > >>> nifi.remote.input.http.enabled=true > >>> nifi.cluster.protocol.is.secure=true > >>> nifi.cluster.is.node=true > >>> > nifi.cluster.node.address=nifi-3.nifi-headless.lizardspock.svc.cluster.local > >>> nifi.cluster.node.protocol.port=6007 > >>> nifi.web.https.host=nifi-3.nifi-headless.lizardspock.svc.cluster.local > >>> nifi.web.https.port=8443 > >>> nifi.security.keystore=./conf/keystore.jks > >>> nifi.security.keystoreType=jks > >>> nifi.security.keystorePasswd= > >>> nifi.security.keyPasswd= > >>> nifi.security.truststore=./conf/truststore.jks > >>> nifi.security.truststoreType=jks > >>> nifi.security.truststorePasswd= > >>> nifi.security.needClientAuth=true > >>> A trusted client cert that worked against the old cluster is getting > the > >>> same trust error (PKIX path building failed). I've verified that the > >>> client cert was issued by an issuer that is definitely in the > >>> ./conf/truststore.jks as a trustedCertEntry. > >>> 2020-02-26 17:11:09,573 WARN [Replicate Request Thread-7] > >>> o.a.n.c.c.h.r.ThreadPoolRequestReplicator > >>> javax.net.ssl.SSLHandshakeException: > >>> sun.security.validator.ValidatorException: PKIX path building failed: > >>> sun.security.provider.certpath.SunCertPathBuilderException: unable to > find > >>> valid certification path to r > >>> equested target > >>> at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) > >>> at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946) > >>> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:316) > >>> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310) > >>> at > >>> > sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639) > >>> at > >>> > sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223) > >>> at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037) > >>> at sun.security.ssl.Handshaker.process_record(Handshaker.java:965) > >>> at > >>> sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1064) > >>> at > >>> > sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367) > >>> at > >>> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395) > >>> at > >>> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379) > >>> at > >>> >
Re: 1.11.3 trust store error
Hi, One additional thing: we encountered something strange as well: on 1.11.2 clustered, kerberized: request replication worked well. on 1.11.3 clustered, kerberized: request replication did not work, unless you specify, and set nifi.security.keyPasswd to the very same password as the nifi.security.keystorePasswd For us this resolved the issue. Best regards, Endre Sent with [ProtonMail](https://protonmail.com) Secure Email. ‐‐‐ Original Message ‐‐‐ On Tuesday, March 3, 2020 12:40 PM, Ryan Ward wrote: > Hi Joe - Did you resolve your issue? If so I am wondering what the fix was as > I'm seeing the same error on my cluster. > > On Thu, Feb 27, 2020 at 3:13 AM Endre Kovacs > wrote: > >> Hi Joe, >> >> 1. Have you tried connecting/debugging with openssl? From one pod to the >> other: >> (openssl s_client -debug -CAfile ca-bundle-signing-node-certificates.crt >> -cert my-client-cert.crt -connect >> nifi-3.nifi-headless.lizardspock.svc.cluster.local:6007) >> >> 2. certs can also be verified by: >> openssl verify -verbose -CAfile ca-bundle.crt my-client-cert.crt >> >> 3. Can you check if no intermediary CAs are missing from the nodes >> truststore? >> >> 4. This exception is coming from inter-node communication (replication of >> request from one node to the other). This means that it is unrelated to >> external user's authentication by client certificate. The question is: is >> your inter node communication secured by the trusted root CA (that you are >> sure that the CA cert is present in the trust store) or is it secured by >> selfsigned CA (which's CA may be lacking from your truststore)? >> >> 5. `nifi.security.needClientAuth` is not part of NiFi properties any more. >> If SSL is turned on, and no `nifi.security.user.login.identity.provider` is >> set, then client cert based auth is the default. But supplying this property >> have no detrimental effect anyhow. >> >> Best regards, >> Endre >> >> Sent with ProtonMail Secure Email. >> >> ‐‐‐ Original Message ‐‐‐ >> On Wednesday, February 26, 2020 6:22 PM, Joe Gresock jgres...@gmail.com >> wrote: >> >>> Were there any changes with how the trust store is used in 1.11.3? I had a >>> 1.11.0 deployment working with the following settings, but when I deployed >>> 1.11.3, the cluster can't seem to replicate requests to itself: >>> nifi.remote.input.host= >>> nifi.remote.input.secure=true >>> nifi.remote.input.socket.port=32440 >>> nifi.remote.input.http.enabled=true >>> nifi.cluster.protocol.is.secure=true >>> nifi.cluster.is.node=true >>> nifi.cluster.node.address=nifi-3.nifi-headless.lizardspock.svc.cluster.local >>> nifi.cluster.node.protocol.port=6007 >>> nifi.web.https.host=nifi-3.nifi-headless.lizardspock.svc.cluster.local >>> nifi.web.https.port=8443 >>> nifi.security.keystore=./conf/keystore.jks >>> nifi.security.keystoreType=jks >>> nifi.security.keystorePasswd= >>> nifi.security.keyPasswd= >>> nifi.security.truststore=./conf/truststore.jks >>> nifi.security.truststoreType=jks >>> nifi.security.truststorePasswd= >>> nifi.security.needClientAuth=true >>> A trusted client cert that worked against the old cluster is getting the >>> same trust error (PKIX path building failed). I've verified that the >>> client cert was issued by an issuer that is definitely in the >>> ./conf/truststore.jks as a trustedCertEntry. >>> 2020-02-26 17:11:09,573 WARN [Replicate Request Thread-7] >>> o.a.n.c.c.h.r.ThreadPoolRequestReplicator >>> javax.net.ssl.SSLHandshakeException: >>> sun.security.validator.ValidatorException: PKIX path building failed: >>> sun.security.provider.certpath.SunCertPathBuilderException: unable to find >>> valid certification path to r >>> equested target >>> at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) >>> at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946) >>> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:316) >>> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310) >>> at >>> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639) >>> at >>> sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223) >>> at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037) >>> at sun.security.ssl.Handshaker.process_record(Handshaker.java:965) >>> at >>> sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1064) >>> at >>> sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367) >>> at >>> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395) >>> at >>> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379) >>> at >>> okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:302) >>> at >>> okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:270) >>> at >>> okhttp3.internal.connection.RealConnection.connect(RealConnection.java:162) >>> at >>>
Re: 1.11.3 trust store error
Hi Joe - Did you resolve your issue? If so I am wondering what the fix was as I'm seeing the same error on my cluster. On Thu, Feb 27, 2020 at 3:13 AM Endre Kovacs wrote: > Hi Joe, > > 1. Have you tried connecting/debugging with openssl? From one pod to the > other: > (openssl s_client -debug -CAfile > ca-bundle-signing-node-certificates.crt -cert my-client-cert.crt -connect > nifi-3.nifi-headless.lizardspock.svc.cluster.local:6007) > > > 2. certs can also be verified by: > openssl verify -verbose -CAfile ca-bundle.crt my-client-cert.crt > > 3. Can you check if no intermediary CAs are missing from the nodes > truststore? > > 4. This exception is coming from inter-node communication (replication of > request from one node to the other). This means that it is unrelated to > external user's authentication by client certificate. The question is: is > your inter node communication secured by the trusted root CA (that you are > sure that the CA cert is present in the trust store) or is it secured by > selfsigned CA (which's CA may be lacking from your truststore)? > > 5. `nifi.security.needClientAuth` is not part of NiFi properties any > more. If SSL is turned on, and no > `nifi.security.user.login.identity.provider` is set, then client cert based > auth is the default. But supplying this property have no detrimental effect > anyhow. > > Best regards, > Endre > > > Sent with ProtonMail Secure Email. > > ‐‐‐ Original Message ‐‐‐ > On Wednesday, February 26, 2020 6:22 PM, Joe Gresock > jgres...@gmail.com wrote: > > > > Were there any changes with how the trust store is used in 1.11.3? I had > a > > 1.11.0 deployment working with the following settings, but when I > deployed > > 1.11.3, the cluster can't seem to replicate requests to itself: > > nifi.remote.input.host= > > nifi.remote.input.secure=true > > nifi.remote.input.socket.port=32440 > > nifi.remote.input.http.enabled=true > > nifi.cluster.protocol.is.secure=true > > nifi.cluster.is.node=true > > > nifi.cluster.node.address=nifi-3.nifi-headless.lizardspock.svc.cluster.local > > nifi.cluster.node.protocol.port=6007 > > nifi.web.https.host=nifi-3.nifi-headless.lizardspock.svc.cluster.local > > nifi.web.https.port=8443 > > nifi.security.keystore=./conf/keystore.jks > > nifi.security.keystoreType=jks > > nifi.security.keystorePasswd= > > nifi.security.keyPasswd= > > nifi.security.truststore=./conf/truststore.jks > > nifi.security.truststoreType=jks > > nifi.security.truststorePasswd= > > nifi.security.needClientAuth=true > > A trusted client cert that worked against the old cluster is getting the > > same trust error (PKIX path building failed). I've verified that the > > client cert was issued by an issuer that is definitely in the > > ./conf/truststore.jks as a trustedCertEntry. > > 2020-02-26 17:11:09,573 WARN [Replicate Request Thread-7] > > o.a.n.c.c.h.r.ThreadPoolRequestReplicator > > javax.net.ssl.SSLHandshakeException: > > sun.security.validator.ValidatorException: PKIX path building failed: > > sun.security.provider.certpath.SunCertPathBuilderException: unable to > find > > valid certification path to r > > equested target > > at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) > > at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946) > > at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:316) > > at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310) > > at > > > sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639) > > at > > > sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223) > > at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037) > > at sun.security.ssl.Handshaker.process_record(Handshaker.java:965) > > at > > sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1064) > > at > > > sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367) > > at > > sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395) > > at > > sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379) > > at > > > okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:302) > > at > > > okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:270) > > at > > > okhttp3.internal.connection.RealConnection.connect(RealConnection.java:162) > > at > > > okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:257) > > at > > > okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:135) > > at > > > okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:114) > > at > > > okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42) > > at > > > okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) > > at > > > okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) > > at > > >
nifi connection with hive
please I want to ask if nifi version 1.9.2 puthivestreaming processor compataible with hive version 2.3.2 , info: I used puthiveql and it works well , but with using puthivestreaming it connect to metastore and create delta files but always empty and errors appear , please any help?
