Re: 1.11.3 trust store error

[Joe Gresock]

The nifi.security.keyPasswd was not filled in, so it looked like this
(which is the default configuration):

nifi.security.keyPasswd=

On Wed, Mar 4, 2020 at 11:36 AM Endre Kovacs
 wrote:

> Hi Nathan,
>
> There is already a ticket about this:
> https://issues.apache.org/jira/browse/NIFI-7219
>
> Best regards,
> Endre
>
> Sent with ProtonMail Secure Email.
>
> ‐‐‐ Original Message ‐‐‐
> On Wednesday, March 4, 2020 5:25 PM, Nathan Gough 
> wrote:
>
> > I've opened https://issues.apache.org/jira/browse/NIFI-7223 to track and
> > I'm working on a fix for this.
> >
> > Nathan
> >
> > On Tue, Mar 3, 2020 at 6:17 PM Nathan Gough thena...@gmail.com wrote:
> >
> > > Hi Joe,
> > > Just to confirm here - was the nifi.security.keyPasswd not defined at
> all
> > > in your nifi.properties? Did you have to add the property and give it
> the
> > > correct value? Or was it in the nifi.properties file but blank? Or
> were the
> > > keyPasswd and keystorePasswd different values?
> > > Thanks,
> > > Nathan
> > > On Tue, Mar 3, 2020 at 3:38 PM Joe Gresock jgres...@gmail.com wrote:
> > >
> > > > Yep, setting the nifi.security.keyPasswd to the same as
> > > > nifi.security.keystorePasswd fixed it. Thanks for the insight, Endre!
> > > > On Tue, Mar 3, 2020 at 2:01 PM Joe Witt joe.w...@gmail.com wrote:
> > > >
> > > > > relevant change I believe is here:
> > > >
> > > >
> https://github.com/apache/nifi/commit/46d3b6b0dc28f04da124be7685f82bec52e88775
> > > >
> > > > > and
> > > > > is from https://issues.apache.org/jira/browse/NIFI-6927
> > > > > It looks to me like this was fixing an improper naming/usage issue
> > > > > that
> > > > > has been present but if so we probably should have addressed not
> in this
> > > > > bug fix line. Will defer to Troy/Andy for more context and next
> steps
> > > > > On Tue, Mar 3, 2020 at 5:53 AM Joe Witt joe.w...@gmail.com wrote:
> > > > >
> > > > > > If accurateWe need to look into whether this was a mistake
> and
> > > > > > fix it
> > > > >
> > > > > > if so. And we need to reflect this in the migration guide
> > > > > > On Tue, Mar 3, 2020 at 4:40 AM Ryan Ward ryan.wa...@gmail.com
> > > > > > wrote:
> > > > >
> > > > > > > Endre - thanks that was it
> > > > > > > On Tue, Mar 3, 2020 at 6:50 AM Endre Kovacs
> > > > > > > andrewsmit...@protonmail.com.invalid wrote:
> > > > > > >
> > > > > > > > Hi,
> > > > > > > > One additional thing:
> > > > > > > > we encountered something strange as well:
> > > > > > > > on 1.11.2 clustered, kerberized: request replication worked
> well.
> > > > > > > > on 1.11.3 clustered, kerberized: request replication did not
> work,
> > > > > > > > unless
> > > > > > > > you specify, and set
> > > > > > > > nifi.security.keyPasswd
> > > > > > > > to the very same password as the
> > > > > > > > nifi.security.keystorePasswd
> > > > > > > > For us this resolved the issue.
> > > > > > > > Best regards,
> > > > > > > > Endre
> > > > > > > > Sent with ProtonMail Secure Email.
> > > > > > > > ‐‐‐ Original Message ‐‐‐
> > > > > > > > On Tuesday, March 3, 2020 12:40 PM, Ryan Ward <
> > > > > > > > ryan.wa...@gmail.com>
> > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Joe - Did you resolve your issue? If so I am wondering
> what
> > > > > > > > > the
> > > > > > > > > fix
> > > > >
> > > > > > > > was as I'm seeing the same error on my cluster.
> > > > > > > >
> > > > > > > > > On Thu, Feb 27, 2020 at 3:13 AM Endre Kovacs <
> > > > > > > > > andrewsmit...@protonmail.com.invalid> wrote:
> > > > > > > > >
> > > > > > > > > > Hi Joe,
> > > > > > > > > >
> > > > > > > > > > 1.  Have you tried connecting/debugging with openssl?
> From one
> > > > > > > > > > pod
> > > > > > > > > > to
> > > > > > > > > >
> > > > >
> > > > > > > > the other:
> > > > > > > >
> > > > > > > > > > (openssl s_client -debug -CAfile
> > > > > > > > > >
> > > > > > > >
> > > > > > > > ca-bundle-signing-node-certificates.crt -cert
> my-client-cert.crt
> > > > > > > > -connect
> > > > > > > > nifi-3.nifi-headless.lizardspock.svc.cluster.local:6007)
> > > > > > > >
> > > > > > > > > > 2.  certs can also be verified by:
> > > > > > > > > > openssl verify -verbose -CAfile ca-bundle.crt
> > > > > > > > > > my-client-cert.crt
> > > > > > > > > >
> > > > >
> > > > > > > > > > 3.  Can you check if no intermediary CAs are missing
> from the
> > > > > > > > > > nodes
> > > > > > > > > >
> > > > >
> > > > > > > > truststore?
> > > > > > > >
> > > > > > > > > > 4.  This exception is coming from inter-node
> communication
> > > > > > > > > > (replication
> > > > > > > > > > of request from one node to the other). This means
> that it is
> > > > > > > > > > unrelated
> > > > > > > > > >
> > > > > >
> > > > > > > to
> > > > > > >
> > > > > > > > external user's authentication by client certificate. The
> question
> > > > > > > > is:
> > > > >
> > > > > > > is
> > > > > > >
> > > > > > > > your inter node communication secured by the trusted root 

Re: 1.11.3 trust store error

[Endre Kovacs]

Hi Nathan,

There is already a ticket about this: 
https://issues.apache.org/jira/browse/NIFI-7219

Best regards,
Endre

Sent with ProtonMail Secure Email.

‐‐‐ Original Message ‐‐‐
On Wednesday, March 4, 2020 5:25 PM, Nathan Gough  wrote:

> I've opened https://issues.apache.org/jira/browse/NIFI-7223 to track and
> I'm working on a fix for this.
>
> Nathan
>
> On Tue, Mar 3, 2020 at 6:17 PM Nathan Gough thena...@gmail.com wrote:
>
> > Hi Joe,
> > Just to confirm here - was the nifi.security.keyPasswd not defined at all
> > in your nifi.properties? Did you have to add the property and give it the
> > correct value? Or was it in the nifi.properties file but blank? Or were the
> > keyPasswd and keystorePasswd different values?
> > Thanks,
> > Nathan
> > On Tue, Mar 3, 2020 at 3:38 PM Joe Gresock jgres...@gmail.com wrote:
> >
> > > Yep, setting the nifi.security.keyPasswd to the same as
> > > nifi.security.keystorePasswd fixed it. Thanks for the insight, Endre!
> > > On Tue, Mar 3, 2020 at 2:01 PM Joe Witt joe.w...@gmail.com wrote:
> > >
> > > > relevant change I believe is here:
> > >
> > > https://github.com/apache/nifi/commit/46d3b6b0dc28f04da124be7685f82bec52e88775
> > >
> > > > and
> > > > is from https://issues.apache.org/jira/browse/NIFI-6927
> > > > It looks to me like this was fixing an improper naming/usage issue
> > > > that
> > > > has been present but if so we probably should have addressed not in this
> > > > bug fix line. Will defer to Troy/Andy for more context and next steps
> > > > On Tue, Mar 3, 2020 at 5:53 AM Joe Witt joe.w...@gmail.com wrote:
> > > >
> > > > > If accurateWe need to look into whether this was a mistake and
> > > > > fix it
> > > >
> > > > > if so. And we need to reflect this in the migration guide
> > > > > On Tue, Mar 3, 2020 at 4:40 AM Ryan Ward ryan.wa...@gmail.com
> > > > > wrote:
> > > >
> > > > > > Endre - thanks that was it
> > > > > > On Tue, Mar 3, 2020 at 6:50 AM Endre Kovacs
> > > > > > andrewsmit...@protonmail.com.invalid wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > > One additional thing:
> > > > > > > we encountered something strange as well:
> > > > > > > on 1.11.2 clustered, kerberized: request replication worked well.
> > > > > > > on 1.11.3 clustered, kerberized: request replication did not work,
> > > > > > > unless
> > > > > > > you specify, and set
> > > > > > > nifi.security.keyPasswd
> > > > > > > to the very same password as the
> > > > > > > nifi.security.keystorePasswd
> > > > > > > For us this resolved the issue.
> > > > > > > Best regards,
> > > > > > > Endre
> > > > > > > Sent with ProtonMail Secure Email.
> > > > > > > ‐‐‐ Original Message ‐‐‐
> > > > > > > On Tuesday, March 3, 2020 12:40 PM, Ryan Ward <
> > > > > > > ryan.wa...@gmail.com>
> > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Joe - Did you resolve your issue? If so I am wondering what
> > > > > > > > the
> > > > > > > > fix
> > > >
> > > > > > > was as I'm seeing the same error on my cluster.
> > > > > > >
> > > > > > > > On Thu, Feb 27, 2020 at 3:13 AM Endre Kovacs <
> > > > > > > > andrewsmit...@protonmail.com.invalid> wrote:
> > > > > > > >
> > > > > > > > > Hi Joe,
> > > > > > > > >
> > > > > > > > > 1.  Have you tried connecting/debugging with openssl? From one
> > > > > > > > > pod
> > > > > > > > > to
> > > > > > > > >
> > > >
> > > > > > > the other:
> > > > > > >
> > > > > > > > > (openssl s_client -debug -CAfile
> > > > > > > > >
> > > > > > >
> > > > > > > ca-bundle-signing-node-certificates.crt -cert my-client-cert.crt
> > > > > > > -connect
> > > > > > > nifi-3.nifi-headless.lizardspock.svc.cluster.local:6007)
> > > > > > >
> > > > > > > > > 2.  certs can also be verified by:
> > > > > > > > > openssl verify -verbose -CAfile ca-bundle.crt
> > > > > > > > > my-client-cert.crt
> > > > > > > > >
> > > >
> > > > > > > > > 3.  Can you check if no intermediary CAs are missing from the
> > > > > > > > > nodes
> > > > > > > > >
> > > >
> > > > > > > truststore?
> > > > > > >
> > > > > > > > > 4.  This exception is coming from inter-node communication
> > > > > > > > > (replication
> > > > > > > > > of request from one node to the other). This means that 
> > > > > > > > > it is
> > > > > > > > > unrelated
> > > > > > > > >
> > > > >
> > > > > > to
> > > > > >
> > > > > > > external user's authentication by client certificate. The question
> > > > > > > is:
> > > >
> > > > > > is
> > > > > >
> > > > > > > your inter node communication secured by the trusted root CA (that
> > > > > > > you
> > > >
> > > > > > are
> > > > > >
> > > > > > > sure that the CA cert is present in the trust store) or is it
> > > > > > > secured
> > > > > > > by
> > > >
> > > > > > > selfsigned CA (which's CA may be lacking from your truststore)?
> > > > > > >
> > > > > > > > > 5.  `nifi.security.needClientAuth` is not part of NiFi
> > > > > > > > > properties
> > > > > > > > > any
> > > > > > > > >
> > > >
> > > > > > > 

Re: 1.11.3 trust store error

[Nathan Gough]

I've opened https://issues.apache.org/jira/browse/NIFI-7223 to track and
I'm working on a fix for this.

Nathan

On Tue, Mar 3, 2020 at 6:17 PM Nathan Gough  wrote:

> Hi Joe,
>
> Just to confirm here - was the nifi.security.keyPasswd not defined at all
> in your nifi.properties? Did you have to add the property and give it the
> correct value? Or was it in the nifi.properties file but blank? Or were the
> keyPasswd and keystorePasswd different values?
>
> Thanks,
> Nathan
>
> On Tue, Mar 3, 2020 at 3:38 PM Joe Gresock  wrote:
>
>> Yep, setting the nifi.security.keyPasswd to the same as
>> nifi.security.keystorePasswd fixed it.  Thanks for the insight, Endre!
>>
>> On Tue, Mar 3, 2020 at 2:01 PM Joe Witt  wrote:
>>
>> > relevant change I believe is here:
>> >
>> >
>> https://github.com/apache/nifi/commit/46d3b6b0dc28f04da124be7685f82bec52e88775
>> > and
>> > is from https://issues.apache.org/jira/browse/NIFI-6927
>> >
>> > It *looks* to me like this was fixing an improper naming/usage issue
>> that
>> > has been present but if so we probably should have addressed not in this
>> > bug fix line.  Will defer to Troy/Andy for more context and next steps
>> >
>> > On Tue, Mar 3, 2020 at 5:53 AM Joe Witt  wrote:
>> >
>> > > If accurateWe need to look into whether this was a mistake and
>> fix it
>> > > if so.  And we need to reflect this in the migration guide
>> > >
>> > > On Tue, Mar 3, 2020 at 4:40 AM Ryan Ward 
>> wrote:
>> > >
>> > >> Endre  - thanks that was it
>> > >>
>> > >> On Tue, Mar 3, 2020 at 6:50 AM Endre Kovacs
>> > >>  wrote:
>> > >>
>> > >> > Hi,
>> > >> >
>> > >> > One additional thing:
>> > >> >
>> > >> > we encountered something strange as well:
>> > >> >
>> > >> > on 1.11.2 clustered, kerberized: request replication worked well.
>> > >> >
>> > >> > on 1.11.3 clustered, kerberized: request replication did not work,
>> > >> unless
>> > >> > you specify, and set
>> > >> > nifi.security.keyPasswd
>> > >> >
>> > >> > to the very same password as the
>> > >> >
>> > >> > nifi.security.keystorePasswd
>> > >> >
>> > >> > For us this resolved the issue.
>> > >> >
>> > >> > Best regards,
>> > >> > Endre
>> > >> >
>> > >> > Sent with [ProtonMail](https://protonmail.com) Secure Email.
>> > >> >
>> > >> > ‐‐‐ Original Message ‐‐‐
>> > >> > On Tuesday, March 3, 2020 12:40 PM, Ryan Ward <
>> ryan.wa...@gmail.com>
>> > >> > wrote:
>> > >> >
>> > >> > > Hi Joe - Did you resolve your issue? If so I am wondering what
>> the
>> > fix
>> > >> > was as I'm seeing the same error on my cluster.
>> > >> > >
>> > >> > > On Thu, Feb 27, 2020 at 3:13 AM Endre Kovacs <
>> > >> > andrewsmit...@protonmail.com.invalid> wrote:
>> > >> > >
>> > >> > >> Hi Joe,
>> > >> > >>
>> > >> > >> 1.  Have you tried connecting/debugging with openssl? From one
>> pod
>> > to
>> > >> > the other:
>> > >> > >> (openssl s_client -debug -CAfile
>> > >> > ca-bundle-signing-node-certificates.crt -cert my-client-cert.crt
>> > >> -connect
>> > >> > nifi-3.nifi-headless.lizardspock.svc.cluster.local:6007)
>> > >> > >>
>> > >> > >> 2. certs can also be verified by:
>> > >> > >>  openssl verify -verbose -CAfile ca-bundle.crt
>> my-client-cert.crt
>> > >> > >>
>> > >> > >> 3.  Can you check if no intermediary CAs are missing from the
>> nodes
>> > >> > truststore?
>> > >> > >>
>> > >> > >> 4.  This exception is coming from inter-node communication
>> > >> (replication
>> > >> > of request from one node to the other). This means that it is
>> > unrelated
>> > >> to
>> > >> > external user's authentication by client certificate. The question
>> is:
>> > >> is
>> > >> > your inter node communication secured by the trusted root CA (that
>> you
>> > >> are
>> > >> > sure that the CA cert is present in the trust store) or is it
>> secured
>> > by
>> > >> > selfsigned CA (which's CA may be lacking from your truststore)?
>> > >> > >>
>> > >> > >> 5.  `nifi.security.needClientAuth` is not part of NiFi
>> properties
>> > any
>> > >> > more. If SSL is turned on, and no
>> > >> > `nifi.security.user.login.identity.provider` is set, then client
>> cert
>> > >> based
>> > >> > auth is the default. But supplying this property have no
>> detrimental
>> > >> effect
>> > >> > anyhow.
>> > >> > >>
>> > >> > >> Best regards,
>> > >> > >> Endre
>> > >> > >>
>> > >> > >> Sent with ProtonMail Secure Email.
>> > >> > >>
>> > >> > >> ‐‐‐ Original Message ‐‐‐
>> > >> > >> On Wednesday, February 26, 2020 6:22 PM, Joe Gresock
>> > >> > jgres...@gmail.com wrote:
>> > >> > >>
>> > >> > >>> Were there any changes with how the trust store is used in
>> > 1.11.3? I
>> > >> > had a
>> > >> > >>> 1.11.0 deployment working with the following settings, but
>> when I
>> > >> > deployed
>> > >> > >>> 1.11.3, the cluster can't seem to replicate requests to itself:
>> > >> > >>> nifi.remote.input.host=
>> > >> > >>> nifi.remote.input.secure=true
>> > >> > >>> nifi.remote.input.socket.port=32440
>> > >> > >>> nifi.remote.input.http.enabled=true
>> > >> > 

Re: 1.11.3 trust store error

[Nathan Gough]

Hi Joe,

Just to confirm here - was the nifi.security.keyPasswd not defined at all
in your nifi.properties? Did you have to add the property and give it the
correct value? Or was it in the nifi.properties file but blank? Or were the
keyPasswd and keystorePasswd different values?

Thanks,
Nathan

On Tue, Mar 3, 2020 at 3:38 PM Joe Gresock  wrote:

> Yep, setting the nifi.security.keyPasswd to the same as
> nifi.security.keystorePasswd fixed it.  Thanks for the insight, Endre!
>
> On Tue, Mar 3, 2020 at 2:01 PM Joe Witt  wrote:
>
> > relevant change I believe is here:
> >
> >
> https://github.com/apache/nifi/commit/46d3b6b0dc28f04da124be7685f82bec52e88775
> > and
> > is from https://issues.apache.org/jira/browse/NIFI-6927
> >
> > It *looks* to me like this was fixing an improper naming/usage issue that
> > has been present but if so we probably should have addressed not in this
> > bug fix line.  Will defer to Troy/Andy for more context and next steps
> >
> > On Tue, Mar 3, 2020 at 5:53 AM Joe Witt  wrote:
> >
> > > If accurateWe need to look into whether this was a mistake and fix
> it
> > > if so.  And we need to reflect this in the migration guide
> > >
> > > On Tue, Mar 3, 2020 at 4:40 AM Ryan Ward  wrote:
> > >
> > >> Endre  - thanks that was it
> > >>
> > >> On Tue, Mar 3, 2020 at 6:50 AM Endre Kovacs
> > >>  wrote:
> > >>
> > >> > Hi,
> > >> >
> > >> > One additional thing:
> > >> >
> > >> > we encountered something strange as well:
> > >> >
> > >> > on 1.11.2 clustered, kerberized: request replication worked well.
> > >> >
> > >> > on 1.11.3 clustered, kerberized: request replication did not work,
> > >> unless
> > >> > you specify, and set
> > >> > nifi.security.keyPasswd
> > >> >
> > >> > to the very same password as the
> > >> >
> > >> > nifi.security.keystorePasswd
> > >> >
> > >> > For us this resolved the issue.
> > >> >
> > >> > Best regards,
> > >> > Endre
> > >> >
> > >> > Sent with [ProtonMail](https://protonmail.com) Secure Email.
> > >> >
> > >> > ‐‐‐ Original Message ‐‐‐
> > >> > On Tuesday, March 3, 2020 12:40 PM, Ryan Ward  >
> > >> > wrote:
> > >> >
> > >> > > Hi Joe - Did you resolve your issue? If so I am wondering what the
> > fix
> > >> > was as I'm seeing the same error on my cluster.
> > >> > >
> > >> > > On Thu, Feb 27, 2020 at 3:13 AM Endre Kovacs <
> > >> > andrewsmit...@protonmail.com.invalid> wrote:
> > >> > >
> > >> > >> Hi Joe,
> > >> > >>
> > >> > >> 1.  Have you tried connecting/debugging with openssl? From one
> pod
> > to
> > >> > the other:
> > >> > >> (openssl s_client -debug -CAfile
> > >> > ca-bundle-signing-node-certificates.crt -cert my-client-cert.crt
> > >> -connect
> > >> > nifi-3.nifi-headless.lizardspock.svc.cluster.local:6007)
> > >> > >>
> > >> > >> 2. certs can also be verified by:
> > >> > >>  openssl verify -verbose -CAfile ca-bundle.crt my-client-cert.crt
> > >> > >>
> > >> > >> 3.  Can you check if no intermediary CAs are missing from the
> nodes
> > >> > truststore?
> > >> > >>
> > >> > >> 4.  This exception is coming from inter-node communication
> > >> (replication
> > >> > of request from one node to the other). This means that it is
> > unrelated
> > >> to
> > >> > external user's authentication by client certificate. The question
> is:
> > >> is
> > >> > your inter node communication secured by the trusted root CA (that
> you
> > >> are
> > >> > sure that the CA cert is present in the trust store) or is it
> secured
> > by
> > >> > selfsigned CA (which's CA may be lacking from your truststore)?
> > >> > >>
> > >> > >> 5.  `nifi.security.needClientAuth` is not part of NiFi properties
> > any
> > >> > more. If SSL is turned on, and no
> > >> > `nifi.security.user.login.identity.provider` is set, then client
> cert
> > >> based
> > >> > auth is the default. But supplying this property have no detrimental
> > >> effect
> > >> > anyhow.
> > >> > >>
> > >> > >> Best regards,
> > >> > >> Endre
> > >> > >>
> > >> > >> Sent with ProtonMail Secure Email.
> > >> > >>
> > >> > >> ‐‐‐ Original Message ‐‐‐
> > >> > >> On Wednesday, February 26, 2020 6:22 PM, Joe Gresock
> > >> > jgres...@gmail.com wrote:
> > >> > >>
> > >> > >>> Were there any changes with how the trust store is used in
> > 1.11.3? I
> > >> > had a
> > >> > >>> 1.11.0 deployment working with the following settings, but when
> I
> > >> > deployed
> > >> > >>> 1.11.3, the cluster can't seem to replicate requests to itself:
> > >> > >>> nifi.remote.input.host=
> > >> > >>> nifi.remote.input.secure=true
> > >> > >>> nifi.remote.input.socket.port=32440
> > >> > >>> nifi.remote.input.http.enabled=true
> > >> > >>> nifi.cluster.protocol.is.secure=true
> > >> > >>> nifi.cluster.is.node=true
> > >> > >>>
> > >> >
> > >>
> >
> nifi.cluster.node.address=nifi-3.nifi-headless.lizardspock.svc.cluster.local
> > >> > >>> nifi.cluster.node.protocol.port=6007
> > >> > >>>
> > >> nifi.web.https.host=nifi-3.nifi-headless.lizardspock.svc.cluster.local
> > >> > >>> 

Re: 1.11.3 trust store error

[Joe Gresock]

Yep, setting the nifi.security.keyPasswd to the same as
nifi.security.keystorePasswd fixed it.  Thanks for the insight, Endre!

On Tue, Mar 3, 2020 at 2:01 PM Joe Witt  wrote:

> relevant change I believe is here:
>
> https://github.com/apache/nifi/commit/46d3b6b0dc28f04da124be7685f82bec52e88775
> and
> is from https://issues.apache.org/jira/browse/NIFI-6927
>
> It *looks* to me like this was fixing an improper naming/usage issue that
> has been present but if so we probably should have addressed not in this
> bug fix line.  Will defer to Troy/Andy for more context and next steps
>
> On Tue, Mar 3, 2020 at 5:53 AM Joe Witt  wrote:
>
> > If accurateWe need to look into whether this was a mistake and fix it
> > if so.  And we need to reflect this in the migration guide
> >
> > On Tue, Mar 3, 2020 at 4:40 AM Ryan Ward  wrote:
> >
> >> Endre  - thanks that was it
> >>
> >> On Tue, Mar 3, 2020 at 6:50 AM Endre Kovacs
> >>  wrote:
> >>
> >> > Hi,
> >> >
> >> > One additional thing:
> >> >
> >> > we encountered something strange as well:
> >> >
> >> > on 1.11.2 clustered, kerberized: request replication worked well.
> >> >
> >> > on 1.11.3 clustered, kerberized: request replication did not work,
> >> unless
> >> > you specify, and set
> >> > nifi.security.keyPasswd
> >> >
> >> > to the very same password as the
> >> >
> >> > nifi.security.keystorePasswd
> >> >
> >> > For us this resolved the issue.
> >> >
> >> > Best regards,
> >> > Endre
> >> >
> >> > Sent with [ProtonMail](https://protonmail.com) Secure Email.
> >> >
> >> > ‐‐‐ Original Message ‐‐‐
> >> > On Tuesday, March 3, 2020 12:40 PM, Ryan Ward 
> >> > wrote:
> >> >
> >> > > Hi Joe - Did you resolve your issue? If so I am wondering what the
> fix
> >> > was as I'm seeing the same error on my cluster.
> >> > >
> >> > > On Thu, Feb 27, 2020 at 3:13 AM Endre Kovacs <
> >> > andrewsmit...@protonmail.com.invalid> wrote:
> >> > >
> >> > >> Hi Joe,
> >> > >>
> >> > >> 1.  Have you tried connecting/debugging with openssl? From one pod
> to
> >> > the other:
> >> > >> (openssl s_client -debug -CAfile
> >> > ca-bundle-signing-node-certificates.crt -cert my-client-cert.crt
> >> -connect
> >> > nifi-3.nifi-headless.lizardspock.svc.cluster.local:6007)
> >> > >>
> >> > >> 2. certs can also be verified by:
> >> > >>  openssl verify -verbose -CAfile ca-bundle.crt my-client-cert.crt
> >> > >>
> >> > >> 3.  Can you check if no intermediary CAs are missing from the nodes
> >> > truststore?
> >> > >>
> >> > >> 4.  This exception is coming from inter-node communication
> >> (replication
> >> > of request from one node to the other). This means that it is
> unrelated
> >> to
> >> > external user's authentication by client certificate. The question is:
> >> is
> >> > your inter node communication secured by the trusted root CA (that you
> >> are
> >> > sure that the CA cert is present in the trust store) or is it secured
> by
> >> > selfsigned CA (which's CA may be lacking from your truststore)?
> >> > >>
> >> > >> 5.  `nifi.security.needClientAuth` is not part of NiFi properties
> any
> >> > more. If SSL is turned on, and no
> >> > `nifi.security.user.login.identity.provider` is set, then client cert
> >> based
> >> > auth is the default. But supplying this property have no detrimental
> >> effect
> >> > anyhow.
> >> > >>
> >> > >> Best regards,
> >> > >> Endre
> >> > >>
> >> > >> Sent with ProtonMail Secure Email.
> >> > >>
> >> > >> ‐‐‐ Original Message ‐‐‐
> >> > >> On Wednesday, February 26, 2020 6:22 PM, Joe Gresock
> >> > jgres...@gmail.com wrote:
> >> > >>
> >> > >>> Were there any changes with how the trust store is used in
> 1.11.3? I
> >> > had a
> >> > >>> 1.11.0 deployment working with the following settings, but when I
> >> > deployed
> >> > >>> 1.11.3, the cluster can't seem to replicate requests to itself:
> >> > >>> nifi.remote.input.host=
> >> > >>> nifi.remote.input.secure=true
> >> > >>> nifi.remote.input.socket.port=32440
> >> > >>> nifi.remote.input.http.enabled=true
> >> > >>> nifi.cluster.protocol.is.secure=true
> >> > >>> nifi.cluster.is.node=true
> >> > >>>
> >> >
> >>
> nifi.cluster.node.address=nifi-3.nifi-headless.lizardspock.svc.cluster.local
> >> > >>> nifi.cluster.node.protocol.port=6007
> >> > >>>
> >> nifi.web.https.host=nifi-3.nifi-headless.lizardspock.svc.cluster.local
> >> > >>> nifi.web.https.port=8443
> >> > >>> nifi.security.keystore=./conf/keystore.jks
> >> > >>> nifi.security.keystoreType=jks
> >> > >>> nifi.security.keystorePasswd=
> >> > >>> nifi.security.keyPasswd=
> >> > >>> nifi.security.truststore=./conf/truststore.jks
> >> > >>> nifi.security.truststoreType=jks
> >> > >>> nifi.security.truststorePasswd=
> >> > >>> nifi.security.needClientAuth=true
> >> > >>> A trusted client cert that worked against the old cluster is
> getting
> >> > the
> >> > >>> same trust error (PKIX path building failed). I've verified that
> the
> >> > >>> client cert was issued by an issuer that is definitely in the
> 

Re: 1.11.3 trust store error

[Joe Witt]

relevant change I believe is here:
https://github.com/apache/nifi/commit/46d3b6b0dc28f04da124be7685f82bec52e88775
and
is from https://issues.apache.org/jira/browse/NIFI-6927

It *looks* to me like this was fixing an improper naming/usage issue that
has been present but if so we probably should have addressed not in this
bug fix line.  Will defer to Troy/Andy for more context and next steps

On Tue, Mar 3, 2020 at 5:53 AM Joe Witt  wrote:

> If accurateWe need to look into whether this was a mistake and fix it
> if so.  And we need to reflect this in the migration guide
>
> On Tue, Mar 3, 2020 at 4:40 AM Ryan Ward  wrote:
>
>> Endre  - thanks that was it
>>
>> On Tue, Mar 3, 2020 at 6:50 AM Endre Kovacs
>>  wrote:
>>
>> > Hi,
>> >
>> > One additional thing:
>> >
>> > we encountered something strange as well:
>> >
>> > on 1.11.2 clustered, kerberized: request replication worked well.
>> >
>> > on 1.11.3 clustered, kerberized: request replication did not work,
>> unless
>> > you specify, and set
>> > nifi.security.keyPasswd
>> >
>> > to the very same password as the
>> >
>> > nifi.security.keystorePasswd
>> >
>> > For us this resolved the issue.
>> >
>> > Best regards,
>> > Endre
>> >
>> > Sent with [ProtonMail](https://protonmail.com) Secure Email.
>> >
>> > ‐‐‐ Original Message ‐‐‐
>> > On Tuesday, March 3, 2020 12:40 PM, Ryan Ward 
>> > wrote:
>> >
>> > > Hi Joe - Did you resolve your issue? If so I am wondering what the fix
>> > was as I'm seeing the same error on my cluster.
>> > >
>> > > On Thu, Feb 27, 2020 at 3:13 AM Endre Kovacs <
>> > andrewsmit...@protonmail.com.invalid> wrote:
>> > >
>> > >> Hi Joe,
>> > >>
>> > >> 1.  Have you tried connecting/debugging with openssl? From one pod to
>> > the other:
>> > >> (openssl s_client -debug -CAfile
>> > ca-bundle-signing-node-certificates.crt -cert my-client-cert.crt
>> -connect
>> > nifi-3.nifi-headless.lizardspock.svc.cluster.local:6007)
>> > >>
>> > >> 2. certs can also be verified by:
>> > >>  openssl verify -verbose -CAfile ca-bundle.crt my-client-cert.crt
>> > >>
>> > >> 3.  Can you check if no intermediary CAs are missing from the nodes
>> > truststore?
>> > >>
>> > >> 4.  This exception is coming from inter-node communication
>> (replication
>> > of request from one node to the other). This means that it is unrelated
>> to
>> > external user's authentication by client certificate. The question is:
>> is
>> > your inter node communication secured by the trusted root CA (that you
>> are
>> > sure that the CA cert is present in the trust store) or is it secured by
>> > selfsigned CA (which's CA may be lacking from your truststore)?
>> > >>
>> > >> 5.  `nifi.security.needClientAuth` is not part of NiFi properties any
>> > more. If SSL is turned on, and no
>> > `nifi.security.user.login.identity.provider` is set, then client cert
>> based
>> > auth is the default. But supplying this property have no detrimental
>> effect
>> > anyhow.
>> > >>
>> > >> Best regards,
>> > >> Endre
>> > >>
>> > >> Sent with ProtonMail Secure Email.
>> > >>
>> > >> ‐‐‐ Original Message ‐‐‐
>> > >> On Wednesday, February 26, 2020 6:22 PM, Joe Gresock
>> > jgres...@gmail.com wrote:
>> > >>
>> > >>> Were there any changes with how the trust store is used in 1.11.3? I
>> > had a
>> > >>> 1.11.0 deployment working with the following settings, but when I
>> > deployed
>> > >>> 1.11.3, the cluster can't seem to replicate requests to itself:
>> > >>> nifi.remote.input.host=
>> > >>> nifi.remote.input.secure=true
>> > >>> nifi.remote.input.socket.port=32440
>> > >>> nifi.remote.input.http.enabled=true
>> > >>> nifi.cluster.protocol.is.secure=true
>> > >>> nifi.cluster.is.node=true
>> > >>>
>> >
>> nifi.cluster.node.address=nifi-3.nifi-headless.lizardspock.svc.cluster.local
>> > >>> nifi.cluster.node.protocol.port=6007
>> > >>>
>> nifi.web.https.host=nifi-3.nifi-headless.lizardspock.svc.cluster.local
>> > >>> nifi.web.https.port=8443
>> > >>> nifi.security.keystore=./conf/keystore.jks
>> > >>> nifi.security.keystoreType=jks
>> > >>> nifi.security.keystorePasswd=
>> > >>> nifi.security.keyPasswd=
>> > >>> nifi.security.truststore=./conf/truststore.jks
>> > >>> nifi.security.truststoreType=jks
>> > >>> nifi.security.truststorePasswd=
>> > >>> nifi.security.needClientAuth=true
>> > >>> A trusted client cert that worked against the old cluster is getting
>> > the
>> > >>> same trust error (PKIX path building failed). I've verified that the
>> > >>> client cert was issued by an issuer that is definitely in the
>> > >>> ./conf/truststore.jks as a trustedCertEntry.
>> > >>> 2020-02-26 17:11:09,573 WARN [Replicate Request Thread-7]
>> > >>> o.a.n.c.c.h.r.ThreadPoolRequestReplicator
>> > >>> javax.net.ssl.SSLHandshakeException:
>> > >>> sun.security.validator.ValidatorException: PKIX path building
>> failed:
>> > >>> sun.security.provider.certpath.SunCertPathBuilderException: unable
>> to
>> > find
>> > >>> valid certification path to r
>> > >>> equested 

Re: 1.11.3 trust store error

[Joe Witt]

If accurateWe need to look into whether this was a mistake and fix it
if so.  And we need to reflect this in the migration guide

On Tue, Mar 3, 2020 at 4:40 AM Ryan Ward  wrote:

> Endre  - thanks that was it
>
> On Tue, Mar 3, 2020 at 6:50 AM Endre Kovacs
>  wrote:
>
> > Hi,
> >
> > One additional thing:
> >
> > we encountered something strange as well:
> >
> > on 1.11.2 clustered, kerberized: request replication worked well.
> >
> > on 1.11.3 clustered, kerberized: request replication did not work, unless
> > you specify, and set
> > nifi.security.keyPasswd
> >
> > to the very same password as the
> >
> > nifi.security.keystorePasswd
> >
> > For us this resolved the issue.
> >
> > Best regards,
> > Endre
> >
> > Sent with [ProtonMail](https://protonmail.com) Secure Email.
> >
> > ‐‐‐ Original Message ‐‐‐
> > On Tuesday, March 3, 2020 12:40 PM, Ryan Ward 
> > wrote:
> >
> > > Hi Joe - Did you resolve your issue? If so I am wondering what the fix
> > was as I'm seeing the same error on my cluster.
> > >
> > > On Thu, Feb 27, 2020 at 3:13 AM Endre Kovacs <
> > andrewsmit...@protonmail.com.invalid> wrote:
> > >
> > >> Hi Joe,
> > >>
> > >> 1.  Have you tried connecting/debugging with openssl? From one pod to
> > the other:
> > >> (openssl s_client -debug -CAfile
> > ca-bundle-signing-node-certificates.crt -cert my-client-cert.crt -connect
> > nifi-3.nifi-headless.lizardspock.svc.cluster.local:6007)
> > >>
> > >> 2. certs can also be verified by:
> > >>  openssl verify -verbose -CAfile ca-bundle.crt my-client-cert.crt
> > >>
> > >> 3.  Can you check if no intermediary CAs are missing from the nodes
> > truststore?
> > >>
> > >> 4.  This exception is coming from inter-node communication
> (replication
> > of request from one node to the other). This means that it is unrelated
> to
> > external user's authentication by client certificate. The question is: is
> > your inter node communication secured by the trusted root CA (that you
> are
> > sure that the CA cert is present in the trust store) or is it secured by
> > selfsigned CA (which's CA may be lacking from your truststore)?
> > >>
> > >> 5.  `nifi.security.needClientAuth` is not part of NiFi properties any
> > more. If SSL is turned on, and no
> > `nifi.security.user.login.identity.provider` is set, then client cert
> based
> > auth is the default. But supplying this property have no detrimental
> effect
> > anyhow.
> > >>
> > >> Best regards,
> > >> Endre
> > >>
> > >> Sent with ProtonMail Secure Email.
> > >>
> > >> ‐‐‐ Original Message ‐‐‐
> > >> On Wednesday, February 26, 2020 6:22 PM, Joe Gresock
> > jgres...@gmail.com wrote:
> > >>
> > >>> Were there any changes with how the trust store is used in 1.11.3? I
> > had a
> > >>> 1.11.0 deployment working with the following settings, but when I
> > deployed
> > >>> 1.11.3, the cluster can't seem to replicate requests to itself:
> > >>> nifi.remote.input.host=
> > >>> nifi.remote.input.secure=true
> > >>> nifi.remote.input.socket.port=32440
> > >>> nifi.remote.input.http.enabled=true
> > >>> nifi.cluster.protocol.is.secure=true
> > >>> nifi.cluster.is.node=true
> > >>>
> >
> nifi.cluster.node.address=nifi-3.nifi-headless.lizardspock.svc.cluster.local
> > >>> nifi.cluster.node.protocol.port=6007
> > >>>
> nifi.web.https.host=nifi-3.nifi-headless.lizardspock.svc.cluster.local
> > >>> nifi.web.https.port=8443
> > >>> nifi.security.keystore=./conf/keystore.jks
> > >>> nifi.security.keystoreType=jks
> > >>> nifi.security.keystorePasswd=
> > >>> nifi.security.keyPasswd=
> > >>> nifi.security.truststore=./conf/truststore.jks
> > >>> nifi.security.truststoreType=jks
> > >>> nifi.security.truststorePasswd=
> > >>> nifi.security.needClientAuth=true
> > >>> A trusted client cert that worked against the old cluster is getting
> > the
> > >>> same trust error (PKIX path building failed). I've verified that the
> > >>> client cert was issued by an issuer that is definitely in the
> > >>> ./conf/truststore.jks as a trustedCertEntry.
> > >>> 2020-02-26 17:11:09,573 WARN [Replicate Request Thread-7]
> > >>> o.a.n.c.c.h.r.ThreadPoolRequestReplicator
> > >>> javax.net.ssl.SSLHandshakeException:
> > >>> sun.security.validator.ValidatorException: PKIX path building failed:
> > >>> sun.security.provider.certpath.SunCertPathBuilderException: unable to
> > find
> > >>> valid certification path to r
> > >>> equested target
> > >>> at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
> > >>> at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946)
> > >>> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:316)
> > >>> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310)
> > >>> at
> > >>>
> >
> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639)
> > >>> at
> > >>>
> >
> sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223)
> > >>> at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037)
> > >>> at 

Re: 1.11.3 trust store error

[Ryan Ward]

Endre  - thanks that was it

On Tue, Mar 3, 2020 at 6:50 AM Endre Kovacs
 wrote:

> Hi,
>
> One additional thing:
>
> we encountered something strange as well:
>
> on 1.11.2 clustered, kerberized: request replication worked well.
>
> on 1.11.3 clustered, kerberized: request replication did not work, unless
> you specify, and set
> nifi.security.keyPasswd
>
> to the very same password as the
>
> nifi.security.keystorePasswd
>
> For us this resolved the issue.
>
> Best regards,
> Endre
>
> Sent with [ProtonMail](https://protonmail.com) Secure Email.
>
> ‐‐‐ Original Message ‐‐‐
> On Tuesday, March 3, 2020 12:40 PM, Ryan Ward 
> wrote:
>
> > Hi Joe - Did you resolve your issue? If so I am wondering what the fix
> was as I'm seeing the same error on my cluster.
> >
> > On Thu, Feb 27, 2020 at 3:13 AM Endre Kovacs <
> andrewsmit...@protonmail.com.invalid> wrote:
> >
> >> Hi Joe,
> >>
> >> 1.  Have you tried connecting/debugging with openssl? From one pod to
> the other:
> >> (openssl s_client -debug -CAfile
> ca-bundle-signing-node-certificates.crt -cert my-client-cert.crt -connect
> nifi-3.nifi-headless.lizardspock.svc.cluster.local:6007)
> >>
> >> 2. certs can also be verified by:
> >>  openssl verify -verbose -CAfile ca-bundle.crt my-client-cert.crt
> >>
> >> 3.  Can you check if no intermediary CAs are missing from the nodes
> truststore?
> >>
> >> 4.  This exception is coming from inter-node communication (replication
> of request from one node to the other). This means that it is unrelated to
> external user's authentication by client certificate. The question is: is
> your inter node communication secured by the trusted root CA (that you are
> sure that the CA cert is present in the trust store) or is it secured by
> selfsigned CA (which's CA may be lacking from your truststore)?
> >>
> >> 5.  `nifi.security.needClientAuth` is not part of NiFi properties any
> more. If SSL is turned on, and no
> `nifi.security.user.login.identity.provider` is set, then client cert based
> auth is the default. But supplying this property have no detrimental effect
> anyhow.
> >>
> >> Best regards,
> >> Endre
> >>
> >> Sent with ProtonMail Secure Email.
> >>
> >> ‐‐‐ Original Message ‐‐‐
> >> On Wednesday, February 26, 2020 6:22 PM, Joe Gresock
> jgres...@gmail.com wrote:
> >>
> >>> Were there any changes with how the trust store is used in 1.11.3? I
> had a
> >>> 1.11.0 deployment working with the following settings, but when I
> deployed
> >>> 1.11.3, the cluster can't seem to replicate requests to itself:
> >>> nifi.remote.input.host=
> >>> nifi.remote.input.secure=true
> >>> nifi.remote.input.socket.port=32440
> >>> nifi.remote.input.http.enabled=true
> >>> nifi.cluster.protocol.is.secure=true
> >>> nifi.cluster.is.node=true
> >>>
> nifi.cluster.node.address=nifi-3.nifi-headless.lizardspock.svc.cluster.local
> >>> nifi.cluster.node.protocol.port=6007
> >>> nifi.web.https.host=nifi-3.nifi-headless.lizardspock.svc.cluster.local
> >>> nifi.web.https.port=8443
> >>> nifi.security.keystore=./conf/keystore.jks
> >>> nifi.security.keystoreType=jks
> >>> nifi.security.keystorePasswd=
> >>> nifi.security.keyPasswd=
> >>> nifi.security.truststore=./conf/truststore.jks
> >>> nifi.security.truststoreType=jks
> >>> nifi.security.truststorePasswd=
> >>> nifi.security.needClientAuth=true
> >>> A trusted client cert that worked against the old cluster is getting
> the
> >>> same trust error (PKIX path building failed). I've verified that the
> >>> client cert was issued by an issuer that is definitely in the
> >>> ./conf/truststore.jks as a trustedCertEntry.
> >>> 2020-02-26 17:11:09,573 WARN [Replicate Request Thread-7]
> >>> o.a.n.c.c.h.r.ThreadPoolRequestReplicator
> >>> javax.net.ssl.SSLHandshakeException:
> >>> sun.security.validator.ValidatorException: PKIX path building failed:
> >>> sun.security.provider.certpath.SunCertPathBuilderException: unable to
> find
> >>> valid certification path to r
> >>> equested target
> >>> at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
> >>> at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946)
> >>> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:316)
> >>> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310)
> >>> at
> >>>
> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639)
> >>> at
> >>>
> sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223)
> >>> at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037)
> >>> at sun.security.ssl.Handshaker.process_record(Handshaker.java:965)
> >>> at
> >>> sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1064)
> >>> at
> >>>
> sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367)
> >>> at
> >>> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395)
> >>> at
> >>> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379)
> >>> at
> >>>
> 

Re: 1.11.3 trust store error

[Endre Kovacs]

Hi,

One additional thing:

we encountered something strange as well:

on 1.11.2 clustered, kerberized: request replication worked well.

on 1.11.3 clustered, kerberized: request replication did not work, unless you 
specify, and set
nifi.security.keyPasswd

to the very same password as the

nifi.security.keystorePasswd

For us this resolved the issue.

Best regards,
Endre

Sent with [ProtonMail](https://protonmail.com) Secure Email.

‐‐‐ Original Message ‐‐‐
On Tuesday, March 3, 2020 12:40 PM, Ryan Ward  wrote:

> Hi Joe - Did you resolve your issue? If so I am wondering what the fix was as 
> I'm seeing the same error on my cluster.
>
> On Thu, Feb 27, 2020 at 3:13 AM Endre Kovacs 
>  wrote:
>
>> Hi Joe,
>>
>> 1.  Have you tried connecting/debugging with openssl? From one pod to the 
>> other:
>> (openssl s_client -debug -CAfile ca-bundle-signing-node-certificates.crt 
>> -cert my-client-cert.crt -connect 
>> nifi-3.nifi-headless.lizardspock.svc.cluster.local:6007)
>>
>> 2. certs can also be verified by:
>>  openssl verify -verbose -CAfile ca-bundle.crt my-client-cert.crt
>>
>> 3.  Can you check if no intermediary CAs are missing from the nodes 
>> truststore?
>>
>> 4.  This exception is coming from inter-node communication (replication of 
>> request from one node to the other). This means that it is unrelated to 
>> external user's authentication by client certificate. The question is: is 
>> your inter node communication secured by the trusted root CA (that you are 
>> sure that the CA cert is present in the trust store) or is it secured by 
>> selfsigned CA (which's CA may be lacking from your truststore)?
>>
>> 5.  `nifi.security.needClientAuth` is not part of NiFi properties any more. 
>> If SSL is turned on, and no `nifi.security.user.login.identity.provider` is 
>> set, then client cert based auth is the default. But supplying this property 
>> have no detrimental effect anyhow.
>>
>> Best regards,
>> Endre
>>
>> Sent with ProtonMail Secure Email.
>>
>> ‐‐‐ Original Message ‐‐‐
>> On Wednesday, February 26, 2020 6:22 PM, Joe Gresock jgres...@gmail.com 
>> wrote:
>>
>>> Were there any changes with how the trust store is used in 1.11.3? I had a
>>> 1.11.0 deployment working with the following settings, but when I deployed
>>> 1.11.3, the cluster can't seem to replicate requests to itself:
>>> nifi.remote.input.host=
>>> nifi.remote.input.secure=true
>>> nifi.remote.input.socket.port=32440
>>> nifi.remote.input.http.enabled=true
>>> nifi.cluster.protocol.is.secure=true
>>> nifi.cluster.is.node=true
>>> nifi.cluster.node.address=nifi-3.nifi-headless.lizardspock.svc.cluster.local
>>> nifi.cluster.node.protocol.port=6007
>>> nifi.web.https.host=nifi-3.nifi-headless.lizardspock.svc.cluster.local
>>> nifi.web.https.port=8443
>>> nifi.security.keystore=./conf/keystore.jks
>>> nifi.security.keystoreType=jks
>>> nifi.security.keystorePasswd=
>>> nifi.security.keyPasswd=
>>> nifi.security.truststore=./conf/truststore.jks
>>> nifi.security.truststoreType=jks
>>> nifi.security.truststorePasswd=
>>> nifi.security.needClientAuth=true
>>> A trusted client cert that worked against the old cluster is getting the
>>> same trust error (PKIX path building failed). I've verified that the
>>> client cert was issued by an issuer that is definitely in the
>>> ./conf/truststore.jks as a trustedCertEntry.
>>> 2020-02-26 17:11:09,573 WARN [Replicate Request Thread-7]
>>> o.a.n.c.c.h.r.ThreadPoolRequestReplicator
>>> javax.net.ssl.SSLHandshakeException:
>>> sun.security.validator.ValidatorException: PKIX path building failed:
>>> sun.security.provider.certpath.SunCertPathBuilderException: unable to find
>>> valid certification path to r
>>> equested target
>>> at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
>>> at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946)
>>> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:316)
>>> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310)
>>> at
>>> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639)
>>> at
>>> sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223)
>>> at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037)
>>> at sun.security.ssl.Handshaker.process_record(Handshaker.java:965)
>>> at
>>> sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1064)
>>> at
>>> sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367)
>>> at
>>> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395)
>>> at
>>> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379)
>>> at
>>> okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:302)
>>> at
>>> okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:270)
>>> at
>>> okhttp3.internal.connection.RealConnection.connect(RealConnection.java:162)
>>> at
>>> 

Re: 1.11.3 trust store error

[Ryan Ward]

Hi Joe - Did you resolve your issue? If so I am wondering what the fix was
as I'm seeing the same error on my cluster.

On Thu, Feb 27, 2020 at 3:13 AM Endre Kovacs
 wrote:

> Hi Joe,
>
> 1.  Have you tried connecting/debugging with openssl? From one pod to the
> other:
> (openssl s_client -debug -CAfile
> ca-bundle-signing-node-certificates.crt -cert my-client-cert.crt -connect
> nifi-3.nifi-headless.lizardspock.svc.cluster.local:6007)
>
>
> 2. certs can also be verified by:
>  openssl verify -verbose -CAfile ca-bundle.crt my-client-cert.crt
>
> 3.  Can you check if no intermediary CAs are missing from the nodes
> truststore?
>
> 4.  This exception is coming from inter-node communication (replication of
> request from one node to the other). This means that it is unrelated to
> external user's authentication by client certificate. The question is: is
> your inter node communication secured by the trusted root CA (that you are
> sure that the CA cert is present in the trust store) or is it secured by
> selfsigned CA (which's CA may be lacking from your truststore)?
>
> 5.  `nifi.security.needClientAuth` is not part of NiFi properties any
> more. If SSL is turned on, and no
> `nifi.security.user.login.identity.provider` is set, then client cert based
> auth is the default. But supplying this property have no detrimental effect
> anyhow.
>
> Best regards,
> Endre
>
>
> Sent with ProtonMail Secure Email.
>
> ‐‐‐ Original Message ‐‐‐
> On Wednesday, February 26, 2020 6:22 PM, Joe Gresock
> jgres...@gmail.com wrote:
>
>
> > Were there any changes with how the trust store is used in 1.11.3? I had
> a
> > 1.11.0 deployment working with the following settings, but when I
> deployed
> > 1.11.3, the cluster can't seem to replicate requests to itself:
> > nifi.remote.input.host=
> > nifi.remote.input.secure=true
> > nifi.remote.input.socket.port=32440
> > nifi.remote.input.http.enabled=true
> > nifi.cluster.protocol.is.secure=true
> > nifi.cluster.is.node=true
> >
> nifi.cluster.node.address=nifi-3.nifi-headless.lizardspock.svc.cluster.local
> > nifi.cluster.node.protocol.port=6007
> > nifi.web.https.host=nifi-3.nifi-headless.lizardspock.svc.cluster.local
> > nifi.web.https.port=8443
> > nifi.security.keystore=./conf/keystore.jks
> > nifi.security.keystoreType=jks
> > nifi.security.keystorePasswd=
> > nifi.security.keyPasswd=
> > nifi.security.truststore=./conf/truststore.jks
> > nifi.security.truststoreType=jks
> > nifi.security.truststorePasswd=
> > nifi.security.needClientAuth=true
> > A trusted client cert that worked against the old cluster is getting the
> > same trust error (PKIX path building failed). I've verified that the
> > client cert was issued by an issuer that is definitely in the
> > ./conf/truststore.jks as a trustedCertEntry.
> > 2020-02-26 17:11:09,573 WARN [Replicate Request Thread-7]
> > o.a.n.c.c.h.r.ThreadPoolRequestReplicator
> > javax.net.ssl.SSLHandshakeException:
> > sun.security.validator.ValidatorException: PKIX path building failed:
> > sun.security.provider.certpath.SunCertPathBuilderException: unable to
> find
> > valid certification path to r
> > equested target
> > at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
> > at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946)
> > at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:316)
> > at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310)
> > at
> >
> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639)
> > at
> >
> sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223)
> > at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037)
> > at sun.security.ssl.Handshaker.process_record(Handshaker.java:965)
> > at
> > sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1064)
> > at
> >
> sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367)
> > at
> > sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395)
> > at
> > sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379)
> > at
> >
> okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:302)
> > at
> >
> okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:270)
> > at
> >
> okhttp3.internal.connection.RealConnection.connect(RealConnection.java:162)
> > at
> >
> okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:257)
> > at
> >
> okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:135)
> > at
> >
> okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:114)
> > at
> >
> okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42)
> > at
> >
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
> > at
> >
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
> > at
> >
> 

Re: 1.11.3 trust store error

[Endre Kovacs]

Hi Joe,

1.  Have you tried connecting/debugging with openssl? From one pod to the other:
(openssl s_client -debug -CAfile ca-bundle-signing-node-certificates.crt 
-cert my-client-cert.crt -connect 
nifi-3.nifi-headless.lizardspock.svc.cluster.local:6007)


2. certs can also be verified by:
 openssl verify -verbose -CAfile ca-bundle.crt my-client-cert.crt

3.  Can you check if no intermediary CAs are missing from the nodes truststore?

4.  This exception is coming from inter-node communication (replication of 
request from one node to the other). This means that it is unrelated to 
external user's authentication by client certificate. The question is: is your 
inter node communication secured by the trusted root CA (that you are sure that 
the CA cert is present in the trust store) or is it secured by selfsigned CA 
(which's CA may be lacking from your truststore)?

5.  `nifi.security.needClientAuth` is not part of NiFi properties any more. If 
SSL is turned on, and no `nifi.security.user.login.identity.provider` is set, 
then client cert based auth is the default. But supplying this property have no 
detrimental effect anyhow.

Best regards,
Endre


Sent with ProtonMail Secure Email.

‐‐‐ Original Message ‐‐‐
On Wednesday, February 26, 2020 6:22 PM, Joe Gresock jgres...@gmail.com 
wrote:


> Were there any changes with how the trust store is used in 1.11.3? I had a
> 1.11.0 deployment working with the following settings, but when I deployed
> 1.11.3, the cluster can't seem to replicate requests to itself:
> nifi.remote.input.host=
> nifi.remote.input.secure=true
> nifi.remote.input.socket.port=32440
> nifi.remote.input.http.enabled=true
> nifi.cluster.protocol.is.secure=true
> nifi.cluster.is.node=true
> nifi.cluster.node.address=nifi-3.nifi-headless.lizardspock.svc.cluster.local
> nifi.cluster.node.protocol.port=6007
> nifi.web.https.host=nifi-3.nifi-headless.lizardspock.svc.cluster.local
> nifi.web.https.port=8443
> nifi.security.keystore=./conf/keystore.jks
> nifi.security.keystoreType=jks
> nifi.security.keystorePasswd=
> nifi.security.keyPasswd=
> nifi.security.truststore=./conf/truststore.jks
> nifi.security.truststoreType=jks
> nifi.security.truststorePasswd=
> nifi.security.needClientAuth=true
> A trusted client cert that worked against the old cluster is getting the
> same trust error (PKIX path building failed). I've verified that the
> client cert was issued by an issuer that is definitely in the
> ./conf/truststore.jks as a trustedCertEntry.
> 2020-02-26 17:11:09,573 WARN [Replicate Request Thread-7]
> o.a.n.c.c.h.r.ThreadPoolRequestReplicator
> javax.net.ssl.SSLHandshakeException:
> sun.security.validator.ValidatorException: PKIX path building failed:
> sun.security.provider.certpath.SunCertPathBuilderException: unable to find
> valid certification path to r
> equested target
> at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
> at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946)
> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:316)
> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310)
> at
> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639)
> at
> sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223)
> at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037)
> at sun.security.ssl.Handshaker.process_record(Handshaker.java:965)
> at
> sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1064)
> at
> sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367)
> at
> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395)
> at
> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379)
> at
> okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:302)
> at
> okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:270)
> at
> okhttp3.internal.connection.RealConnection.connect(RealConnection.java:162)
> at
> okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:257)
> at
> okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:135)
> at
> okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:114)
> at
> okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42)
> at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
> at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
> at
> okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
> at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
> at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
> at
> okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
> at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
> at
> 

Re: 1.11.3 trust store error

[Andy LoPresto]

You can post them on a temporary file sharing service, post them in the Apache 
NiFi Slack [1], or email them to me directly at alopre...@apache.org 
. The mailing list software tends to strip 
attachments. 

[1] https://apachenifi.slack.com 

Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Feb 26, 2020, at 12:21 PM, Joe Gresock  wrote:
> 
> Good question -- I can't share these keystores and truststores, but I'll
> see if I can generate some test ones tomorrow.  How should I send them to
> you?
> 
> On Wed, Feb 26, 2020 at 8:01 PM Andy LoPresto  wrote:
> 
>> Joe,
>> 
>> Can you share the keystores and truststores you are using? I understand
>> the issue you’re encountering but we haven’t yet been able to reproduce it
>> locally running with certs that work on 1.11.1. Please DO NOT share actual
>> keystores if they contain real private keys, only if these are dev
>> instances you are comfortable sharing.
>> 
>> If you cannot share these, can you please try generating a new set using
>> the TLS Toolkit and verify that they also fail on 1.11.3 in your
>> environment?
>> 
>> 
>> Andy LoPresto
>> alopre...@apache.org
>> alopresto.apa...@gmail.com
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>> 
>>> On Feb 26, 2020, at 11:07 AM, Joe Gresock  wrote:
>>> 
>>> Ok, I added all the server certs and my administrator's client cert to
>> the
>>> trust store, and they all still got PKIX path building failed.  So I
>>> redeployed nifi 1.11.1, and now it works again.
>>> 
>>> Joe
>>> 
>>> On Wed, Feb 26, 2020 at 6:21 PM Joe Gresock  wrote:
>>> 
 Yes, on Kubernetes.
 
 FWIW, I do see changes to
 
>> nifi-commons/nifi-security-utils/src/main/java/org/apache/nifi/security/util/SslContextFactory.java
 involving a new function createTrustSslContextWithTrustManagers(), among
 other related changes.
 
 I'm going to try directly adding the client certs to my trust store to
>> see
 if that works (instead of just adding the client certs' issuer).
 
 On Wed, Feb 26, 2020 at 6:08 PM Joe Witt  wrote:
 
> on kubernetes is a key detail here too...
> 
> On Wed, Feb 26, 2020 at 10:01 AM Joe Gresock 
>> wrote:
> 
>> Java 8
>> 
>> On Wed, Feb 26, 2020 at 5:59 PM Nathan Gough 
> wrote:
>> 
>>> Hi Joe,
>>> 
>>> I just set up a secure cluster with NiFi 1.11.3 and am not seeing any
>>> issues like you describe.
>>> 
>>> Are you running Java 8 or Java 11?
>>> 
>>> Nathan
>>> 
>>> On Wed, Feb 26, 2020 at 12:22 PM Joe Gresock 
> wrote:
>>> 
 Were there any changes with how the trust store is used in 1.11.3?
> I
>>> had a
 1.11.0 deployment working with the following settings, but when I
>>> deployed
 1.11.3, the cluster can't seem to replicate requests to itself:
 
 nifi.remote.input.host=
 nifi.remote.input.secure=true
 nifi.remote.input.socket.port=32440
 nifi.remote.input.http.enabled=true
 
 nifi.cluster.protocol.is.secure=true
 nifi.cluster.is.node=true
 
 
>>> 
>> 
> 
>> nifi.cluster.node.address=nifi-3.nifi-headless.lizardspock.svc.cluster.local
 nifi.cluster.node.protocol.port=6007
 
 
> nifi.web.https.host=nifi-3.nifi-headless.lizardspock.svc.cluster.local
 nifi.web.https.port=8443
 
 nifi.security.keystore=./conf/keystore.jks
 nifi.security.keystoreType=jks
 nifi.security.keystorePasswd=
 nifi.security.keyPasswd=
 nifi.security.truststore=./conf/truststore.jks
 nifi.security.truststoreType=jks
 nifi.security.truststorePasswd=
 nifi.security.needClientAuth=true
 
 A trusted client cert that worked against the old cluster is getting
>> the
 same trust error (PKIX path building failed).  I've verified that
> the
 client cert was issued by an issuer that is definitely in the
 ./conf/truststore.jks as a trustedCertEntry.
 
 2020-02-26 17:11:09,573 WARN [Replicate Request Thread-7]
 o.a.n.c.c.h.r.ThreadPoolRequestReplicator
 javax.net.ssl.SSLHandshakeException:
 sun.security.validator.ValidatorException: PKIX path building
> failed:
 sun.security.provider.certpath.SunCertPathBuilderException: unable
> to
>>> find
 valid certification path to r
 equested target
   at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
   at
>> sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946)
   at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:316)
   at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310)
   at

Re: 1.11.3 trust store error

[Joe Gresock]

Good question -- I can't share these keystores and truststores, but I'll
see if I can generate some test ones tomorrow.  How should I send them to
you?

On Wed, Feb 26, 2020 at 8:01 PM Andy LoPresto  wrote:

> Joe,
>
> Can you share the keystores and truststores you are using? I understand
> the issue you’re encountering but we haven’t yet been able to reproduce it
> locally running with certs that work on 1.11.1. Please DO NOT share actual
> keystores if they contain real private keys, only if these are dev
> instances you are comfortable sharing.
>
> If you cannot share these, can you please try generating a new set using
> the TLS Toolkit and verify that they also fail on 1.11.3 in your
> environment?
>
>
> Andy LoPresto
> alopre...@apache.org
> alopresto.apa...@gmail.com
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> > On Feb 26, 2020, at 11:07 AM, Joe Gresock  wrote:
> >
> > Ok, I added all the server certs and my administrator's client cert to
> the
> > trust store, and they all still got PKIX path building failed.  So I
> > redeployed nifi 1.11.1, and now it works again.
> >
> > Joe
> >
> > On Wed, Feb 26, 2020 at 6:21 PM Joe Gresock  wrote:
> >
> >> Yes, on Kubernetes.
> >>
> >> FWIW, I do see changes to
> >>
> nifi-commons/nifi-security-utils/src/main/java/org/apache/nifi/security/util/SslContextFactory.java
> >> involving a new function createTrustSslContextWithTrustManagers(), among
> >> other related changes.
> >>
> >> I'm going to try directly adding the client certs to my trust store to
> see
> >> if that works (instead of just adding the client certs' issuer).
> >>
> >> On Wed, Feb 26, 2020 at 6:08 PM Joe Witt  wrote:
> >>
> >>> on kubernetes is a key detail here too...
> >>>
> >>> On Wed, Feb 26, 2020 at 10:01 AM Joe Gresock 
> wrote:
> >>>
>  Java 8
> 
>  On Wed, Feb 26, 2020 at 5:59 PM Nathan Gough 
> >>> wrote:
> 
> > Hi Joe,
> >
> > I just set up a secure cluster with NiFi 1.11.3 and am not seeing any
> > issues like you describe.
> >
> > Are you running Java 8 or Java 11?
> >
> > Nathan
> >
> > On Wed, Feb 26, 2020 at 12:22 PM Joe Gresock 
> >>> wrote:
> >
> >> Were there any changes with how the trust store is used in 1.11.3?
> >>> I
> > had a
> >> 1.11.0 deployment working with the following settings, but when I
> > deployed
> >> 1.11.3, the cluster can't seem to replicate requests to itself:
> >>
> >> nifi.remote.input.host=
> >> nifi.remote.input.secure=true
> >> nifi.remote.input.socket.port=32440
> >> nifi.remote.input.http.enabled=true
> >>
> >> nifi.cluster.protocol.is.secure=true
> >> nifi.cluster.is.node=true
> >>
> >>
> >
> 
> >>>
> nifi.cluster.node.address=nifi-3.nifi-headless.lizardspock.svc.cluster.local
> >> nifi.cluster.node.protocol.port=6007
> >>
> >>
> >>> nifi.web.https.host=nifi-3.nifi-headless.lizardspock.svc.cluster.local
> >> nifi.web.https.port=8443
> >>
> >> nifi.security.keystore=./conf/keystore.jks
> >> nifi.security.keystoreType=jks
> >> nifi.security.keystorePasswd=
> >> nifi.security.keyPasswd=
> >> nifi.security.truststore=./conf/truststore.jks
> >> nifi.security.truststoreType=jks
> >> nifi.security.truststorePasswd=
> >> nifi.security.needClientAuth=true
> >>
> >> A trusted client cert that worked against the old cluster is getting
>  the
> >> same trust error (PKIX path building failed).  I've verified that
> >>> the
> >> client cert was issued by an issuer that is definitely in the
> >> ./conf/truststore.jks as a trustedCertEntry.
> >>
> >> 2020-02-26 17:11:09,573 WARN [Replicate Request Thread-7]
> >> o.a.n.c.c.h.r.ThreadPoolRequestReplicator
> >> javax.net.ssl.SSLHandshakeException:
> >> sun.security.validator.ValidatorException: PKIX path building
> >>> failed:
> >> sun.security.provider.certpath.SunCertPathBuilderException: unable
> >>> to
> > find
> >> valid certification path to r
> >> equested target
> >>at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
> >>at
>  sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946)
> >>at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:316)
> >>at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310)
> >>at
> >>
> >>
> >
> 
> >>>
> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639)
> >>at
> >>
> >
> 
> >>>
> sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223)
> >>at
>  sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037)
> >>at
> > sun.security.ssl.Handshaker.process_record(Handshaker.java:965)
> >>at
> >> sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1064)
> >>at
> >>
> >>
> 

Re: 1.11.3 trust store error

[Andy LoPresto]

Joe, 

Can you share the keystores and truststores you are using? I understand the 
issue you’re encountering but we haven’t yet been able to reproduce it locally 
running with certs that work on 1.11.1. Please DO NOT share actual keystores if 
they contain real private keys, only if these are dev instances you are 
comfortable sharing. 

If you cannot share these, can you please try generating a new set using the 
TLS Toolkit and verify that they also fail on 1.11.3 in your environment?


Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Feb 26, 2020, at 11:07 AM, Joe Gresock  wrote:
> 
> Ok, I added all the server certs and my administrator's client cert to the
> trust store, and they all still got PKIX path building failed.  So I
> redeployed nifi 1.11.1, and now it works again.
> 
> Joe
> 
> On Wed, Feb 26, 2020 at 6:21 PM Joe Gresock  wrote:
> 
>> Yes, on Kubernetes.
>> 
>> FWIW, I do see changes to
>> nifi-commons/nifi-security-utils/src/main/java/org/apache/nifi/security/util/SslContextFactory.java
>> involving a new function createTrustSslContextWithTrustManagers(), among
>> other related changes.
>> 
>> I'm going to try directly adding the client certs to my trust store to see
>> if that works (instead of just adding the client certs' issuer).
>> 
>> On Wed, Feb 26, 2020 at 6:08 PM Joe Witt  wrote:
>> 
>>> on kubernetes is a key detail here too...
>>> 
>>> On Wed, Feb 26, 2020 at 10:01 AM Joe Gresock  wrote:
>>> 
 Java 8
 
 On Wed, Feb 26, 2020 at 5:59 PM Nathan Gough 
>>> wrote:
 
> Hi Joe,
> 
> I just set up a secure cluster with NiFi 1.11.3 and am not seeing any
> issues like you describe.
> 
> Are you running Java 8 or Java 11?
> 
> Nathan
> 
> On Wed, Feb 26, 2020 at 12:22 PM Joe Gresock 
>>> wrote:
> 
>> Were there any changes with how the trust store is used in 1.11.3?
>>> I
> had a
>> 1.11.0 deployment working with the following settings, but when I
> deployed
>> 1.11.3, the cluster can't seem to replicate requests to itself:
>> 
>> nifi.remote.input.host=
>> nifi.remote.input.secure=true
>> nifi.remote.input.socket.port=32440
>> nifi.remote.input.http.enabled=true
>> 
>> nifi.cluster.protocol.is.secure=true
>> nifi.cluster.is.node=true
>> 
>> 
> 
 
>>> nifi.cluster.node.address=nifi-3.nifi-headless.lizardspock.svc.cluster.local
>> nifi.cluster.node.protocol.port=6007
>> 
>> 
>>> nifi.web.https.host=nifi-3.nifi-headless.lizardspock.svc.cluster.local
>> nifi.web.https.port=8443
>> 
>> nifi.security.keystore=./conf/keystore.jks
>> nifi.security.keystoreType=jks
>> nifi.security.keystorePasswd=
>> nifi.security.keyPasswd=
>> nifi.security.truststore=./conf/truststore.jks
>> nifi.security.truststoreType=jks
>> nifi.security.truststorePasswd=
>> nifi.security.needClientAuth=true
>> 
>> A trusted client cert that worked against the old cluster is getting
 the
>> same trust error (PKIX path building failed).  I've verified that
>>> the
>> client cert was issued by an issuer that is definitely in the
>> ./conf/truststore.jks as a trustedCertEntry.
>> 
>> 2020-02-26 17:11:09,573 WARN [Replicate Request Thread-7]
>> o.a.n.c.c.h.r.ThreadPoolRequestReplicator
>> javax.net.ssl.SSLHandshakeException:
>> sun.security.validator.ValidatorException: PKIX path building
>>> failed:
>> sun.security.provider.certpath.SunCertPathBuilderException: unable
>>> to
> find
>> valid certification path to r
>> equested target
>>at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
>>at
 sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946)
>>at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:316)
>>at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310)
>>at
>> 
>> 
> 
 
>>> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639)
>>at
>> 
> 
 
>>> sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223)
>>at
 sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037)
>>at
> sun.security.ssl.Handshaker.process_record(Handshaker.java:965)
>>at
>> sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1064)
>>at
>> 
>> 
> 
 
>>> sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367)
>>at
>> 
>>> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395)
>>at
>> 
>>> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379)
>>at
>> 
>> 
> 
 
>>> okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:302)
>>at
>> 

Re: 1.11.3 trust store error

[Joe Gresock]

Ok, I added all the server certs and my administrator's client cert to the
trust store, and they all still got PKIX path building failed.  So I
redeployed nifi 1.11.1, and now it works again.

Joe

On Wed, Feb 26, 2020 at 6:21 PM Joe Gresock  wrote:

> Yes, on Kubernetes.
>
> FWIW, I do see changes to
> nifi-commons/nifi-security-utils/src/main/java/org/apache/nifi/security/util/SslContextFactory.java
> involving a new function createTrustSslContextWithTrustManagers(), among
> other related changes.
>
> I'm going to try directly adding the client certs to my trust store to see
> if that works (instead of just adding the client certs' issuer).
>
> On Wed, Feb 26, 2020 at 6:08 PM Joe Witt  wrote:
>
>> on kubernetes is a key detail here too...
>>
>> On Wed, Feb 26, 2020 at 10:01 AM Joe Gresock  wrote:
>>
>> > Java 8
>> >
>> > On Wed, Feb 26, 2020 at 5:59 PM Nathan Gough 
>> wrote:
>> >
>> > > Hi Joe,
>> > >
>> > > I just set up a secure cluster with NiFi 1.11.3 and am not seeing any
>> > > issues like you describe.
>> > >
>> > > Are you running Java 8 or Java 11?
>> > >
>> > > Nathan
>> > >
>> > > On Wed, Feb 26, 2020 at 12:22 PM Joe Gresock 
>> wrote:
>> > >
>> > > > Were there any changes with how the trust store is used in 1.11.3?
>> I
>> > > had a
>> > > > 1.11.0 deployment working with the following settings, but when I
>> > > deployed
>> > > > 1.11.3, the cluster can't seem to replicate requests to itself:
>> > > >
>> > > > nifi.remote.input.host=
>> > > > nifi.remote.input.secure=true
>> > > > nifi.remote.input.socket.port=32440
>> > > > nifi.remote.input.http.enabled=true
>> > > >
>> > > > nifi.cluster.protocol.is.secure=true
>> > > > nifi.cluster.is.node=true
>> > > >
>> > > >
>> > >
>> >
>> nifi.cluster.node.address=nifi-3.nifi-headless.lizardspock.svc.cluster.local
>> > > > nifi.cluster.node.protocol.port=6007
>> > > >
>> > > >
>> nifi.web.https.host=nifi-3.nifi-headless.lizardspock.svc.cluster.local
>> > > > nifi.web.https.port=8443
>> > > >
>> > > > nifi.security.keystore=./conf/keystore.jks
>> > > > nifi.security.keystoreType=jks
>> > > > nifi.security.keystorePasswd=
>> > > > nifi.security.keyPasswd=
>> > > > nifi.security.truststore=./conf/truststore.jks
>> > > > nifi.security.truststoreType=jks
>> > > > nifi.security.truststorePasswd=
>> > > > nifi.security.needClientAuth=true
>> > > >
>> > > > A trusted client cert that worked against the old cluster is getting
>> > the
>> > > > same trust error (PKIX path building failed).  I've verified that
>> the
>> > > > client cert was issued by an issuer that is definitely in the
>> > > > ./conf/truststore.jks as a trustedCertEntry.
>> > > >
>> > > > 2020-02-26 17:11:09,573 WARN [Replicate Request Thread-7]
>> > > > o.a.n.c.c.h.r.ThreadPoolRequestReplicator
>> > > > javax.net.ssl.SSLHandshakeException:
>> > > > sun.security.validator.ValidatorException: PKIX path building
>> failed:
>> > > > sun.security.provider.certpath.SunCertPathBuilderException: unable
>> to
>> > > find
>> > > > valid certification path to r
>> > > > equested target
>> > > > at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
>> > > > at
>> > sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946)
>> > > > at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:316)
>> > > > at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310)
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639)
>> > > > at
>> > > >
>> > >
>> >
>> sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223)
>> > > > at
>> > sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037)
>> > > > at
>> > > sun.security.ssl.Handshaker.process_record(Handshaker.java:965)
>> > > > at
>> > > > sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1064)
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367)
>> > > > at
>> > > >
>> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395)
>> > > > at
>> > > >
>> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379)
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:302)
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:270)
>> > > > at
>> > > >
>> > >
>> >
>> okhttp3.internal.connection.RealConnection.connect(RealConnection.java:162)
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:257)
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:135)
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> 

Re: 1.11.3 trust store error

[Joe Gresock]

Yes, on Kubernetes.

FWIW, I do see changes to
nifi-commons/nifi-security-utils/src/main/java/org/apache/nifi/security/util/SslContextFactory.java
involving a new function createTrustSslContextWithTrustManagers(), among
other related changes.

I'm going to try directly adding the client certs to my trust store to see
if that works (instead of just adding the client certs' issuer).

On Wed, Feb 26, 2020 at 6:08 PM Joe Witt  wrote:

> on kubernetes is a key detail here too...
>
> On Wed, Feb 26, 2020 at 10:01 AM Joe Gresock  wrote:
>
> > Java 8
> >
> > On Wed, Feb 26, 2020 at 5:59 PM Nathan Gough  wrote:
> >
> > > Hi Joe,
> > >
> > > I just set up a secure cluster with NiFi 1.11.3 and am not seeing any
> > > issues like you describe.
> > >
> > > Are you running Java 8 or Java 11?
> > >
> > > Nathan
> > >
> > > On Wed, Feb 26, 2020 at 12:22 PM Joe Gresock 
> wrote:
> > >
> > > > Were there any changes with how the trust store is used in 1.11.3?  I
> > > had a
> > > > 1.11.0 deployment working with the following settings, but when I
> > > deployed
> > > > 1.11.3, the cluster can't seem to replicate requests to itself:
> > > >
> > > > nifi.remote.input.host=
> > > > nifi.remote.input.secure=true
> > > > nifi.remote.input.socket.port=32440
> > > > nifi.remote.input.http.enabled=true
> > > >
> > > > nifi.cluster.protocol.is.secure=true
> > > > nifi.cluster.is.node=true
> > > >
> > > >
> > >
> >
> nifi.cluster.node.address=nifi-3.nifi-headless.lizardspock.svc.cluster.local
> > > > nifi.cluster.node.protocol.port=6007
> > > >
> > > >
> nifi.web.https.host=nifi-3.nifi-headless.lizardspock.svc.cluster.local
> > > > nifi.web.https.port=8443
> > > >
> > > > nifi.security.keystore=./conf/keystore.jks
> > > > nifi.security.keystoreType=jks
> > > > nifi.security.keystorePasswd=
> > > > nifi.security.keyPasswd=
> > > > nifi.security.truststore=./conf/truststore.jks
> > > > nifi.security.truststoreType=jks
> > > > nifi.security.truststorePasswd=
> > > > nifi.security.needClientAuth=true
> > > >
> > > > A trusted client cert that worked against the old cluster is getting
> > the
> > > > same trust error (PKIX path building failed).  I've verified that the
> > > > client cert was issued by an issuer that is definitely in the
> > > > ./conf/truststore.jks as a trustedCertEntry.
> > > >
> > > > 2020-02-26 17:11:09,573 WARN [Replicate Request Thread-7]
> > > > o.a.n.c.c.h.r.ThreadPoolRequestReplicator
> > > > javax.net.ssl.SSLHandshakeException:
> > > > sun.security.validator.ValidatorException: PKIX path building failed:
> > > > sun.security.provider.certpath.SunCertPathBuilderException: unable to
> > > find
> > > > valid certification path to r
> > > > equested target
> > > > at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
> > > > at
> > sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946)
> > > > at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:316)
> > > > at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310)
> > > > at
> > > >
> > > >
> > >
> >
> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639)
> > > > at
> > > >
> > >
> >
> sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223)
> > > > at
> > sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037)
> > > > at
> > > sun.security.ssl.Handshaker.process_record(Handshaker.java:965)
> > > > at
> > > > sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1064)
> > > > at
> > > >
> > > >
> > >
> >
> sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367)
> > > > at
> > > >
> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395)
> > > > at
> > > >
> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379)
> > > > at
> > > >
> > > >
> > >
> >
> okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:302)
> > > > at
> > > >
> > > >
> > >
> >
> okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:270)
> > > > at
> > > >
> > >
> >
> okhttp3.internal.connection.RealConnection.connect(RealConnection.java:162)
> > > > at
> > > >
> > > >
> > >
> >
> okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:257)
> > > > at
> > > >
> > > >
> > >
> >
> okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:135)
> > > > at
> > > >
> > > >
> > >
> >
> okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:114)
> > > > at
> > > >
> > > >
> > >
> >
> okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42)
> > > > at
> > > >
> > > >
> > >
> >
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
> > > > at
> > > >
> > > >
> > >
> >
> 

Re: 1.11.3 trust store error

[Joe Witt]

on kubernetes is a key detail here too...

On Wed, Feb 26, 2020 at 10:01 AM Joe Gresock  wrote:

> Java 8
>
> On Wed, Feb 26, 2020 at 5:59 PM Nathan Gough  wrote:
>
> > Hi Joe,
> >
> > I just set up a secure cluster with NiFi 1.11.3 and am not seeing any
> > issues like you describe.
> >
> > Are you running Java 8 or Java 11?
> >
> > Nathan
> >
> > On Wed, Feb 26, 2020 at 12:22 PM Joe Gresock  wrote:
> >
> > > Were there any changes with how the trust store is used in 1.11.3?  I
> > had a
> > > 1.11.0 deployment working with the following settings, but when I
> > deployed
> > > 1.11.3, the cluster can't seem to replicate requests to itself:
> > >
> > > nifi.remote.input.host=
> > > nifi.remote.input.secure=true
> > > nifi.remote.input.socket.port=32440
> > > nifi.remote.input.http.enabled=true
> > >
> > > nifi.cluster.protocol.is.secure=true
> > > nifi.cluster.is.node=true
> > >
> > >
> >
> nifi.cluster.node.address=nifi-3.nifi-headless.lizardspock.svc.cluster.local
> > > nifi.cluster.node.protocol.port=6007
> > >
> > > nifi.web.https.host=nifi-3.nifi-headless.lizardspock.svc.cluster.local
> > > nifi.web.https.port=8443
> > >
> > > nifi.security.keystore=./conf/keystore.jks
> > > nifi.security.keystoreType=jks
> > > nifi.security.keystorePasswd=
> > > nifi.security.keyPasswd=
> > > nifi.security.truststore=./conf/truststore.jks
> > > nifi.security.truststoreType=jks
> > > nifi.security.truststorePasswd=
> > > nifi.security.needClientAuth=true
> > >
> > > A trusted client cert that worked against the old cluster is getting
> the
> > > same trust error (PKIX path building failed).  I've verified that the
> > > client cert was issued by an issuer that is definitely in the
> > > ./conf/truststore.jks as a trustedCertEntry.
> > >
> > > 2020-02-26 17:11:09,573 WARN [Replicate Request Thread-7]
> > > o.a.n.c.c.h.r.ThreadPoolRequestReplicator
> > > javax.net.ssl.SSLHandshakeException:
> > > sun.security.validator.ValidatorException: PKIX path building failed:
> > > sun.security.provider.certpath.SunCertPathBuilderException: unable to
> > find
> > > valid certification path to r
> > > equested target
> > > at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
> > > at
> sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946)
> > > at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:316)
> > > at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310)
> > > at
> > >
> > >
> >
> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639)
> > > at
> > >
> >
> sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223)
> > > at
> sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037)
> > > at
> > sun.security.ssl.Handshaker.process_record(Handshaker.java:965)
> > > at
> > > sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1064)
> > > at
> > >
> > >
> >
> sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367)
> > > at
> > > sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395)
> > > at
> > > sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379)
> > > at
> > >
> > >
> >
> okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:302)
> > > at
> > >
> > >
> >
> okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:270)
> > > at
> > >
> >
> okhttp3.internal.connection.RealConnection.connect(RealConnection.java:162)
> > > at
> > >
> > >
> >
> okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:257)
> > > at
> > >
> > >
> >
> okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:135)
> > > at
> > >
> > >
> >
> okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:114)
> > > at
> > >
> > >
> >
> okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42)
> > > at
> > >
> > >
> >
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
> > > at
> > >
> > >
> >
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
> > > at
> > >
> >
> okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
> > > at
> > >
> > >
> >
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
> > > at
> > >
> > >
> >
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
> > > at
> > >
> > >
> >
> okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
> > > at
> > >
> > >
> >
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
> > > at
> > >
> > >
> >
> okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:126)
> > > at
> > >
> 

Re: 1.11.3 trust store error

[Joe Gresock]

Java 8

On Wed, Feb 26, 2020 at 5:59 PM Nathan Gough  wrote:

> Hi Joe,
>
> I just set up a secure cluster with NiFi 1.11.3 and am not seeing any
> issues like you describe.
>
> Are you running Java 8 or Java 11?
>
> Nathan
>
> On Wed, Feb 26, 2020 at 12:22 PM Joe Gresock  wrote:
>
> > Were there any changes with how the trust store is used in 1.11.3?  I
> had a
> > 1.11.0 deployment working with the following settings, but when I
> deployed
> > 1.11.3, the cluster can't seem to replicate requests to itself:
> >
> > nifi.remote.input.host=
> > nifi.remote.input.secure=true
> > nifi.remote.input.socket.port=32440
> > nifi.remote.input.http.enabled=true
> >
> > nifi.cluster.protocol.is.secure=true
> > nifi.cluster.is.node=true
> >
> >
> nifi.cluster.node.address=nifi-3.nifi-headless.lizardspock.svc.cluster.local
> > nifi.cluster.node.protocol.port=6007
> >
> > nifi.web.https.host=nifi-3.nifi-headless.lizardspock.svc.cluster.local
> > nifi.web.https.port=8443
> >
> > nifi.security.keystore=./conf/keystore.jks
> > nifi.security.keystoreType=jks
> > nifi.security.keystorePasswd=
> > nifi.security.keyPasswd=
> > nifi.security.truststore=./conf/truststore.jks
> > nifi.security.truststoreType=jks
> > nifi.security.truststorePasswd=
> > nifi.security.needClientAuth=true
> >
> > A trusted client cert that worked against the old cluster is getting the
> > same trust error (PKIX path building failed).  I've verified that the
> > client cert was issued by an issuer that is definitely in the
> > ./conf/truststore.jks as a trustedCertEntry.
> >
> > 2020-02-26 17:11:09,573 WARN [Replicate Request Thread-7]
> > o.a.n.c.c.h.r.ThreadPoolRequestReplicator
> > javax.net.ssl.SSLHandshakeException:
> > sun.security.validator.ValidatorException: PKIX path building failed:
> > sun.security.provider.certpath.SunCertPathBuilderException: unable to
> find
> > valid certification path to r
> > equested target
> > at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
> > at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946)
> > at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:316)
> > at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310)
> > at
> >
> >
> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639)
> > at
> >
> sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223)
> > at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037)
> > at
> sun.security.ssl.Handshaker.process_record(Handshaker.java:965)
> > at
> > sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1064)
> > at
> >
> >
> sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367)
> > at
> > sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395)
> > at
> > sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379)
> > at
> >
> >
> okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:302)
> > at
> >
> >
> okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:270)
> > at
> >
> okhttp3.internal.connection.RealConnection.connect(RealConnection.java:162)
> > at
> >
> >
> okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:257)
> > at
> >
> >
> okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:135)
> > at
> >
> >
> okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:114)
> > at
> >
> >
> okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42)
> > at
> >
> >
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
> > at
> >
> >
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
> > at
> >
> okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
> > at
> >
> >
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
> > at
> >
> >
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
> > at
> >
> >
> okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
> > at
> >
> >
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
> > at
> >
> >
> okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:126)
> > at
> >
> >
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
> > at
> >
> >
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
> > at
> > okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:200)
> > at okhttp3.RealCall.execute(RealCall.java:77)
> > at
> >
> >
> 

Re: 1.11.3 trust store error

[Nathan Gough]

Hi Joe,

I just set up a secure cluster with NiFi 1.11.3 and am not seeing any
issues like you describe.

Are you running Java 8 or Java 11?

Nathan

On Wed, Feb 26, 2020 at 12:22 PM Joe Gresock  wrote:

> Were there any changes with how the trust store is used in 1.11.3?  I had a
> 1.11.0 deployment working with the following settings, but when I deployed
> 1.11.3, the cluster can't seem to replicate requests to itself:
>
> nifi.remote.input.host=
> nifi.remote.input.secure=true
> nifi.remote.input.socket.port=32440
> nifi.remote.input.http.enabled=true
>
> nifi.cluster.protocol.is.secure=true
> nifi.cluster.is.node=true
>
> nifi.cluster.node.address=nifi-3.nifi-headless.lizardspock.svc.cluster.local
> nifi.cluster.node.protocol.port=6007
>
> nifi.web.https.host=nifi-3.nifi-headless.lizardspock.svc.cluster.local
> nifi.web.https.port=8443
>
> nifi.security.keystore=./conf/keystore.jks
> nifi.security.keystoreType=jks
> nifi.security.keystorePasswd=
> nifi.security.keyPasswd=
> nifi.security.truststore=./conf/truststore.jks
> nifi.security.truststoreType=jks
> nifi.security.truststorePasswd=
> nifi.security.needClientAuth=true
>
> A trusted client cert that worked against the old cluster is getting the
> same trust error (PKIX path building failed).  I've verified that the
> client cert was issued by an issuer that is definitely in the
> ./conf/truststore.jks as a trustedCertEntry.
>
> 2020-02-26 17:11:09,573 WARN [Replicate Request Thread-7]
> o.a.n.c.c.h.r.ThreadPoolRequestReplicator
> javax.net.ssl.SSLHandshakeException:
> sun.security.validator.ValidatorException: PKIX path building failed:
> sun.security.provider.certpath.SunCertPathBuilderException: unable to find
> valid certification path to r
> equested target
> at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
> at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946)
> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:316)
> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310)
> at
>
> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639)
> at
> sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223)
> at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037)
> at sun.security.ssl.Handshaker.process_record(Handshaker.java:965)
> at
> sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1064)
> at
>
> sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367)
> at
> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395)
> at
> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379)
> at
>
> okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:302)
> at
>
> okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:270)
> at
> okhttp3.internal.connection.RealConnection.connect(RealConnection.java:162)
> at
>
> okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:257)
> at
>
> okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:135)
> at
>
> okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:114)
> at
>
> okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42)
> at
>
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
> at
>
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
> at
> okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
> at
>
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
> at
>
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
> at
>
> okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
> at
>
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
> at
>
> okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:126)
> at
>
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
> at
>
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
> at
> okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:200)
> at okhttp3.RealCall.execute(RealCall.java:77)
> at
>
> org.apache.nifi.cluster.coordination.http.replication.okhttp.OkHttpReplicationClient.replicate(OkHttpReplicationClient.java:143)
> at
>
> org.apache.nifi.cluster.coordination.http.replication.okhttp.OkHttpReplicationClient.replicate(OkHttpReplicationClient.java:137)
> at
>
> 

1.11.3 trust store error

[Joe Gresock]

Were there any changes with how the trust store is used in 1.11.3?  I had a
1.11.0 deployment working with the following settings, but when I deployed
1.11.3, the cluster can't seem to replicate requests to itself:

nifi.remote.input.host=
nifi.remote.input.secure=true
nifi.remote.input.socket.port=32440
nifi.remote.input.http.enabled=true

nifi.cluster.protocol.is.secure=true
nifi.cluster.is.node=true
nifi.cluster.node.address=nifi-3.nifi-headless.lizardspock.svc.cluster.local
nifi.cluster.node.protocol.port=6007

nifi.web.https.host=nifi-3.nifi-headless.lizardspock.svc.cluster.local
nifi.web.https.port=8443

nifi.security.keystore=./conf/keystore.jks
nifi.security.keystoreType=jks
nifi.security.keystorePasswd=
nifi.security.keyPasswd=
nifi.security.truststore=./conf/truststore.jks
nifi.security.truststoreType=jks
nifi.security.truststorePasswd=
nifi.security.needClientAuth=true

A trusted client cert that worked against the old cluster is getting the
same trust error (PKIX path building failed).  I've verified that the
client cert was issued by an issuer that is definitely in the
./conf/truststore.jks as a trustedCertEntry.

2020-02-26 17:11:09,573 WARN [Replicate Request Thread-7]
o.a.n.c.c.h.r.ThreadPoolRequestReplicator
javax.net.ssl.SSLHandshakeException:
sun.security.validator.ValidatorException: PKIX path building failed:
sun.security.provider.certpath.SunCertPathBuilderException: unable to find
valid certification path to r
equested target
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:316)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310)
at
sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639)
at
sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223)
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037)
at sun.security.ssl.Handshaker.process_record(Handshaker.java:965)
at
sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1064)
at
sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367)
at
sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395)
at
sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379)
at
okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:302)
at
okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:270)
at
okhttp3.internal.connection.RealConnection.connect(RealConnection.java:162)
at
okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:257)
at
okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:135)
at
okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:114)
at
okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42)
at
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
at
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
at
okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
at
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
at
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
at
okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
at
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
at
okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:126)
at
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
at
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
at
okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:200)
at okhttp3.RealCall.execute(RealCall.java:77)
at
org.apache.nifi.cluster.coordination.http.replication.okhttp.OkHttpReplicationClient.replicate(OkHttpReplicationClient.java:143)
at
org.apache.nifi.cluster.coordination.http.replication.okhttp.OkHttpReplicationClient.replicate(OkHttpReplicationClient.java:137)
at
org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator.replicateRequest(ThreadPoolRequestReplicator.java:647)
at
org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator$NodeHttpRequest.run(ThreadPoolRequestReplicator.java:839)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at