This ticket could use some additional feedback
https://github.com/apache/nifi/pull/4104 It started out as an exercise in adding Decimal128 support to Mongo and started integrating support for Avro Decimal into the support for Kudu, Hive3, etc. Overall LGTM, but it'd be good to have one or two other folks look over those changes as well. Thanks, Mike
Re: 1.11.3 trust store error
The nifi.security.keyPasswd was not filled in, so it looked like this (which is the default configuration): nifi.security.keyPasswd= On Wed, Mar 4, 2020 at 11:36 AM Endre Kovacs wrote: > Hi Nathan, > > There is already a ticket about this: > https://issues.apache.org/jira/browse/NIFI-7219 > > Best regards, > Endre > > Sent with ProtonMail Secure Email. > > ‐‐‐ Original Message ‐‐‐ > On Wednesday, March 4, 2020 5:25 PM, Nathan Gough > wrote: > > > I've opened https://issues.apache.org/jira/browse/NIFI-7223 to track and > > I'm working on a fix for this. > > > > Nathan > > > > On Tue, Mar 3, 2020 at 6:17 PM Nathan Gough thena...@gmail.com wrote: > > > > > Hi Joe, > > > Just to confirm here - was the nifi.security.keyPasswd not defined at > all > > > in your nifi.properties? Did you have to add the property and give it > the > > > correct value? Or was it in the nifi.properties file but blank? Or > were the > > > keyPasswd and keystorePasswd different values? > > > Thanks, > > > Nathan > > > On Tue, Mar 3, 2020 at 3:38 PM Joe Gresock jgres...@gmail.com wrote: > > > > > > > Yep, setting the nifi.security.keyPasswd to the same as > > > > nifi.security.keystorePasswd fixed it. Thanks for the insight, Endre! > > > > On Tue, Mar 3, 2020 at 2:01 PM Joe Witt joe.w...@gmail.com wrote: > > > > > > > > > relevant change I believe is here: > > > > > > > > > https://github.com/apache/nifi/commit/46d3b6b0dc28f04da124be7685f82bec52e88775 > > > > > > > > > and > > > > > is from https://issues.apache.org/jira/browse/NIFI-6927 > > > > > It looks to me like this was fixing an improper naming/usage issue > > > > > that > > > > > has been present but if so we probably should have addressed not > in this > > > > > bug fix line. Will defer to Troy/Andy for more context and next > steps > > > > > On Tue, Mar 3, 2020 at 5:53 AM Joe Witt joe.w...@gmail.com wrote: > > > > > > > > > > > If accurateWe need to look into whether this was a mistake > and > > > > > > fix it > > > > > > > > > > > if so. And we need to reflect this in the migration guide > > > > > > On Tue, Mar 3, 2020 at 4:40 AM Ryan Ward ryan.wa...@gmail.com > > > > > > wrote: > > > > > > > > > > > > Endre - thanks that was it > > > > > > > On Tue, Mar 3, 2020 at 6:50 AM Endre Kovacs > > > > > > > andrewsmit...@protonmail.com.invalid wrote: > > > > > > > > > > > > > > > Hi, > > > > > > > > One additional thing: > > > > > > > > we encountered something strange as well: > > > > > > > > on 1.11.2 clustered, kerberized: request replication worked > well. > > > > > > > > on 1.11.3 clustered, kerberized: request replication did not > work, > > > > > > > > unless > > > > > > > > you specify, and set > > > > > > > > nifi.security.keyPasswd > > > > > > > > to the very same password as the > > > > > > > > nifi.security.keystorePasswd > > > > > > > > For us this resolved the issue. > > > > > > > > Best regards, > > > > > > > > Endre > > > > > > > > Sent with ProtonMail Secure Email. > > > > > > > > ‐‐‐ Original Message ‐‐‐ > > > > > > > > On Tuesday, March 3, 2020 12:40 PM, Ryan Ward < > > > > > > > > ryan.wa...@gmail.com> > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi Joe - Did you resolve your issue? If so I am wondering > what > > > > > > > > > the > > > > > > > > > fix > > > > > > > > > > > > > was as I'm seeing the same error on my cluster. > > > > > > > > > > > > > > > > > On Thu, Feb 27, 2020 at 3:13 AM Endre Kovacs < > > > > > > > > > andrewsmit...@protonmail.com.invalid> wrote: > > > > > > > > > > > > > > > > > > > Hi Joe, > > > > > > > > > > > > > > > > > > > > 1. Have you tried connecting/debugging with openssl? > From one > > > > > > > > > > pod > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > > > the other: > > > > > > > > > > > > > > > > > > (openssl s_client -debug -CAfile > > > > > > > > > > > > > > > > > > > > > > > > > > ca-bundle-signing-node-certificates.crt -cert > my-client-cert.crt > > > > > > > > -connect > > > > > > > > nifi-3.nifi-headless.lizardspock.svc.cluster.local:6007) > > > > > > > > > > > > > > > > > > 2. certs can also be verified by: > > > > > > > > > > openssl verify -verbose -CAfile ca-bundle.crt > > > > > > > > > > my-client-cert.crt > > > > > > > > > > > > > > > > > > > > > > > > > 3. Can you check if no intermediary CAs are missing > from the > > > > > > > > > > nodes > > > > > > > > > > > > > > > > > > > > > > > truststore? > > > > > > > > > > > > > > > > > > 4. This exception is coming from inter-node > communication > > > > > > > > > > (replication > > > > > > > > > > of request from one node to the other). This means > that it is > > > > > > > > > > unrelated > > > > > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > external user's authentication by client certificate. The > question > > > > > > > > is: > > > > > > > > > > > > is > > > > > > > > > > > > > > > your inter node communication secured by the trusted root
Re: nifi connection with hive
Aya, The Hive components included with Apache NiFi are not compatible with Hive 2.x, they are built with Hive 1.2.x. There is a Jira to add Hive 2 support [1] but it is not yet in NiFi. Regards, Matt [1] https://issues.apache.org/jira/browse/NIFI-6456 On Tue, Mar 3, 2020 at 6:35 AM Aya Asfour wrote: > > please I want to ask if nifi version 1.9.2 puthivestreaming processor > compataible with hive version 2.3.2 , info: I used puthiveql and it > works well , but with using puthivestreaming it connect to metastore and > create delta files but always empty and errors appear , please any help? >
Re: 1.11.3 trust store error
Hi Nathan, There is already a ticket about this: https://issues.apache.org/jira/browse/NIFI-7219 Best regards, Endre Sent with ProtonMail Secure Email. ‐‐‐ Original Message ‐‐‐ On Wednesday, March 4, 2020 5:25 PM, Nathan Gough wrote: > I've opened https://issues.apache.org/jira/browse/NIFI-7223 to track and > I'm working on a fix for this. > > Nathan > > On Tue, Mar 3, 2020 at 6:17 PM Nathan Gough thena...@gmail.com wrote: > > > Hi Joe, > > Just to confirm here - was the nifi.security.keyPasswd not defined at all > > in your nifi.properties? Did you have to add the property and give it the > > correct value? Or was it in the nifi.properties file but blank? Or were the > > keyPasswd and keystorePasswd different values? > > Thanks, > > Nathan > > On Tue, Mar 3, 2020 at 3:38 PM Joe Gresock jgres...@gmail.com wrote: > > > > > Yep, setting the nifi.security.keyPasswd to the same as > > > nifi.security.keystorePasswd fixed it. Thanks for the insight, Endre! > > > On Tue, Mar 3, 2020 at 2:01 PM Joe Witt joe.w...@gmail.com wrote: > > > > > > > relevant change I believe is here: > > > > > > https://github.com/apache/nifi/commit/46d3b6b0dc28f04da124be7685f82bec52e88775 > > > > > > > and > > > > is from https://issues.apache.org/jira/browse/NIFI-6927 > > > > It looks to me like this was fixing an improper naming/usage issue > > > > that > > > > has been present but if so we probably should have addressed not in this > > > > bug fix line. Will defer to Troy/Andy for more context and next steps > > > > On Tue, Mar 3, 2020 at 5:53 AM Joe Witt joe.w...@gmail.com wrote: > > > > > > > > > If accurateWe need to look into whether this was a mistake and > > > > > fix it > > > > > > > > > if so. And we need to reflect this in the migration guide > > > > > On Tue, Mar 3, 2020 at 4:40 AM Ryan Ward ryan.wa...@gmail.com > > > > > wrote: > > > > > > > > > > Endre - thanks that was it > > > > > > On Tue, Mar 3, 2020 at 6:50 AM Endre Kovacs > > > > > > andrewsmit...@protonmail.com.invalid wrote: > > > > > > > > > > > > > Hi, > > > > > > > One additional thing: > > > > > > > we encountered something strange as well: > > > > > > > on 1.11.2 clustered, kerberized: request replication worked well. > > > > > > > on 1.11.3 clustered, kerberized: request replication did not work, > > > > > > > unless > > > > > > > you specify, and set > > > > > > > nifi.security.keyPasswd > > > > > > > to the very same password as the > > > > > > > nifi.security.keystorePasswd > > > > > > > For us this resolved the issue. > > > > > > > Best regards, > > > > > > > Endre > > > > > > > Sent with ProtonMail Secure Email. > > > > > > > ‐‐‐ Original Message ‐‐‐ > > > > > > > On Tuesday, March 3, 2020 12:40 PM, Ryan Ward < > > > > > > > ryan.wa...@gmail.com> > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Hi Joe - Did you resolve your issue? If so I am wondering what > > > > > > > > the > > > > > > > > fix > > > > > > > > > > > was as I'm seeing the same error on my cluster. > > > > > > > > > > > > > > > On Thu, Feb 27, 2020 at 3:13 AM Endre Kovacs < > > > > > > > > andrewsmit...@protonmail.com.invalid> wrote: > > > > > > > > > > > > > > > > > Hi Joe, > > > > > > > > > > > > > > > > > > 1. Have you tried connecting/debugging with openssl? From one > > > > > > > > > pod > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > the other: > > > > > > > > > > > > > > > > (openssl s_client -debug -CAfile > > > > > > > > > > > > > > > > > > > > > > > ca-bundle-signing-node-certificates.crt -cert my-client-cert.crt > > > > > > > -connect > > > > > > > nifi-3.nifi-headless.lizardspock.svc.cluster.local:6007) > > > > > > > > > > > > > > > > 2. certs can also be verified by: > > > > > > > > > openssl verify -verbose -CAfile ca-bundle.crt > > > > > > > > > my-client-cert.crt > > > > > > > > > > > > > > > > > > > > > > 3. Can you check if no intermediary CAs are missing from the > > > > > > > > > nodes > > > > > > > > > > > > > > > > > > > > truststore? > > > > > > > > > > > > > > > > 4. This exception is coming from inter-node communication > > > > > > > > > (replication > > > > > > > > > of request from one node to the other). This means that > > > > > > > > > it is > > > > > > > > > unrelated > > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > external user's authentication by client certificate. The question > > > > > > > is: > > > > > > > > > > is > > > > > > > > > > > > > your inter node communication secured by the trusted root CA (that > > > > > > > you > > > > > > > > > > are > > > > > > > > > > > > > sure that the CA cert is present in the trust store) or is it > > > > > > > secured > > > > > > > by > > > > > > > > > > > selfsigned CA (which's CA may be lacking from your truststore)? > > > > > > > > > > > > > > > > 5. `nifi.security.needClientAuth` is not part of NiFi > > > > > > > > > properties > > > > > > > > > any > > > > > > > > > > > > > > > > > > > >
Re: 1.11.3 trust store error
I've opened https://issues.apache.org/jira/browse/NIFI-7223 to track and I'm working on a fix for this. Nathan On Tue, Mar 3, 2020 at 6:17 PM Nathan Gough wrote: > Hi Joe, > > Just to confirm here - was the nifi.security.keyPasswd not defined at all > in your nifi.properties? Did you have to add the property and give it the > correct value? Or was it in the nifi.properties file but blank? Or were the > keyPasswd and keystorePasswd different values? > > Thanks, > Nathan > > On Tue, Mar 3, 2020 at 3:38 PM Joe Gresock wrote: > >> Yep, setting the nifi.security.keyPasswd to the same as >> nifi.security.keystorePasswd fixed it. Thanks for the insight, Endre! >> >> On Tue, Mar 3, 2020 at 2:01 PM Joe Witt wrote: >> >> > relevant change I believe is here: >> > >> > >> https://github.com/apache/nifi/commit/46d3b6b0dc28f04da124be7685f82bec52e88775 >> > and >> > is from https://issues.apache.org/jira/browse/NIFI-6927 >> > >> > It *looks* to me like this was fixing an improper naming/usage issue >> that >> > has been present but if so we probably should have addressed not in this >> > bug fix line. Will defer to Troy/Andy for more context and next steps >> > >> > On Tue, Mar 3, 2020 at 5:53 AM Joe Witt wrote: >> > >> > > If accurateWe need to look into whether this was a mistake and >> fix it >> > > if so. And we need to reflect this in the migration guide >> > > >> > > On Tue, Mar 3, 2020 at 4:40 AM Ryan Ward >> wrote: >> > > >> > >> Endre - thanks that was it >> > >> >> > >> On Tue, Mar 3, 2020 at 6:50 AM Endre Kovacs >> > >> wrote: >> > >> >> > >> > Hi, >> > >> > >> > >> > One additional thing: >> > >> > >> > >> > we encountered something strange as well: >> > >> > >> > >> > on 1.11.2 clustered, kerberized: request replication worked well. >> > >> > >> > >> > on 1.11.3 clustered, kerberized: request replication did not work, >> > >> unless >> > >> > you specify, and set >> > >> > nifi.security.keyPasswd >> > >> > >> > >> > to the very same password as the >> > >> > >> > >> > nifi.security.keystorePasswd >> > >> > >> > >> > For us this resolved the issue. >> > >> > >> > >> > Best regards, >> > >> > Endre >> > >> > >> > >> > Sent with [ProtonMail](https://protonmail.com) Secure Email. >> > >> > >> > >> > ‐‐‐ Original Message ‐‐‐ >> > >> > On Tuesday, March 3, 2020 12:40 PM, Ryan Ward < >> ryan.wa...@gmail.com> >> > >> > wrote: >> > >> > >> > >> > > Hi Joe - Did you resolve your issue? If so I am wondering what >> the >> > fix >> > >> > was as I'm seeing the same error on my cluster. >> > >> > > >> > >> > > On Thu, Feb 27, 2020 at 3:13 AM Endre Kovacs < >> > >> > andrewsmit...@protonmail.com.invalid> wrote: >> > >> > > >> > >> > >> Hi Joe, >> > >> > >> >> > >> > >> 1. Have you tried connecting/debugging with openssl? From one >> pod >> > to >> > >> > the other: >> > >> > >> (openssl s_client -debug -CAfile >> > >> > ca-bundle-signing-node-certificates.crt -cert my-client-cert.crt >> > >> -connect >> > >> > nifi-3.nifi-headless.lizardspock.svc.cluster.local:6007) >> > >> > >> >> > >> > >> 2. certs can also be verified by: >> > >> > >> openssl verify -verbose -CAfile ca-bundle.crt >> my-client-cert.crt >> > >> > >> >> > >> > >> 3. Can you check if no intermediary CAs are missing from the >> nodes >> > >> > truststore? >> > >> > >> >> > >> > >> 4. This exception is coming from inter-node communication >> > >> (replication >> > >> > of request from one node to the other). This means that it is >> > unrelated >> > >> to >> > >> > external user's authentication by client certificate. The question >> is: >> > >> is >> > >> > your inter node communication secured by the trusted root CA (that >> you >> > >> are >> > >> > sure that the CA cert is present in the trust store) or is it >> secured >> > by >> > >> > selfsigned CA (which's CA may be lacking from your truststore)? >> > >> > >> >> > >> > >> 5. `nifi.security.needClientAuth` is not part of NiFi >> properties >> > any >> > >> > more. If SSL is turned on, and no >> > >> > `nifi.security.user.login.identity.provider` is set, then client >> cert >> > >> based >> > >> > auth is the default. But supplying this property have no >> detrimental >> > >> effect >> > >> > anyhow. >> > >> > >> >> > >> > >> Best regards, >> > >> > >> Endre >> > >> > >> >> > >> > >> Sent with ProtonMail Secure Email. >> > >> > >> >> > >> > >> ‐‐‐ Original Message ‐‐‐ >> > >> > >> On Wednesday, February 26, 2020 6:22 PM, Joe Gresock >> > >> > jgres...@gmail.com wrote: >> > >> > >> >> > >> > >>> Were there any changes with how the trust store is used in >> > 1.11.3? I >> > >> > had a >> > >> > >>> 1.11.0 deployment working with the following settings, but >> when I >> > >> > deployed >> > >> > >>> 1.11.3, the cluster can't seem to replicate requests to itself: >> > >> > >>> nifi.remote.input.host= >> > >> > >>> nifi.remote.input.secure=true >> > >> > >>> nifi.remote.input.socket.port=32440 >> > >> > >>> nifi.remote.input.http.enabled=true >> > >> >
