Re: Certificate Issue when connecting to NiFi Registry
Hi Bryan and Mark, Thank you for your replies. So after a lot of trial and error, got the connection working between NiFi and the NiFi Registry. Went with the approach of LDAP + local file based. However, it seems like since we’ve installed a wildcard certificate, we need to specify the DN exactly as follows in the Initial User Identity as well as the Node Identity: CN=*domain.com, O="Company", L=Place, ST= Area, C=TH Only then it works. Even if we try adding a subdomain such as nifi.domain.com in the CN, the whole thing stops working. Is this how it’s intended to work? Thank You. Best Regards, Nikhil C. On 17/9/18 20:39, "Bryan Bende" wrote: In addition to what Mark said, the NiFi nodes to be represented as users somewhere in order to create policies that grant them proxy permission. If your authorizer is using the LDAP user group provider then you could create users in LDAP to represent your NiFi nodes. If you don't want to create users in LDAP, then you can use a composite user group provider made up of LDAP + file-based and you can create users in the file-based user group provider using the DNs of the NiFi certificates. On Mon, Sep 17, 2018 at 9:11 AM Mark Payne wrote: > > Nikhil, > > LDAP is used for user authentication. Machine-to-machine communications, though, must use > certificates, and so authentication in this case is done via certificates. In both cases, though, this > is authentication (i.e., identifying who the user is), not authorization (i.e., determining the user's permissions). > You'll need to ensure that you assign appropriate permissions for your users. All NiFi nodes will need the > Proxy permission, I believe. The Registry User Guide [1] will explain the different permissions/policies available > and what they are used for. > > Thanks > -Mark > > > [1] https://nifi.apache.org/docs/nifi-registry-docs/html/user-guide.html#special-privileges > > > On Sep 17, 2018, at 8:34 AM, Nikhil Chaudhary wrote: > > Hi Mark, > > So I added clientAuth and serverAuth both in the certificate and now the error is as follows: > > NiFi Registry Logs: > > 2018-09-17 12:26:07,170 INFO [NiFi Registry Web Server-12] o.a.n.r.w.s.NiFiRegistrySecurityConfig Identity in proxy chain not trusted to act as a proxy: org.apache.nifi.registry.web.security.authentication.exception.UntrustedProxyException: Untrusted proxy [CN=*domain.com, O="Company", L=Place, ST= Area, C=TH]. Returning 403 response. > > > NiFi Logs: > > 2018-09-17 12:26:07,202 INFO [NiFi Web Server-21] o.a.n.w.a.config.NiFiCoreExceptionMapper org.apache.nifi.web.NiFiCoreException: Unable to obtain listing of buckets: org.apache.nifi.registry.client.NiFiRegistryException: Error retrieving all buckets: Untrusted proxy [CN=*domain.com, O="Company", L= Place, ST=Area, C=TH]. Contact the system administrator.. Returning Conflict response. > 2018-09-17 12:26:07,203 DEBUG [NiFi Web Server-21] o.a.n.w.a.config.NiFiCoreExceptionMapper > org.apache.nifi.web.NiFiCoreException: Unable to obtain listing of buckets: org.apache.nifi.registry.client.NiFiRegistryException: Error retrieving all buckets: Untrusted proxy [CN=*domain.com, O="Company", L= Place, ST= Area, C=TH]. Contact the system administrator. > > Both the registry and NiFi are running on the same instance for test purposes and have the same host set up in the configuration: nifi.domain.com > Any idea what could be wrong now? > > The only information I've found online is to add the Node information within the authorizers file, however since we use LDAP, I believe that isn't necessary? Or is it similar to the set up of nifi-to-nifi clustering? > > Cheers, > Nikhil C. > > On Thu, Sep 13, 2018 at 8:40 PM Mark Payne wrote: >> >> Hi Nikhil, >> >> The property that you mention: "nifi.registry.security.needClientAuth" applies only to user logins. >> This allows users to login via certificate or other methods by not requiring that they present a client >> certificate. However, NiFi & registry require mutual authentication for all machine-to-machine interactions. >> So in order to have NiFi talk to the registry, NiFi's cert will need to have client auth usage as well. >> >> Thanks >> -Mark >> >> >> On Sep 13, 2018, at 5:35 AM, Nikhil Chaudhary wrote: >> >> Hey Guys, >> >> Been stumped with a certificate issue. >> A bit of info on the deployment strategy. >> >>
Re: Certificate Issue when connecting to NiFi Registry
Hi Mark, So I added clientAuth and serverAuth both in the certificate and now the error is as follows: *NiFi Registry Logs:* 2018-09-17 12:26:07,170 INFO [NiFi Registry Web Server-12] o.a.n.r.w.s.NiFiRegistrySecurityConfig Identity in proxy chain not trusted to act as a proxy: org.apache.nifi.registry.web.security.authentication.exception.UntrustedProxyException: Untrusted proxy [CN=*domain.com, O="Company", L=Place, ST= Area, C=TH]. Returning 403 response. *NiFi Logs:* 2018-09-17 12:26:07,202 INFO [NiFi Web Server-21] o.a.n.w.a.config.NiFiCoreExceptionMapper org.apache.nifi.web.NiFiCoreException: Unable to obtain listing of buckets: org.apache.nifi.registry.client.NiFiRegistryException: Error retrieving all buckets: Untrusted proxy [CN=*domain.com, O="Company", L= Place, ST=Area, C=TH]. Contact the system administrator.. Returning Conflict response. 2018-09-17 12:26:07,203 DEBUG [NiFi Web Server-21] o.a.n.w.a.config.NiFiCoreExceptionMapper org.apache.nifi.web.NiFiCoreException: Unable to obtain listing of buckets: org.apache.nifi.registry.client.NiFiRegistryException: Error retrieving all buckets: Untrusted proxy [CN=*domain.com, O="Company", L= Place, ST= Area, C=TH]. Contact the system administrator. Both the registry and NiFi are running on the same instance for test purposes and have the same host set up in the configuration: nifi.domain.com Any idea what could be wrong now? The only information I've found online is to add the Node information within the authorizers file, however since we use LDAP, I believe that isn't necessary? Or is it similar to the set up of nifi-to-nifi clustering? Cheers, Nikhil C. On Thu, Sep 13, 2018 at 8:40 PM Mark Payne wrote: > Hi Nikhil, > > The property that you mention: "nifi.registry.security.needClientAuth" > applies only to user logins. > This allows users to login via certificate or other methods by not > requiring that they present a client > certificate. However, NiFi & registry require mutual authentication for > all machine-to-machine interactions. > So in order to have NiFi talk to the registry, NiFi's cert will need to > have client auth usage as well. > > Thanks > -Mark > > > On Sep 13, 2018, at 5:35 AM, Nikhil Chaudhary > wrote: > > Hey Guys, > > Been stumped with a certificate issue. > A bit of info on the deployment strategy. > > NiFi is running with a wildcard certificate in its keystore (*.domain.com) > – It’s a self signed certificate. > We’ve added the Root CA in the truststore of NiFi. > > We’ve used the same keystore to run NiFi registry. > > So installing the Root CA on my laptop, I can access NiFi on HTTPS with no > errors or warnings. > In theory the Root CA within the NiFi truststore should do the same when > accessing NiFi registry, shouldn’t it? > > I enabled debug logs and the error that came up was: *Caused by: > sun.security.validator.ValidatorException: Extended key usage does not > permit use for TLS client authentication* > > The certificate only has serverAuth in it’s extended key usage but > shouldn’t that be enough? > I’ve seen emails and posts online regarding NiFi clustering in which case > clientAuth needs to be enabled but this case seems different? > > ClientAuth in NiFi registry properties file is set as false. > *nifi.registry.security.needClientAuth=false* > > Is there something I’m missing or not doing correctly? > > *Stack Trace:* > > 2018-09-12 11:02:22,581 DEBUG [NiFi Registry Web Server-15] > org.eclipse.jetty.server.HttpConnection > javax.net.ssl.SSLHandshakeException: General SSLEngine problem >at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1529) > ~[na:1.8.0_181] >at > sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535) > ~[na:1.8.0_181] >at > sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:813) > ~[na:1.8.0_181] >at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781) > ~[na:1.8.0_181] >at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) > ~[na:1.8.0_181] >at > org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.fill(SslConnection.java:621) > ~[jetty-io-9.4.3.v20170317.jar:9.4.3.v20170317] >at > org.eclipse.jetty.server.HttpConnection.fillRequestBuffer(HttpConnection.java:322) > [jetty-server-9.4.3.v20170317.jar:9.4.3.v20170317] >at > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:231) > [jetty-server-9.4.3.v20170317.jar:9.4.3.v20170317] >at > org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279) > [jetty-io-9.4.3.v20170317.jar:9.4.3.v20170317] >at > org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110) > [jetty
Certificate Issue when connecting to NiFi Registry
Hey Guys, Been stumped with a certificate issue. A bit of info on the deployment strategy. NiFi is running with a wildcard certificate in its keystore (*.domain.com) – It’s a self signed certificate. We’ve added the Root CA in the truststore of NiFi. We’ve used the same keystore to run NiFi registry. So installing the Root CA on my laptop, I can access NiFi on HTTPS with no errors or warnings. In theory the Root CA within the NiFi truststore should do the same when accessing NiFi registry, shouldn’t it? I enabled debug logs and the error that came up was: Caused by: sun.security.validator.ValidatorException: Extended key usage does not permit use for TLS client authentication The certificate only has serverAuth in it’s extended key usage but shouldn’t that be enough? I’ve seen emails and posts online regarding NiFi clustering in which case clientAuth needs to be enabled but this case seems different? ClientAuth in NiFi registry properties file is set as false. nifi.registry.security.needClientAuth=false Is there something I’m missing or not doing correctly? Stack Trace: 2018-09-12 11:02:22,581 DEBUG [NiFi Registry Web Server-15] org.eclipse.jetty.server.HttpConnection javax.net.ssl.SSLHandshakeException: General SSLEngine problem at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1529) ~[na:1.8.0_181] at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535) ~[na:1.8.0_181] at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:813) ~[na:1.8.0_181] at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781) ~[na:1.8.0_181] at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[na:1.8.0_181] at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.fill(SslConnection.java:621) ~[jetty-io-9.4.3.v20170317.jar:9.4.3.v20170317] at org.eclipse.jetty.server.HttpConnection.fillRequestBuffer(HttpConnection.java:322) [jetty-server-9.4.3.v20170317.jar:9.4.3.v20170317] at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:231) [jetty-server-9.4.3.v20170317.jar:9.4.3.v20170317] at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279) [jetty-io-9.4.3.v20170317.jar:9.4.3.v20170317] at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110) [jetty-io-9.4.3.v20170317.jar:9.4.3.v20170317] at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:258) [jetty-io-9.4.3.v20170317.jar:9.4.3.v20170317] at org.eclipse.jetty.io.ssl.SslConnection$3.succeeded(SslConnection.java:147) [jetty-io-9.4.3.v20170317.jar:9.4.3.v20170317] at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110) [jetty-io-9.4.3.v20170317.jar:9.4.3.v20170317] at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124) [jetty-io-9.4.3.v20170317.jar:9.4.3.v20170317] at org.eclipse.jetty.util.thread.Invocable.invokePreferred(Invocable.java:122) [jetty-util-9.4.3.v20170317.jar:9.4.3.v20170317] at org.eclipse.jetty.util.thread.strategy.ExecutingExecutionStrategy.invoke(ExecutingExecutionStrategy.java:58) [jetty-util-9.4.3.v20170317.jar:9.4.3.v20170317] at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:201) [jetty-util-9.4.3.v20170317.jar:9.4.3.v20170317] at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:133) [jetty-util-9.4.3.v20170317.jar:9.4.3.v20170317] at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:672) [jetty-util-9.4.3.v20170317.jar:9.4.3.v20170317] at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:590) [jetty-util-9.4.3.v20170317.jar:9.4.3.v20170317] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181] Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[na:1.8.0_181] at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728) ~[na:1.8.0_181] at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330) ~[na:1.8.0_181] at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322) ~[na:1.8.0_181] at sun.security.ssl.ServerHandshaker.clientCertificate(ServerHandshaker.java:1979) ~[na:1.8.0_181] at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:237) ~[na:1.8.0_181] at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052) ~[na:1.8.0_181] at sun.security.ssl.Handshaker$1.run(Handshaker.java:992) ~[na:1.8.0_181] at sun.security.ssl.Handshaker$1.run(Handshaker.java:989) ~[na:1.8.0_181] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_181] at
ListS3 Processor State Defaulted
Hi, Has anyone else faced an issue when upgrading NiFi from 1.4.0 to 1.7.1, the ListS3 processor gets defaulted and it starts downloading everything all over again within that S3 bucket? The state management file mentioned at nifi.state.management.configuration.file was kept intact. Is there some other configurations that we need to be aware of to keep the state as-is and only continue onwards? Thank You. Best Regards, Nikhil C.
Re: LDAP Users & Group Sync
Hi Kevin, Noted and thank you very much. Best Regards, Nick On 18 Feb 2018 21:29 +0700, Kevin Doran <kdo...@apache.org>, wrote: > Hi Nick, > > Just wanted to shoot you a quick note mentioning that this issue was resolved > in NiFi 1.5.0 which was released in January. If/when you upgrade, be sure to > change your configuration from 120ms to 120s (or whatever time interval you > prefer to use, it will work correctly now). There is a guard in place that > will protect users upgrading who were using the workaround of setting a tiny > (sub-second) sync interval, but that will just prevent startup with an error > message. You’ll need to change your authorizers.xml sync interval to be > longer to get past that sanity check. > > Regards, > Kevin > > From: Nikhil Chaudhary <nik...@acommerce.asia> > Reply-To: <users@nifi.apache.org> > Date: Friday, December 8, 2017 at 02:23 > To: <users@nifi.apache.org> > Subject: Re: LDAP Users & Group Sync > > Hi Kevin, > > Changing that directly into milliseconds (120 ms) worked perfectly and the > LDAP group and users synced as expected. > > Thank You. > > Cheers, > Nick > > On 8 Dec 2017 11:59 +0700, Kevin Doran <kdoran.apa...@gmail.com>, wrote: > > > Hi Nikhil, > > > > I haven't heard of this issue before. Looking at the NiFi code that is > > interpreting the sync interval property from authorizers.xml [1], I think > > you may have come across a bug. > > > > With just looking at the code, and not having tried to reproduce the issue, > > it appears to me that the sync interval is not being converted correctly in > > the block of code that processes the XML and schedules a background task to > > perform the sync. Specifically, it appears the configured value is read > > into a variable and normalized to *milliseconds*, but then passed into a > > function as *seconds* without value conversion. So in your case, that would > > mean this is happening: > > > > 2 minutes -> 120,000 milliseconds -> 120,000 seconds > > > > If I'm correct, that would certainly explain why you aren't seeing > > synchronizations happening! > > > > I've filed a JIRA bug report [2]. I want to confirm the issue and the fix > > by attempting to reproduce it, after which I can submit a patch to fix this > > in the next regular NiFi release. > > > > In the meantime, can you try this workaround in your authorizers.xml: > > > > > by a factor of 1000, > > so this is actually specifying an interval of 120 seconds, not 120 > > milliseconds. -- > > 120 ms > > > Let me know if that does not work, in which case we can dig into it > > further. Thanks for bringing this issue to everyone's attention! > > > > Kevin > > > > [1] > > https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-ldap-iaa-providers-bundle/nifi-ldap-iaa-providers/src/main/java/org/apache/nifi/ldap/tenants/LdapUserGroupProvider.java#L365 > > [2] https://issues.apache.org/jira/browse/NIFI-4677 > > > > On 12/7/17, 22:31, "Nikhil Chaudhary" <nik...@acommerce.asia> wrote: > > > > Hey Guys, > > > > We’re trying to get NiFi synced to our LDAP directory and the initial sync > > happens successfully, however any changes made within LDAP (adding another > > group to an existing user) does not get synced unless we restart NiFi. > > > > Tried to set the Sync Interval to 2 mins in the authorizers.xml file but > > still nothing. > > > > Is there any other place we need to set the sync interval or something else > > that needs to be done for the sync to happen? > > > > Thank You. > > > > Cheers, > > Nick > > > > > >
LDAP Users & Group Sync
Hey Guys, We’re trying to get NiFi synced to our LDAP directory and the initial sync happens successfully, however any changes made within LDAP (adding another group to an existing user) does not get synced unless we restart NiFi. Tried to set the Sync Interval to 2 mins in the authorizers.xml file but still nothing. Is there any other place we need to set the sync interval or something else that needs to be done for the sync to happen? Thank You. Cheers, Nick
Running TRUNCATE or UPSERT for Data Ingestion on a JDBC Controller (Postgres Database)
Hello, So I'm trying to create a flow wherein one flow will export data from a table to an S3 bucket in avro format and the other flow will import that data into a table. It would be best if we could do an UPSERT, however as there are many tables, was trying to keep the flow as dynamic and small as possible. The schema exists in the avro registry thus the PutDatabaseRecord works perfectly without writing any sql query. Tried creating a flow wherein we could separate each avro record into 1 record and if INSERT fails then update, but with over a million records, that is extremely slow. The last option we have is to TRUNCATE and re-insert all the data, but those get separated into 2 flows, truncate in a separate flow and then insert in a different flow. Has anyone else run into this issue? Would like to know if its possible to combine truncate and insert into one transaction so we are sure there is always data available. Secondly, what is the best way to do an UPSERT if any? The data source doesn't have a timestamp hence it's hard to check what really was created/updated hence the upsert reasoning and the source tables cannot be changed. Best Regards, Nikhil C.
