Re: Two datacenters with NiFi and dedicated zookeeper - which function in which datacenter

[Edward Armes]

Hi Josef,

I would suggest reaching out to the Apache ZooKeeper community who would be
able to give you the best guidance on your ZooKeeper deployments. In the
meantime, I would also suggest reviewing the offical ZooKeeper docs (
https://zookeeper.apache.org/documentation.html) as well.

Edward

On Mon, 17 Jul 2023, 11:44 ,  wrote:

> Hi guys
>
>
>
> Today we have just one datacenter with a few NiFi clusters, so we use a
> dedicated 3-node zookeeper cluster in that datacenter. We are now planning
> to expand to another datacenter, so we would like to split the NiFis as
> well as zookeeper. However 2 zookeeper nodes is not a good quorum number,
> so we had to idea to do the following regarding zookeeper:
>
>
>
>- Datacenter 1: 2 zookeeper nodes
>- Datacenter 2: 2 zookeeper nodes
>- Location 3 (another small DC): 1 zookeper node -> no NiFis
>
>
>
> All locations are connected via dark fiber, however the third location is
> bit more far away from the others. Now, as we anyway split the NiFi
> clusters over the two datacenters. Shall we limit the NiFi zookeeper
> clients (state-management.xml) to the zookeeper nodes located within the
> same datacenter? Any comments to our design idea? How do you deploy NiFis
> over multiple regions/datacenters from zookeeper perspective?
>
>
>
> Cheers Josef
>

Re: Use Kafka processors with MSK/IAM

[Edward Armes]

Hi Max,

Having a quick look through the PR, it looks like it was closed due to
inactivity.

All that probably needs to happen is for someone to check that the change
is still good and then re-open the PR.

If someone can pick this up, the contributor guidelines, (
https://cwiki.apache.org/confluence/plugins/servlet/mobile?contentId=57904904#content/view/57904904),
should cover what needs to be done.

Edward



On Wed, 20 Jul 2022, 06:52 LEZIER Maxime (ITNOVEM), 
wrote:

> Hello,
>
>
>
> I have to consume an MSK (kafka from aws) wich use IAM assume role for
> authentication.
>
>
>
> this does not seem to be supported by NiFi.
>
>
>
> I see there is an pull request for this. :
>
> https://github.com/apache/nifi/pull/5291#discussion_r724428821
>
>
>
> but it look to be closed and the feature not implemented.
>
>
>
> Someone could tell me if this work will be finished ?
>
>
>
> Also, is there any workaround I could use for connect a kafka nifi
> consumer to an msk with iam assume role authentication ?
>
>
>
> Any help will be very usefull .
>
>
>
> Thanks .
>
>
>
> Max L.
>
>
>
> Interne
>
> Interne
> ---
> Ce message et toutes les pièces jointes sont établis à l'intention
> exclusive de ses destinataires et sont confidentiels. L'intégrité de ce
> message n'étant pas assurée sur Internet, la SNCF ne peut être tenue
> responsable des altérations qui pourraient se produire sur son contenu.
> Toute publication, utilisation, reproduction, ou diffusion, même partielle,
> non autorisée préalablement par la SNCF, est strictement interdite. Si vous
> n'êtes pas le destinataire de ce message, merci d'en avertir immédiatement
> l'expéditeur et de le détruire.
> ---
> This message and any attachments are intended solely for the addressees
> and are confidential. SNCF may not be held responsible for their contents
> whose accuracy and completeness cannot be guaranteed over the Internet.
> Unauthorized use, disclosure, distribution, copying, or any part thereof is
> strictly prohibited. If you are not the intended recipient of this message,
> please notify the sender immediately and delete it.
>

Re: Nifi 1.15.2 and 1.15.3 compilation errors

[Edward Armes]

t; >> We also tested -Dhive.version=3.1.2, that is most recent on Apache Hive
> site. Also without success.
> >>
> >> Please note that without this parameter probably hive2 version is
> compiled into nifi and HiveStreaming is not working then.
> >> And hive 3 is most popular novadays.
> >>
> >> Regards,
> >> Mike
> >>
> >> -Original Message-
> >> From: Bryan Bende 
> >> Sent: Wednesday, January 19, 2022 3:03 PM
> >> To: users@nifi.apache.org
> >> Subject: Re: Nifi 1.15.2 and 1.15.3 compilation errors
> >>
> >> We did ban all use of log4j-core in favor of log4j-to-slf4j...
> >>
> >> https://issues.apache.org/jira/browse/NIFI-9483
> >>
> >> With your build specifying the -D versions, it created a different
> classpath during your build than what existed during the normal build of
> the release.
> >>
> >> We should try to fix any issues like this though, so if you can figure
> out which dependency path is the issue, the fix is to add some exclusions
> and then to ensure that log4j-to-slf4j dependency is added.
> >>
> >> On Wed, Jan 19, 2022 at 8:48 AM Michal Tomaszewski <
> michal.tomaszew...@cca.pl> wrote:
> >> >
> >> > OK, but there is information that log4j-core:jar:2.17.1 is blocked:
> >> >
> >> > [WARNING] Rule 0:
> org.apache.maven.plugins.enforcer.BannedDependencies failed with message:
> >> >
> >> > Found Banned Dependency:
> >> > org.apache.logging.log4j:log4j-core:jar:2.17.1
> >> >
> >> > while this version is not impacted by log4j security vulnerability:
> >> >
> >> > https://logging.apache.org/log4j/2.x/security.html
> >> >
> >> >
> >> >
> >> > So for sure it shouldn’t be banned.
> >> >
> >> >
> >> >
> >> > Regards,
> >> >
> >> > Mike
> >> >
> >> > From: Pierre Villard 
> >> > Sent: Wednesday, January 19, 2022 1:11 PM
> >> > To: users@nifi.apache.org
> >> > Subject: Re: Nifi 1.15.2 and 1.15.3 compilation errors
> >> >
> >> >
> >> >
> >> > We recently changed the build to fail if it's including problematic
> transitive dependencies for log4j. I think those are coming with the
> specific versions of HBase/Hive you're using. Not a maven expert but I
> guess there is a way to exclude the step "enforce-banned-dependencies" from
> the build.
> >> >
> >> >
> >> >
> >> > Le mer. 19 janv. 2022 à 12:57, Michal Tomaszewski <
> michal.tomaszew...@cca.pl> a écrit :
> >> >
> >> > Hi Edward,
> >> >
> >> > Full mvn dependency:tree log enclosed.
> >> >
> >> > We’re compiling using command:
> >> >
> >> > mvn -e -X -T C2.0 clean install -DskipTests -Dhive.version=3.1.0
> >> > -Dhbase.version=2.0.2
> >> >
> >> >
> >> >
> >> > Regards,
> >> >
> >> > Mike
> >> >
> >> >
> >> >
> >> > From: Edward Armes 
> >> > Sent: Wednesday, January 19, 2022 12:34 PM
> >> > To: users@nifi.apache.org
> >> > Subject: Re: Nifi 1.15.2 and 1.15.3 compilation errors
> >> >
> >> >
> >> >
> >> > Hi Mike,
> >> >
> >> >
> >> >
> >> > Can you give the full output you get from mvn dependency:tree as we
> 15.2 and 3 was done to remove log4j dependences from Nifi as a safety
> measure due the recent log4shell issues.
> >> >
> >> >
> >> >
> >> > Thanks
> >> >
> >> >
> >> >
> >> > Edward
> >> >
> >> >
> >> >
> >> > On Wed, Jan 19, 2022 at 10:24 AM Michal Tomaszewski <
> michal.tomaszew...@cca.pl> wrote:
> >> >
> >> > Hi,
> >> >
> >> > We are trying to compile nifi from sources.
> >> >
> >> > There is no problem with 1.15.1 compilation but in case of 1.15.2 and
> 1.15.3 there are compilation errors.
> >> >
> >> >
> >> >
> >> > Can you suggest how to solve this problem?
> >> >
> >> >
> >> >
> >> > mvn dependency:tree has no errors:
> >> >
> >> >
> >> >
> >> >
> >

Re: Nifi 1.15.2 and 1.15.3 compilation errors

[Edward Armes]

Hi Mike,

Can you give the full output you get from mvn dependency:tree as we 15.2
and 3 was done to remove log4j dependences from Nifi as a safety measure
due the recent log4shell issues.

Thanks

Edward

On Wed, Jan 19, 2022 at 10:24 AM Michal Tomaszewski <
michal.tomaszew...@cca.pl> wrote:

> Hi,
>
> We are trying to compile nifi from sources.
>
> There is no problem with 1.15.1 compilation but in case of 1.15.2 and
> 1.15.3 there are compilation errors.
>
>
>
> Can you suggest how to solve this problem?
>
>
>
> mvn dependency:tree has no errors:
>
>
>
>
>
> $ tail log1.txt
>
> [INFO] nifi-toolkit-flowfile-repo . SUCCESS [
> 0.005 s]
>
> [INFO] nifi-toolkit-flowanalyzer .. SUCCESS [
> 0.008 s]
>
> [INFO] nifi-toolkit-assembly .. SUCCESS [
> 0.463 s]
>
> [INFO] nifi-toolkit-api ... SUCCESS [
> 0.023 s]
>
> [INFO]
> 
>
> [INFO] BUILD SUCCESS
>
> [INFO]
> 
>
> [INFO] Total time:  02:41 min
>
> [INFO] Finished at: 2022-01-19T08:51:46+01:00
>
>
>
>
>
> Compilation of branch rel/nifi-1.15.3:
>
>
>
> [DEBUG] Adding ERROR message due to exception
>
> org.apache.maven.enforcer.rule.api.EnforcerRuleException: Found Banned
> Dependency: org.apache.logging.log4j:log4j-core:jar:2.17.1
>
> Use 'mvn dependency:tree' to locate the source of the banned dependencies.
>
> at org.apache.maven.plugins.enforcer.AbstractBanDependencies.execute
> (AbstractBanDependencies.java:113)
>
> at org.apache.maven.plugins.enforcer.EnforceMojo.execute
> (EnforceMojo.java:200)
>
> at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo
> (DefaultBuildPluginManager.java:137)
>
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute
> (MojoExecutor.java:210)
>
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute
> (MojoExecutor.java:156)
>
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute
> (MojoExecutor.java:148)
>
> at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject
> (LifecycleModuleBuilder.java:117)
>
> at
> org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call
> (MultiThreadedBuilder.java:196)
>
> at
> org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call
> (MultiThreadedBuilder.java:186)
>
> at java.util.concurrent.FutureTask.run (FutureTask.java:266)
>
> at java.util.concurrent.Executors$RunnableAdapter.call
> (Executors.java:511)
>
> at java.util.concurrent.FutureTask.run (FutureTask.java:266)
>
> at java.util.concurrent.ThreadPoolExecutor.runWorker
> (ThreadPoolExecutor.java:1149)
>
> at java.util.concurrent.ThreadPoolExecutor$Worker.run
> (ThreadPoolExecutor.java:624)
>
> at java.lang.Thread.run (Thread.java:748)
>
> [WARNING] Rule 0: org.apache.maven.plugins.enforcer.BannedDependencies
> failed with message:
>
> Found Banned Dependency: org.apache.logging.log4j:log4j-core:jar:2.17.1
>
> Use 'mvn dependency:tree' to locate the source of the banned dependencies.
>
> [INFO]
>
>
>
> ….
>
>
>
> [INFO]
> 
>
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-enforcer-plugin:3.0.0:enforce
> (enforce-banned-dependencies) on project nifi-kite-processors: Some
> Enforcer rules have failed. Look above for specific messages explaining why
> the rule failed. -> [Help 1]
>
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute
> goal org.apache.maven.plugins:maven-enforcer-plugin:3.0.0:enforce
> (enforce-banned-dependencies) on project nifi-kite-processors: Some
> Enforcer rules have failed. Look above for specific messages explaining why
> the rule failed.
>
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute
> (MojoExecutor.java:215)
>
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute
> (MojoExecutor.java:156)
>
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute
> (MojoExecutor.java:148)
>
> at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject
> (LifecycleModuleBuilder.java:117)
>
> at
> org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call
> (MultiThreadedBuilder.java:196)
>
> at
> org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call
> (MultiThreadedBuilder.java:186)
>
> at java.util.concurrent.FutureTask.run (FutureTask.java:266)
>
> at java.util.concurrent.Executors$RunnableAdapter.call
> (Executors.java:511)
>
> at java.util.concurrent.FutureTask.run (FutureTask.java:266)
>
> at java.util.concurrent.ThreadPoolExecutor.runWorker
> (ThreadPoolExecutor.java:1149)
>
> at java.util.concurrent.ThreadPoolExecutor$Worker.run
> 

Re: NiFi Losing Zookeeper Connection

[Edward Armes]

I've just had a quick look on ZooKeeper JIRA and it looks like there is a
bug around ZooKeeper session closing that was introduced as part of a
change in 3.6 release it looks like it might of been fixed in 3.7 release

https://issues.apache.org/jira/browse/ZOOKEEPER-3822
https://issues.apache.org/jira/browse/ZOOKEEPER-3706
https://issues.apache.org/jira/browse/ZOOKEEPER-3828

Edward

On Mon, Jan 10, 2022 at 6:44 PM Shawn Weeks 
wrote:

> I think I beat you to it. Feel free to mark whichever a dupelicate.
>
>
>
> https://issues.apache.org/jira/browse/NIFI-9559
>
>
>
> Thanks
>
> Shawn
>
>
>
> *From:* Nathan Gough 
> *Sent:* Monday, January 10, 2022 12:37 PM
> *To:* users@nifi.apache.org
> *Subject:* Re: NiFi Losing Zookeeper Connection
>
>
>
> Hi Shawn,
>
>
>
> Thanks for reporting the issue. I'll see if I can reproduce it and figure
> out why it's occurring/what we can do to fix it. I created a Jira issue
> here: https://issues.apache.org/jira/browse/NIFI-9560.
>
>
>
> Nathan
>
>
>
> On Mon, Jan 10, 2022 at 1:24 PM Shawn Weeks 
> wrote:
>
> In my case the IP of Zookeeper is not changing though…
>
>
>
> Thanks
>
> Shawn
>
>
>
> *From:* Shawn Weeks 
> *Sent:* Monday, January 10, 2022 12:23 PM
> *To:* users@nifi.apache.org
> *Subject:* RE: NiFi Losing Zookeeper Connection
>
>
>
> Of note someone on Stack overflow is having this issue with the current
> version of the curator framework. This sounds like the same issue.
>
>
>
>
> https://stackoverflow.com/questions/68215630/why-isnt-curator-recovering-when-zookeeper-is-back-online
>
>
>
> Thanks
>
> Shawn
>
>
>
> *From:* Shawn Weeks 
> *Sent:* Monday, January 10, 2022 12:12 PM
> *To:* users@nifi.apache.org
> *Subject:* NiFi Losing Zookeeper Connection
>
>
>
> I’ve been dealing with a Zookeeper connection issue on NiFi 1.14 for a
> while now and I was wondering if anyone had any ideas. Basic issue is a
> NiFi node will lose its connection to Zookeeper due to network
> interruptions and then it’s never able to get its connection back. Logs
> look like it’s retrying over and over but I suspect it’s not and it’s stuck
> in this mode where the connection is gone but it’s never going to
> reconnect. Only way to resolve the issue is to restart NiFi. Exception in
> the logs starts around 2022-01-10 17:20:55,919 and I’ve cross referenced it
> with some zookeeper logs at the same time. All three zookeeper logs show a
> similar error about this box. In this example 192.168.1.212 is the IP for
> the NiFi instance called nifi0592.example.org. This is running in AWS and
> I’ve reviewed flow logs for REJECT or firewall blocks but nothing. We’re on
> Zookeeper 3.6.3 and I’m seeing this across multiple NiFi instances and
> VPCs. I’ve found mentions of the suspended in a zookeeper ticket but the
> client version that fixed it has been in NiFi for several versions now.
>
>
>
> Thanks
>
> Shawn
>
>
>
> # NiFi Log
>
> 2022-01-10 17:19:57,464 INFO [Write-Ahead Local State Provider
> Maintenance] org.wali.MinimalLockingWriteAheadLog
> org.wali.MinimalLockingWriteAheadLog@718198db checkpointed with 2951
> Records and 0 Swap Files in 19 milliseconds (Stop-the-world time = 11
> milliseconds, Clear Edit Logs time = 1 millis), max Transaction ID 1224814
>
> 2022-01-10 17:19:57,781 WARN [Clustering Tasks Thread-3]
> o.apache.nifi.controller.FlowController Failed to send heartbeat due to:
> org.apache.nifi.cluster.protocol.ProtocolException: Cannot send heartbeat
> because there is no Cluster Coordinator currently elected
>
> 2022-01-10 17:19:57,927 INFO [Timer-Driven Process Thread-13]
> o.a.n.remote.StandardRemoteProcessGroup Successfully refreshed Flow
> Contents for RemoteProcessGroup[https://nifi0590.example.org:8443/nifi];
> updated to reflect 2 Input Ports [InputPort[name=vantage_file_push,
> targetId=51747258-3f23-3cc2-885c-0acf8f94d8dc],
> InputPort[name=incoming_bulletin,
> targetId=45d7c264-3094-352f-9734-7c379d2ec648]] and 0 Output Ports []
>
> 2022-01-10 17:20:05,918 WARN [Curator-ConnectionStateManager-0]
> o.a.c.f.state.ConnectionStateManager Session timeout has elapsed while
> SUSPENDED. Injecting a session expiration. Elapsed ms: 10001. Adjusted
> session timeout ms: 1
>
> 2022-01-10 17:20:12,884 WARN [Clustering Tasks Thread-3]
> o.apache.nifi.controller.FlowController Failed to send heartbeat due to:
> org.apache.nifi.cluster.protocol.ProtocolException: Cannot send heartbeat
> because there is no Cluster Coordinator currently elected
>
> 2022-01-10 17:20:15,918 WARN [Curator-ConnectionStateManager-0]
> o.a.c.f.state.ConnectionStateManager Session timeout has elapsed while
> SUSPENDED. Injecting a session expiration. Elapsed ms: 1. Adjusted
> session timeout ms: 1
>
> 2022-01-10 17:20:16,992 INFO [pool-13-thread-1]
> o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile
> Repository
>
> 2022-01-10 17:20:16,992 INFO [pool-13-thread-1]
> o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile
> Repository with 98 records in 0 

Re: Penalty feature of Processor (Disable)

[Edward Armes]

Sounds good to me, how do we want to expose this in the LookupProcessor
family? I'm conscious of not wanting to add to much overhead for cases
where the processor is never going to fail in this way?

My gut say that we should just root to failure with some kind of attribute
and then let user decide on how to handle it. With either a combination or
RoueOnAttribute attribute and RetryFlowFile processors



On Wed, 27 Oct 2021, 15:56 Kevin Doran,  wrote:

> Seems like a nice improvement. I would lean towards introducing the
> RetryableLookupFailureException. At the same time, I think we should add
> catch blocks for more specific subclasses of SQLException, such as
> SQLTransientException, which would also trigger a
> RetryableLookupFailureException instead of a LookupFailureException.
>
> > On Oct 27, 2021, at 08:55, Bryan Bende  wrote:
> >
> > I'd consider changing DatabaseRecordLookupService.. the way it is
> > currently implemented there is no way to tell the difference between a
> > retryable exception like IOException vs a non-retryable exception like
> > SQLException because the columns are wrong and will never work. We
> > could introduce a new RetryableLookupFailureException and have both
> > services catch IOException and throw the retryable exception, or both
> > services can let IOException be thrown and let the callers decide what
> > to do.
> >
> > On Wed, Oct 27, 2021 at 5:07 AM Edward Armes 
> wrote:
> >>
> >> Hi Bilal,
> >>
> >> Thanks for confirming, it looks like my hunch was correct, and there is
> a discrepancy in the lookup service code itself.
> >>
> >> In DatabaseRecordLookupService we catch an IOException and return a
> lookup failure like is done for an SQLExeception. This isn't done in the
> SimpleDatabaseLookupService. I think in this case its worth adding the
> IOException catch to SimpleDatabaseLookupService to bringing it in line
> with DatabaseRecordLookupService.
> >>
> >> Edward
> >>
> >> On Wed, 27 Oct 2021, 09:11 Bilal Bektas, 
> wrote:
> >>>
> >>> Hi Edward,
> >>>
> >>>
> >>>
> >>> Thank you for helping.
> >>>
> >>>
> >>>
> >>> You can find the information which you want:
> >>>
> >>>
> >>>
> >>> * LookupAttribute processor uses SimpleDatabaseLookupService, bundle
> of which is “org.apache.nifi - nifi-lookup-services-nar”
> >>>
> >>> * SimpleDatabaseLookupService uses DBCPConnectionPool, bundle of which
> is “org.apache.nifi - nifi-dbcp-service-nar”. There is no custom build
> service or processor on NiFi. All are the default bundle.
> >>>
> >>>
> >>> * Teradata JDBC version: 16.20.00.13
> >>>
> >>> * Oracle JDBC version: 12.2.0.1.0
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> In addition, I have done similar test on LookupAttribute (Oracle)
> processor. The same situation happened; flow files were penalized and the
> queue on upstream connection of LookupAttribute (Oracle) increased..
> >>>
> >>>
> >>>
> >>> Thank you in advance,
> >>>
> >>>
> >>>
> >>> --Bilal
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> From: Edward Armes 
> >>> Sent: 26 Ekim 2021 Salı 23:40
> >>> To: users@nifi.apache.org
> >>> Subject: Re: Penalty feature of Processor (Disable)
> >>>
> >>>
> >>>
> >>> Hi Bilal,
> >>>
> >>>
> >>>
> >>> Can you just confirm that your connection to Teradata is done using
> the DatabaseRecordLookupService and is using one of the 2
> DBCPConnectionPools which is using the Teradata JDBC driver or are you
> using custom built service?
> >>>
> >>>
> >>>
> >>> The reason for asking is that I want to do a quick check to make sure
> by were not masking an issue in one of the underlying services that hasn't
> been caught correctly for some reason
> >>>
> >>>
> >>>
> >>> Edward
> >>>
> >>>
> >>>
> >>> On Tue, 26 Oct 2021, 10:29 Bilal Bektas, 
> wrote:
> >>>
> >>> Hi Community,
> >>>
> >>>
> >>>
> >>> Thank you for detailed solutions and analysis.
> >>>
> >>>
> >>>
> >>> @Dev Team, do you

Re: Penalty feature of Processor (Disable)

[Edward Armes]

Hi Bilal,

Thanks for confirming, it looks like my hunch was correct, and there is a
discrepancy in the lookup service code itself.

In DatabaseRecordLookupService we catch an IOException and return a lookup
failure like is done for an SQLExeception. This isn't done in the
SimpleDatabaseLookupService. I think in this case its worth adding the
IOException catch to SimpleDatabaseLookupService to bringing it in line
with DatabaseRecordLookupService.

Edward

On Wed, 27 Oct 2021, 09:11 Bilal Bektas,  wrote:

> Hi Edward,
>
>
>
> Thank you for helping.
>
>
>
> You can find the information which you want:
>
>
>
> * LookupAttribute processor uses SimpleDatabaseLookupService, bundle of
> which is “org.apache.nifi - nifi-lookup-services-nar”
>
> * SimpleDatabaseLookupService uses DBCPConnectionPool, bundle of which is
> “org.apache.nifi - nifi-dbcp-service-nar”. There is no custom build service
> or processor on NiFi. All are the default bundle.
>
>
> * Teradata JDBC version: 16.20.00.13
>
> * Oracle JDBC version: 12.2.0.1.0
>
>
>
>
>
> In addition, I have done similar test on LookupAttribute (Oracle)
> processor. The same situation happened; flow files were penalized and the
> queue on upstream connection of LookupAttribute (Oracle) increased..
>
>
>
> Thank you in advance,
>
>
>
> --Bilal
>
>
>
>
>
> *From:* Edward Armes 
> *Sent:* 26 Ekim 2021 Salı 23:40
> *To:* users@nifi.apache.org
> *Subject:* Re: Penalty feature of Processor (Disable)
>
>
>
> Hi Bilal,
>
>
>
> Can you just confirm that your connection to Teradata is done using the
> DatabaseRecordLookupService and is using one of the 2 DBCPConnectionPools
> which is using the Teradata JDBC driver or are you using custom built
> service?
>
>
>
> The reason for asking is that I want to do a quick check to make sure by
> were not masking an issue in one of the underlying services that hasn't
> been caught correctly for some reason
>
>
>
> Edward
>
>
>
> On Tue, 26 Oct 2021, 10:29 Bilal Bektas,  wrote:
>
> Hi Community,
>
>
>
> Thank you for detailed solutions and analysis.
>
>
>
> @Dev Team, do you think to add a new feature (Rollback On Failure) for
> LookupAttribute processor like PutHiveQL processor?
>
>
>
> Thank you for helping,
>
>
>
> --Bilal
>
>
>
>
>
> *From:* Edward Armes 
> *Sent:* 26 Ekim 2021 Salı 01:32
> *To:* users@nifi.apache.org
> *Subject:* Re: Penalty feature of Processor (Disable)
>
>
>
> Having a quick look at the lookupAttribute code it looks like it takes a
> Optional<> from the call to the configured service. So I wonder if its
> worth adding the logic to service instead so that on erroring it can either
> return a missing value or throw an exception that would trigger the
> roleback. That would achieve the same goal without affecting other users of
> the LookupAttribute processor, where such logic isn't needed or wanted
> (e.g. SimpleLookupService).
>
>
>
> Edward
>
>
>
> On Mon, 25 Oct 2021, 21:54 Matt Burgess,  wrote:
>
> The approach in #1 is already present in a few Put processors like
> PutHive3QL, the property is named "Rollback on Failure" and takes a
> boolean value. The docs explain that if set to false, the flowfile is
> routed to failure, and if true will throw an exception and rollback
> the session. Check RollbackOnFailure.java for more details.
>
> Regards,
> Matt
>
> On Mon, Oct 25, 2021 at 4:46 PM Bryan Bende  wrote:
> >
> > The try/catch for IOException in LookupAttribute is after already
> > calling session.get(), so it is separate from loading a flow file.
> >
> > The SimpleDatabaseLookupService catches SQLException and throws
> > LookupFailureException which is the indicator to route to failure, and
> > it lets IOException be thrown so that callers can decide what to do.
> >
> > Typically IOException would be considered retryable so the current
> > behavior seems reasonable, but in this case the user wants to decide
> > not to retry which currently can't be done.
> >
> > Seems like two options...
> >
> > 1) Introduce a new property like "Communication Error Strategy" with
> > choices of "Rollback" (current) or "Route to Failure" (needed for this
> > case).
> >
> > 2) Introduce a new relationship like "Retry" and instead of throwing
> > ProcessException when catching IOException, instead route to Retry. It
> > is then up to the user to decide if they want to connect Retry back to
> > self to get the current behavior, auto-terminate it, or con

Re: Penalty feature of Processor (Disable)

[Edward Armes]

Hi Bilal,

Can you just confirm that your connection to Teradata is done using the
DatabaseRecordLookupService and is using one of the 2 DBCPConnectionPools
which is using the Teradata JDBC driver or are you using custom built
service?

The reason for asking is that I want to do a quick check to make sure by
were not masking an issue in one of the underlying services that hasn't
been caught correctly for some reason

Edward


On Tue, 26 Oct 2021, 10:29 Bilal Bektas,  wrote:

> Hi Community,
>
>
>
> Thank you for detailed solutions and analysis.
>
>
>
> @Dev Team, do you think to add a new feature (Rollback On Failure) for
> LookupAttribute processor like PutHiveQL processor?
>
>
>
> Thank you for helping,
>
>
>
> --Bilal
>
>
>
>
>
> *From:* Edward Armes 
> *Sent:* 26 Ekim 2021 Salı 01:32
> *To:* users@nifi.apache.org
> *Subject:* Re: Penalty feature of Processor (Disable)
>
>
>
> Having a quick look at the lookupAttribute code it looks like it takes a
> Optional<> from the call to the configured service. So I wonder if its
> worth adding the logic to service instead so that on erroring it can either
> return a missing value or throw an exception that would trigger the
> roleback. That would achieve the same goal without affecting other users of
> the LookupAttribute processor, where such logic isn't needed or wanted
> (e.g. SimpleLookupService).
>
>
>
> Edward
>
>
>
> On Mon, 25 Oct 2021, 21:54 Matt Burgess,  wrote:
>
> The approach in #1 is already present in a few Put processors like
> PutHive3QL, the property is named "Rollback on Failure" and takes a
> boolean value. The docs explain that if set to false, the flowfile is
> routed to failure, and if true will throw an exception and rollback
> the session. Check RollbackOnFailure.java for more details.
>
> Regards,
> Matt
>
> On Mon, Oct 25, 2021 at 4:46 PM Bryan Bende  wrote:
> >
> > The try/catch for IOException in LookupAttribute is after already
> > calling session.get(), so it is separate from loading a flow file.
> >
> > The SimpleDatabaseLookupService catches SQLException and throws
> > LookupFailureException which is the indicator to route to failure, and
> > it lets IOException be thrown so that callers can decide what to do.
> >
> > Typically IOException would be considered retryable so the current
> > behavior seems reasonable, but in this case the user wants to decide
> > not to retry which currently can't be done.
> >
> > Seems like two options...
> >
> > 1) Introduce a new property like "Communication Error Strategy" with
> > choices of "Rollback" (current) or "Route to Failure" (needed for this
> > case).
> >
> > 2) Introduce a new relationship like "Retry" and instead of throwing
> > ProcessException when catching IOException, instead route to Retry. It
> > is then up to the user to decide if they want to connect Retry back to
> > self to get the current behavior, auto-terminate it, or connect it to
> > the next processor like this case wants to do.
> >
> >
> > On Mon, Oct 25, 2021 at 4:01 PM Edward Armes 
> wrote:
> > >
> > > Hmm, it sounds like to me there might be 2 bugs here.
> > >
> > > One in the lookup attribute processor not isolating the loading of
> attributes from a FlowFile which may legitimately cause an IOException that
> would result in the FlowFile needing to be retired. The other in the
> TeradataDB lookup service not returning suitable errors that indicate if
> the issue is transient and a retry is needed or if it's a failure and
> should be routed to the failure queue.
> > >
> > > Edward
> > >
> > > On Mon, 25 Oct 2021, 16:50 Bryan Bende,  wrote:
> > >>
> > >> I'm not 100% sure on this, but I think the issue is that when
> LookupAttribute calls the LookupService, it catches IOException and throws
> a ProcessException, which rolls back the current session and puts the
> incoming flow files back in the preceding queue. The idea is that it would
> then retry the flow files until the comms issue is resolved, but in your
> case you don't want that.
> > >>
> > >> I think there would need to be an enhancement to LookupAttribute that
> adds a property to control the behavior on IOException so that the user can
> decide between rollback vs route to failure.
> > >>
> > >> On Mon, Oct 25, 2021 at 11:29 AM Etienne Jouvin <
> lapinoujou...@gmail.com> wrote:
> > >>>
> > >>> Hello all.
> > >>>
> > >>> You can d

Re: Penalty feature of Processor (Disable)

[Edward Armes]

Having a quick look at the lookupAttribute code it looks like it takes a
Optional<> from the call to the configured service. So I wonder if its
worth adding the logic to service instead so that on erroring it can either
return a missing value or throw an exception that would trigger the
roleback. That would achieve the same goal without affecting other users of
the LookupAttribute processor, where such logic isn't needed or wanted
(e.g. SimpleLookupService).

Edward

On Mon, 25 Oct 2021, 21:54 Matt Burgess,  wrote:

> The approach in #1 is already present in a few Put processors like
> PutHive3QL, the property is named "Rollback on Failure" and takes a
> boolean value. The docs explain that if set to false, the flowfile is
> routed to failure, and if true will throw an exception and rollback
> the session. Check RollbackOnFailure.java for more details.
>
> Regards,
> Matt
>
> On Mon, Oct 25, 2021 at 4:46 PM Bryan Bende  wrote:
> >
> > The try/catch for IOException in LookupAttribute is after already
> > calling session.get(), so it is separate from loading a flow file.
> >
> > The SimpleDatabaseLookupService catches SQLException and throws
> > LookupFailureException which is the indicator to route to failure, and
> > it lets IOException be thrown so that callers can decide what to do.
> >
> > Typically IOException would be considered retryable so the current
> > behavior seems reasonable, but in this case the user wants to decide
> > not to retry which currently can't be done.
> >
> > Seems like two options...
> >
> > 1) Introduce a new property like "Communication Error Strategy" with
> > choices of "Rollback" (current) or "Route to Failure" (needed for this
> > case).
> >
> > 2) Introduce a new relationship like "Retry" and instead of throwing
> > ProcessException when catching IOException, instead route to Retry. It
> > is then up to the user to decide if they want to connect Retry back to
> > self to get the current behavior, auto-terminate it, or connect it to
> > the next processor like this case wants to do.
> >
> >
> > On Mon, Oct 25, 2021 at 4:01 PM Edward Armes 
> wrote:
> > >
> > > Hmm, it sounds like to me there might be 2 bugs here.
> > >
> > > One in the lookup attribute processor not isolating the loading of
> attributes from a FlowFile which may legitimately cause an IOException that
> would result in the FlowFile needing to be retired. The other in the
> TeradataDB lookup service not returning suitable errors that indicate if
> the issue is transient and a retry is needed or if it's a failure and
> should be routed to the failure queue.
> > >
> > > Edward
> > >
> > > On Mon, 25 Oct 2021, 16:50 Bryan Bende,  wrote:
> > >>
> > >> I'm not 100% sure on this, but I think the issue is that when
> LookupAttribute calls the LookupService, it catches IOException and throws
> a ProcessException, which rolls back the current session and puts the
> incoming flow files back in the preceding queue. The idea is that it would
> then retry the flow files until the comms issue is resolved, but in your
> case you don't want that.
> > >>
> > >> I think there would need to be an enhancement to LookupAttribute that
> adds a property to control the behavior on IOException so that the user can
> decide between rollback vs route to failure.
> > >>
> > >> On Mon, Oct 25, 2021 at 11:29 AM Etienne Jouvin <
> lapinoujou...@gmail.com> wrote:
> > >>>
> > >>> Hello all.
> > >>>
> > >>> You can decrease the penalty value on the processor.
> > >>> Set to 0 for example.
> > >>>
> > >>>
> > >>>
> > >>> Le lun. 25 oct. 2021 à 16:22, Bilal Bektas 
> a écrit :
> > >>>>
> > >>>> Hi Community,
> > >>>>
> > >>>>
> > >>>>
> > >>>> We use LookupAttribute processor in order to get lookup value from
> Teradata or Oracle DB. Processors work as follows:
> > >>>>
> > >>>>
> > >>>>
> > >>>> LookupAttribute (Teradata)  ---(failure & unmatched) --->
> LookupAttribute (Oracle)
> > >>>>
> > >>>>
> > >>>>
> > >>>> This flows works well and LookupAttribute (Teradata) penalizes to
> flow files when Teradata DB is down. Therefore, the queue on upstream
> connection of LookupAttribute (Teradata) increases. But, we don

Re: Penalty feature of Processor (Disable)

[Edward Armes]

Hmm, it sounds like to me there might be 2 bugs here.

One in the lookup attribute processor not isolating the loading of
attributes from a FlowFile which may legitimately cause an IOException that
would result in the FlowFile needing to be retired. The other in the
TeradataDB lookup service not returning suitable errors that indicate if
the issue is transient and a retry is needed or if it's a failure and
should be routed to the failure queue.

Edward

On Mon, 25 Oct 2021, 16:50 Bryan Bende,  wrote:

> I'm not 100% sure on this, but I think the issue is that when
> LookupAttribute calls the LookupService, it catches IOException and throws
> a ProcessException, which rolls back the current session and puts the
> incoming flow files back in the preceding queue. The idea is that it would
> then retry the flow files until the comms issue is resolved, but in your
> case you don't want that.
>
> I think there would need to be an enhancement to LookupAttribute that adds
> a property to control the behavior on IOException so that the user can
> decide between rollback vs route to failure.
>
> On Mon, Oct 25, 2021 at 11:29 AM Etienne Jouvin 
> wrote:
>
>> Hello all.
>>
>> You can decrease the penalty value on the processor.
>> Set to 0 for example.
>>
>>
>>
>> Le lun. 25 oct. 2021 à 16:22, Bilal Bektas  a
>> écrit :
>>
>>> Hi Community,
>>>
>>>
>>>
>>> We use LookupAttribute processor in order to get lookup value from
>>> Teradata or Oracle DB. Processors work as follows:
>>>
>>>
>>>
>>> LookupAttribute (Teradata)  ---(failure & unmatched) --->
>>> LookupAttribute (Oracle)
>>>
>>>
>>>
>>> This flows works well and LookupAttribute (Teradata) penalizes to flow
>>> files when Teradata DB is down. Therefore, the queue on upstream connection
>>> of LookupAttribute (Teradata) increases. But, we don't want to that
>>> LookupAttribute (Teradata) penalizes to flow files. We want to that
>>> LookupAttribute (Teradata) processor forwards flow files to failure
>>> downstream connection when all failure situation on LookupAttribute
>>> (Teradata). Thus, LookupAttribute (Oracle) can process flow files which
>>> cannot process on LookupAttribute (Teradata).
>>>
>>>
>>>
>>> Is it possible to disable penalty feature of processor or is there any
>>> solution which you can suggest for this situation.
>>>
>>>
>>>
>>> Thank you in advance,
>>>
>>>
>>>
>>> --Bilal
>>>
>>> obase
>>> TEL: +90216 527 30 00
>>> FAX: +90216 527 31 11
>>>  
>>>  
>>> 
>>>
>>> Bu elektronik posta ve onunla iletilen bütün dosyalar sadece göndericisi
>>> tarafindan almasi amaclanan yetkili gercek ya da tüzel kisinin kullanimi
>>> icindir. Eger söz konusu yetkili alici degilseniz bu elektronik postanin
>>> icerigini aciklamaniz, kopyalamaniz, yönlendirmeniz ve kullanmaniz
>>> kesinlikle yasaktir ve bu elektronik postayi derhal silmeniz gerekmektedir.
>>> OBASE bu mesajin icerdigi bilgilerin doğruluğu veya eksiksiz oldugu
>>> konusunda herhangi bir garanti vermemektedir. Bu nedenle bu bilgilerin ne
>>> sekilde olursa olsun iceriginden, iletilmesinden, alinmasindan ve
>>> saklanmasindan sorumlu degildir. Bu mesajdaki görüsler yalnizca gönderen
>>> kisiye aittir ve OBASE görüslerini yansitmayabilir.
>>>
>>> Bu e-posta bilinen bütün bilgisayar virüslerine karsi taranmistir.
>>>
>>> This e-mail and any files transmitted with it are confidential and
>>> intended solely for the use of the individual or entity to whom they are
>>> addressed. If you are not the intended recipient you are hereby notified
>>> that any dissemination, forwarding, copying or use of any of the
>>> information is strictly prohibited, and the e-mail should immediately be
>>> deleted. OBASE makes no warranty as to the accuracy or completeness of any
>>> information contained in this message and hereby excludes any liability of
>>> any kind for the information contained therein or for the information
>>> transmission, recepxion, storage or use of such in any way whatsoever. The
>>> opinions expressed in this message belong to sender alone and may not
>>> necessarily reflect the opinions of OBASE.
>>>
>>> This e-mail has been scanned for all known computer viruses.
>>>
>>

Re: Scale NiFi cluster for more new data Flows

[Edward Armes]

Hi Thắng,

It feels like to me you may need more nifi nodes in your cluster, as sounds
like he current load is not distributed enough across the cluster. Would
you be able share a few additional pieces of information to help the
community help you? Specificly what version of Nifi you are running, what
version of Java your using this will help us give you more specific advice.

In general a few things to think about that might help you improve the
performance of you Nifi cluster:
- Look at adjusting the scheduling of your less active processors so that
they are concidered for running less often by the framework.
- Look at setting up a reporting task and using that to capture the metrics
from your cluster to external system like Prometheus to give you an idea of
what processors are not performing as expected and where your bottlenecks
are.
- If you are using custom processors check that they are not doing anything
in the background that may result in additional memory consumption.
- Try to reduce the ammount of information you keep in the flow-files as
these are normally kept in memory, where as the content is always kept on
disk (except be used in a processor)
- Investigate can be done with things like queue limits and back preasure
to reduce the ammount of WIP in the cluster.

Hope that helps

Edward

On Tue, 12 Oct 2021, 10:07 Thắng Nguyễn Đình,  wrote:

> Hi everyone,
>
> I have a question about the way a NiFi cluster is scaled when we have more
> new data Flows.
>
> We are having a NiFi cluster with 3 nodes running on production for about
> 2 years.
> Currently, there are 3000 processors running. The UI interaction will slow
> down when more processors are added and more JVM memory required.
>
> If we add more nodes to the cluster, the UI interaction will be slow as
> the replication of Flow configuration and the required JVM memory will not
> decrease as the number of processors remains the same.
>
> Should we set up a separate cluster for new data Flows?
> Could you please give me a suggestion? Thank you so much!
>
> *Our server specs:*
> Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz
> CPU(s):24
> Mem: 62 gb (total), 24 gb (used), 7 gb(free), ... cached
>
> *NiFi instance specs:*
> # Secure setup
> # JVM memory settings
> java.arg.2=-Xms24g
> java.arg.3=-Xmx24g
>
> # Repos
> 50 % of 1.7 Tb HDD disk
> # Network: 10Gb/s bandwidth
>
> Thanks & Regards
>
> Thắng, Nguyễn Đình
>

Re: Environment variables

[Edward Armes]

Hi Dima,

 In that case, I would recommend just deploying the pre-built docker image
in your dev environment instead of running multiple instances together.

I would also suggest as alternative one of 2 things, people experiment
locally on their own instances as Nifi is distributed as an archive it can
just be run locally or have one instance on a server and ask people to only
work in a processes group and not off the root canvas.

Edward

On Fri, 10 Sep 2021, 17:22 Dmitry Stepanov,  wrote:

> Hi Edward,
>
> You're right - we will be using containers for production. I'm just
> testing things out on a sandbox or dev env we already have.
>
> Thank you,
>
> On Fri, Sep 10, 2021 at 12:04 PM Edward Armes 
> wrote:
>
>> Hi Dima,
>>
>> Ideally you should not run anything as the root user as it tends to cause
>> more issues then it solves in the long term.
>>
>> Secondly I would recommend against running more than one Nifi instance
>> concurrently on a host without some sort of isolation like a container or a
>> jail.
>>
>>  If you're looking to create a multi tenancy Nifi environment then that
>> is something that this list should be able to provide guidance on, without
>> having to run multiple Nifi instances per host.
>>
>> Edward
>>
>> On Fri, 10 Sep 2021, 14:47 Dmitry Stepanov,  wrote:
>>
>>> Hi Community,
>>> Question on the env variables - I plan on using multiple service
>>> instances of NiFi running on same server under same user (root).  I'm
>>> mostly concerned about env variables from "nifi-1.14.0/bin/nifi-env.sh"
>>>
>>> Thank you for your help,
>>>
>>> Dima Stepanov
>>>
>>

Re: Environment variables

[Edward Armes]

Hi Dima,

Ideally you should not run anything as the root user as it tends to cause
more issues then it solves in the long term.

Secondly I would recommend against running more than one Nifi instance
concurrently on a host without some sort of isolation like a container or a
jail.

 If you're looking to create a multi tenancy Nifi environment then that is
something that this list should be able to provide guidance on, without
having to run multiple Nifi instances per host.

Edward

On Fri, 10 Sep 2021, 14:47 Dmitry Stepanov,  wrote:

> Hi Community,
> Question on the env variables - I plan on using multiple service instances
> of NiFi running on same server under same user (root).  I'm mostly
> concerned about env variables from "nifi-1.14.0/bin/nifi-env.sh"
>
> Thank you for your help,
>
> Dima Stepanov
>

Re: PutHDFS Datanode communication issue

[Edward Armes]

Hi,

Looking at the error I would guess that for some reason the PutHDFS
processor isn't able to resolve the data node in HDFS.

Do you have any additional infornation around HDFS in your Nifi app log or
any information in the HDFS logs?

Otherwise I would suggest lowering lowering the log level for the HDFS
processor which can be configured in conf/logback.xml. That may give you
some more information.

Hope that helps

Edward


On Wed, 4 Nov 2020, 09:07 varuntango,  wrote:

> Hi,
>
> Im trying to put data into hdfs filesystem, but i didnt work properly. I
> have configured both 50010, 8020 ports telnet everything working from nifi
> server but still i cant be able to resolve this issue, Please anyone help
> me
> out from this issue, it creates file in hdfs location but it didnt write
> the
> content. Here is complete stacktrace;
>
> 2020-11-04 08:58:29,915 INFO [Flow Service Tasks Thread-2]
> o.a.nifi.controller.StandardFlowService Saved flow controller
> org.apache.nifi.controller.FlowController@1bc8715e // Another save
> pending =
> false
> 2020-11-04 08:58:33,594 INFO [NiFi Web Server-338]
> o.a.n.c.queue.AbstractFlowFileQueue Initiating drop of FlowFiles from
> FlowFileQueue[id=8e56b08a-0175-1000-32c1-44e0a0a2eb7c] on behalf of
> anonymous (request identifier=927aa8fa-0175-1000-0956-7f6ab2993387)
> 2020-11-04 08:58:33,596 INFO [Drop FlowFiles for Connection
> 8e56b08a-0175-1000-32c1-44e0a0a2eb7c] o.a.n.c.queue.SwappablePriorityQueue
> Successfully dropped 1 FlowFiles (11851 bytes) from Connection with ID
> 8e56b08a-0175-1000-32c1-44e0a0a2eb7c on behalf of anonymous
> 2020-11-04 08:58:38,507 INFO [NiFi Web Server-340]
> o.a.n.c.s.StandardProcessScheduler Starting
> PutHDFS[id=8e0a7636-0175-1000-810b-e0cb6cb164e0]
> 2020-11-04 08:58:38,528 INFO [Timer-Driven Process Thread-10]
> o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled
> PutHDFS[id=8e0a7636-0175-1000-810b-e0cb6cb164e0] to run with 1 threads
> 2020-11-04 08:58:38,564 WARN [Thread-416]
> org.apache.hadoop.hdfs.DataStreamer DataStreamer Exception
> java.nio.channels.UnresolvedAddressException: null
> at sun.nio.ch.Net.checkAddress(Net.java:104)
> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:621)
> at
> org.apache.hadoop.net
> .SocketIOWithTimeout.connect(SocketIOWithTimeout.java:192)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
> at
>
> org.apache.hadoop.hdfs.DataStreamer.createSocketForPipeline(DataStreamer.java:253)
> at
>
> org.apache.hadoop.hdfs.DataStreamer.createBlockOutputStream(DataStreamer.java:1725)
> at
>
> org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1679)
> at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:716)
> 2020-11-04 08:58:39,492 INFO [Flow Service Tasks Thread-1]
> o.a.nifi.controller.StandardFlowService Saved flow controller
> org.apache.nifi.controller.FlowController@1bc8715e // Another save
> pending =
> false
>
>
>
>
> --
> Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/
>

Re: Combine Attributes & Content

[Edward Armes]

Hi Sharma,

An alternative could be, duplicate the FlowFile from the HandleHTTPRequest
Processor have one wait while the other goes through an AttributesToJSON
and stores the JSON in a lookup cache. The other FlowFile then grabs that
json from the lookup cache and then use ReplaceText processor mode to add
that specific attribute into the content.

Edward

On Thu, May 21, 2020 at 10:11 AM Dweep Sharma 
wrote:

> Thanks Matt, UpdateRecord works however, all headers must be added
> manually (unlike AttributesToJson). Since it is one time effort, it will do
> for now.
>
> -Dweep
>
> On Tue, May 19, 2020 at 9:16 PM Matt Burgess  wrote:
>
>> Dweep,
>>
>> Depending on how complex the content JSON is, you might be able to use
>> ReplaceText to smuggle the attributes into the text, but this can be
>> tricky as you need to match on the opening JSON and the rest, and then
>> replace it with the opening JSON, the attributes, then the rest in
>> order to preserve valid JSON.
>>
>> You may also be able to use UpdateRecord to add fields to your
>> content, I believe if the Replacement Value Strategy property is set
>> to "Literal Value" you can use Expression Language which may be able
>> access flow file attributes.
>>
>> An alternative is to use a scripted processor if you're comfortable
>> with a scripting language such as Groovy, as it can "slurp" JSON into
>> a Plain Old Groovy Object (POGO), then you can add fields and
>> serialize back out to JSON. I have a somewhat similar example here
>> [1].
>>
>> Regards,
>> Matt
>>
>> [1]
>> http://funnifi.blogspot.com/2016/02/executescript-json-to-json-conversion.html
>>
>> On Tue, May 19, 2020 at 10:46 AM Dweep Sharma 
>> wrote:
>> >
>> > Hi All,
>> >
>> > Does anyone have a strategy to combine Attributes & Content into a
>> single Flowfile?
>> >
>> > AttributesToJson seems like the best way convert all attributes to
>> content -Json but I
>> >
>> > lose the original content.
>> >
>> > Basically, I have a HandleHTTPRequest Processor and headers and body
>> must be combined,
>> >
>> > currently Headers are stored as Attributes in a flowfile
>> >
>> > -Dweep
>> >
>> >
>> > ::DISCLAIMER::
>> >
>> 
>> >
>> > The contents of this e-mail and any attachments are confidential and
>> intended for the named recipient(s) only.E-mail transmission is not
>> guaranteed to be secure or error-free as information could be intercepted,
>> corrupted,lost, destroyed, arrive late or incomplete, or may contain
>> viruses in transmission. The e mail and its contents(with or without
>> referred errors) shall therefore not attach any liability on the originator
>> or redBus.com. Views or opinions, if any, presented in this email are
>> solely those of the author and may not necessarily reflect the views or
>> opinions of redBus.com. Any form of reproduction, dissemination, copying,
>> disclosure, modification,distribution and / or publication of this message
>> without the prior written consent of authorized representative of
>> redbus.com is strictly prohibited. If you have received this email in
>> error please delete it and notify the sender immediately.Before opening any
>> email and/or attachments, please check them for viruses and other defects.
>>
>
>
>
>
> *::DISCLAIMER::The
> contents of this e-mail and any attachments are confidential and intended
> for the named recipient(s) only.E-mail transmission is not guaranteed to be
> secure or error-free as information could be intercepted, corrupted,lost,
> destroyed, arrive late or incomplete, or may contain viruses in
> transmission. The e mail and its contents(with or without referred errors)
> shall therefore not attach any liability on the originator or redBus.com.
> Views or opinions, if any, presented in this email are solely those of the
> author and may not necessarily reflect the views or opinions of redBus.com.
> Any form of reproduction, dissemination, copying, disclosure,
> modification,distribution and / or publication of this message without the
> prior written consent of authorized representative of redbus.
> com is strictly prohibited. If you have received this
> email in error please delete it and notify the sender immediately.Before
> opening any email and/or attachments, please check them for viruses and
> other defects.*

Re: Nifi - how to achieve a concurrent development and CI/CD

[Edward Armes]

Hi Eric,

So I looked into this September last year. I found that a variant of master
driven development was the best course of action, for collaboration on a
single Nifi flow.

As far as I can tell the git integration deliberately designed to be as
simple a s possible, the down side of this is that git functionality like
branching just doesn't work with the git integration.

I briefly looked into seeing if I could sidestep a few of the issues by
changing how the flow version information was stored, but I ran into
several road blocks around adding support for history re-writes...


Edward


On Fri, 15 May 2020, 01:17 Eric Secules,  wrote:

> Hi Ami,
>
> I'm testing our NiFi application using a blackbox approach. We have a test
> harness that stages input files and waits for the corresponding outputs,
> then validates the content of the output. NiFi runs in a docker container
> (see https://hub.docker.com/r/apache/nifi/) and you can automate starting
> it up and orchestrating a test harness for your application in any
> competent CI/CD pipeline.
>
> Jorge, I don't understand your comment. NiFi registry can be synced to a
> git repo as a backup mechanism, but you can't simply make branches, work
> and merge them back, because even the smallest change in a flow will create
> a massive diff in the Registry's backup git repo. It would be great if NiFi
> had a way of highlighting the differences between flow versions on the
> canvas and allow you to choose which elements to keep and which to discard
> when resolving a merge conflict.
>
> -Eric
>
> On Thu, May 14, 2020 at 12:00 PM Jorge Machado  wrote:
>
>> I think we could improve Nifi by hooking up to GitHub somehow. So that we
>> don’t need the registry…
>>
>> On 14. May 2020, at 18:49, Ami Goldenberg  wrote:
>>
>> Hi Eric,
>> Would love to know, what kind of tests do you write/run for NiFi? We were
>> just researching this topic.
>>
>> On Thu, May 14, 2020 at 6:38 PM Eric Secules  wrote:
>>
>>> Hi Michal,
>>>
>>> I'm also using a single registry for development and production. It
>>> doesn't help with collaborating on the same process group as there is way
>>> for it to reconcile merge conflicts. Instead, the registry will earn you
>>> that you're about to overwrite someone else's changes. Another pain of
>>> concurrent development is there's no concept of a PR and no visual diff of
>>> your local changes making review difficult. I've backed our registry up to
>>> git I've set up a CI pipeline in Azure Devops which runs our tests every
>>> time a new version of a process group is checked in. It's better than
>>> nothing but I'd rather nifi implemented git flow. Developing as a team on
>>> nifi and nifi registry is like your whole team developing and pushing
>>> directly to master.
>>>
>>>
>>> -Eric
>>>
>>> On Thu., May 14, 2020, 7:02 a.m. Jorge Machado,  wrote:
>>>
 Hi,

 Managing xml is always hard I think. Last time I need to do something
 similar we used https://nifi.apache.org/registry.html
 Works pretty well It was already 2 Years ago. Maybe now there is
 something better

 On 14. May 2020, at 15:57, Michal Slama (DHL IT Services), external <
 michal.sl...@dhl.com> wrote:

 Hello,

 may I ask you for recommendations for development and CI/CD in NiFi?
 Pls let me describe our situation…I am a developer from DHL currently
 working on project including NiFi. It is part of our core and it is
 responsible for handling incoming data streams, data transormation and
 put it into various elastic search indexes (and queues).

 Up to now development was quite straightforward as only one developer
 did it. But we have extended our team recently and now we face problems how
 to correctly maintain development for more developers working in
 parallel and then CI/CD of it as we have classical dev/test/uat/project
  env. structure.

 Pls do you have any recommadation how to achive it? For now in general
 is enought.  Its good to mention that currently we work with NiFi version
  1.8…tried to upgrade to 1.9. but some of components failed so the
 upgrade was postponed. But with new features in 1.10. and 1.11. we head
 to uprade to these versions.

 Maybe if we can arrange a call it would be great!

 With regards,
 Michal Sláma



 *This message is from DHL Information Services (Europe) s.r.o. and may
 contain confidential business information. It is intended solely for the
 use of the individual to whom it is addressed. If you are not the intended
 recipient please contact the sender and delete this message and any
 attachment from your system. Unauthorized publication, use, dissemination,
 forwarding, printing or copying of this email and its attachments is
 strictly prohibited.*



>>

Re: OIDC Redirect loop

[Edward Armes]

Hi Ami,

Biased on the error you've got in the user log it looks like you've got 
a local trust issue. If you could tell us what you've already tried, 
someone might be able to help you a bit more.


Edward

On 27/04/2020 05:36, Ami Goldenberg wrote:

Hi,

We are trying to deploy NiFi on kubernetes after successfully using it 
for a while.
The issue we are having is that every time we enter our nifi URL it 
will redirect us to Google and once we sign in we just get 
redirected again.


_The error I see on users.log is:_
o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (token>) GET https://XXX.XXX./nifi-api/flow/current-user (source 
ip: 172.32.34.99)
2020-04-25T19:48:06.256605759Z 2020-04-25 19:48:05,983 ERROR [NiFi 
Web Server-16] o.a.nifi.web.security.jwt.JwtService There was an error 
validating the JWT
2020-04-25T19:48:06.256610178Z 2020-04-25 19:48:05,983 ERROR [NiFi Web 
Server-16] o.a.nifi.web.security.jwt.JwtService Unable to validate 
the access token.
2020-04-25T19:48:06.256613727Z Caused by: JWT signature does not 
match locally computed signature. JWT validity cannot be asserted and 
should not be trusted.
2020-04-25T19:48:06.256617124Z 2020-04-25 19:48:05,984 WARN [NiFi Web 
Server-16] o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web 
api:Unable to validate the access token.


_We're trying to follow practices from blogs and pvillard's repo:_

  * 
https://github.com/pvillard31/nifi-gcp-terraform/tree/master/gcp-cluster-secured-nifi-oidc
  * https://bryanbende.com/development/2017/10/03/apache-nifi-openid-connect
  * https://medium.com/swlh/operationalising-nifi-on-kubernetes-1a8e0ae16a6c

_Our set up is as such:_

  * OIDC provider is Google
  * TLS-toolkit running in server mode inside k8s
  * StatefulSet of 3 replicas
  * Zookeeper in K8s
  * Ingress that is set up to create a load balancer in AWS - with
sticky sessions (based on cookie)
  * Service that is set up with sessionAffinity: ClientIP


Any idea which direction I should be checking next?anks!

Re: Apache NiFi 1.9.2 InferAvroSchema on csv file header with :

[Edward Armes]

Hi Jouvin,

I believe you are correct that the inferAvroSchema and the convert record
processor do work differently. I believe this is because the
inferAvroSchema uses Apache Kite and the convert record derives the schema
from the record reader itself.

As an aside I have also noticed that when you use a validateRecord with a
different types of reader and writer record handlers (i.e. json in avro
out). You get different results l, while I'm not surprised by this I think
it's worth just flagging up, for future reference.

Edward

On Wed, 11 Mar 2020, 09:35 Etienne Jouvin,  wrote:

> Hello all.
>
> Just in case someone "can test".
>
> I have NiFi 1.9.2 and need to convert CSV to JSON. I do not planned to
> upgrade for now (because of deployment procedure)
> In the CSV, I have a column with value like prop:Name
>
> i set true for the property Get CSV Header Definition From Data
>
> The processor failed because of the name.
>
> But if I use a convertRecord with a CSV Reader, that infer schema, and a
> JSON writer, this is working fine.
>
> Not the same algorithm to get infer schema from InferAvroSchema and the
> reader ?
>
> Regards
>
> Etienne Jouvin
>
>
>
> 
>  Garanti
> sans virus. www.avast.com
> 
> <#m_1566800260412186955_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>

Re: Listing a folder with millions of files

[Edward Armes]

Hi Jeremy,

In this case I don't think there is an easy answer here.

You may have some luck with adjusting the max runtime of the processor but
without checking the the processors implementation I couldn't know for
certain if that would have any effect.

Edward

On Mon, 9 Mar 2020, 06:34 Jeremy Pemberton-Pigott, 
wrote:

> Hi,
>
> I need to list a sub-set (few 100,000) of files in a folder with millions
> of files (to do some historical processing).  What's the best way I can do
> that?  ListFiles is taking way too long and seems to try to dump the entire
> list to the flow when I test it on a smaller folder list.  It would be good
> if the listing emitted files in smaller chunks so that the flow can start
> working on them.
>
> Regards,
>
> Jeremy
>

Re: How to deploy NiFi processors change to multiple NiFi instances?

[Edward Armes]

Hi Lei,

As far as I'm aware there currently isn't an out of the box method to do
this
 However I would look at using a combination of the nifi registry and tools
in nifi toolkit to roll your own.

Also, there have been multiple questions to both this and the dev mailing
list, asking what could be done. I would also suggest searching the list
archives and see what others did and didn't do as well.

Edward

On Mon, 28 Oct 2019, 08:14 wangl...@geekplus.com.cn, <
wangl...@geekplus.com.cn> wrote:

>
> We have many standalone NiFi instances and all running the same  NiFi
> Flow.
> If the flow changes,  how to deploy the change to all NiFi instances
> automatically?
>
> Thanks,
> Lei
>
>
> --
> wangl...@geekplus.com.cn
>

Re: My nifi no more serve admin interface

[Edward Armes]

Hmm, I wonder if there's a change that could be made to expose this error
so its a bit more obvious, maybe one for the Dev mailing list?

Edward

On Wed, Aug 14, 2019 at 3:12 PM Pierre Villard 
wrote:

> Glad you sorted it out and thanks for letting us know!
> In case you missed it, you might be interested by the NiFi toolkit [1]
> containing a TLS toolkit to help you with certificates [2].
>
> [1] https://nifi.apache.org/download.html
> [2]
> https://nifi.apache.org/docs/nifi-docs/html/toolkit-guide.html#tls_toolkit
>
> Le mer. 14 août 2019 à 15:54, Nicolas Delsaux  a
> écrit :
>
>> Oh damn
>>
>> It appeared (after a long search) that my keystore was incorrectly built.
>>
>> Indeed, it contained the server certificate as a trusted certificate,
>> where it should had been a key pair (with both private and public keys in)
>> as is explained in Jetty documentation (
>> https://www.eclipse.org/jetty/documentation/9.4.19.v20190610/configuring-ssl.html#understanding-certificates-and-keys
>> - see part Layout of keystore and truststore). And this happened because
>> I'm really bad at certificates.
>>
>> Sorry to have consumed some of your time, you all.
>> Le 13/08/2019 à 16:21, Nicolas Delsaux a écrit :
>>
>> oh, sorry, I forgot to mention i use the nifi docker image, with
>> configuration
>> services:
>> nifi-runner:
>> hostname: nifi-psh.adeo.com
>> image: apache/nifi:1.9.2
>> ports:
>> - "38080:8443"
>> - "5000:8000"
>> volumes:
>> -
>> ${project.basedir}/target/docker-compose/includes/nifi/node/conf:/opt/nifi/nifi-current/conf
>> -
>> ${project.basedir}/target/docker-compose/includes/nifi/node/cacerts.jks:/opt/certs/cacerts.jks
>> -
>> ${project.basedir}/target/docker-compose/includes/nifi/node/https_certificates.pkcs:/opt/certs/https_certificates.pkcs
>>
>> And port 8443 is standard http port, I guess (the port 8000 is the
>> standard debug one)
>>
>>
>> Le 13/08/2019 à 16:10, Pierre Villard a écrit :
>>
>> Might be a dumb question but I'm wondering why you're trying with port
>> 38080? Did you change the configuration to use that specific port with a
>> secured instance?
>>
>> Pierre
>>
>> Le mar. 13 août 2019 à 16:00, Nicolas Delsaux  a
>> écrit :
>>
>>> To go a little further, a test with openssl s_client gives the following
>>>
>>> nicolas-delsaux@NICOLASDELSAUX C:\Users\nicolas-delsaux
>>> $ openssl s_client -host localhost -port 38080
>>> CONNECTED(0164)
>>> 416:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake
>>> failure:ssl\record\rec_layer_s3.c:1399:SSL alert number 40
>>> ---
>>> no peer certificate available
>>> ---
>>> No client certificate CA names sent
>>> ---
>>> SSL handshake has read 7 bytes and written 176 bytes
>>> Verification: OK
>>> ---
>>> New, (NONE), Cipher is (NONE)
>>> Secure Renegotiation IS NOT supported
>>> Compression: NONE
>>> Expansion: NONE
>>> No ALPN negotiated
>>> SSL-Session:
>>>  Protocol  : TLSv1.2
>>>  Cipher: 
>>>  Session-ID:
>>>  Session-ID-ctx:
>>>  Master-Key:
>>>  PSK identity: None
>>>  PSK identity hint: None
>>>  SRP username: None
>>>  Start Time: 1565704262
>>>  Timeout   : 7200 (sec)
>>>  Verify return code: 0 (ok)
>>>  Extended master secret: no
>>> ---
>>>
>>>
>>> Which i weird considering nifi outputs in its startup log the lines
>>>
>>> nifi-runner_1  | 2019-08-13 13:37:52,315 INFO [main]
>>> o.e.jetty.server.handler.ContextHandler Started
>>> o.e.j.w.WebAppContext@7cb81ae{nifi-error,/,
>>> file:///opt/nifi/nifi-current/work/jetty/nifi-web-error-1.9.2.war/webapp/,AVAILABLE
>>> }{./work/nar/framework/nifi-framework-nar-1.9.2.nar-unpacked/NAR-INF/bundled-dependencies/nifi-web-error-1.9.2.war}
>>> nifi-runner_1  | 2019-08-13 13:37:52,490 INFO [main]
>>> o.e.jetty.util.ssl.SslContextFactory
>>> x509=X509@3d94d7f3(nifi-psh.adeo.com (adeo
>>> ca),h=[nifi-psh.adeo.com],w=[]) for
>>> SslContextFactory@da1abd6[provider=null,keyStore=
>>> file:///opt/certs/https_certificates.pkcs,trustStore=file:///opt/certs/cacerts.jks
>>> ]
>>> nifi-runner_1  | 2019-08-13 13:37:52,510 INFO [main]
>>> o.eclipse.jetty.server.AbstractConnector Started
>>> ServerConnector@2066f0d3{SSL,[ssl, http/1.1]}{0.0.0.0:8443}
>>>
>>>
>>> which seems to indicate Jetty is able to listen for https connections on
>>> port 8443 using certificates described in SslContextFactory. No ?
>>>
>>> Le 13/08/2019 à 15:40, Nicolas Delsaux a écrit :
>>> > I'm currently trying to implement ldap user group authorization in
>>> nifi.
>>> >
>>> > For that, I've deployed nifi docker image with configuration files
>>> > containing required config elements (a ldap identity provider, a ldap
>>> > user group provider).
>>> >
>>> > I've also configured https with a keystore/truststore that are injected
>>> > into docker container through volumes.
>>> >
>>> > Once all is configured, i've taken the time to do some debug session to
>>> > make sure tue FileAccessPolicyProvider correctly loads my user from
>>> > ldap, and it works ok.
>>> >

Re: My nifi no more serve admin interface

[Edward Armes]

Hi Nicolas,

This is another dump question. As I've only ever seen this before when I've
accidentally connect to a secured Nifi cluster over HTTP and not HTTPS.
>From I've seen Nifi won't ask your browser to do a connection upgrade (HTTP
-> HTTPS),

When you type in the address are you sure your browser is not cutting of
the *s* from the *https* in the URL you entering?

Edward

On Tue, Aug 13, 2019 at 3:22 PM Nicolas Delsaux 
wrote:

> oh, sorry, I forgot to mention i use the nifi docker image, with
> configuration
> services:
> nifi-runner:
> hostname: nifi-psh.adeo.com
> image: apache/nifi:1.9.2
> ports:
> - "38080:8443"
> - "5000:8000"
> volumes:
> -
> ${project.basedir}/target/docker-compose/includes/nifi/node/conf:/opt/nifi/nifi-current/conf
> -
> ${project.basedir}/target/docker-compose/includes/nifi/node/cacerts.jks:/opt/certs/cacerts.jks
> -
> ${project.basedir}/target/docker-compose/includes/nifi/node/https_certificates.pkcs:/opt/certs/https_certificates.pkcs
>
> And port 8443 is standard http port, I guess (the port 8000 is the
> standard debug one)
>
>
> Le 13/08/2019 à 16:10, Pierre Villard a écrit :
>
> Might be a dumb question but I'm wondering why you're trying with port
> 38080? Did you change the configuration to use that specific port with a
> secured instance?
>
> Pierre
>
> Le mar. 13 août 2019 à 16:00, Nicolas Delsaux  a
> écrit :
>
>> To go a little further, a test with openssl s_client gives the following
>>
>> nicolas-delsaux@NICOLASDELSAUX C:\Users\nicolas-delsaux
>> $ openssl s_client -host localhost -port 38080
>> CONNECTED(0164)
>> 416:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake
>> failure:ssl\record\rec_layer_s3.c:1399:SSL alert number 40
>> ---
>> no peer certificate available
>> ---
>> No client certificate CA names sent
>> ---
>> SSL handshake has read 7 bytes and written 176 bytes
>> Verification: OK
>> ---
>> New, (NONE), Cipher is (NONE)
>> Secure Renegotiation IS NOT supported
>> Compression: NONE
>> Expansion: NONE
>> No ALPN negotiated
>> SSL-Session:
>>  Protocol  : TLSv1.2
>>  Cipher: 
>>  Session-ID:
>>  Session-ID-ctx:
>>  Master-Key:
>>  PSK identity: None
>>  PSK identity hint: None
>>  SRP username: None
>>  Start Time: 1565704262
>>  Timeout   : 7200 (sec)
>>  Verify return code: 0 (ok)
>>  Extended master secret: no
>> ---
>>
>>
>> Which i weird considering nifi outputs in its startup log the lines
>>
>> nifi-runner_1  | 2019-08-13 13:37:52,315 INFO [main]
>> o.e.jetty.server.handler.ContextHandler Started
>> o.e.j.w.WebAppContext@7cb81ae{nifi-error,/,
>> file:///opt/nifi/nifi-current/work/jetty/nifi-web-error-1.9.2.war/webapp/,AVAILABLE
>> }{./work/nar/framework/nifi-framework-nar-1.9.2.nar-unpacked/NAR-INF/bundled-dependencies/nifi-web-error-1.9.2.war}
>> nifi-runner_1  | 2019-08-13 13:37:52,490 INFO [main]
>> o.e.jetty.util.ssl.SslContextFactory
>> x509=X509@3d94d7f3(nifi-psh.adeo.com (adeo
>> ca),h=[nifi-psh.adeo.com],w=[]) for
>> SslContextFactory@da1abd6[provider=null,keyStore=
>> file:///opt/certs/https_certificates.pkcs,trustStore=file:///opt/certs/cacerts.jks
>> ]
>> nifi-runner_1  | 2019-08-13 13:37:52,510 INFO [main]
>> o.eclipse.jetty.server.AbstractConnector Started
>> ServerConnector@2066f0d3{SSL,[ssl, http/1.1]}{0.0.0.0:8443}
>>
>>
>> which seems to indicate Jetty is able to listen for https connections on
>> port 8443 using certificates described in SslContextFactory. No ?
>>
>> Le 13/08/2019 à 15:40, Nicolas Delsaux a écrit :
>> > I'm currently trying to implement ldap user group authorization in nifi.
>> >
>> > For that, I've deployed nifi docker image with configuration files
>> > containing required config elements (a ldap identity provider, a ldap
>> > user group provider).
>> >
>> > I've also configured https with a keystore/truststore that are injected
>> > into docker container through volumes.
>> >
>> > Once all is configured, i've taken the time to do some debug session to
>> > make sure tue FileAccessPolicyProvider correctly loads my user from
>> > ldap, and it works ok.
>> >
>> > Unfortunatly, now, when i try to load Nifi admin interface, I get a
>> > strange http response containing only the string "   �  P".
>> >
>> > In other words,
>> >
>> >
>> > nicolas-delsaux@NICOLASDELSAUX C:\Users\nicolas-delsaux
>> > $ curl -v -H "Host: nifi-psh.adeo.com" http://localhost:38080/
>> --output -
>> > *   Trying ::1...
>> > * TCP_NODELAY set
>> > * Connected to localhost (::1) port 38080 (#0)
>> > > GET / HTTP/1.1
>> > > Host: nifi-psh.adeo.com
>> > > User-Agent: curl/7.55.1
>> > > Accept: */*
>> > >
>> > §♥♥ ☻☻P* Connection #0 to host localhost left intact
>> >
>> >
>> > http does not work (which i expects, since I've configured
>> > authentication/authorization
>> >
>> > nicolas-delsaux@NICOLASDELSAUX C:\Users\nicolas-delsaux
>> > $ curl -v -H "Host: nifi-psh.adeo.com" https://localhost:38080/
>> > --output -
>> > *   Trying ::1...
>> > * TCP_NODELAY set
>> > 

Re: Anti-Virus Scanning

[Edward Armes]

Hi Jason,

This isn't explicitly documented anywhere, however the locations of all the
key paths for Nifi can be found in the documentation in general. Hopefully
a combination of this email thread and official documentation should be
enough for your client to give the AV exemptions you need.

Getting information added to the documentation is simple in one aspect as
ticket just has to be raised on the ASF Nifi JIRA. However, given that
large elements of Nifi are configurable via plugins it would be difficult
to give definitive advise, as different plugins will have different AV
requirements.

In the meantime I have created NIFI-6553
 to look at improving the
documentation around this as we are missing other bits of information like
SELinux configuration that I think would also be useful to have in the
documentation

Edward

On Tue, Aug 13, 2019 at 7:31 PM Jason Csencsits <
jcsencs...@technicallycreative.com> wrote:

> Joe,
>
> Thank you for the information.  Is this documented anywhere as I have a
> client looking for it from Apache?.
>
>
>
> Thank you,
>
> :::
>
> *Jason Csencsits *
>
> Manager of Technical Operations
>
> Technically Creative Inc.
>
> *Simplifying IT Solutions*
>
>
>
> Office: 845.725.7883
>
> jcsencs...@technicallycreative.com
>
> www.TechnicallyCreative.com
> 
>
>
>
> ::
>
>
>
> *From:* Joe Witt 
> *Sent:* Tuesday, August 13, 2019 2:27 PM
> *To:* users@nifi.apache.org
> *Subject:* Re: Anti-Virus Scanning
>
>
>
> Jason
>
>
>
> The work dir gets created at startup and possible as new nars are loaded.
> I think you'd be ok to scan this.
>
>
>
> The flowfile and content repository and provenance directories as
> configured should be skipped. The logs dir should be skipped.  The state
> directory should be skipped.  All else I believe would be fair game.
>
>
>
> Thanks
>
>
>
> On Tue, Aug 13, 2019 at 2:24 PM Jason Csencsits <
> jcsencs...@technicallycreative.com> wrote:
>
> What is the recommended anti virus scanning exclusions from active scans.
> Can not find anything in the documents. Need to make sure my linux redhat
> scans do not compromise the flow files or anything else.
>
>
>
> Thank you,
>
> :::
>
> *Jason Csencsits *
>
> Manager of Technical Operations
>
> Technically Creative Inc.
>
> *Simplifying IT Solutions*
>
>
>
> Office: 845.725.7883
>
> jcsencs...@technicallycreative.com
>
> www.TechnicallyCreative.com
> 
>
>
>
> ::
>
>
>
>

Re: data metrics / data monitoring

[Edward Armes]

Hi Peter,

I think this depends on where this data is stored. If this data is avaiable
as metrics record by Nifi, then a reporting task would be the best way
forward. However if this is data that is recorded in your FlowFiles as part
of your flow then I think you're looking at either collecting in a KeyValue
store of sorts and exposing it via a Web Server pattern or forwarding the
metrics contained in the FlowFIle via a message bus, database or flow file
reciever of some description.

As for displaying your metrics there are a lot of options out there that
can recieve and processes data in various forms and it really depends on
what is the best fit for your orginisation.

Personaly I would work out how and what you use display the data and from
there use that to influence how you export it out from Nifi.

Edward

On Mon, 12 Aug 2019, 22:54 Peter Piehler,  wrote:

> Hello,
>
> does anyone have a tip for me on how I can provide metrics about data
> processed in nifi in a web UI?
>
> I process XML files with nifi. for each file I calculate how many new,
> modified, unmodified, and deleted records are contained. for each record
> checks are still made. For example, whether values are in the value range.
> I would like to create an evaluation which shows me how the data
> properties are. For example yesterday I had 5 files, one of them with
> 1000 deletions, but the average is only 10 deleted records per file, on
> average we process 500 files per day.
>
> I'm currently looking for ideas on how to do this. I think it would be
> useful to export this data and then evaluate it in an external
> application. I am grateful for every hint.
>
> Thx,
> Peter
>
>

Re: Optimizing Performance of Apache NiFi's Network Listening Processors

[Edward Armes]

HI Clay,

So as Bryan has said the actual connection is managed by a selector and all
this does is goes through each connection and once that connection has data
to receive it the selector then hands that over to a thread in the TCP
receiving thread pool which does then some basic TCP processing and puts it
into a buffer for an instance of associated ListenSyslog processor to
processes, when the framework executes an instance of that processor.

Just so you're aware while setting the maximum number of connections does
create a thread pool of 4,000 threads. In reality these threads don't
really exist until one is created by the selector to run on the pool. So in
short unless a single Nifi server gets 4,000 syslog messages in a very
short space time (< 1 micro-second) I can't see it being an issue.

Edward

On Fri, Aug 2, 2019 at 2:06 PM Bryan Bende  wrote:

> The actual connections themselves are managed with a selector, so if
> all the connections are idle there should only be one thread for the
> socket.
>
> As soon as a connection has something available to read then a thread
> is spawned to start reading the connection until either no matter is
> available, or it is closed.
>
> On Fri, Aug 2, 2019 at 7:18 AM Clay Teahouse 
> wrote:
> >
> > Hello Edward,
> > So, if have of to listen to 32,000 tcp connections and I have only 80
> cores, and I configure each ListenSyslog instance for 4,000 connections,
> doesn't each spawn 4,000 threads behind the scene? The tcp connections will
> be idle most of the time.
> >
> > thanks
> > Clay
> >
> >
> > On Fri, Aug 2, 2019 at 6:10 AM Edward Armes 
> wrote:
> >>
> >> Hi Clay,
> >>
> >> Because Nifi underneath uses a thread pool for it's own threading
> underneath, and each instance processor runs does so in it's own thread, I
> don't see any reason why not. One thing to note that the way the ListenTCP
> processor appears to have been written such that it gets all the requests
> that have been received on that socket and processes them until either it
> has no more requests left or process or that instance of the processor is
> no longer scheduled to run.
> >>
> >> Hope that helps
> >>
> >> Edward
> >>
> >> On Fri, Aug 2, 2019 at 11:28 AM Clay Teahouse 
> wrote:
> >>>
> >>> Hello All,
> >>>
> >>> I need to listen to and process thousands of persistent TCP
> connections. I have 10 nodes, each having 8 cores.
> >>> My understanding is that with existing NiFi listening processors, such
> as ListnSyslog, a thread is utilized for each TCP connection. Does this
> scale? Do I need to write a custom processor that utilizes a thread pool
> for reading the data from the socket and processing them?
> >>>
> >>> thanks
> >>> Clay
>

Re: Optimizing Performance of Apache NiFi's Network Listening Processors

[Edward Armes]

Hi Clay,

Because Nifi underneath uses a thread pool for it's own threading
underneath, and each instance processor runs does so in it's own thread, I
don't see any reason why not. One thing to note that the way the ListenTCP
processor appears to have been written such that it gets all the requests
that have been received on that socket and processes them until either it
has no more requests left or process or that instance of the processor is
no longer scheduled to run.

Hope that helps

Edward

On Fri, Aug 2, 2019 at 11:28 AM Clay Teahouse 
wrote:

> Hello All,
>
> I need to listen to and process thousands of persistent TCP connections. I
> have 10 nodes, each having 8 cores.
> My understanding is that with existing NiFi listening processors, such as
> ListnSyslog, a thread is utilized for each TCP connection. Does this scale?
> Do I need to write a custom processor that utilizes a thread pool for
> reading the data from the socket and processing them?
>
> thanks
> Clay
>

Re: Continuing my LDAP auth adventures

[Edward Armes]

So when you use the LDAP provider the initial admin identity still needs to
be set. However as your using the LDAP plugin, it does need to be the full
DN of the user who is going to connect to cluster the first time to setup
all the user permissions for all the other users.

This site gives a good example and a break down on how to specify a unique
user by attributes with a DN:
http://www.zytrax.com/books/ldap/apa/dn-rdn.html

Edward

On Fri, Jul 19, 2019 at 2:59 PM Nicolas Delsaux 
wrote:

> And indeed, it changed the error
>
>
> nifi-runner_1  | Caused by:
> org.springframework.beans.factory.BeanCreationException: Error creating
> bean with name 'authorizer': FactoryBean threw exception on object
> creation; nested exception is
> org.apache.nifi.authorization.exception.AuthorizerCreationException:
> org.apache.nifi.authorization.exception.AuthorizerCreationException: Unable
> to locate initial admin a_dn to seed policies
> nifi-runner_1  |at
> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:185)
> nifi-runner_1  |at
> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103)
> nifi-runner_1  |at
> org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1640)
> nifi-runner_1  |at
> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:323)
> nifi-runner_1  |at
> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
> nifi-runner_1  |at
> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
> nifi-runner_1  |... 96 common frames omitted
> nifi-runner_1  | Caused by:
> org.apache.nifi.authorization.exception.AuthorizerCreationException:
> org.apache.nifi.authorization.exception.AuthorizerCreationException: Unable
> to locate initial admin a_dn to seed policies
> nifi-runner_1  |at
> org.apache.nifi.authorization.FileAccessPolicyProvider.onConfigured(FileAccessPolicyProvider.java:263)
> nifi-runner_1  |at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> nifi-runner_1  |at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> nifi-runner_1  |at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> nifi-runner_1  |at java.lang.reflect.Method.invoke(Method.java:498)
> nifi-runner_1  |at
> org.apache.nifi.authorization.AccessPolicyProviderInvocationHandler.invoke(AccessPolicyProviderInvocationHandler.java:54)
> nifi-runner_1  |at com.sun.proxy.$Proxy78.onConfigured(Unknown
> Source)
> nifi-runner_1  |at
> org.apache.nifi.authorization.AuthorizerFactoryBean.getObject(AuthorizerFactoryBean.java:153)
> nifi-runner_1  |at
> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:178)
> nifi-runner_1  |... 101 common frames omitted
> nifi-runner_1  | Caused by:
> org.apache.nifi.authorization.exception.AuthorizerCreationException: Unable
> to locate initial admin a_dn to seed policies
> nifi-runner_1  |at
> org.apache.nifi.authorization.FileAccessPolicyProvider.populateInitialAdmin(FileAccessPolicyProvider.java:598)
> nifi-runner_1  |at
> org.apache.nifi.authorization.FileAccessPolicyProvider.load(FileAccessPolicyProvider.java:541)
> nifi-runner_1  |at
> org.apache.nifi.authorization.FileAccessPolicyProvider.onConfigured(FileAccessPolicyProvider.java:254)
> nifi-runner_1  |... 109 common frames omitted
>
> which seems to indicate that on startup, the FileAccessPolicyProvider will
> try to get informations for the manager dn in the file (which, as far as a
> I understand, is not yet loaded)
>
> .
>
> So there must be some weird back-and-forth dance between the ldap user
> group provider and the file policy provider ... But I don't understand the
> dance in question
> Le 19/07/2019 à 15:38, Edward Armes a écrit :
>
> Hi Nicolas,
>
> In your actual configuration, is this the actual entry and not sanitized
> version?
>
> This attribute doesn't exist
> to make sure no grouping is done
>
> If so I think this is the problem. As I what I think is happening Nifi is
> trying to interpret this value as a DN and failing, if you only need the
> users returned from the LDAP search to be the list of valid users then this
> field can just be left blank, if however you need a list of valid (not
> necessarily authorized) 

Re: Continuing my LDAP auth adventures

[Edward Armes]

Hi Nicolas,

In your actual configuration, is this the actual entry and not sanitized
version?

This attribute doesn't exist to
make sure no grouping is done

If so I think this is the problem. As I what I think is happening Nifi is
trying to interpret this value as a DN and failing, if you only need the
users returned from the LDAP search to be the list of valid users then this
field can just be left blank, if however you need a list of valid (not
necessarily authorized) users to be filtered to be a member of a specific
LDAP group then you can specify the DN for that group here.

I would change it to:



and see if that works

Edward

On Fri, Jul 19, 2019 at 2:04 PM Nicolas Delsaux 
wrote:

> Here is the full version (with obvious replacements for manager dn,
> manager password, ldap server url, and other "sensitive" informations
>
>
> 
> 
> 
> 
> 
> 
> 
> 
> ldap-user-group-provider
> org.apache.nifi.ldap.tenants.LdapUserGroupProvider
> LDAPS
> a_dn
> a_password
> 
> 
> 
> /opt/certs/cacerts.jks
> changeit
> JKS
> 
> TLSv1
> 
> FOLLOW
> 10 secs
> 10 secs
> ldaps://myserver.mycompany.com:636
> 
> 30 mins
> ou=people,o=mycompany.com
> privPerson
> SUBTREE
> 
> uid
> This attribute doesn't exist
> to make sure no grouping is done
>  property>
> 
> group
> ONE_LEVEL
> 
> 
> 
>  property>
> 
> 
> 
> 
> 
> 
> 
> file-access-policy-provider
> org.apache.nifi.authorization.FileAccessPolicyProvider
> ldap-user-group-provider
> ./conf/authorizations.xml
> 
> 
> 
> 
> 
> 
> 
> managed-authorizer
> org.apache.nifi.authorization.StandardManagedAuthorizer
> file-access-policy-provider property>
> 
> 
> 
> 
> Le 19/07/2019 à 12:03, Pierre Villard a écrit :
>
> Hi Nicolas,
>
> Could you share the full content of your authorizers.xml file? Sometimes
> it's just a matter of references not being in the right "order".
>
> Le ven. 19 juil. 2019 à 11:59, Edward Armes  a
> écrit :
>
>> I wasn't able to find any single good way, I don't know if switching the
>> logs down to debug or trace might give you a bit more info though . In the
>> end I just went through a worked it out by hand using a combination of
>> manual checking against an alternative tool (i.e. an LDAP browser), file
>> format checkers, or just commenting things out by hand.
>>
>> I did sometimes find that white space character (new line etc...) can
>> occasionally cause a problem with the Spring loading.
>>
>> Edward
>>
>> On Fri, Jul 19, 2019 at 10:45 AM Nicolas Delsaux 
>> wrote:
>>
>>> Is there any way to get a better error ?
>>> Le 19/07/2019 à 11:36, Edward Armes a écrit :
>>>
>>> Hi Nicolas,
>>>
>>> This one is a bit of a Spring special. The actual cause here is that the
>>> Spring Bean that is being created from this file has silently failed, and
>>> thus the auto-wiring has failed as well. The result is you get this lovely
>>> misleading error. The normal reason for the bean not being created I found
>>> was because I made a typo in the configuration file(s).
>>>
>>> Edward
>>>
>>> On Fri, Jul 19, 2019 at 10:21 AM Nicolas Delsaux 
>>> wrote:
>>>
>>>> Hi all
>>>>
>>>> Now I know how to connect to my LDAP directory, i now have a strange
>>>> error
>>>>
>>>>
>>>> nifi-runner_1  |
>>>> org.springframework.beans.factory.UnsatisfiedDependencyException: Error
>>>> creating bean with name
>>>> 'org.springframework.security.config.annotation.web.configuration.WebSecurityConfiguration':
>>>> Unsatisfied dependency expressed through method
>>>> 'setFilterChainProxySecurityConfigurer' parameter 1; nested exception is
>>>> org.springframework.beans.factory.BeanExpressionException: Expression
>>>> parsing failed; nested exception is
>>>> org.springframework.beans.factory.UnsatisfiedDependencyException: Error
>>>> creating bean with name
>>>> 'org.apache.nifi.web.NiFiWebApiSecurityConfiguration': Unsatisfied
>>>> dependency expressed through method 'setJwtAuthenticationProvider'
>>>> parameter 0; nested exception is
>>>> org.springframework.beans.factory.BeanCreationException: Error creating
>>>> bean with name 'jwtAuthenticationProvider' defined in class path resource
>>>> [nifi-web-security-context.xml]: Cannot resolve reference to 

Re: Continuing my LDAP auth adventures

[Edward Armes]

I wasn't able to find any single good way, I don't know if switching the
logs down to debug or trace might give you a bit more info though . In the
end I just went through a worked it out by hand using a combination of
manual checking against an alternative tool (i.e. an LDAP browser), file
format checkers, or just commenting things out by hand.

I did sometimes find that white space character (new line etc...) can
occasionally cause a problem with the Spring loading.

Edward

On Fri, Jul 19, 2019 at 10:45 AM Nicolas Delsaux 
wrote:

> Is there any way to get a better error ?
> Le 19/07/2019 à 11:36, Edward Armes a écrit :
>
> Hi Nicolas,
>
> This one is a bit of a Spring special. The actual cause here is that the
> Spring Bean that is being created from this file has silently failed, and
> thus the auto-wiring has failed as well. The result is you get this lovely
> misleading error. The normal reason for the bean not being created I found
> was because I made a typo in the configuration file(s).
>
> Edward
>
> On Fri, Jul 19, 2019 at 10:21 AM Nicolas Delsaux 
> wrote:
>
>> Hi all
>>
>> Now I know how to connect to my LDAP directory, i now have a strange error
>>
>>
>> nifi-runner_1  |
>> org.springframework.beans.factory.UnsatisfiedDependencyException: Error
>> creating bean with name
>> 'org.springframework.security.config.annotation.web.configuration.WebSecurityConfiguration':
>> Unsatisfied dependency expressed through method
>> 'setFilterChainProxySecurityConfigurer' parameter 1; nested exception is
>> org.springframework.beans.factory.BeanExpressionException: Expression
>> parsing failed; nested exception is
>> org.springframework.beans.factory.UnsatisfiedDependencyException: Error
>> creating bean with name
>> 'org.apache.nifi.web.NiFiWebApiSecurityConfiguration': Unsatisfied
>> dependency expressed through method 'setJwtAuthenticationProvider'
>> parameter 0; nested exception is
>> org.springframework.beans.factory.BeanCreationException: Error creating
>> bean with name 'jwtAuthenticationProvider' defined in class path resource
>> [nifi-web-security-context.xml]: Cannot resolve reference to bean
>> 'authorizer' while setting constructor argument; nested exception is
>> org.springframework.beans.factory.BeanCreationException: Error creating
>> bean with name 'authorizer': FactoryBean threw exception on object
>> creation; nested exception is java.lang.Exception: The specified authorizer
>> 'ldap-user-group-provider' could not be found.
>>
>> [... let me just skip the uninteresting Spring stack ...]
>>
>> nifi-runner_1  | Caused by:
>> org.springframework.beans.factory.BeanCreationException: Error creating
>> bean with name 'authorizer': FactoryBean threw exception on object
>> creation; nested exception is java.lang.Exception: The specified authorizer
>> 'ldap-user-group-provider' could not be found.
>> nifi-runner_1  |at
>> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:185)
>> nifi-runner_1  |at
>> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103)
>> nifi-runner_1  |at
>> org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1640)
>> nifi-runner_1  |at
>> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:323)
>> nifi-runner_1  |at
>> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
>> nifi-runner_1  |at
>> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
>> nifi-runner_1  |... 96 common frames omitted
>> nifi-runner_1  | Caused by: java.lang.Exception: The specified authorizer
>> 'ldap-user-group-provider' could not be found.
>> nifi-runner_1  |at
>> org.apache.nifi.authorization.AuthorizerFactoryBean.getObject(AuthorizerFactoryBean.java:175)
>> nifi-runner_1  |at
>> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:178)
>>
>> From what I understand, it seems like the AuthorizerFactoryBean tries to
>> read my user-group-provider from the authorizers.xml file.
>>
>>
>> I have such an user group provider, which is a ldap one :
>> 
>> 
>> ldap-user-group-provider
>> org.apache.nifi.ldap.tenants.LdapUserGroupProvider
>> LDAPS
>> a_dn

Re: Continuing my LDAP auth adventures

[Edward Armes]

Hi Nicolas,

This one is a bit of a Spring special. The actual cause here is that the
Spring Bean that is being created from this file has silently failed, and
thus the auto-wiring has failed as well. The result is you get this lovely
misleading error. The normal reason for the bean not being created I found
was because I made a typo in the configuration file(s).

Edward

On Fri, Jul 19, 2019 at 10:21 AM Nicolas Delsaux 
wrote:

> Hi all
>
> Now I know how to connect to my LDAP directory, i now have a strange error
>
>
> nifi-runner_1  |
> org.springframework.beans.factory.UnsatisfiedDependencyException: Error
> creating bean with name
> 'org.springframework.security.config.annotation.web.configuration.WebSecurityConfiguration':
> Unsatisfied dependency expressed through method
> 'setFilterChainProxySecurityConfigurer' parameter 1; nested exception is
> org.springframework.beans.factory.BeanExpressionException: Expression
> parsing failed; nested exception is
> org.springframework.beans.factory.UnsatisfiedDependencyException: Error
> creating bean with name
> 'org.apache.nifi.web.NiFiWebApiSecurityConfiguration': Unsatisfied
> dependency expressed through method 'setJwtAuthenticationProvider'
> parameter 0; nested exception is
> org.springframework.beans.factory.BeanCreationException: Error creating
> bean with name 'jwtAuthenticationProvider' defined in class path resource
> [nifi-web-security-context.xml]: Cannot resolve reference to bean
> 'authorizer' while setting constructor argument; nested exception is
> org.springframework.beans.factory.BeanCreationException: Error creating
> bean with name 'authorizer': FactoryBean threw exception on object
> creation; nested exception is java.lang.Exception: The specified authorizer
> 'ldap-user-group-provider' could not be found.
>
> [... let me just skip the uninteresting Spring stack ...]
>
> nifi-runner_1  | Caused by:
> org.springframework.beans.factory.BeanCreationException: Error creating
> bean with name 'authorizer': FactoryBean threw exception on object
> creation; nested exception is java.lang.Exception: The specified authorizer
> 'ldap-user-group-provider' could not be found.
> nifi-runner_1  |at
> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:185)
> nifi-runner_1  |at
> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103)
> nifi-runner_1  |at
> org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1640)
> nifi-runner_1  |at
> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:323)
> nifi-runner_1  |at
> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
> nifi-runner_1  |at
> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
> nifi-runner_1  |... 96 common frames omitted
> nifi-runner_1  | Caused by: java.lang.Exception: The specified authorizer
> 'ldap-user-group-provider' could not be found.
> nifi-runner_1  |at
> org.apache.nifi.authorization.AuthorizerFactoryBean.getObject(AuthorizerFactoryBean.java:175)
> nifi-runner_1  |at
> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:178)
>
> From what I understand, it seems like the AuthorizerFactoryBean tries to
> read my user-group-provider from the authorizers.xml file.
>
>
> I have such an user group provider, which is a ldap one :
> 
> 
> ldap-user-group-provider
> org.apache.nifi.ldap.tenants.LdapUserGroupProvider
> LDAPS
> a_dn
> a_password
> 
> 
> 
> /opt/certs/cacerts.jks
> another
> JKS
> 
> TLSv1
> 
> FOLLOW
> 10 secs
> 10 secs
> ldaps://myserver.mycompany.com:636
> 
> 30 mins
> ou=people,o=mycompany.com
> privPerson
> SUBTREE
> 
> uid
> This attribute doesn't exist
> to make sure no grouping is done
>  property>
> 
> group
> ONE_LEVEL
> 
> 
> 
>  property>
> 
>
> So why can't it be loaded ?
>
> Because I don't see any other exception (typically, I would expect a
> search fail exception, but it seems to work).
>

Re: DistributeLoad across a NiFi cluster

[Edward Armes]

Hi Andrew,

Is this functionality documented anywhere do you know? As I've had a quick
look through the documentation and I haven't seen this.

Edward

On Tue, Jul 2, 2019 at 5:33 PM James McMahon  wrote:

> Excellent - thanks very much Andrew. This is my first crack at working
> with a clustered configuration, and I guess that shows by my question.
> Outstanding - thanks again.
>
> On Tue, Jul 2, 2019 at 12:29 PM Andrew Grande  wrote:
>
>> Jim,
>>
>> There's a better solution in NiFi. Right click on the connection between
>> ListFile and FetchFile and select a cluster distribution strategy in
>> options. That's it :)
>>
>> Andrew
>>
>> On Tue, Jul 2, 2019, 7:37 AM James McMahon  wrote:
>>
>>> We would like to employ a DistributeLoad processor, restricted to run on
>>> the primary node of our cluster. Is there a recommended approach employed
>>> to efficiently distribute across nodes in the cluster?
>>>
>>> As I understand it, and using a FetchFile running in "all nodes" as the
>>> first processor following the DistributeLoad, I can have it distribute by
>>> round robin, next available, or load distribution service.  Can anyone
>>> provide a link to an example that employs the load distribution service? Is
>>> that the recommended distribution approach when running in clustered mode?
>>>
>>> I am interested in maintaining load balance across my cluster nodes when
>>> running at high flowfile volumes. Flow files will vary greatly in contents,
>>> so I'd like to design with an approach that helps me balance processing
>>> distribution.
>>>
>>> Thanks very much in advance. -Jim
>>>
>>