Re: Error on first launch (probable newbie misconfiguration)

[Mark Payne]

2.jar:/opt/nifi-2.0.0-M2/./lib/nifi-stateless-bootstrap-2.0.0-M2.jar:/opt/nifi-2.0.0-M2/./lib/logback-classic
>> -1.4.14.jar:/opt/nifi-2.0.0-M2/./lib/logback-core-1.4.14.jar:/opt/nifi-2.0.0-M2/./lib/nifi-nar-utils-2.0.0-M2.jar:/opt/nifi-2.0.0-M2/./lib/slf4j-a
>> pi-2.0.11.jar:/opt/nifi-2.0.0-M2/./lib/nifi-api-2.0.0-M2.jar:/opt/nifi-2.0.0-M2/./lib/nifi-stateless-api-2.0.0-M2.jar:/opt/nifi-2.0.0-M2/./lib/nif
>> i-properties-2.0.0-M2.jar -Xmx1g -Djava.awt.headless=true -Xms1g 
>> -Djavax.security.auth.useSubjectCredsOnly=true 
>> -Dsun.net.http.allowRestrictedHead
>> ers=true -Djava.protocol.handler.pkgs=sun.net.www.protocol 
>> -Dcurator-log-only-first-connection-issue-as-error-level=true 
>> -Dnifi.properties.file.pa
>> th=/opt/nifi-2.0.0-M2/./conf/nifi.properties 
>> -Dnifi.bootstrap.listen.port=37235 -Dapp=NiFi 
>> -Dorg.apache.nifi.bootstrap.config.log.dir=/opt/nifi-2.
>> 0.0-M2/logs org.apache.nifi.NiFi  
>> 2024-03-07 08:05:28,757 INFO [main] org.apache.nifi.bootstrap.Command 
>> Application Process [330] launched 
>> 2024-03-07 08:05:29,087 INFO [NiFi Bootstrap Command Listener] 
>> org.apache.nifi.bootstrap.RunNiFi Apache NiFi now running and listening for 
>> Bootstr
>> ap requests on port 40615 
>> 2024-03-07 08:05:36,535 ERROR [NiFi logging handler] org.apache.nifi.StdErr 
>> Failed to start web server: Error creating bean with name 'org.springf
>> ramework.security.config.annotation.web.configuration.WebSecurityConfiguration':
>>  Unsatisfied dependency expressed through method 'setFilterChains'
>> parameter 0: Error creating bean with name 'securityFilterChain' defined in 
>> org.apache.nifi.web.security.configuration.WebSecurityConfiguration: 
>> Unsatisfied dependency expressed through method 'securityFilterChain' 
>> parameter 2: Error creating bean with name 'org.apache.nifi.web.security.con
>> figuration.JwtAuthenticationSecurityConfiguration': Unsatisfied dependency 
>> expressed through constructor parameter 2: Error creating bean with nam
>> e 'flowController' defined in class path resource [nifi-context.xml]: Cannot 
>> resolve reference to bean 'propertyEncryptor' while setting bean prop
>> erty 'encryptor' 
>> 2024-03-07 08:05:36,535 ERROR [NiFi logging handler] org.apache.nifi.StdErr 
>> Shutting down... 
>> 2024-03-07 08:05:37,759 INFO [main] org.apache.nifi.bootstrap.RunNiFi NiFi 
>> never started. Will not restart NiFi
>> 
>> 
>> Regards
>> Franck
>> 
>> 
>> Le mercredi 6 mars 2024, 19:55:36 CET Mark Payne a écrit :
>>> Hey there Franck,
>>> 
>>> Can you provide the full error message with stack trace that gets printed?
>>> 
>>> Thanks
>>> -Mark
>>> 
>>> 
>>> On Mar 6, 2024, at 1:26 PM, Franck Routier via users 
>>>  wrote:
>>> 
>>> Hi,
>>> 
>>> I'm a first time wanabe user of Nifi, and I followed the installation 
>>> guide, until the point when I try to launch Nifi:
>>> 
>>> - install ubuntu 22.04 (in an incus container)
>>> - install java (21.0.2)
>>> - download and install nifi
>>> - set mandatory properties in conf/nifi.properties
>>> - and run:
>>> 
>>> ../bin/nifi.sh run
>>> but it returns quickly, and the logs tells me that...:
>>> 
>>> Error creating bean with name 
>>> 'org.springframework.security.config.annotation.web.configuration.WebSecurityConfiguration':
>>>  Unsatisfied dependency expressed through method 'setFilterChains' 
>>> parameter 0: Error creating bean with name
>>> 'securityFilterChain' defined in 
>>> org.apache.nifi.web.security.configuration.WebSecurityConfiguration: 
>>> Unsatisfied dependency expressed through method 'securityFilterChain' 
>>> parameter 2: Error creating bean with name 
>>> 'org.apache.nifi.web.security.configuration.JwtAuthenticationSecurityConfiguration':
>>>  Unsatisfied dependency expressed thr
>>> ough constructor parameter 2: Error creating bean with name 
>>> 'flowController' defined in class path resource [nifi-context.xml]: Cannot 
>>> resolve reference to bean 'propertyEncryptor' while setting bean property 
>>> 'encryptor'
>>> 
>>> So, I guess a configuration is missing somewhere, but I have no clue where.
>>> (I checked that nifi.sensitive.props.key is set, it, is, and now I'm lost).
>>> 
>>> Thanks
>>> Franck
>>> 
>>> 
>> 
> 
> 
> 

Re: Why are my journal files so large on node 2 of cluster?

[Mark Payne]

David,

Makes sense. Large values should not be added as attributes. Attributes are 
designed for String values. Think 100-200 characters, generally. A couple KB 
can be fine, as well, but it can significantly reduce performance. If the 
intent is to “stash the content” so that you can change it and perform 
enrichment, you should take a look at ForkEnrichment / JoinEnrichment 
processors.

Thanks
-Mark

On Mar 11, 2024, at 2:05 PM, David Early  wrote:

Mark,

Yes, it was the flowfile repository.

Of all your points, the large attributes is most likely our issue.  One of our 
folks was caching the flowfile (which can be large occasionally) in an 
attribute ahead of a DB lookup (which would overwrite the contents) and then 
reinstating the content after merging with the DB lookup.

The attribute was not removed after the merge. We have added a couple of items 
to remove the attribute this morning, but the mere presence of it briefly may 
be enough to cause the spikes.

I have since attached a very large disk and I can see the occasionally spikes:


At 22% on a 512G disk, that is over 110G.  What isn't clear is why it is not 
consistently spiking.

We have made some changes to the how long the attribute lives and will monitor 
over the next couple of days, but likely we will need to cache the contents 
somewhere and retrieve them later unless someone knows of a better solution 
here.

Thanks for the guidance

Dave


On Fri, Mar 8, 2024 at 7:05 AM Mark Payne 
mailto:marka...@hotmail.com>> wrote:
Dave,

When you say that the journal files are huge, I presume you mean the FlowFile 
repository?

There are generally 4 things that can cause this:
- OutOfMemoryError causing the FlowFile repo not to properly checkpoint
- Out of Disk Space causing the FlowFile repo not to properly checkpoint
- Out of open file handles causing the FlowFile repo not to properly checkpoint
- Creating a lot of huge attributes on your FlowFiles.

The first 3 situations can be identified by looking for errors in the logs.
For the third one, you need to understand whether or not you’re creating huge 
FlowFile attributes. Generally, attributes should be very small - 100-200 
characters or less, ideally. It’s possible that you have a flow that creates 
huge attributes but the flow is only running on the Primary Node, and Node 2 is 
your Primary Node, which would cause this to occur only on this node.

Thanks
-Mark


> On Mar 7, 2024, at 9:24 PM, David Early via users 
> mailto:users@nifi.apache.org>> wrote:
>
> I have a massive issue: I have a 2 node cluster (using 5 external zookeepers 
> on other boxes), and for some reason on node 2 I have MASSIVE journal files.
>
> I am round robbining data between the nodes, but for some reason node 2 just 
> fills up.  This is the second time this has happened this week.
>
> What should I do?  nifi.properties are the same on both systems (except for 
> local host names)..
>
> Any ideas of what might be causing one node to overload?
>
> Dave
>
>



--
David Early, Ph.D.
david.ea...@grokstream.com<mailto:david.ea...@grokstream.com>
720-470-7460 Cell
[https://ci3.googleusercontent.com/mail-sig/AIorK4ytFrueqWyKLKu2TrMCXdoDWTMEnQxLcsSDLlHSBOyzXbaaJq-i2giAs6TarzTUtUl8iUVecLU]

Re: Why are my journal files so large on node 2 of cluster?

[Mark Payne]

Dave,

When you say that the journal files are huge, I presume you mean the FlowFile 
repository?

There are generally 4 things that can cause this:
- OutOfMemoryError causing the FlowFile repo not to properly checkpoint
- Out of Disk Space causing the FlowFile repo not to properly checkpoint
- Out of open file handles causing the FlowFile repo not to properly checkpoint
- Creating a lot of huge attributes on your FlowFiles.

The first 3 situations can be identified by looking for errors in the logs.
For the third one, you need to understand whether or not you’re creating huge 
FlowFile attributes. Generally, attributes should be very small - 100-200 
characters or less, ideally. It’s possible that you have a flow that creates 
huge attributes but the flow is only running on the Primary Node, and Node 2 is 
your Primary Node, which would cause this to occur only on this node.

Thanks
-Mark


> On Mar 7, 2024, at 9:24 PM, David Early via users  
> wrote:
> 
> I have a massive issue: I have a 2 node cluster (using 5 external zookeepers 
> on other boxes), and for some reason on node 2 I have MASSIVE journal files.  
> 
> I am round robbining data between the nodes, but for some reason node 2 just 
> fills up.  This is the second time this has happened this week.
> 
> What should I do?  nifi.properties are the same on both systems (except for 
> local host names)..
> 
> Any ideas of what might be causing one node to overload?
> 
> Dave
> 
> 

Re: Error on first launch (probable newbie misconfiguration)

[Mark Payne]

Hey there Franck,

Can you provide the full error message with stack trace that gets printed?

Thanks
-Mark


On Mar 6, 2024, at 1:26 PM, Franck Routier via users  
wrote:

Hi,

I'm a first time wanabe user of Nifi, and I followed the installation guide, 
until the point when I try to launch Nifi:

- install ubuntu 22.04 (in an incus container)
- install java (21.0.2)
- download and install nifi
- set mandatory properties in conf/nifi.properties
- and run:

../bin/nifi.sh run
but it returns quickly, and the logs tells me that...:

Error creating bean with name 
'org.springframework.security.config.annotation.web.configuration.WebSecurityConfiguration':
 Unsatisfied dependency expressed through method 'setFilterChains' parameter 0: 
Error creating bean with name
'securityFilterChain' defined in 
org.apache.nifi.web.security.configuration.WebSecurityConfiguration: 
Unsatisfied dependency expressed through method 'securityFilterChain' parameter 
2: Error creating bean with name 
'org.apache.nifi.web.security.configuration.JwtAuthenticationSecurityConfiguration':
 Unsatisfied dependency expressed thr
ough constructor parameter 2: Error creating bean with name 'flowController' 
defined in class path resource [nifi-context.xml]: Cannot resolve reference to 
bean 'propertyEncryptor' while setting bean property 'encryptor'

So, I guess a configuration is missing somewhere, but I have no clue where.
(I checked that nifi.sensitive.props.key is set, it, is, and now I'm lost).

Thanks
Franck

Re: InvokeHttp - Provenance Events

[Mark Payne]

Greg,

Yes, the Relationship is generally only populated for ROUTE events, such as 
RouteOnAttribute.

Thanks
-Mark


> On Feb 26, 2024, at 11:34 AM, Gregory Foreman 
>  wrote:
> 
> Hello:
> 
> I am having an issue with the InvokeHttp processor provenance events.  No 
> Relationship field is populated in the Provenance Event / Details page.  Is 
> this expected?  I am trying to query provenance for Relationship other than 
> “Response" so that I can view failed flow files.
> 
> The system is running Nifi 1.23.2, the nifi.properties is properly set to 
> index this Field, and the RouteOnAttribute processor is setting this Field as 
> expected.
> 
> Thanks,
> Greg

Re: expression language/groovy date handling

[Mark Payne]

Richard,

It sounds like your scripted reader is responsible for parsing the Avro? In 
short, the Record appears to have an Avro Utf8 value, not a String, in the 
field you’re looking at. You could call .toString() on that Utf8 object, or you 
could configure the Avro reader to return Strings instead of Utf8 objects.

Thanks
-Mark

On Feb 15, 2024, at 5:52 AM, Richard Beare  wrote:

Hi,
This is a test pipeline reading pdf files from disk. It begins with a GetFile 
processor supplying a ConvertRecord processor with a scripted reader input and 
an avrorecordsetwriter, generic output.

The scripted reader places the file content in a "content" field:

 List recordFields = []
recordFields.add(new RecordField("content", 
RecordFieldType.ARRAY.getArrayDataType(RecordFieldType.BYTE.getDataType(
schema = new SimpleRecordSchema(recordFields)

This bit seems OK.

Next step is update record which adds other fields to mimic the real case of 
pulling out of a DB - Age, gender etc, all of which are dummies and a timestamp 
based on the filename by the following expression language:

${filename:substringBeforeLast('.'):substringAfterLast('_'):toDate('MMdd'):format("-MM-dd
 HH:mm:ss")}

If I explicitly set the  schema for the record writer to include
 {"name":"Visit_DateTime","type": {"type" : "long", "logicalType" : 
"timestamp-millis"}},

then I can get the following converter, a groovy script, which converts to json 
for transmission to an web service, to deal with the dates as follows:

Date VisitTimeValue = null

VisitTimeValue = new Date(currRecord.get(TimeStampFieldName))


I guess I thought this approach was overly complex. Given that I'm using Date 
functions in the expression language I hoped that the generic avro writer would 
correctly infer the schema so that I didn't have to explicitly provide one. Is 
this approach the right one? Is there a way I can isolate the expectation of a 
date component inside the groovy file only?

I hope this is clear.
Thanks for your help.


On Thu, Feb 15, 2024 at 9:38 AM Mark Payne 
mailto:marka...@hotmail.com>> wrote:
Hey Richard,

I think you’d need to explain more about what you’re doing in your groovy 
script. What processor are you using? What’s the script doing? Is it parsing 
Avro data?

On Jan 29, 2024, at 12:26 AM, Richard Beare 
mailto:richard.be...@gmail.com>> wrote:

Anyone able to offer assistance with this?

I think my problem relates to correctly specifying types using expression 
languages and using schema inference from groovy.

On Tue, Jan 23, 2024 at 2:20 PM Richard Beare 
mailto:richard.be...@gmail.com>> wrote:
Hi,
What is the right way to deal with dates in the following context.

I'm using the updaterecord processor to add a datestamp field to a record 
(derived from a filename attribute inserted by the getfile processor).

/Visit_DateTime.  
${filename:substringBeforeLast('.'):substringAfterLast('_'):toDate('MMdd'):format('-MM-dd'T'HH:mm:ss'Z'")

Inside the groovy script I'm attempting to convert to date as follows:

VisitTimeValue = new Date(currRecord.get(Visit_DateTime as String))

However I always get messages about "could not find matching constructor for 
java.util.Date(org.apackge.avro.util.Utf8)"

I have a previously working version, from a slightly different context which 
did a cast to long: Date((long)currRecord.get). In that case the record was 
created by a database query.

The eventual use of VisitTimeValue is to dump it into a flowfile attribute.

It seems to me that the type of the date field is not being correctly inferred 
by the avro reader/writers after I create it with the expression language. 
Alternatively, perhaps I should be using different date handling tools inside 
groovy.

All advice welcome.
Thanks

Re: Can we access Queued Duration as an attribute?

[Mark Payne]

Jim,

You can actually reference “lastQueueDate” in Expression Language. It is 
formatted as number of milliseconds since epoch.

So you might have a RouteOnAttribute that has a property named “old” with a 
value of:
${lastQueueDate:lt( ${now():minus(1)} )}

So any FlowFile that has been queued for more than 10 seconds would be routed 
to “old”, anything else to “unmatched”

Thanks
-Mark


On Feb 15, 2024, at 10:18 AM, James McMahon  wrote:

That would work - what a good suggestion. I'll do that. I can format the 
resulting number and then RouteOnAttribute by the desired subset of the result.
Something like this to set attribute dt.failure:
${now():toNumber():toDate("-MM-ddHH:mm:ss"):format("MMddHHmmss","EST")}
Then I can effectively route the files.
Thank you Jim S.

On Thu, Feb 15, 2024 at 9:55 AM Jim Steinebrey 
mailto:jrsteineb...@gmail.com>> wrote:
You could add an UpdateAttribute processor first in the failure path to add a 
new attribute which contains the time the error occurred by using the ${now()} 
or ${now():toNumber()} expression language function.
https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#now

Then later on in the flow you can compare current time to the saved error time 
to see how much time has elapsed.

— Jim


On Feb 15, 2024, at 9:44 AM, James McMahon 
mailto:jsmcmah...@gmail.com>> wrote:

As it turns out lineageStartDate and Queued Duration are very different. 
Without being able to get at Queued Duration as an attribute, it appears we 
cannot RouteOnAttribute to filter thousands in a queue by anything like hours 
they have been in queue.
Why would this be helpful? Let us say we have an InvokeHttp processor making 
calls to a REST endpoint. We leave for a weekend and return to find 5000 files 
in the Failure queue from this processor. It would be most helpful to identify 
the start time and end time of these 5000 failures. We can't do that reviewing 
only the first 100 flowfiles in the queue from the UI.
One can make an assumption that all of these 5000 flowfiles that failed 
InvokeHttp share a similar range of lineageStartDate, but that will not 
necessarily be true depending on flow complexity.

On Wed, Feb 14, 2024 at 9:49 AM James McMahon 
mailto:jsmcmah...@gmail.com>> wrote:
What a great workaround, thank you once again Mike. I'll put this in and use it 
now.
Jim

On Tue, Feb 13, 2024 at 4:41 PM Michael Moser 
mailto:moser...@gmail.com>> wrote:
Hello James,

I'm not aware of a way to access Queued Duration using expression language, but 
you can access the Lineage Duration information.  The Getting Started Guide 
mentions both entryDate and lineageStartDate as immutable attributes on all 
flowfiles.  These are numbers of milliseconds since epoch.  If you need them in 
a readable format, you can use the format() function.

simple examples:
${entryDate} = 1707859943778
${lineageStartDate} = 1707859943778
${lineageStartDate:format("-MM-dd HH:mm:ss.SSS")} = 2024-02-13 21:32:23.778

-- Mike


On Mon, Feb 12, 2024 at 11:38 AM James McMahon 
mailto:jsmcmah...@gmail.com>> wrote:
When we examine the contents of a queue through the UI and select a flowfile 
from the resulting list, we see FlowFile Details in the Details tab. Are those 
key/values accessible from nifi expression language? I would like to access 
Queued Duration. I have a queue that holds flowfiles with non-successful return 
codes for calls to REST services, and I want to route depending on how long 
these flowfiles have been sitting in my error queue to isolate the window when 
the REST service was unavailable.
Thank you for any examples that show how we can access these keys and values.

Re: expression language/groovy date handling

[Mark Payne]

Hey Richard,

I think you’d need to explain more about what you’re doing in your groovy 
script. What processor are you using? What’s the script doing? Is it parsing 
Avro data?

On Jan 29, 2024, at 12:26 AM, Richard Beare  wrote:

Anyone able to offer assistance with this?

I think my problem relates to correctly specifying types using expression 
languages and using schema inference from groovy.

On Tue, Jan 23, 2024 at 2:20 PM Richard Beare 
mailto:richard.be...@gmail.com>> wrote:
Hi,
What is the right way to deal with dates in the following context.

I'm using the updaterecord processor to add a datestamp field to a record 
(derived from a filename attribute inserted by the getfile processor).

/Visit_DateTime.  
${filename:substringBeforeLast('.'):substringAfterLast('_'):toDate('MMdd'):format('-MM-dd'T'HH:mm:ss'Z'")

Inside the groovy script I'm attempting to convert to date as follows:

VisitTimeValue = new Date(currRecord.get(Visit_DateTime as String))

However I always get messages about "could not find matching constructor for 
java.util.Date(org.apackge.avro.util.Utf8)"

I have a previously working version, from a slightly different context which 
did a cast to long: Date((long)currRecord.get). In that case the record was 
created by a database query.

The eventual use of VisitTimeValue is to dump it into a flowfile attribute.

It seems to me that the type of the date field is not being correctly inferred 
by the avro reader/writers after I create it with the expression language. 
Alternatively, perhaps I should be using different date handling tools inside 
groovy.

All advice welcome.
Thanks

Re: Finding slow down in processing

[Mark Payne]

Aaron,

It doesn’t sound like you’re back to the drawing board at all - sounds like you 
have the solution in hand. Just increase the size of your Timer Driven Thread 
Pool and leave it there.

Thanks
-Mark


On Jan 15, 2024, at 11:16 AM, Aaron Rich  wrote:

@Mark - thanks for that note. I hadn't tried restarting. When I did that, the 
performance dropped back down. So I'm back to the drawing board.

@Phillip - I didn't have any other services/components/dataflows going. It was 
just those 2 processors going (I tried to remove every variable I could to make 
it as controlled as possible). And during the week I ran that test, there 
wasn't any slow down at all. Even when I turned on the rest of the dataflows 
(~2500 components total) everything was performing as expected. There is very, 
very little variability in data volumes so I don't have any reason to believe 
that is the cause of the slow down.

I'm going to try to see what kind of the nifi diagnostics and such I can get.

Is there anywhere that explains the output of nifi.sh dump and nifi.sh 
diagnostics?

Thanks all for the help.

-Aaron

On Fri, Jan 12, 2024 at 11:45 AM Phillip Lord 
mailto:phillord0...@gmail.com>> wrote:
Ditto...

@Aaron... so outside of the GenerateFlowFile -> PutFile, were there additional 
components/dataflows handling data at the same time as the "stress-test".  
These will all share the same thread-pool.  So depending upon your dataflow 
footprint and any variability regarding data volumes... 20 timer-driven threads 
could be exhausted pretty quickly.  This might cause not only your 
"stress-test" to slow down but your other flows as well as components might be 
waiting for available threads to do their jobs.

Thanks,
Phil

On Thu, Jan 11, 2024 at 3:44 PM Mark Payne 
mailto:marka...@hotmail.com>> wrote:
Aaron,

Interestingly, up to version 1.21 of NiFi, if you increase the size of the 
thread pool, it increased immediately. But if you decreased the size of the 
thread pool, the decrease didn’t take effect until you restart NiFi. So that’s 
probably why you’re seeing the behavior you are. Even though you reset it to 10 
or 20, it’s still running at 40.

This was done to issues with Java many years ago, where it caused problems to 
decrease the thread pool size.  So just recently we updated NiFi to immediately 
scale down the thread pools as well.

Thanks
-Mark


On Jan 11, 2024, at 1:35 PM, Aaron Rich 
mailto:aaron.r...@gmail.com>> wrote:

So the good news is it's working now. I know what I did but I don't know why it 
worked so I'm hoping others can enlighten me based on what I did.

TL;DR - "turn it off/turn in on" for Max Timer Driven Thread Count fixed 
performance. Max Timer Driven Thread Count was set to 20. I changed it to 30 - 
performance increased. I changed to more to 40 - it increased. I moved it back 
to 20 - performance was still up and what it originally was before ever slowing 
down.

(this is long to give background and details)
NiFi version: 1.19.1

NiFi was deployed into a Kubernetes cluster as a single instance - no NiFi 
clustering. We would set a CPU request of 4, and limit of 8, memory request of 
8, limit of 12. The repos are all volumed mounted out to ssd.

The original deployment was as described above and Max Timer Driven Thread 
Count was set to 20. We ran a very simple data flow (generatoeFile->PutFile) 
AFAP to try to stress as much as possible before starting our other data flows. 
That ran for a week with no issue doing 20K/5m.
We turned on the other data flows and everything was processing as expected, 
good throughput rates and things were happy.
Then the throughput dropped DRAMATICALLY to (instead of 11K/5m in an 
UpdateAttribute, it went to 350/5m) after 3 days. The data being processed did 
not change in volume/cadence/velocity/etc.
Rancher Cluster explorer dashboards didn't show resources standing out as 
limiting or constraining.
Tried restarting workload in Kubernetes, and data flows were slow right from 
start - so there wasn't a ramp up or any degradation over time - it was just 
slow to begin.
Tried removing all the repos/state so NiFi came up clean incase it was the 
historical data that was issue - still slow from start.
Tried changing node in Kube Cluster incase node was bad - still slow from start.
Removed CPU limit (allowing NiFi to potentially use all 16 cores on node) from 
deployment to see if there was CPU throttling happening that I wasn't able to 
see on the Grafana dashboards - still slow from start.
While NiFi was running, I changed the Max Timer Driven Thread Count from 
20->30, performance picked up. Changed it again from 30->40, performance picked 
up. I changed from 40->10, performance stayed up. I changed from 10-20, 
performance stayed up and was at the original amount before slow down every 
happened.

So end of the day, the Max Timer Driven Thread Count is at exactly what it was 
before but the 

Re: Finding slow down in processing

[Mark Payne]

Aaron,

Interestingly, up to version 1.21 of NiFi, if you increase the size of the 
thread pool, it increased immediately. But if you decreased the size of the 
thread pool, the decrease didn’t take effect until you restart NiFi. So that’s 
probably why you’re seeing the behavior you are. Even though you reset it to 10 
or 20, it’s still running at 40.

This was done to issues with Java many years ago, where it caused problems to 
decrease the thread pool size.  So just recently we updated NiFi to immediately 
scale down the thread pools as well.

Thanks
-Mark


On Jan 11, 2024, at 1:35 PM, Aaron Rich  wrote:

So the good news is it's working now. I know what I did but I don't know why it 
worked so I'm hoping others can enlighten me based on what I did.

TL;DR - "turn it off/turn in on" for Max Timer Driven Thread Count fixed 
performance. Max Timer Driven Thread Count was set to 20. I changed it to 30 - 
performance increased. I changed to more to 40 - it increased. I moved it back 
to 20 - performance was still up and what it originally was before ever slowing 
down.

(this is long to give background and details)
NiFi version: 1.19.1

NiFi was deployed into a Kubernetes cluster as a single instance - no NiFi 
clustering. We would set a CPU request of 4, and limit of 8, memory request of 
8, limit of 12. The repos are all volumed mounted out to ssd.

The original deployment was as described above and Max Timer Driven Thread 
Count was set to 20. We ran a very simple data flow (generatoeFile->PutFile) 
AFAP to try to stress as much as possible before starting our other data flows. 
That ran for a week with no issue doing 20K/5m.
We turned on the other data flows and everything was processing as expected, 
good throughput rates and things were happy.
Then the throughput dropped DRAMATICALLY to (instead of 11K/5m in an 
UpdateAttribute, it went to 350/5m) after 3 days. The data being processed did 
not change in volume/cadence/velocity/etc.
Rancher Cluster explorer dashboards didn't show resources standing out as 
limiting or constraining.
Tried restarting workload in Kubernetes, and data flows were slow right from 
start - so there wasn't a ramp up or any degradation over time - it was just 
slow to begin.
Tried removing all the repos/state so NiFi came up clean incase it was the 
historical data that was issue - still slow from start.
Tried changing node in Kube Cluster incase node was bad - still slow from start.
Removed CPU limit (allowing NiFi to potentially use all 16 cores on node) from 
deployment to see if there was CPU throttling happening that I wasn't able to 
see on the Grafana dashboards - still slow from start.
While NiFi was running, I changed the Max Timer Driven Thread Count from 
20->30, performance picked up. Changed it again from 30->40, performance picked 
up. I changed from 40->10, performance stayed up. I changed from 10-20, 
performance stayed up and was at the original amount before slow down every 
happened.

So end of the day, the Max Timer Driven Thread Count is at exactly what it was 
before but the performance changed. It's like something was "stuck". It's very, 
very odd to me to see things be fine, degrade for days and through multiple 
environment changes/debugging, and then return to fine when I change a 
parameter to a different value->back to original value. Effectively, I "turned 
it off/turned it on" with the Max Timer Driven Thread Count value.

My question is - what is happening under the hood when the Max Timer Driven 
Thread Count is changed? What does that affect? Is there something I could look 
at from Kubernetes' side potentially that would relate to that value?

Could an internal NiFi thread gotten stuck and changing that value rebuilt the 
thread pool? If that is even possible? If that is even possible, is any way to 
know what caused the thread to "get stuck" in the first place?

Any insight would be greatly appreciated!

Thanks so much for all the suggestions and help on this.

-Aaron



On Wed, Jan 10, 2024 at 1:54 PM Aaron Rich 
mailto:aaron.r...@gmail.com>> wrote:
Hi Joe,

Nothing is load balanced- it's all basic queues.

Mark,
I'm using NiFi 1.19.1.

nifi.performance.tracking.percentage sounds exactly what I might need. I'll 
give that a shot.

Richard,
I hadn't looked at the rotating logs and/or cleared them out. I'll give that a 
shot too.

Thank you all. Please keep the suggestions coming.

-Aaron

On Wed, Jan 10, 2024 at 1:34 PM Richard Beare 
mailto:richard.be...@gmail.com>> wrote:
I had a similar sounding issue, although not in a Kube cluster. Nifi was 
running in a docker container and the issue was the log rotation interacting 
with the log file being mounted from the host. The mounted log file was not 
deleted on rotation, meaning that once rotation was triggered by log file size 
it would be continually triggered because the new log file was never emptied. 
The clue was that the content of rotated logfiles was mostly the same, with 
only a 

Re: Finding slow down in processing

[Mark Payne]

Aaron,

What version of NiFi are you running? One thing that you can look into if 
you’re running a pretty recent version, (though the user friendliness is not 
great) is to update nifi.properties and set the 
“nifi.performance.tracking.percentage” property from 0 to something like 5 or 
10. Restart NiFi and let it run for a while.

Then you can run “bin/nifi.sh diagnostics diagnostics1.txt”
In that diagnostics1.txt it will give a rather detailed breakdown of where 
you’re spending your time. Each processor will show how much of your CPU it’s 
using, as well as how much CPU time it’s using. It’ll also show how much time 
was spent committing transactions, reading from disk, writing to disk, how much 
time that processor was paused for garbage collection.  Lots of really detailed 
metrics in there. That might help you to have an “aha” moment as to what 
exactly the resource is that’s being causing poor performance.

Though I will warn you that setting the value above 0, in and of itself, might 
make the system slower if you have a huge graph with many processors. But it 
should definitely help you to narrow down what the resource constraint is. You 
can then turn it back off if necessary.

Thanks
-Mark


On Jan 10, 2024, at 3:13 PM, Aaron Rich  wrote:

Hi Joe,

It's a pretty fixed size objects at a fixed interval- One 5mb-ish file, we 
break down to individual rows.

I went so far as to create a "stress test" where I have a generateFlow( 
creating a fix, 100k fille, in batches of 1000, every .1s) feeding right into a 
putFile. I wanted to see the sustained max. It was very stable, fast for over a 
week running - but now it's extremely slow. That was able as simple of a data 
flow I could think of to hit all the different resources (CPU, memory

I was thinking too, maybe it was memory but it's slow right at the start when 
starting NiFi. I would expect the memory to cause it to be slower over time, 
and the stress test showed it wasn't something that was fluenting over time.

I'm happy to make other flows that anyone can suggest to help troubleshoot, 
diagnose issue.

Lars,

We haven't changed it between when performance was good and now when it's slow. 
That is what is throwing me - nothing changed from NiFi configuration standby.
My guess is we are having some throttling/resource contention from our provider 
but I can't determine what/where/how. The Grafana cluster dashboards I have 
don't indicate issues. If there are suggestions for specific cluster metrics to 
plot/dashboards to use, I'm happy to build them and contribute them back (I do 
have a dashboard I need to figure out how to share for creating the "status 
history" plots in Grafana).
The repos aren't full and I tried even blowing them away just to see if that 
made a difference.
I'm not seeing anything new in the logs that indicate an issue...but maybe I'm 
missing it so I will try to look again

By chance, are there any low level debugging metrics/observability/etc that 
would show how long things like writing to the repository disks is taking? 
There is a part of me that feels this could be a Disk I/O resource issue but I 
don't know how I can verify that is/isn't the issue.

Thank you all for the help and suggestions - please keep them coming as I'm 
grasping at straws right now.

-Aaron


On Wed, Jan 10, 2024 at 10:10 AM Joe Witt 
mailto:joe.w...@gmail.com>> wrote:
Aaron,

The usual suspects are memory consumption leading to high GC leading to lower 
performance over time, or back pressure in the flow, etc.. But your description 
does not really fit either exactly.  Does your flow see a mix of large objects 
and smaller objects?

Thanks

On Wed, Jan 10, 2024 at 10:07 AM Aaron Rich 
mailto:aaron.r...@gmail.com>> wrote:
Hi all,

I’m running into an odd issue and hoping someone can point me in the right 
direction.

I have NiFi 1.19 deployed in a Kube cluster with all the repositories volume 
mounted out. It was processing great with processors like UpdateAttribute 
sending through 15K/5m PutFile sending through 3K/5m.

With nothing changing in the deployment, the performance has dropped to 
UpdateAttribute doing 350/5m and Putfile to 200/5m.

I’m trying to determine what resource is suddenly dropping our performance like 
this. I don’t see anything on the Kube monitoring that stands out and I have 
restarted, cleaned repos, changed nodes but nothing is helping.

I was hoping there is something from the NiFi POV that can help identify the 
limiting resource. I'm not sure if there is additional diagnostic/debug/etc 
information available beyond the node status graphs.

Any help would be greatly appreciated.

Thanks.

-Aaron

Re: Hardware requirement for NIFI instance

[Mark Payne]

Thanks for following up. That actually makes sense. I don’t think Output Batch 
Size will play a very big role here. But Fetch Size, if I understand correctly, 
is essentially telling the JDBC Driver “Here’s how many rows you should pull 
back at once.” And so it’s going to buffer all of those rows into memory until 
it has written out all of them.

So if you set Fetch Size = 0, it’s going to pull back all rows in your database 
into memory. To be honest, I cannot imagine a single scenario where that’s 
desirable. We should probably set the default to something reasonable like 
1,000 or 10,000 at most. And in 2.0, where we have the ability to migrate old 
configurations we should automatically change any config that has Fetch Size of 
0 to the default value.

@Matt Burgess, et al., any concerns with that?

Thanks
-Mark


On Jan 5, 2024, at 9:45 AM, e-soci...@gmx.fr wrote:

So after some tests, here the result perhaps could help someone.

With nifi (2CPU / 8Go Ram)

I have tested with these couples properties :

> 1 executeSQL with "select * from table"
Output Batch Size : 1
Fetch Size : 10

> 2 executeSQL with "select * from table"
Output Batch Size : 1
Fetch Size : 20

> 2 executeSQL with "select * from table"
Output Batch Size : 1
Fetch Size : 40
and started 5 executeSQL in the same time

The 5 processors work perfectly and receive 5 avro files with same size.
And during the test, the memory is stable and the Web UI works perfectly


FAILED TEST "OUT OF MEMORY" if the properties are :

> 1 executeSQL with "select * from table"
Output Batch Size : 0
Fetch Size : 0
Regards


Envoyé: vendredi 5 janvier 2024 à 08:12
De: "Matt Burgess" 
À: users@nifi.apache.org
Objet: Re: Hardware requirement for NIFI instance
You may not need to merge if your Fetch Size is set appropriately. For
your case I don't recommend setting Max Rows Per Flow File because you
still have to wait for all the results to be processed before the
FlowFile(s) get sent "downstream". Also if you set Output Batch Size
you can't use Merge downstream as ExecuteSQL will send FlowFiles
downstream before it knows the total count.

If you have a NiFi cluster and not a standalone instance you MIGHT be
able to represent your complex query using GenerateTableFetch and use
a load-balanced connection to grab different "pages" of the table in
parallel with ExecuteSQL. Those can be merged later as long as you get
all the FlowFiles back to a single node. Depending on how complex your
query is then it's a long shot but I thought I'd mention it just in
case.

Regards,
Matt


On Thu, Jan 4, 2024 at 1:41 PM Pierre Villard
 wrote:
>
> You can merge multiple Avro flow files with MergeRecord with an Avro Reader 
> and an Avro Writer
>
> Le jeu. 4 janv. 2024 à 22:05,  a écrit :
>>
>> And the important thing for us it has only one avro file by table.
>>
>> So it is possible to merge avro files to one avro file ?
>>
>> Regards
>>
>>
>> Envoyé: jeudi 4 janvier 2024 à 19:01
>> De: e-soci...@gmx.fr
>> À: users@nifi.apache.org
>> Cc: users@nifi.apache.org
>> Objet: Re: Hardware requirement for NIFI instance
>>
>> Hello all,
>>
>> Thanks a lot for the reply.
>>
>> So for more details.
>>
>> All the properties for the ExecuteSQL are set by default, except "Set Auto 
>> Commit: false".
>>
>> The sql command could not be more simple than "select * from 
>> ${db.table.fullname}"
>>
>> The nifi version is 1.16.3 and 1.23.2
>>
>> I have also test the same sql command in the another nifi (8 cores/ 16G Ram) 
>> and it is working.
>> The result is the avro file with 1.6GB
>>
>> The detail about the output flowfile :
>>
>> executesql.query.duration
>> 245118
>> executesql.query.executiontime
>> 64122
>> executesql.query.fetchtime
>> 180996
>> executesql.resultset.index
>> 0
>> executesql.row.count
>> 14961077
>>
>> File Size
>> 1.62 GB
>>
>> Regards
>>
>> Minh
>>
>>
>> Envoyé: jeudi 4 janvier 2024 à 17:18
>> De: "Matt Burgess" 
>> À: users@nifi.apache.org
>> Objet: Re: Hardware requirement for NIFI instance
>> If I remember correctly, the default Fetch Size for Postgresql is to
>> get all the rows at once, which can certainly cause the problem.
>> Perhaps try setting Fetch Size to something like 1000 or so and see if
>> that alleviates the problem.
>>
>> Regards,
>> Matt
>>
>> On Thu, Jan 4, 2024 at 8:48 AM Etienne Jouvin  
>> wrote:
>> >
>> > Hello.
>> >
>> > I also think the problem is more about the processor, I guess ExecuteSQL.
>> >
>> > Should play with batch configuration and commit flag to commit 
>> > intermediate FlowFile.
>> >
>> > The out of memory exception makes me believe the full table is retrieved, 
>> > and if it is huge the FlowFile content is very large.
>> >
>> >
>> >
>> >
>> > Le jeu. 4 janv. 2024 à 14:37, Pierre Villard  
>> > a écrit :
>> >>
>> >> It should be memory efficient so I think this is likely a configuration 
>> >> aspect of your processor. Can you share the configuration for all 
>> >> properties?
>> >> As a side note: if NiFi ran out 

Re: Nifi - Content-repo on AWS-EBS volumes

[Mark Payne]

Greg,

Whether or not multiple content repos will have any impact depends very much on 
where your system’s bottleneck is. If your bottleneck is disk I/O, it will 
absolutely help. If your bottleneck is CPU, it won’t. If, for example, you’re 
running on bare metal and have 48 cores on your machine and you’re running with 
spinning disks, you’ll definitely want to use multiple spinning disks. But if 
you’re running in AWS on a VM that has 4 cores and you’re using gp3 EBS 
volumes, it’s unlikely that multiple content repos will help.

Thanks
-Mark



> On Dec 15, 2023, at 3:25 PM, Gregory M. Foreman 
>  wrote:
> 
> Mark:
> 
> I was just discussing multiple content repos on EBS volumes with a colleague. 
>  I found your post from a long time ago:
> 
> https://lists.apache.org/thread/nq3mpry0wppzrodmldrcfnxwzp3n1cjv
> 
> “Re #2: I don't know that i've used any SAN to back my repositories other 
> than the EBS provided by Amazon EC2. In that environment, I found that having 
> one or having multiple repos was essentially equivalent.”
> 
> Does that statement still hold true today?  Essentially there is no real 
> performance benefit to having multiple content repos on multiple EBS volumes?
> 
> Thanks,
> Greg
> 
> 
> 
>> On Dec 11, 2023, at 8:50 PM, Mark Payne  wrote:
>> 
>> Hey Phil,
>> 
>> NiFi will not spread the content of a single file over multiple partitions. 
>> It will write the content of FlowFile 1 to content repo 1, then write the 
>> next FlowFile to repo 2, etc. so it does round-robin but does not spread a 
>> single FlowFile across multiple repos.
>> 
>> Thanks
>> -Mark
>> 
>> Sent from my iPhone
>> 
>>> On Dec 11, 2023, at 8:45 PM, Phillip Lord  wrote:
>>> 
>>> 
>>> Hello Nifi comrades,
>>> 
>>> Here's my scenario...
>>> Let's say I have a Nifi cluster running on EC2 instances with attached EBS 
>>> volumes serving as their repos.  They've split up their content-repos into 
>>> three content-repos per node(cont1, cont2, cont3).  Each being a dedicated 
>>> EBS volume.  My understanding is that the content-claims for a single file 
>>> can potentially span across more than one of these repos.(correct me if 
>>> I've lost my mind over the years)
>>> For instance if you have a 1 MB file, and lets say your 
>>> max.content.claim.size is 100KB, that's 10 - 100KB claims(ish) potentially 
>>> split up across the 3 EBS volumes.  So if Nifi is trying to move that file 
>>> to S3 or something for instance... it needs to be read from each of the 
>>> volumes.  
>>> Whereas if it was a single EBS volume for the cont-repo... it would read 
>>> from the single volume, which I would think would be more performant?  Or 
>>> does spreading out any IO contention across volumes provide more of a 
>>> benefit?
>>> I know there's different levels of EBS volumes... but not factoring that in 
>>> for right now.
>>> 
>>> Appreciate any insight... trying to determine the best configuration.  
>>> 
>>> Thanks,
>>> Phil
>>> 
>>> 
> 

Re: ConsumeKafka to PublishKafka doesn't keep the order of the messages in the destination topic

[Mark Payne]

Edi,

Looking at your config again, you’ll want to also ensure that on the Publisher, 
you set the partition to `${kafka.partition}` so that it goes to the same 
partition on the destination system. You’ll also went to ensure that you set 
“Failure Strategy” to “Rollback” - otherwise any failure would route to 
‘failure’ relationship and change the ordering. You’ll also need to limit the 
concurrent tasks on the publisher to 1 concurrent task, to ensure that you’re 
not sending multiple FlowFiles out of order.

Thanks
-Mark


On Dec 15, 2023, at 2:26 AM, edi mari  wrote:

Hi Mark,
I tried the combination of FIFO and setting the back pressure to 10k, but it 
didn't preserve the order.

Thanks
Edi

On Wed, Dec 13, 2023 at 3:47 PM Mark Payne 
mailto:marka...@hotmail.com>> wrote:
Hey Edi,

By default, nifi doesn’t preserve ordering but you can have it do so by 
updating the connection’s configuration and adding the First In First Out 
Prioritizer.

Also of note you will want to keep the backpressure threshold set to 10,000 
objects rather than increasing it as shown in the image.

Thanks
Mark


Sent from my iPhone

On Dec 13, 2023, at 8:19 AM, edi mari 
mailto:edim2...@gmail.com>> wrote:



Hello ,
I'm using NIFI v1.20.0 to replicate 250 million messages between Kafka topics.
The problem is that NIFI replicates messages in a non-sequential order, 
resulting in the destination topic storing messages differently than the source 
topic.

for example
source topic - partition 0
offset:5 key:a value:v1
offset:6 key:a value:v2
offset:7 key:a value:v3

destination topic - partition 0
offset:5 key:a value:v2
offset:6 key:a value:v1
offset:7 key:a value:v3

The topics are configured with a cleanup policy: compact.

I'm using ConsumeKafka and PublishKafka processors to replicate topics.











Thanks
Edi

Re: Nifi - Content-repo on AWS-EBS volumes

[Mark Payne]

Hey Phil,

NiFi will not spread the content of a single file over multiple partitions. It 
will write the content of FlowFile 1 to content repo 1, then write the next 
FlowFile to repo 2, etc. so it does round-robin but does not spread a single 
FlowFile across multiple repos.

Thanks
-Mark

Sent from my iPhone

> On Dec 11, 2023, at 8:45 PM, Phillip Lord  wrote:
> 
> 
> Hello Nifi comrades,
> 
> Here's my scenario...
> Let's say I have a Nifi cluster running on EC2 instances with attached EBS 
> volumes serving as their repos.  They've split up their content-repos into 
> three content-repos per node(cont1, cont2, cont3).  Each being a dedicated 
> EBS volume.  My understanding is that the content-claims for a single file 
> can potentially span across more than one of these repos.(correct me if I've 
> lost my mind over the years)
> For instance if you have a 1 MB file, and lets say your 
> max.content.claim.size is 100KB, that's 10 - 100KB claims(ish) potentially 
> split up across the 3 EBS volumes.  So if Nifi is trying to move that file to 
> S3 or something for instance... it needs to be read from each of the volumes. 
>  
> Whereas if it was a single EBS volume for the cont-repo... it would read from 
> the single volume, which I would think would be more performant?  Or does 
> spreading out any IO contention across volumes provide more of a benefit?
> I know there's different levels of EBS volumes... but not factoring that in 
> for right now.
> 
> Appreciate any insight... trying to determine the best configuration.  
> 
> Thanks,
> Phil
> 
> 

Re: Configuring ExecuteStreamCommand on jar flowfiles

[Mark Payne]

Jim,

UnpackContent does just that. It does not write to an external directory.

Thanks
Mark

Sent from my iPhone

On Dec 3, 2023, at 5:18 PM, James McMahon  wrote:


UnpackContent examples seem to require that I output the results of the unpack 
to a directory outside of the nifi flow. Is it possible to unpack the jar in 
the flow, keeping the results as new flowfiles in the output stream?

On Sun, Dec 3, 2023 at 1:23 PM James Srinivasan 
mailto:james.sriniva...@gmail.com>> wrote:
Since a jar file is mostly just a standard zip file, can you use a built in 
processor instead?

On Sun, 3 Dec 2023, 15:36 James McMahon, 
mailto:jsmcmah...@gmail.com>> wrote:
I have a large volume of a wide variety of incoming data files. A subset of 
these are jar files. Can the ExecuteStreamCommand be configured to run the 
equivalent of

jar -xf ${flowfile}

and will that automatically direct each output file to a new flowfile, or does 
ESC need to be told to direct each output file from jar standard out to the 
Success path out of ESC?

Thank you in advance for any assistance.

Re: ListSFTP Processor CRON doesn't start

[Mark Payne]

GCS Processors are under org.apache.nifi.processors.gcp.storage


On Nov 14, 2023, at 10:51 AM, e-soci...@gmx.fr<mailto:e-soci...@gmx.fr> wrote:

Hello

Thanks Mark

What could be a good way to debug "google processor"

 (not working)


Thanks


Envoyé: mardi 14 novembre 2023 à 16:11
De: "Mark Payne" mailto:marka...@hotmail.com>>
À: "users@nifi.apache.org<mailto:users@nifi.apache.org>" 
mailto:users@nifi.apache.org>>
Objet: Re: ListSFTP Processor CRON doesn't start
Hi Minh,

No - you can configure logging for any Java class, more or less. So that would 
equate to probably tens of thousands of possible loggers that you could 
configure.
Of course, they are hierarchical, though, so you could configure, for example, 
“org.apache.nifi.processors” and that should affect all processors. You could 
also go another level down, and configure perhaps for 
“org.apache.nifi.processors.aws” or “org.apache.nifi.processors.aws.s3”.

Thanks
-Mark


On Nov 14, 2023, at 9:37 AM, e-soci...@gmx.fr<mailto:e-soci...@gmx.fr> wrote:


Hello Mark,

Have we got the documentation about exhaustive list about logger we got have in 
NIFI ?

Regards

Minh
Envoyé: mardi 14 novembre 2023 à 15:25
De: "Mark Payne" mailto:marka...@hotmail.com>>
À: "users" mailto:users@nifi.apache.org>>
Objet: Re: ListSFTP Processor CRON doesn't start
Hi Quentin,

What is the CRON schedule that you configured? What version of NiFi are you 
running?

You’ll not see any debug related logs for that Processor by changing its log 
level, as the Processor is not responsible for scheduling itself. But you can 
enable DEBUG level logs for 
org.apache.nifi.controller.scheduling.QuartzSchedulingAgent and that will 
provide a debug log each time the Processor runs, indicating when it’s expected 
to run again.

Thanks
-Mark


> On Nov 14, 2023, at 2:28 AM, Quentin HORNEMAN GUTTON 
> mailto:qhornemangut...@gmail.com>> wrote:
>
> Hello,
>
> I am facing an issue that I am having difficulty resolving. I have a ListSFTP 
> processor that is supposed to run every day at 12:15 AM, but it is not 
> launching. I added TRACE logs to this type of processor, but since it is not 
> launching, I cannot determine what is happening. If I change the launch time 
> of the processor (for example, to 04:00 PM), it launches successfully. This 
> is a NiFi cluster running on Redhat. Does anyone have an idea of how I can 
> identify the root cause of this processor not launching ?
>
> Best regards,

Re: ListSFTP Processor CRON doesn't start

[Mark Payne]

Hi Minh,

No - you can configure logging for any Java class, more or less. So that would 
equate to probably tens of thousands of possible loggers that you could 
configure.
Of course, they are hierarchical, though, so you could configure, for example, 
“org.apache.nifi.processors” and that should affect all processors. You could 
also go another level down, and configure perhaps for 
“org.apache.nifi.processors.aws” or “org.apache.nifi.processors.aws.s3”.

Thanks
-Mark


On Nov 14, 2023, at 9:37 AM, e-soci...@gmx.fr<mailto:e-soci...@gmx.fr> wrote:


Hello Mark,

Have we got the documentation about exhaustive list about logger we got have in 
NIFI ?

Regards

Minh
Envoyé: mardi 14 novembre 2023 à 15:25
De: "Mark Payne" mailto:marka...@hotmail.com>>
À: "users" mailto:users@nifi.apache.org>>
Objet: Re: ListSFTP Processor CRON doesn't start
Hi Quentin,

What is the CRON schedule that you configured? What version of NiFi are you 
running?

You’ll not see any debug related logs for that Processor by changing its log 
level, as the Processor is not responsible for scheduling itself. But you can 
enable DEBUG level logs for 
org.apache.nifi.controller.scheduling.QuartzSchedulingAgent and that will 
provide a debug log each time the Processor runs, indicating when it’s expected 
to run again.

Thanks
-Mark


> On Nov 14, 2023, at 2:28 AM, Quentin HORNEMAN GUTTON 
> mailto:qhornemangut...@gmail.com>> wrote:
>
> Hello,
>
> I am facing an issue that I am having difficulty resolving. I have a ListSFTP 
> processor that is supposed to run every day at 12:15 AM, but it is not 
> launching. I added TRACE logs to this type of processor, but since it is not 
> launching, I cannot determine what is happening. If I change the launch time 
> of the processor (for example, to 04:00 PM), it launches successfully. This 
> is a NiFi cluster running on Redhat. Does anyone have an idea of how I can 
> identify the root cause of this processor not launching ?
>
> Best regards,

Re: ListSFTP Processor CRON doesn't start

[Mark Payne]

Hi Quentin,

What is the CRON schedule that you configured? What version of NiFi are you 
running?

You’ll not see any debug related logs for that Processor by changing its log 
level, as the Processor is not responsible for scheduling itself. But you can 
enable DEBUG level logs for 
org.apache.nifi.controller.scheduling.QuartzSchedulingAgent and that will 
provide a debug log each time the Processor runs, indicating when it’s expected 
to run again.

Thanks
-Mark


> On Nov 14, 2023, at 2:28 AM, Quentin HORNEMAN GUTTON 
>  wrote:
> 
> Hello,
> 
> I am facing an issue that I am having difficulty resolving. I have a ListSFTP 
> processor that is supposed to run every day at 12:15 AM, but it is not 
> launching. I added TRACE logs to this type of processor, but since it is not 
> launching, I cannot determine what is happening. If I change the launch time 
> of the processor (for example, to 04:00 PM), it launches successfully. This 
> is a NiFi cluster running on Redhat. Does anyone have an idea of how I can 
> identify the root cause of this processor not launching ?
> 
> Best regards,

Re: Expression Language does not work within QueryNifiReportingTask

[Mark Payne]

You’re right, Dogukan. It looks like the “SQL Query” property is documented as 
supporting Expression Language, but the EL is never evaluated. I filed a JIRA 
[1] for the issue.

Thanks
-Mark


[1] https://issues.apache.org/jira/browse/NIFI-12314


On Nov 3, 2023, at 5:09 AM, Doğukan Levendoğlu | Obase 
mailto:dogukan.levendo...@obase.com>> wrote:

Apologies fort he horrible image quality:


From: Doğukan Levendoğlu | Obase
Sent: 03 November 2023 12:06
To: 'users@nifi.apache.org' 
mailto:users@nifi.apache.org>>
Subject: Expression Language does not work within QueryNifiReportingTask

Hello,

I’m trying to add additional fields to the query results obtained by 
QueryNifiReportingTask like below:


SQL Query property in QueryNifiReportingTask indicates that it supports the 
expression language. My understanding is that the query needs to be evaluated 
before execution. However I am getting this error ( which tells me that’s not 
what’s happening):

QueryNiFiReportingTask[id=8fbb9a3a-018b-1000--bb48d6d7] Error 
processing the query due to java.sql.SQLException: Error while preparing 
statement [SELECT
   *,
   'myCluster' as clusterName,
   ${hostname(true)} as 'hostname'
FROM PROCESSOR_STATUS]: 
org.apache.nifi.reporting.sql.MetricsSqlQueryService$PreparedStatementException:
 java.sql.SQLException: Error while preparing statement [SELECT
   *,
   'myCluster' as clusterName,
   ${hostname(true)} as 'hostname'
FROM PROCESSOR_STATUS]
- Caused by: java.sql.SQLException: Error while preparing statement [SELECT
   *,
   'myCluster' as clusterName,
   ${hostname(true)} as 'hostname'
FROM PROCESSOR_STATUS]
- Caused by: java.lang.RuntimeException: parse failed: Encountered "$" at line 
4, column 2.
Was expecting one of:
"ABS" ...
"ARRAY" ...
"AVG" ...
"CARDINALITY" ...
"CASE" ...
"CAST" ...
"CEIL" ...
"CEILING" ...
"CHAR" ...
.
.
.

We want to be able to monitor some processors on a per node basis. Is there a 
cleaner way to do this? I am on version 1.23.2.

Thank you,
Dogukan

Re: NiFi hanging during large sql query

[Mark Payne]

Thanks for sharing the solution Mike. Is there something we need to update in 
nifi to prevent this from biting others?

Thanks
Mark

Sent from my iPhone

On Sep 2, 2023, at 9:48 AM, Joe Witt  wrote:


Nice.  Gald you found it.

On Sat, Sep 2, 2023 at 5:07 AM Mike Thomsen 
mailto:mikerthom...@gmail.com>> wrote:
It was the PostgreSQL JDBC driver. If you don't paginate the query 
aggressively, it will try to load a significant chunk of the table into memory 
rather than just pulling chunks, even with fetchSize set low.

On Fri, Sep 1, 2023 at 6:01 PM Mike Thomsen 
mailto:mikerthom...@gmail.com>> wrote:
I have a three node cluster with an executesqlrecord processor with primary 
execution only. The sql it runs is a straight forward select on a table with 
about 44m records. If I leave it running, after about 10 min the node becomes 
unresponsive and leaves the cluster. The query runs just fine in jetbrains data 
grip on that postgresql server, so I don’t think it’s anything weird with the 
db or query. Any ideas about what could be causing this? Even with a high limit 
like 5m records the query doesn’t lock up the NiFi node.

Sent from my iPhone

Re: SOLVED - Re: Putemail - subject contains utf-8 chars - the result is lot of question marks

[Mark Payne]

Fantastic! Thanks for confirming, István. I filed a Jira [1] to add this by 
default to bootstrap.conf.

Thanks
-Mark

[1] https://issues.apache.org/jira/browse/NIFI-11977




On Aug 22, 2023, at 10:00 AM, Pongrácz István 
mailto:pongracz.ist...@gmail.com>> wrote:

Hello Mark,

Thank you very much for your quick reply!
I confirm, your solution fixed the problem, now I got beautiful utf-8 
characters in the subject, too :)

Probably include this trick in the documentation [1]  would be nice and/or the 
default bootstrap.conf could include this settings.

Regarding to the question about the mail body:
In general, it was working with my characters.
The content type did not matter, text/plain or text/html, both of them are 
working.
(Please note, I did not write 'text/plain;charset=utf-8'  nor  
'text/html;charset=utf-8' )

The html content contains the , but I do not think it does matter.



Thank you!
István

[1] 
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.23.1/org.apache.nifi.processors.standard.PutEmail/index.html



2023. 08. 22, kedd keltezéssel 13.29-kor Mark Payne ezt írta:
Hey István,

The PutEmail processor is using Jakarta Mail as the underlying library. Based 
on some googling, I found a Stack Overflow issue [1]
with similar concerns. The recommendation there is to set a system property to 
specify mail.mime.charset. Can you try doing that?
To do so, you’d update conf/bootstrap.conf and add a new line to the bottom of 
the file:

java.arg.mail.charset=-Dmail.mime.charset=UTF-8

Unfortunately, that does require a restart of NiFi. Am interested to know if 
that helps, though.

One question for you: you mentioned that the subject is incorrect, but what 
about the message body? Do you have UTF-8 characters in the message body
that are also incorrect? Or does the message body appear to be fine?

Thanks
-Mark

[1] 
https://stackoverflow.com/questions/15044027/utf-8-charset-doesnt-work-with-javax-mail


On Aug 22, 2023, at 8:59 AM, Pongrácz István 
mailto:pongracz.ist...@gmail.com>> wrote:

Hi,

I would like to send out email with utf-8 chars in the subject, usingputemail.
Technically it is working fine, except the utf-8 chars in the subject changed 
to '?' characters.

Example:
I wrote the following into the subject: Hűvös van íűáéúőóüö

The result is:
H?v?s van ?

The source of the receiverd email looks like this:

Subject: =?ANSI_X3.4-1968?Q?H=3Fv=3Fs_van_=3F=3F=3F=3F=3F=3F=3F=3F=3F?=

This seems a little bit unprofessional.

Do I miss something trivial? I guess, I just missed something with the 
environment (locales?).

I tried to find expression language possibility to do something with the 
subject or character encoding, but I failed. As I remember, nifi uses utf-8 and 
doesn't really care about character encoding conversion.

I did not change too much in the environment (nothing, barebone debian 11).

I use the following:
Debian 11 + Nifi 1.20


root@nifi10<mailto:root@nifi10>:~# locale -a

C

C.UTF-8

POSIX

root@nifi10<mailto:root@nifi10>:~# java -version

openjdk version "11.0.16" 2022-07-19

OpenJDK Runtime Environment (build 11.0.16+8-post-Debian-1deb11u1)

OpenJDK 64-Bit Server VM (build 11.0.16+8-post-Debian-1deb11u1, mixed mode, 
sharing)


Could you give me a hint, where to find a solution? I would like to use utf-8 
chars instead of ascii.

Thank you!

István

Re: Putemail - subject contains utf-8 chars - the result is lot of question marks

[Mark Payne]

Hey István,

The PutEmail processor is using Jakarta Mail as the underlying library. Based 
on some googling, I found a Stack Overflow issue [1]
with similar concerns. The recommendation there is to set a system property to 
specify mail.mime.charset. Can you try doing that?
To do so, you’d update conf/bootstrap.conf and add a new line to the bottom of 
the file:

java.arg.mail.charset=-Dmail.mime.charset=UTF-8

Unfortunately, that does require a restart of NiFi. Am interested to know if 
that helps, though.

One question for you: you mentioned that the subject is incorrect, but what 
about the message body? Do you have UTF-8 characters in the message body
that are also incorrect? Or does the message body appear to be fine?

Thanks
-Mark

[1] 
https://stackoverflow.com/questions/15044027/utf-8-charset-doesnt-work-with-javax-mail


On Aug 22, 2023, at 8:59 AM, Pongrácz István 
mailto:pongracz.ist...@gmail.com>> wrote:

Hi,

I would like to send out email with utf-8 chars in the subject, using putemail.
Technically it is working fine, except the utf-8 chars in the subject changed 
to '?' characters.

Example:
I wrote the following into the subject: Hűvös van íűáéúőóüö

The result is:
H?v?s van ?

The source of the receiverd email looks like this:

Subject: =?ANSI_X3.4-1968?Q?H=3Fv=3Fs_van_=3F=3F=3F=3F=3F=3F=3F=3F=3F?=

This seems a little bit unprofessional.

Do I miss something trivial? I guess, I just missed something with the 
environment (locales?).

I tried to find expression language possibility to do something with the 
subject or character encoding, but I failed. As I remember, nifi uses utf-8 and 
doesn't really care about character encoding conversion.

I did not change too much in the environment (nothing, barebone debian 11).

I use the following:
Debian 11 + Nifi 1.20


root@nifi10:~# locale -a

C

C.UTF-8

POSIX

root@nifi10:~# java -version

openjdk version "11.0.16" 2022-07-19

OpenJDK Runtime Environment (build 11.0.16+8-post-Debian-1deb11u1)

OpenJDK 64-Bit Server VM (build 11.0.16+8-post-Debian-1deb11u1, mixed mode, 
sharing)


Could you give me a hint, where to find a solution? I would like to use utf-8 
chars instead of ascii.

Thank you!

István

Re: UI SocketTimeoutException - heavy IO

[Mark Payne]

Joe,

The way that the processor works is that it adds an attribute for every 
“Capturing Group” in the regular expression.
This includes a “Capturing Group” 0, which contains the entire value that the 
regex was run against.
You can actually disable capturing this as an attribute by setting the “Include 
Capture Group 0” property to false.

Thanks
-Mark


On Jul 12, 2023, at 12:36 PM, Joe Obernberger  
wrote:


Thank you Mark - it looks like attributes is to blame.  I'm adding lots of 
UpdateAttribute to delete them as soon as they are not needed and disk IO has 
dropped.
Right now, it's all going to 'spinning rust' - soon to all new SSDs, but either 
way, this needed addressing.

One oddity, is when I do ExtractText to a property (call it value) of (.*), 
I'll see value, and value.0, value.1 in the attributes list.  Not sure why it 
makes multiple copies.

-Joe

On 7/12/2023 11:27 AM, Mark Payne wrote:
Joe,

How many FlowFiles are you processing here? Let’s say, per second? How many 
processors are in those flows?

Is the FlowFile Repo a spinning disk, SSD, or NAS?

You said you’re using ExtractText to pull 10 KB into an attribute. I presume 
you’re then doing something with it. So maybe you’re extracting a few parts of 
it using jsonPath in expression language or whatever the case may be. So that 
one 10KB attribute is not the only attribute you have. So theoretically, let’s 
consider:

- Total of all attributes for a FlowFiles is 20 KB
- You process an average of 1,000 FlowFiles per second
- Each FlowFile goes through 15 processors, each of which modifies at least 
attribute.

That means you’re writing to the flowfile repository about 20 KB * 1000 * 15 
per second - or about 300 MB/sec.
This is why we constantly warn against creating large attributes. Attributes 
are meant to be on the order of say 100-200 characters - not 10 KB.
If you’re processing a few thousand FlowFiles per hour then 10 KB is fine, but 
if you’re processing a bunch of FlowFiles it adds up very quickly.

Thanks
-Mark


On Jul 12, 2023, at 11:16 AM, Joe Obernberger 
<mailto:joseph.obernber...@gmail.com> wrote:


Thank you Joe -
The content repo doesn't seem to be the issue - it's the flowfile repo.
Here is the section from one of the nodes:

nifi.content.repository.implementation=org.apache.nifi.controller.repository.FileSystemRepository
nifi.content.claim.max.appendable.size=50 KB
nifi.content.repository.directory.default=/data/4/nifi_content_repository
nifi.content.repository.archive.max.retention.period=2 days
nifi.content.repository.archive.max.usage.percentage=50%
nifi.content.repository.archive.enabled=false
nifi.content.repository.always.sync=false
nifi.content.viewer.url=../nifi-content-viewer/

nifi.flowfile.repository.implementation=org.apache.nifi.controller.repository.WriteAheadFlowFileRepository
nifi.flowfile.repository.wal.implementation=org.apache.nifi.wali.SequentialAccessWriteAheadLog
nifi.flowfile.repository.directory=/data/5/nifi_flowfile_repository
nifi.flowfile.repository.checkpoint.interval=300 secs
nifi.flowfile.repository.always.sync=false
nifi.flowfile.repository.retain.orphaned.flowfiles=true


-Joe

On 7/12/2023 11:07 AM, Joe Witt wrote:
Joe

I dont recall the specific version in which we got it truly sorted but there 
was an issue with our default settings for an important content repo property 
and how we handled mixture of large/small flowfiles written within the same 
underlying slab/claim in the content repository.

Please check what you have for conf/nifi.properties
  nifi.content.claim.max.appendable.size=

What value do you have there?  I recommend reducing it to 50KB and restarting.

Can you show your full 'nifi.content' section from the nifi.properties?

Thanks

On Wed, Jul 12, 2023 at 7:54 AM Joe Obernberger 
mailto:joseph.obernber...@gmail.com>> wrote:

Raising this thread from the dead...
Having issues with IO to the flowfile repository.  NiFi will show 500k flow 
files and a size of ~1.7G - but the size on disk on each of the 4 nodes is 
massive - over 100G, and disk IO to the flowfile spindle is just pegged doing 
writes.

I do have ExtractText processors that take the flowfile content (.*) and put it 
into an attribute, but the sizes of these is maybe in the 10k at most size.  
How can I find out what module (there are some 2200) is causing the issue?  I 
think I'm doing something fundamentally wrong with NiFi.  :)
Perhaps I should change the size of all the queues to something less than 
10k/1G?

Under cluster/FLOWFILE STORAGE, one of the nodes shows 3.74TBytes of usage, but 
it's actually ~150G on disk.  The other nodes are correct.

Ideas on what to debug?
Thank you!

-Joe (NiFi 1.18)

On 3/22/2023 12:49 PM, Mark Payne wrote:
OK. So changing the checkpoint internal to 300 seconds might help reduce IO a 
bit. But it will cause the repo to become much larger, and it will take much 
longer to startup whenever you restart NiFi.

The variance in size between nodes is likel

Re: UI SocketTimeoutException - heavy IO

[Mark Payne]

Joe,

How many FlowFiles are you processing here? Let’s say, per second? How many 
processors are in those flows?

Is the FlowFile Repo a spinning disk, SSD, or NAS?

You said you’re using ExtractText to pull 10 KB into an attribute. I presume 
you’re then doing something with it. So maybe you’re extracting a few parts of 
it using jsonPath in expression language or whatever the case may be. So that 
one 10KB attribute is not the only attribute you have. So theoretically, let’s 
consider:

- Total of all attributes for a FlowFiles is 20 KB
- You process an average of 1,000 FlowFiles per second
- Each FlowFile goes through 15 processors, each of which modifies at least 
attribute.

That means you’re writing to the flowfile repository about 20 KB * 1000 * 15 
per second - or about 300 MB/sec.
This is why we constantly warn against creating large attributes. Attributes 
are meant to be on the order of say 100-200 characters - not 10 KB.
If you’re processing a few thousand FlowFiles per hour then 10 KB is fine, but 
if you’re processing a bunch of FlowFiles it adds up very quickly.

Thanks
-Mark


On Jul 12, 2023, at 11:16 AM, Joe Obernberger  
wrote:


Thank you Joe -
The content repo doesn't seem to be the issue - it's the flowfile repo.
Here is the section from one of the nodes:

nifi.content.repository.implementation=org.apache.nifi.controller.repository.FileSystemRepository
nifi.content.claim.max.appendable.size=50 KB
nifi.content.repository.directory.default=/data/4/nifi_content_repository
nifi.content.repository.archive.max.retention.period=2 days
nifi.content.repository.archive.max.usage.percentage=50%
nifi.content.repository.archive.enabled=false
nifi.content.repository.always.sync=false
nifi.content.viewer.url=../nifi-content-viewer/

nifi.flowfile.repository.implementation=org.apache.nifi.controller.repository.WriteAheadFlowFileRepository
nifi.flowfile.repository.wal.implementation=org.apache.nifi.wali.SequentialAccessWriteAheadLog
nifi.flowfile.repository.directory=/data/5/nifi_flowfile_repository
nifi.flowfile.repository.checkpoint.interval=300 secs
nifi.flowfile.repository.always.sync=false
nifi.flowfile.repository.retain.orphaned.flowfiles=true


-Joe

On 7/12/2023 11:07 AM, Joe Witt wrote:
Joe

I dont recall the specific version in which we got it truly sorted but there 
was an issue with our default settings for an important content repo property 
and how we handled mixture of large/small flowfiles written within the same 
underlying slab/claim in the content repository.

Please check what you have for conf/nifi.properties
  nifi.content.claim.max.appendable.size=

What value do you have there?  I recommend reducing it to 50KB and restarting.

Can you show your full 'nifi.content' section from the nifi.properties?

Thanks

On Wed, Jul 12, 2023 at 7:54 AM Joe Obernberger 
mailto:joseph.obernber...@gmail.com>> wrote:

Raising this thread from the dead...
Having issues with IO to the flowfile repository.  NiFi will show 500k flow 
files and a size of ~1.7G - but the size on disk on each of the 4 nodes is 
massive - over 100G, and disk IO to the flowfile spindle is just pegged doing 
writes.

I do have ExtractText processors that take the flowfile content (.*) and put it 
into an attribute, but the sizes of these is maybe in the 10k at most size.  
How can I find out what module (there are some 2200) is causing the issue?  I 
think I'm doing something fundamentally wrong with NiFi.  :)
Perhaps I should change the size of all the queues to something less than 
10k/1G?

Under cluster/FLOWFILE STORAGE, one of the nodes shows 3.74TBytes of usage, but 
it's actually ~150G on disk.  The other nodes are correct.

Ideas on what to debug?
Thank you!

-Joe (NiFi 1.18)

On 3/22/2023 12:49 PM, Mark Payne wrote:
OK. So changing the checkpoint internal to 300 seconds might help reduce IO a 
bit. But it will cause the repo to become much larger, and it will take much 
longer to startup whenever you restart NiFi.

The variance in size between nodes is likely due to how recently it’s 
checkpointed. If it stays large like 31 GB while the other stay small, that 
would be interesting to know.

Thanks
-Mark


On Mar 22, 2023, at 12:45 PM, Joe Obernberger 
<mailto:joseph.obernber...@gmail.com> wrote:


Thanks for this Mark.  I'm not seeing any large attributes at the moment but 
will go through this and verify - but I did have one queue that was set to 100k 
instead of 10k.
I set the nifi.cluster.node.connection.timeout to 30 seconds (up from 5) and 
the nifi.flowfile.repository.checkpoint.interval to 300 seconds (up from 20).

While it's running the size of the flowfile repo varies (wildly?) on each of 
the nodes from 1.5G to over 30G.  Disk IO is still very high, but it's running 
now and I can use the UI.  Interestingly at this point the UI shows 677k files 
and 1.5G of flow.  But disk usage on the flowfile repo is 31G, 3.7G, and 2.6G 
on the 3 nodes.  I'd love to throw some SSDs at this problem.  I 

Re: BufferedReader best option to search through large flowfiles?

[Mark Payne]

Jim,

Take a look at RouteText.

Thanks
-Mark


> On Jun 5, 2023, at 8:09 AM, James McMahon  wrote:
> 
> Hello. I have a requirement to scan for multiple regex patterns in very large 
> flowfiles. Given that my flowfiles can be very large, I think my best 
> approach is to employ an ExecuteGroovyScript processor and a script using a 
> BufferedReader to scan the file one line at a time. 
> 
> I am concerned that I might exhaust jvm resources trying to otherwise process 
> large content if I try to handle it all at once. Is a BufferedReader the 
> right call? Does anyone recommend a better approach?
> 
> Thanks in advance,
> Jim

Re: [EMBEDDED ZOOKEEPER] Encrypt password

[Mark Payne]

Quentin,

I don’t believe there’s any way to encrypt the properties in 
zookeeper.properties. I will note, though, that the embedded zookeeper should 
not be used for production use. It’s convenient for testing, proofs of concept, 
etc. But for any production deployment an external ZooKeeper should be used.

Thanks
-Mark


> On Apr 13, 2023, at 5:41 AM, Quentin HORNEMAN GUTTON 
>  wrote:
> 
> Hello,
> 
> I'm using Apache NiFi 1.13.2 cluster with an embedded secure Zookeeper.
> I would like to encrypt the keystore and truststore password in 
> zookeeper.properties file but I can't find information about it. Am I missing 
> something ?
> 
> Best Regards,

Re: UI SocketTimeoutException - heavy IO

[Mark Payne]

OK. So changing the checkpoint internal to 300 seconds might help reduce IO a 
bit. But it will cause the repo to become much larger, and it will take much 
longer to startup whenever you restart NiFi.

The variance in size between nodes is likely due to how recently it’s 
checkpointed. If it stays large like 31 GB while the other stay small, that 
would be interesting to know.

Thanks
-Mark


On Mar 22, 2023, at 12:45 PM, Joe Obernberger  
wrote:


Thanks for this Mark.  I'm not seeing any large attributes at the moment but 
will go through this and verify - but I did have one queue that was set to 100k 
instead of 10k.
I set the nifi.cluster.node.connection.timeout to 30 seconds (up from 5) and 
the nifi.flowfile.repository.checkpoint.interval to 300 seconds (up from 20).

While it's running the size of the flowfile repo varies (wildly?) on each of 
the nodes from 1.5G to over 30G.  Disk IO is still very high, but it's running 
now and I can use the UI.  Interestingly at this point the UI shows 677k files 
and 1.5G of flow.  But disk usage on the flowfile repo is 31G, 3.7G, and 2.6G 
on the 3 nodes.  I'd love to throw some SSDs at this problem.  I can add more 
nifi nodes.

-Joe

On 3/22/2023 11:08 AM, Mark Payne wrote:
Joe,

The errors noted are indicating that NiFi cannot communicate with registry. 
Either the registry is offline, NiFi’s Registry Client is not configured 
properly, there’s a firewall in the way, etc.

A FlowFile repo of 35 GB is rather huge. This would imply one of 3 things:
- You have a huge number of FlowFiles (doesn’t seem to be the case)
- FlowFiles have a huge number of attributes
or
- FlowFiles have 1 or more huge attribute values.

Typically, FlowFile attribute should be kept minimal and should never contain 
chunks of contents from the FlowFile content. Often when we see this type of 
behavior it’s due to using something like ExtractText or EvaluateJsonPath to 
put large blocks of content into attributes.

And in this case, setting Backpressure Threshold above 10,000 is even more 
concerning, as it means even greater disk I/O.

Thanks
-Mark


On Mar 22, 2023, at 11:01 AM, Joe Obernberger 
<mailto:joseph.obernber...@gmail.com> wrote:


Thank you Mark.  These are SATA drives - but there's no way for the flowfile 
repo to be on multiple spindles.  It's not huge - maybe 35G per node.
I do see a lot of messages like this in the log:

2023-03-22 10:52:13,960 ERROR [Timer-Driven Process Thread-62] 
o.a.nifi.groups.StandardProcessGroup Failed to synchronize 
StandardProcessGroup[identifier=861d3b27-aace-186d-bbb7-870c6fa65243,name=TIKA 
Handle Extract Metadata] with Flow Registry because could not retrieve version 
1 of flow with identifier d64e72b5-16ea-4a87-af09-72c5bbcd82bf in bucket 
736a8f4b-19be-4c01-b2c3-901d9538c5ef due to: Connection refused (Connection 
refused)
2023-03-22 10:52:13,960 ERROR [Timer-Driven Process Thread-62] 
o.a.nifi.groups.StandardProcessGroup Failed to synchronize 
StandardProcessGroup[identifier=bcc23c03-49ef-1e41-83cb-83f22630466d,name=WriteDB]
 with Flow Registry because could not retrieve version 2 of flow with 
identifier ff197063-af31-45df-9401-e9f8ba2e4b2b in bucket 
736a8f4b-19be-4c01-b2c3-901d9538c5ef due to: Connection refused (Connection 
refused)
2023-03-22 10:52:13,960 ERROR [Timer-Driven Process Thread-62] 
o.a.nifi.groups.StandardProcessGroup Failed to synchronize 
StandardProcessGroup[identifier=bc913ff1-06b1-1b76-a548-7525a836560a,name=TIKA 
Handle Extract Metadata] with Flow Registry because could not retrieve version 
1 of flow with identifier d64e72b5-16ea-4a87-af09-72c5bbcd82bf in bucket 
736a8f4b-19be-4c01-b2c3-901d9538c5ef due to: Connection refused (Connection 
refused)
2023-03-22 10:52:13,960 ERROR [Timer-Driven Process Thread-62] 
o.a.nifi.groups.StandardProcessGroup Failed to synchronize 
StandardProcessGroup[identifier=920c3600-2954-1c8e-b121-6d7d3d393de6,name=Save 
Binary Data] with Flow Registry because could not retrieve version 1 of flow 
with identifier 7a8c82be-1707-4e7d-a5e7-bb3825e0a38f in bucket 
736a8f4b-19be-4c01-b2c3-901d9538c5ef due to: Connection refused (Connection 
refused)

A clue?

-joe

On 3/22/2023 10:49 AM, Mark Payne wrote:
Joe,

1.8 million FlowFiles is not a concern. But when you say “Should I reduce the 
queue sizes?” it makes me wonder if they’re all in a single queue?
Generally, you should leave the backpressure threshold at the default 10,000 
FlowFile max. Increasing this can lead to huge amounts of swapping, which will 
drastically reduce performance and increase disk utilization very significantly.

Also from the diagnostics, it looks like you’ve got a lot of CPU cores, but 
you’re not using much. And based on the amount of disk space available and the 
fact that you’re seeing 100% utilization, I’m wondering if you’re using 
spinning disks, rather than SSDs? I would highly recommend always running NiFi 
with ssd/nvme drives. Absent that, if you have multiple disk drives, you could 
also con

Re: UI SocketTimeoutException - heavy IO

[Mark Payne]

Joe,

The errors noted are indicating that NiFi cannot communicate with registry. 
Either the registry is offline, NiFi’s Registry Client is not configured 
properly, there’s a firewall in the way, etc.

A FlowFile repo of 35 GB is rather huge. This would imply one of 3 things:
- You have a huge number of FlowFiles (doesn’t seem to be the case)
- FlowFiles have a huge number of attributes
or
- FlowFiles have 1 or more huge attribute values.

Typically, FlowFile attribute should be kept minimal and should never contain 
chunks of contents from the FlowFile content. Often when we see this type of 
behavior it’s due to using something like ExtractText or EvaluateJsonPath to 
put large blocks of content into attributes.

And in this case, setting Backpressure Threshold above 10,000 is even more 
concerning, as it means even greater disk I/O.

Thanks
-Mark


On Mar 22, 2023, at 11:01 AM, Joe Obernberger  
wrote:


Thank you Mark.  These are SATA drives - but there's no way for the flowfile 
repo to be on multiple spindles.  It's not huge - maybe 35G per node.
I do see a lot of messages like this in the log:

2023-03-22 10:52:13,960 ERROR [Timer-Driven Process Thread-62] 
o.a.nifi.groups.StandardProcessGroup Failed to synchronize 
StandardProcessGroup[identifier=861d3b27-aace-186d-bbb7-870c6fa65243,name=TIKA 
Handle Extract Metadata] with Flow Registry because could not retrieve version 
1 of flow with identifier d64e72b5-16ea-4a87-af09-72c5bbcd82bf in bucket 
736a8f4b-19be-4c01-b2c3-901d9538c5ef due to: Connection refused (Connection 
refused)
2023-03-22 10:52:13,960 ERROR [Timer-Driven Process Thread-62] 
o.a.nifi.groups.StandardProcessGroup Failed to synchronize 
StandardProcessGroup[identifier=bcc23c03-49ef-1e41-83cb-83f22630466d,name=WriteDB]
 with Flow Registry because could not retrieve version 2 of flow with 
identifier ff197063-af31-45df-9401-e9f8ba2e4b2b in bucket 
736a8f4b-19be-4c01-b2c3-901d9538c5ef due to: Connection refused (Connection 
refused)
2023-03-22 10:52:13,960 ERROR [Timer-Driven Process Thread-62] 
o.a.nifi.groups.StandardProcessGroup Failed to synchronize 
StandardProcessGroup[identifier=bc913ff1-06b1-1b76-a548-7525a836560a,name=TIKA 
Handle Extract Metadata] with Flow Registry because could not retrieve version 
1 of flow with identifier d64e72b5-16ea-4a87-af09-72c5bbcd82bf in bucket 
736a8f4b-19be-4c01-b2c3-901d9538c5ef due to: Connection refused (Connection 
refused)
2023-03-22 10:52:13,960 ERROR [Timer-Driven Process Thread-62] 
o.a.nifi.groups.StandardProcessGroup Failed to synchronize 
StandardProcessGroup[identifier=920c3600-2954-1c8e-b121-6d7d3d393de6,name=Save 
Binary Data] with Flow Registry because could not retrieve version 1 of flow 
with identifier 7a8c82be-1707-4e7d-a5e7-bb3825e0a38f in bucket 
736a8f4b-19be-4c01-b2c3-901d9538c5ef due to: Connection refused (Connection 
refused)

A clue?

-joe

On 3/22/2023 10:49 AM, Mark Payne wrote:
Joe,

1.8 million FlowFiles is not a concern. But when you say “Should I reduce the 
queue sizes?” it makes me wonder if they’re all in a single queue?
Generally, you should leave the backpressure threshold at the default 10,000 
FlowFile max. Increasing this can lead to huge amounts of swapping, which will 
drastically reduce performance and increase disk utilization very significantly.

Also from the diagnostics, it looks like you’ve got a lot of CPU cores, but 
you’re not using much. And based on the amount of disk space available and the 
fact that you’re seeing 100% utilization, I’m wondering if you’re using 
spinning disks, rather than SSDs? I would highly recommend always running NiFi 
with ssd/nvme drives. Absent that, if you have multiple disk drives, you could 
also configure the content repository to span multiple disks, in order to 
spread that load.

Thanks
-Mark

On Mar 22, 2023, at 10:41 AM, Joe Obernberger 
<mailto:joseph.obernber...@gmail.com> wrote:


Thank you.  Was able to get in.
Currently there are 1.8 million flow files and 3.2G.  Is this too much for a 3 
node cluster with mutliple spindles each (SATA drives)?
Should I reduce the queue sizes?

-Joe

On 3/22/2023 10:23 AM, Phillip Lord wrote:
Joe,

If you need the UI to come back up, try setting the autoresume setting in 
nifi.properties to false and restart node(s).
This will bring up every component/controllerService up stopped/disabled and 
may provide some breathing room for the UI to become available again.

Phil
On Mar 22, 2023 at 10:20 AM -0400, Joe Obernberger 
<mailto:joseph.obernber...@gmail.com>, wrote:
atop shows the disk as being all red with IO - 100% utilization. There
are a lot of flowfiles currently trying to run through, but I can't
monitor it becauseUI wont' load.

-Joe

On 3/22/2023 10:16 AM, Mark Payne wrote:
Joe,

I’d recommend taking a look at garbage collection. It is far more likely the 
culprit than disk I/O.

Thanks
-Mark

On Mar 22, 2023, at 10:12 AM, Joe Obernberger 
<mailto:joseph.obernber...@gmail.com> w

Re: UI SocketTimeoutException - heavy IO

[Mark Payne]

Joe,

1.8 million FlowFiles is not a concern. But when you say “Should I reduce the 
queue sizes?” it makes me wonder if they’re all in a single queue?
Generally, you should leave the backpressure threshold at the default 10,000 
FlowFile max. Increasing this can lead to huge amounts of swapping, which will 
drastically reduce performance and increase disk utilization very significantly.

Also from the diagnostics, it looks like you’ve got a lot of CPU cores, but 
you’re not using much. And based on the amount of disk space available and the 
fact that you’re seeing 100% utilization, I’m wondering if you’re using 
spinning disks, rather than SSDs? I would highly recommend always running NiFi 
with ssd/nvme drives. Absent that, if you have multiple disk drives, you could 
also configure the content repository to span multiple disks, in order to 
spread that load.

Thanks
-Mark

On Mar 22, 2023, at 10:41 AM, Joe Obernberger  
wrote:


Thank you.  Was able to get in.
Currently there are 1.8 million flow files and 3.2G.  Is this too much for a 3 
node cluster with mutliple spindles each (SATA drives)?
Should I reduce the queue sizes?

-Joe

On 3/22/2023 10:23 AM, Phillip Lord wrote:
Joe,

If you need the UI to come back up, try setting the autoresume setting in 
nifi.properties to false and restart node(s).
This will bring up every component/controllerService up stopped/disabled and 
may provide some breathing room for the UI to become available again.

Phil
On Mar 22, 2023 at 10:20 AM -0400, Joe Obernberger 
<mailto:joseph.obernber...@gmail.com>, wrote:
atop shows the disk as being all red with IO - 100% utilization. There
are a lot of flowfiles currently trying to run through, but I can't
monitor it becauseUI wont' load.

-Joe

On 3/22/2023 10:16 AM, Mark Payne wrote:
Joe,

I’d recommend taking a look at garbage collection. It is far more likely the 
culprit than disk I/O.

Thanks
-Mark

On Mar 22, 2023, at 10:12 AM, Joe Obernberger 
<mailto:joseph.obernber...@gmail.com> wrote:

I'm getting "java.net.SocketTimeoutException: timeout" from the user interface 
of NiFi when load is heavy. This is 1.18.0 running on a 3 node cluster. Disk IO 
is high and when that happens, I can't get into the UI to stop any of the 
processors.
Any ideas?

I have put the flowfile repository and content repository on different disks on 
the 3 nodes, but disk usage is still so high that I can't get in.
Thank you!

-Joe


--
This email has been checked for viruses by AVG antivirus software.
www.avg.com<http://www.avg.com/>

[https://s-install.avcdn.net/ipm/preview/icons/icon-envelope-tick-green-avg-v1.png]<http://www.avg.com/email-signature?utm_medium=email_source=link_campaign=sig-email_content=emailclient>
 
Virus-free.www.avg.com<http://www.avg.com/email-signature?utm_medium=email_source=link_campaign=sig-email_content=emailclient>

Re: UI SocketTimeoutException - heavy IO

[Mark Payne]

Joe,

I’d recommend taking a look at garbage collection. It is far more likely the 
culprit than disk I/O.

Thanks
-Mark

> On Mar 22, 2023, at 10:12 AM, Joe Obernberger  
> wrote:
> 
> I'm getting "java.net.SocketTimeoutException: timeout" from the user 
> interface of NiFi when load is heavy.  This is 1.18.0 running on a 3 node 
> cluster.  Disk IO is high and when that happens, I can't get into the UI to 
> stop any of the processors.
> Any ideas?
> 
> I have put the flowfile repository and content repository on different disks 
> on the 3 nodes, but disk usage is still so high that I can't get in.
> Thank you!
> 
> -Joe
> 
> 
> -- 
> This email has been checked for viruses by AVG antivirus software.
> www.avg.com

Re: Processor with cron scheduling in middle of flow

[Mark Payne]

Jens,

In this case it would make perfect sense to use Cron for the ListFile. 
FetchFile would then be Timer Driven. The point here is around CRON driven 
processors in the middle of the flow.

Thanks
Mark

Sent from my iPhone

On Feb 22, 2023, at 10:17 AM, Jens M. Kofoed  wrote:


Hi Mark
We have many List/Get processors which is running via cron. Some systems export 
data to disk every hour, but the systems can't block read acces to the files 
while writing them. So NiFi can pull the same file multiple times and tries to 
delete it while the file is written. But we know that the export only takes 10 
minutes. Therefore we use a CRON to get files between 0 0 15-55 * *
We have similar issues with other systems only providing data or are accessibly 
at specific time slots.

To John:
Could you use a Notify/Wait gate function. Where a wait processor is blocking 
flowfiles to the mergeContent processor. And in another flow use a 
generateFlowfile and a notify process to open the gate (wait process). After 
the mergeContent you could have a notify process to close the gate again.
In this way, you would get many flowfile into the mergeContent process at the 
same time.

Kind regards
Jens M. Kofoed

Den 22. feb. 2023 kl. 15.24 skrev Mark Payne 
mailto:marka...@hotmail.com>>:

Interesting. Thanks for that feedback Harald. It might make sense to be more 
surgical about this, disabling it for MergeContent, for example, instead of all 
interflow processors.

Thanks
-Mark


On Feb 22, 2023, at 5:42 AM, Dobbernack, Harald (Key-Work) 
mailto:harald.dobbern...@key-work.de>> wrote:

Just responding to this part:
You should not be using CRON driven for any processors in the middle of a flow. 
In fact, we really
should probably just disable that all together.
Please don't disable this! We actually use CRON for some of our PutSFTP 
Processors as there are servicetimes of these SFTP we are supposed to respect 
and not use them or the SFTP will actually not be available... Of course we can 
also use a routing to a wait processor if we have arrived at a time where the 
destination should not be called, but it is so more simpler being able to tell 
the processor in the middle of the flow when not to run.

-Ursprüngliche Nachricht-----
Von: Mark Payne mailto:marka...@hotmail.com>>
Gesendet: Dienstag, 21. Februar 2023 21:37
An: users@nifi.apache.org<mailto:users@nifi.apache.org>; John McGinn 
mailto:amruginn-n...@yahoo.com>>
Betreff: Re: Processor with cron scheduling in middle of flow

Key-Work IT-Sicherheit: Es handelt sich um eine externe E-Mail. Bitte nur auf 
Links oder Anhänge klicken, sofern die Echtheit der Nachricht klar ist.

John,

You should not be using CRON driven for any processors in the middle of a flow. 
In fact, we really should probably just disable that all together.
In fact, it’s exceedingly rare that you’d want anything other than Timer-Driven 
with a Run Schedule of 0 sec.
MergeContent will not create any merged output on its first iteration after 
it’s scheduled to run. It requires at least a second iteration before anything 
is transferred. Its algorithm has evolved over time, and it may well have 
happened to work previously but it’s really not being configured as intended.

When you say that you’re retrieving data from a few sources and then “merges 
that all back into a single file” - does that mean that you started with one 
FlowFile, split it up, and then want to re-assemble the data after performing 
enrichment? If so you’ll want to use a Merge Strategy of Defragment.

If you’re trying to just bring in some data and merge it together by 
correlation attribute, then Bin Packing makes sense. Here, you have a few 
properties that you can use to try to get the best bin packing. In short, a bin 
will be created when any of these conditions is met:

- The Minimum Group Size is reached AND the Minimum Number of Entries is met
- The Maximum Group Size OR the Maximum Number of Entries is met
- A bin has been sitting for “Max Bin Age” amount of time
- If a correlation attribute is used, and a FlowFile comes in that can’t go 
into any bin, it will evict the oldest.

If you’re seeing bins smaller than expected, you can look at the Data 
Provenance for the merged FlowFile, and it will tell you exactly which of the 
conditions above triggered the data to be merged. This may help to adjust these 
settings.

Hope this is helpful.

Thanks
-Mark


On Feb 17, 2023, at 1:39 PM, John McGinn via users 
mailto:users@nifi.apache.org>> wrote:

Hello,

NiFi 1.19.0 - I need some help in trying to make my idea work, or figure out 
the better way to do this.

I've got a flow that retrieves data from a few data sources, enhances 
individual flow files, converts attributes to CSV and then merges that all back 
into a single file. It takes roughly 20 minutes for the process to run from 
start to the MergeContent part, so when I do it manually, I stop the 
MergeContent processor until al

Re: Processor with cron scheduling in middle of flow

[Mark Payne]

Interesting. Thanks for that feedback Harald. It might make sense to be more 
surgical about this, disabling it for MergeContent, for example, instead of all 
interflow processors.

Thanks
-Mark


> On Feb 22, 2023, at 5:42 AM, Dobbernack, Harald (Key-Work) 
>  wrote:
> 
> Just responding to this part:
>> You should not be using CRON driven for any processors in the middle of a 
>> flow. In fact, we really
>> should probably just disable that all together.
> Please don't disable this! We actually use CRON for some of our PutSFTP 
> Processors as there are servicetimes of these SFTP we are supposed to respect 
> and not use them or the SFTP will actually not be available... Of course we 
> can also use a routing to a wait processor if we have arrived at a time where 
> the destination should not be called, but it is so more simpler being able to 
> tell the processor in the middle of the flow when not to run.
> 
> -Ursprüngliche Nachricht-
> Von: Mark Payne 
> Gesendet: Dienstag, 21. Februar 2023 21:37
> An: users@nifi.apache.org; John McGinn 
> Betreff: Re: Processor with cron scheduling in middle of flow
> 
> Key-Work IT-Sicherheit: Es handelt sich um eine externe E-Mail. Bitte nur auf 
> Links oder Anhänge klicken, sofern die Echtheit der Nachricht klar ist.
> 
> John,
> 
> You should not be using CRON driven for any processors in the middle of a 
> flow. In fact, we really should probably just disable that all together.
> In fact, it’s exceedingly rare that you’d want anything other than 
> Timer-Driven with a Run Schedule of 0 sec.
> MergeContent will not create any merged output on its first iteration after 
> it’s scheduled to run. It requires at least a second iteration before 
> anything is transferred. Its algorithm has evolved over time, and it may well 
> have happened to work previously but it’s really not being configured as 
> intended.
> 
> When you say that you’re retrieving data from a few sources and then “merges 
> that all back into a single file” - does that mean that you started with one 
> FlowFile, split it up, and then want to re-assemble the data after performing 
> enrichment? If so you’ll want to use a Merge Strategy of Defragment.
> 
> If you’re trying to just bring in some data and merge it together by 
> correlation attribute, then Bin Packing makes sense. Here, you have a few 
> properties that you can use to try to get the best bin packing. In short, a 
> bin will be created when any of these conditions is met:
> 
> - The Minimum Group Size is reached AND the Minimum Number of Entries is met
> - The Maximum Group Size OR the Maximum Number of Entries is met
> - A bin has been sitting for “Max Bin Age” amount of time
> - If a correlation attribute is used, and a FlowFile comes in that can’t go 
> into any bin, it will evict the oldest.
> 
> If you’re seeing bins smaller than expected, you can look at the Data 
> Provenance for the merged FlowFile, and it will tell you exactly which of the 
> conditions above triggered the data to be merged. This may help to adjust 
> these settings.
> 
> Hope this is helpful.
> 
> Thanks
> -Mark
> 
> 
>> On Feb 17, 2023, at 1:39 PM, John McGinn via users  
>> wrote:
>> 
>> Hello,
>> 
>> NiFi 1.19.0 - I need some help in trying to make my idea work, or figure out 
>> the better way to do this.
>> 
>> I've got a flow that retrieves data from a few data sources, enhances 
>> individual flow files, converts attributes to CSV and then merges that all 
>> back into a single file. It takes roughly 20 minutes for the process to run 
>> from start to the MergeContent part, so when I do it manually, I stop the 
>> MergeContent processor until all flowfiles are in the queue waiting, and 
>> then I start the MergeContent processor. (Run One Time doesn't work for some 
>> reason.) That works fine, manually.
>> 
>> When I try to put cron scheduling in, it never kicks off. For instance, the 
>> initial processor in the flow has a cron schedule of the top of the hour. (0 
>> 0 * * * ?) I then put 25 past the hour for Merge Content (0 25 * * * ?). 
>> When I start the flow, the flowfiles are generated and queue up in front of 
>> MergeContent by 25 minutes past the hour, but the MergeContent never kicks 
>> off.
>> 
>> I added a correlation attribute recently and removed the cron entry, but the 
>> MergeContent just creates small bunches of merged files.
>> 
>> I even attempted to put a cron on the AttributesToCSV with a maximum bin age 
>> on the Merge Content, since it takes less than a minute for the 
>> AttribuesToCSV to process the flowfiles at that point, but the cron didn't 
&g

Re: Processor with cron scheduling in middle of flow

[Mark Payne]

John,

You should not be using CRON driven for any processors in the middle of a flow. 
In fact, we really should probably just disable that all together.
In fact, it’s exceedingly rare that you’d want anything other than Timer-Driven 
with a Run Schedule of 0 sec.
MergeContent will not create any merged output on its first iteration after 
it’s scheduled to run. It requires at least a second iteration before anything 
is transferred. Its algorithm has evolved over time, and it may well have 
happened to work previously but it’s really not being configured as intended.

When you say that you’re retrieving data from a few sources and then “merges 
that all back into a single file” - does that mean that you started with one 
FlowFile, split it up, and then want to re-assemble the data after performing 
enrichment? If so you’ll want to use a Merge Strategy of Defragment.

If you’re trying to just bring in some data and merge it together by 
correlation attribute, then Bin Packing makes sense. Here, you have a few 
properties that you can use to try to get the best bin packing. In short, a bin 
will be created when any of these conditions is met:

- The Minimum Group Size is reached AND the Minimum Number of Entries is met
- The Maximum Group Size OR the Maximum Number of Entries is met
- A bin has been sitting for “Max Bin Age” amount of time
- If a correlation attribute is used, and a FlowFile comes in that can’t go 
into any bin, it will evict the oldest.

If you’re seeing bins smaller than expected, you can look at the Data 
Provenance for the merged FlowFile, and it will tell you exactly which of the 
conditions above triggered the data to be merged. This may help to adjust these 
settings.

Hope this is helpful.

Thanks
-Mark


> On Feb 17, 2023, at 1:39 PM, John McGinn via users  
> wrote:
> 
> Hello,
> 
> NiFi 1.19.0 - I need some help in trying to make my idea work, or figure out 
> the better way to do this.
> 
> I've got a flow that retrieves data from a few data sources, enhances 
> individual flow files, converts attributes to CSV and then merges that all 
> back into a single file. It takes roughly 20 minutes for the process to run 
> from start to the MergeContent part, so when I do it manually, I stop the 
> MergeContent processor until all flowfiles are in the queue waiting, and then 
> I start the MergeContent processor. (Run One Time doesn't work for some 
> reason.) That works fine, manually. 
> 
> When I try to put cron scheduling in, it never kicks off. For instance, the 
> initial processor in the flow has a cron schedule of the top of the hour. (0 
> 0 * * * ?) I then put 25 past the hour for Merge Content (0 25 * * * ?). When 
> I start the flow, the flowfiles are generated and queue up in front of 
> MergeContent by 25 minutes past the hour, but the MergeContent never kicks 
> off.
> 
> I added a correlation attribute recently and removed the cron entry, but the 
> MergeContent just creates small bunches of merged files.
> 
> I even attempted to put a cron on the AttributesToCSV with a maximum bin age 
> on the Merge Content, since it takes less than a minute for the 
> AttribuesToCSV to process the flowfiles at that point, but the cron didn't 
> kick off there either.
> 
> Any ideas on how to get this to work?
> 
> Thanks,
> John

Re: How to cherry pick a specific line from a flowfile?

[Mark Payne]

James,

Have a look at the RouteText processor. I wrote a blog post recently on using 
it: 
https://medium.com/cloudera-inc/building-an-effective-nifi-flow-routetext-5068a3b4efb3

Thanks
Mark

Sent from my iPhone

On Feb 9, 2023, at 8:06 PM, James McMahon  wrote:


My version of nifi does not have Range Sampling unfortunately.
If I get the flowfile through a session as done in the Cookbook, does anyone 
know of an approach in Groovy to grab line N and avoid loading the entire CSV 
file into string variable text?

On Thu, Feb 9, 2023 at 7:18 PM Matt Burgess 
mailto:mattyb...@gmail.com>> wrote:
I’m AFK ATM but Range Sampling was added into the SampleRecord processor 
(https://issues.apache.org/jira/browse/NIFI-9814), the Jira doesn’t say which 
version it went into but it is definitely in 1.19.1+. If that’s available to 
you then you can just specify “2” as the range and it will only return that 
line.

For total record count without loading the whole thing into memory, there’s 
probably a more efficient way but you could use ConvertRecord and convert it 
from CSV to CSV and it should write out the “record.count” attribute. I think 
some/most/all record processors write this attribute, and they work record by 
record so they don’t load the whole thing into memory. Even SampleRecord adds a 
record.count attribute but if you specify one line the value will be 1 :)

Regards,
Matt


On Feb 9, 2023, at 6:57 PM, James McMahon 
mailto:jsmcmah...@gmail.com>> wrote:


Hello. I am trying to identify a header line and a data line count from a 
flowfile that is in csv format.

Most of us are familiar with Matt B's outstanding Cookbook series, and I am 
trying to use that as my starting point. Here is my Groovy code:

import org.apache.commons.io.IOUtils
import java.nio.charset.StandardCharsets
def ff=session.get()
if(!ff)return
try {
 def text = ''
 // Cast a closure with an inputStream parameter to InputStreamCallback
 session.read(ff, {inputStream ->
  text = IOUtils.toString(inputStream, StandardCharsets.UTF_8)
  // Do something with text here
  // get header from the second line of the flowfile
  // set datacount as the total line count of the file - 2
  ...
  ff = session.putAttribute(ff, 'mdb.table.header', header)
  ff = session.putAttribute(ff, 'mdb.table.datarecords', datacount)
 } as InputStreamCallback)
 session.transfer(flowFile, REL_SUCCESS)
} catch(e) {
 log.error('Error occurred identifying tables in mdb file', e)
 session.transfer(ff, REL_FAILURE)
}

I want to avoid using that line in red, because as Matt cautions in his 
cookbook, our csv files are too large. I do not want to read in the entire file 
to variable text. It's going to be a problem.

How in Groovy can I cherry pick only the line I want from the stream (line #2 
in this case)?

Also, how can I get a count of the total lines without loading them all into 
text?

Thanks in advance for your help.

Re: DistributedMapCacheClient corrupting dataflow

[Mark Payne]

Hey Paul,

Looks like you’re running into 
https://issues.apache.org/jira/browse/NIFI-10246, which was addressed in 1.17.0
When NiFi gets to that point, the service is either in an ENABLING state (even 
though it says it’s enabled, it may just be enabling) or it’s decided that it 
needs to disable the service in order to ensure that the flow is in sync with 
the cluster.

Unfortunately, there was an issue where it would not properly wait for the 
service to be completely DISABLED. It would asynchronously disable the service 
and then go ahead without waiting for that action to complete. That produced 
the issue that you’re seeing in the stack trace.

Thanks
-Mark



On Jan 12, 2023, at 11:36 AM, Paul Riddle  wrote:

Good Morning NiFi Folks!

NiFi Version: 1.16.3

Issue: Enabling DistributedMapCacheClient is causing clustered nodes to not be 
able to startup or rejoin cluster after NiFi stop/restart.

Scenario:  I am using a DetectDuplicate processor on a 3 node cluster.  I have 
a DistributedMapCacheServer controller service running on one of the 3 nodes.  
It enables and works fine.

I also have a DistributeMapCacheClient controller service configured.  It 
appears to enable and work fine.   I run data through DetectDuplicate.  It 
finds duplicates as I would expect it to do.

An issue presents itself, however, when NiFi is restarted for any reason.  If, 
and only if, the DistributedMapCacheClient service (I verified each other 
service) is set to ENABLED when NiFi initially starts up or tries to restart, 
NiFi  fails to start up and I receive a stack trace in the logs that says:

-
“2023-01-12 16:06:59,386 ERROR [main] o.a.nifi.controller.StandardFlowService 
Failed to load flow from cluster due to: 
org.apache.nifi.controller.serialization.FlowSynchronizationException: Failed 
to connect node to cluster because local

   flow controller partially 
updated. Administrator should disconnect node and review flow for corruption.
org.apache.nifi.controller.serialization.FlowSynchronizationException: Failed 
to connect node to cluster because local flow controller partially updated. 
Administrator should disconnect node and review flow for corruption.
at 
org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:1061)
at 
org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:524)
at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:1086)
at org.apache.nifi.NiFi.(NiFi.java:170)
at org.apache.nifi.NiFi.(NiFi.java:82)
at org.apache.nifi.NiFi.main(NiFi.java:330)
Caused by: 
org.apache.nifi.controller.serialization.FlowSynchronizationException: 
java.lang.IllegalStateException: Cannot modify Controller Service configuration 
because it is currently enabled. Please disable the Controller Service firs 


 t.
at 
org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.synchronizeFlow(VersionedFlowSynchronizer.java:370)
at 
org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.sync(VersionedFlowSynchronizer.java:188)
at 
org.apache.nifi.controller.serialization.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:43)
at 
org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1524)
at 
org.apache.nifi.persistence.StandardFlowConfigurationDAO.load(StandardFlowConfigurationDAO.java:107)
at 
org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:819)
at 
org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:1030)”

-

At first glance it seems that NiFi is trying to modify something within the 
DistributedMapCacheClient service on startup after it is in an ENABLED state 
for some reason.  It appears to be an order of operations issue.

Is this a known bug?  Is there some sort of workaround other than disabling the 
controller service every time before I shut NiFi down?

Thanks for the Help!
Paul

Re: Issue with removal and re-add of a cluster node

[Mark Payne]

David,

I think you’re running into https://issues.apache.org/jira/browse/NIFI-10453, 
which was fixed in 1.19.

It results in the "Cannot set AnnotationData while processor is running” error.

Recommend upgrading to 1.19. In the meantime, though, you should be okay to 
shutdown node 3, delete the conf/flow.xml.gz and conf/flow.json.gz and restart
That will rejoin the cluster and inherit whatever the cluster’s flow is.

Thanks
-Mark


On Dec 9, 2022, at 2:18 PM, David Early via users  wrote:

Forgot my version: 1.16.3

Dave

On Fri, Dec 9, 2022 at 11:22 AM David Early 
mailto:david.ea...@grokstream.com>> wrote:
Hi all,

I have a major issue and am not sure what to do about it.

We have a 3 node cluster.  I was working on a one-off load for some data we 
were doing out of sequence and it resulted in build-up of some flowfiles in a 
queue.  In order to prevent a backpressure situation, I cleared one of the 
holding queues that had about 69k flow files.

During the clear operation the node I was on (node 3 UI in this case) returned 
and stated that the node was no longer part of the cluster.  Not clear why that 
happened.

This, by itself, is not really an issue.  Looking at the logs (at bottom of 
this note), you can see theflowfile drop and immediate adjustment to the node 3 
to state of CONNECTING to the cluster.  Subsequently, an error occurred:  
"Disconnecting node due to Failed to properly handle Reconnection request due 
to org.apache.nifi.controller.serialization.FlowSynchronizationException: 
Failed to connect node to cluster because local flow controller partially 
updated. Administrator should disconnect node and review flow for corruption".

When I attempted to readd the node from the UI, it repeated this error.

I compared users.xml and authroizations.xml on all three nodes, textually the 
same and identical md5sum on all (issues with users.xml and authorizations.xml 
were listed online as usual suspects).

I then offloaded the node via the UI to make sure I didn't have anything stuck 
in queue on node 3 and hoped it would allow the node to rejoin.  After 
offloading, I attempted to reconnect and what happened next gave me a heart 
attack:  Node 3 now showed as connected but in the UI (accessed via node 1), 
ALL PROCESSORS WERE SHOWN AS STOPPED.

A quick review showed that traffic was actually flowing (View status history 
showed flowfiles moving, observing some of our observation queues showed 
traffic on nodes 1 and 2).  Removing node 3 from the cluster restored the UI to 
show everything running, but adding it back in showed everything as stopped.

I tried to start some processors while node 3 was connected and while I could 
start individual processors, I could not do a "global" start by right clicking 
on canvas and trying "start".  I set up a sample processor to generate a file 
on all 3 nodes and it did generate a new flowfile on node 3.  All of that 
worked fine.

We have 400+ processors that I would need to hand start and I am super nervous 
about the cluster deciding to make node 3 the primary which would affect some 
timed process that we are running on the primary node.  As long as I don't 
restart the http input feed, I COULD restart all the processors, but this seems 
like the wrong process.

Anyone have any idea what I did wrong and how to fix it?  The errors show in 
the log attached happened before any offloading, but I wondered if the 
offloading caused part of this issue.

Is there anything I can do to readd the node without having to restart all the 
processors manually?

Should I clean up the node and add it as a "new" node and let it completely 
sync?

Thanks for any insight!


Dave


---
Log:
---
2022-12-08 22:26:20,706 INFO [Drop FlowFiles for Connection 
8b0ee741-0183-1000--68704c93] o.a.n.c.queue.SwappablePriorityQueue 
Successfully dropped 69615 FlowFiles (35496003 bytes) from Connection with ID 
8b0ee741-0183-1000--68704c93 on behalf of 
u...@org.com
2022-12-08 22:26:20,707 INFO [Process Cluster Protocol Request-29] 
o.a.n.c.c.node.NodeClusterCoordinator Status of prod-stsc2-3:8443 changed from 
NodeConnectionStatus[nodeId=prod-stsc2-3:8443, state=CONNECTED, updateId=108] 
to NodeConnectionStatus[nodeId=prod-stsc2-3:8443, state=CONNECTING, 
updateId=114]
2022-12-08 22:26:20,707 INFO [Process Cluster Protocol Request-29] 
o.a.n.c.p.impl.SocketProtocolListener Finished processing request 
070fe65c-4a77-41d0-9d7f-8f08ede6ac71 (type=NODE_STATUS_CHANGE, length=1217 
bytes) from 
prod-stsc2-1.internal.cloudapp.net 
in 10 seconds, 842 millis
2022-12-08 22:26:20,750 INFO [Reconnect to Cluster] 
o.a.nifi.controller.StandardFlowService Setting Flow Controller's Node ID: 
prod-stsc2-3:8443
2022-12-08 22:26:20,751 INFO [Reconnect to Cluster] 
o.a.n.c.s.VersionedFlowSynchronizer Synchronizing FlowController with proposed 
flow: Controller 

Re: Expected mergerecord performance

[Mark Payne]

> Is there something about this structure that is likely to be causing the 
> problem? Could there be other issues with the avro generated by the script?

I don’t think the structure should matter. And as long as the avro produced is 
proper Avro, I don’t think it should matter. Unless perhaps there’s some issue 
with the Avro library itself that’s causing it to take a really long time to 
parse the Avro or something? I’d be curious - if you take the output of your 
script and then you run it through a ConvertRecord (Avro Reader -> Json Writer) 
is the ConvertRecord fast? Or is it really slow to process it?

On Dec 5, 2022, at 5:58 AM, Richard Beare  wrote:

Further - I performed another test in which I replaced the custom json to avro 
script with a ConvertRecord processor - merge record appears to work as 
expected in that case.

Output of convertrecord looks like this:

[ {
  "text" : "  No Alert Found \n\n",
  "metadata" : {
"X_TIKA_Parsed_By" : null,
"X_OCR_Applied" : null,
"Content_Type" : null
  },
  "success" : true,
  "timestamp" : "2022-12-05T10:49:18.568Z",
  "processingElapsedTime" : 0,
  "doc_id" : "5.60178607E8"
} ]

while the output of the script looks like:

[ {
  "doc_id" : "5.61996505E8",
  "doc_text" : "  No Alert Found \n\n",
  "processing_timestamp" : "2022-11-28T01:16:46.775Z",
  "metadata_x_ocr_applied" : true,
  "metadata_x_parsed_by" : 
"org.apache.tika.parser.DefaultParser;org.apache.tika.parser.microsoft.rtf.RTFParser;org.apache.tika.parser.AutoDetectParser",
  "metadata_content_type" : "application/rtf",
  "metadata_page_count" : null,
  "metadata_creation_date" : null,
  "metadata_last_modified" : null
} ]

Is there something about this structure that is likely to be causing the 
problem? Could there be other issues with the avro generated by the script?

On Mon, Dec 5, 2022 at 9:31 PM Richard Beare 
mailto:richard.be...@gmail.com>> wrote:
I've reset the backpressure to the default

This remains something of a mystery. The merge with synthetic data happily 
creates flowfiles with 100 records, and the join says "Records merged due to: 
Bin is full" or "Records merged due to: Bin is full enough". No timeouts in 
that case, even with the max bin age at 4.5 seconds. The resulting flowfiles 
were about 300K.

The real data is doing much the same as before, producing flowfiles of about 
30K, with 7 records or so. If I increase the maximum bin age to 30 seconds the 
size and record count is higher - 12 to 15. Nothing like the behaviour with 
synthetic data, where the 100 record flowfiles are created almost instantly. 
Joins are always due to bin age.

Can the problem relate to the structure of the avro files? Any way to dive into 
that? Everything else about the mergerecord settings appear the same, so I 
can't see an explanation as to why the behaviour is different on the same 
hardware.








On Mon, Dec 5, 2022 at 2:09 AM Mark Payne 
mailto:marka...@hotmail.com>> wrote:
Hey Richard,

So a few things that I’ve done/looked at.

I generated some Avro data (random JSON that I downloaded from a Random JSON 
Generator and then converted to Avro).

I then ran this avro data into both the MergeRecord processors.

Firstly, I noticed that both are very slow. Found that was because Run Schedule 
was set to 5 seconds. This should *ALWAYS* be 0 secs for MergeRecord. And 
really for basically all processors except for the first one in the flow.

I also notice that you have backpressure set on your connections to 40,000 
FlowFiles and 4 GB. This can significantly slow things down. If you have 
performance concerns you definitely want backpressure set back to the default 
of 10,000 FlowFiles. Otherwise, as the queues fill up they start “swapping out” 
FlowFiles to disk, and this can significantly slow things down.

I noticed that MergeRecord is set to 1 concurrent task. Probably worth 
considering increasing that, if performance is a concern.

That said, I am seeing nice, full bins of 100 records merged from each of the 
MergeRecord processors.
So it is certainly possible that if you’re seeing smaller bins it’s becuase 
you’re timing out. The 4.5 seconds timeout is quite short. Have you tried 
increasing that to say 30 seconds to see if it gives you larger bins?
I also recommend that you take a look at the data provenance to see why it’s 
creating the bins.

If unclear how to do that:
Right-click on the MergeRecord processor
Go to View data provenance
Scroll down the list until you see a “JOIN” event type. You can ignore the 
ATTRIBUTES_MODIFIED and DROP events for now.
Click the ‘i’ icon on the left-hand side.
This will show you details about the merge. In the Details tab, if you s

Re: NiFi 1.18.0 Sensitive Property broken after Upgrade

[Mark Payne]

I’ve not followed everything goin on in this thread. But just to offer a bit of 
clarification: If the flow.json.gz exists then the flow.xml.gz is ignored. But 
if the flow.json.gz file is not found, it will automatically fall back to the 
flow.xml.gz

So if you were to remove/rename the flow.json.gz it would definitely pick up 
the XML file and process that. Otherwise, it will ignore it.

Thanks
-Mark


On Dec 7, 2022, at 4:24 AM, Tiago Luís Sebastião (DSI) 
 wrote:

Hi Isha,

I did not delete the flow.xml.gz because I think that since the version 1.16.0, 
if I’m not mistaken, when the flow.json.xml was released it replaced completely 
the flow.xml.gz, so any changes made on flow.xml.gz wouldn’t make any 
difference, so knowing that, I assumed there shouldn’t be anything 
configured/dependent on the xml file. Also, the error didn’t seem related at 
all to the flow, it seemed related to not being able to decrypt the 
flow.json.gz with the old algorithm and encrypt with the new algorithm.

Thanks for the help.

Regards.

Tiago Sebastião

From: Isha Lamboo [mailto:isha.lam...@virtualsciences.nl]
Sent: 7 de dezembro de 2022 08:42
To: users@nifi.apache.org
Subject: RE: NiFi 1.18.0 Sensitive Property broken after Upgrade

*** ATENÇÃO: esta mensagem de e-mail tem origem externa!
A cibersegurança é uma responsabilidade partilhada. Não aceda a links nem 
anexos de mensagens suspeitas ou inesperadas.
CSIRT CGD ***


Hi Tiago,

Thanks for updating and sharing your solution.

I noticed you only encrypted the flow.json.gz file. Did you also delete the xml 
version in previous attempts?
It makes me wonder if the migrating fails for the flow.xml.gz specifically, 
then deleting the xml during migration might be an easier fix.

Regards,

Isha


Van: Tiago Luís Sebastião (DSI) 
mailto:tiago.luis.sebast...@cgd.pt>>
Verzonden: dinsdag 6 december 2022 17:24
Aan: users@nifi.apache.org
Onderwerp: RE: NiFi 1.18.0 Sensitive Property broken after Upgrade

Hi Isha,

I had already tried that and didn’t work also.

But I tried the last thing I wrote on the previous email and it worked… Not 
happy with this solution but the issue (encryption/warnings) is solved I guess.

Having Nifi.properties:
nifi.sensitive.props.key=PasswordUsedOnNifiProperties
nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL

With nifi Stopped:

  1.  Encrypt new files on the side:

/apps/nifi-toolkit-1.18.0/bin/encrypt-config.sh -f 
/apps/nifi-1.18.0/conf/flow.json.gz -g /apps/nifi-1.18.0/conf/flow2.json.gz -n 
/apps/nifi-1.18.0/conf/nifi.properties -o 
/apps/nifi-1.18.0/conf/nifi2.properties -A NIFI_PBKDF2_AES_GCM_256 
-sPasswordUsedOnNifiProperties -x -v


  1.  Backup the flow.json.gz and rename the new one:

mv /apps/nifi-1.18.0/conf/flow.json.gz 
/apps/nifi-1.18.0/conf/flow_bk202212.json.gz
mv /apps/nifi-1.18.0/conf/flow2.json.gz /apps/nifi-1.18.0/conf/flow.json.gz


  1.  Change nifi.properties file:

nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
to
nifi.sensitive.props.algorithm=NIFI_PBKDF2_AES_GCM_256


  1.  Start nifi


No errors found and no warnings…

Tiago Sebastião

From: Isha Lamboo [mailto:isha.lam...@virtualsciences.nl]
Sent: 25 de novembro de 2022 09:48
To: users@nifi.apache.org
Subject: RE: NiFi 1.18.0 Sensitive Property broken after Upgrade

*** ATENÇÃO: esta mensagem de e-mail tem origem externa!
A cibersegurança é uma responsabilidade partilhada. Não aceda a links nem 
anexos de mensagens suspeitas ou inesperadas.
CSIRT CGD ***


Hi Tiago,

I’ve had a similar experience with migrating the flow encryption algorithm and 
in fact, some of them are still on the old one. The nifi.sh commands to update 
the sensitive properties key and algorithm are very tricky to use, because they 
update the nifi.properties file even if the migration fails for the flow.xml.gz 
and flow.json.gz.
It took me a while to realize my first failed attempt caused all the following 
ones to fail because it tried to decrypt using the new algorithm. I needed to 
reset the nifi.properties file everytime.

Another thing I’ve noticed is that it doesn’t support the property file 
protection scheme that NiFi has. If your sensitive props key is encrypted you 
need to enter the raw value and make sure the 
nifi.sensitive.props.key.protected is empty. You can re-encrypt afterwards.

These steps have mostly worked for me:


  1.  Backup your conf dir and flow.xml.gz/flow.json.gz if they are in another 
dir
  2.  Unprotect the sensitive properties key:

 *   Replace the encrypted key with the raw one
 *   Empty this property: nifi.sensitive.props.key.protected

  1.  Check that the algorithm is still the old one: 
nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
  2.  Check the length of the raw key, it needs to be 12 characters or longer 
to migrate to the new 

Re: Expected mergerecord performance

[Mark Payne]

Hey Richard,

So a few things that I’ve done/looked at.

I generated some Avro data (random JSON that I downloaded from a Random JSON 
Generator and then converted to Avro).

I then ran this avro data into both the MergeRecord processors.

Firstly, I noticed that both are very slow. Found that was because Run Schedule 
was set to 5 seconds. This should *ALWAYS* be 0 secs for MergeRecord. And 
really for basically all processors except for the first one in the flow.

I also notice that you have backpressure set on your connections to 40,000 
FlowFiles and 4 GB. This can significantly slow things down. If you have 
performance concerns you definitely want backpressure set back to the default 
of 10,000 FlowFiles. Otherwise, as the queues fill up they start “swapping out” 
FlowFiles to disk, and this can significantly slow things down.

I noticed that MergeRecord is set to 1 concurrent task. Probably worth 
considering increasing that, if performance is a concern.

That said, I am seeing nice, full bins of 100 records merged from each of the 
MergeRecord processors.
So it is certainly possible that if you’re seeing smaller bins it’s becuase 
you’re timing out. The 4.5 seconds timeout is quite short. Have you tried 
increasing that to say 30 seconds to see if it gives you larger bins?
I also recommend that you take a look at the data provenance to see why it’s 
creating the bins.

If unclear how to do that:
Right-click on the MergeRecord processor
Go to View data provenance
Scroll down the list until you see a “JOIN” event type. You can ignore the 
ATTRIBUTES_MODIFIED and DROP events for now.
Click the ‘i’ icon on the left-hand side.
This will show you details about the merge. In the Details tab, if you scroll 
down, it will show you a Details field, which tells you why the data was 
merged. It should either say: "Records Merged due to: Bin has reached Max Bin 
Age” or “ Records Merged due to: Bin is full”

If it is due to Max Bin Age reached, then I’d recommend increasing number of 
concurrent tasks, reducing backpressure to no more than 10,000 FlowFiles in the 
queue, and/or increasing the Max Bin Age.
Also worth asking - what kind of machines is this running on? A 64 core VM with 
1 TB volume will, of course, run MUCH differently than a 4 core VM with a 10 GB 
volume, especially in the cloud.

If still having trouble, let me know what the provenance tells you about the 
reason for merging the data, and we can go from there.

Thanks!
-Mark


On Dec 3, 2022, at 4:38 PM, Mark Payne  wrote:

Richard,

I think just the flow structure shoudl be sufficient.

Thanks
-Mark


On Dec 3, 2022, at 4:32 PM, Richard Beare  wrote:

Thanks for responding,
I re-tested with max bins = 2, but the behaviour remained the same. I can 
easily share a version of the functioning workflow (and data), which is part of 
a public project. The problem workflow (which shares many of the same 
components) is part of a health research project, so more difficult. I 
definitely can't share any data from that one. Do you need to see the data or 
is the overall structure sufficient at this point? Happy to demonstrate via 
video conference too.

Thanks

On Sun, Dec 4, 2022 at 1:37 AM Mark Payne 
mailto:marka...@hotmail.com>> wrote:
Hi Richard,

Can you try increasing the Maximum Number of Bins? I think there was an issue 
that was recently addressed in which the merge processors had an issue when Max 
Number of Bins = 1.

If you still see the same issue, please provide a copy of the flow that can be 
used to replicate the issue.

Thanks
-Mark


On Dec 3, 2022, at 5:21 AM, Richard Beare 
mailto:richard.be...@gmail.com>> wrote:

Hi,

Pretty much the same - I seem to end up with flowfiles containing about 7 
records, presumably always triggered by the timeout.

I had thought the timeout needed to be less than the run schedule, but it looks 
like it can be the same.

Here's a debug dump

10:13:43 UTC
DEBUG
99ae16aa-0184-1000-8ccc-ee1f2ee77ca1

MergeRecord[id=99ae16aa-0184-1000-8ccc-ee1f2ee77ca1] Migrating id=1066297 to 
RecordBin[size=4, full=false, isComplete=false, id=4021]

10:13:43 UTC
DEBUG
99ae16aa-0184-1000-8ccc-ee1f2ee77ca1

MergeRecord[id=99ae16aa-0184-1000-8ccc-ee1f2ee77ca1] Transferred id=1066297 to 
RecordBin[size=5, full=false, isComplete=false, id=4021]

10:13:44 UTC
DEBUG
99ae16aa-0184-1000-8ccc-ee1f2ee77ca1

MergeRecord[id=99ae16aa-0184-1000-8ccc-ee1f2ee77ca1] Got Group ID 
{"type":"record","name":"document","fields":[{"name":"doc_id","type":"string"},{"name":"doc_text","type":"string","default":""},{"name":"processing_timestamp","type":{"type":"string","logicalType":"timestamp-millis"}},{"name":"metadata_x_ocr_applied","type":"boolean"},{"name":&q

Re: Expected mergerecord performance

[Mark Payne]

Richard,

I think just the flow structure shoudl be sufficient.

Thanks
-Mark


On Dec 3, 2022, at 4:32 PM, Richard Beare  wrote:

Thanks for responding,
I re-tested with max bins = 2, but the behaviour remained the same. I can 
easily share a version of the functioning workflow (and data), which is part of 
a public project. The problem workflow (which shares many of the same 
components) is part of a health research project, so more difficult. I 
definitely can't share any data from that one. Do you need to see the data or 
is the overall structure sufficient at this point? Happy to demonstrate via 
video conference too.

Thanks

On Sun, Dec 4, 2022 at 1:37 AM Mark Payne 
mailto:marka...@hotmail.com>> wrote:
Hi Richard,

Can you try increasing the Maximum Number of Bins? I think there was an issue 
that was recently addressed in which the merge processors had an issue when Max 
Number of Bins = 1.

If you still see the same issue, please provide a copy of the flow that can be 
used to replicate the issue.

Thanks
-Mark


On Dec 3, 2022, at 5:21 AM, Richard Beare 
mailto:richard.be...@gmail.com>> wrote:

Hi,

Pretty much the same - I seem to end up with flowfiles containing about 7 
records, presumably always triggered by the timeout.

I had thought the timeout needed to be less than the run schedule, but it looks 
like it can be the same.

Here's a debug dump

10:13:43 UTC
DEBUG
99ae16aa-0184-1000-8ccc-ee1f2ee77ca1

MergeRecord[id=99ae16aa-0184-1000-8ccc-ee1f2ee77ca1] Migrating id=1066297 to 
RecordBin[size=4, full=false, isComplete=false, id=4021]

10:13:43 UTC
DEBUG
99ae16aa-0184-1000-8ccc-ee1f2ee77ca1

MergeRecord[id=99ae16aa-0184-1000-8ccc-ee1f2ee77ca1] Transferred id=1066297 to 
RecordBin[size=5, full=false, isComplete=false, id=4021]

10:13:44 UTC
DEBUG
99ae16aa-0184-1000-8ccc-ee1f2ee77ca1

MergeRecord[id=99ae16aa-0184-1000-8ccc-ee1f2ee77ca1] Got Group ID 
{"type":"record","name":"document","fields":[{"name":"doc_id","type":"string"},{"name":"doc_text","type":"string","default":""},{"name":"processing_timestamp","type":{"type":"string","logicalType":"timestamp-millis"}},{"name":"metadata_x_ocr_applied","type":"boolean"},{"name":"metadata_x_parsed_by","type":"string"},{"name":"metadata_content_type","type":["null","string"],"default":null},{"name":"metadata_page_count","type":["null","int"],"default":null},{"name":"metadata_creation_date","type":["null",{"type":"string","logicalType":"timestamp-millis"}],"default":null},{"name":"metadata_last_modified","type":["null",{"type":"string","logicalType":"timestamp-millis"}],"default":null}]}
 for FlowFile[filename=9e9908f6-b28e-4615-b6c8-4bd163a3bc00]

10:13:44 UTC
DEBUG
99ae16aa-0184-1000-8ccc-ee1f2ee77ca1

MergeRecord[id=99ae16aa-0184-1000-8ccc-ee1f2ee77ca1] Migrating id=1066372 to 
RecordBin[size=5, full=false, isComplete=false, id=4021]

10:13:44 UTC
DEBUG
99ae16aa-0184-1000-8ccc-ee1f2ee77ca1

MergeRecord[id=99ae16aa-0184-1000-8ccc-ee1f2ee77ca1] Transferred id=1066372 to 
RecordBin[size=6, full=false, isComplete=false, id=4021]

10:13:45 UTC
DEBUG
99ae16aa-0184-1000-8ccc-ee1f2ee77ca1

MergeRecord[id=99ae16aa-0184-1000-8ccc-ee1f2ee77ca1] Transferred id=1066575 to 
RecordBin[size=7, full=false, isComplete=true, id=4021]

10:13:46 UTC
DEBUG
99ae16aa-0184-1000-8ccc-ee1f2ee77ca1

MergeRecord[id=99ae16aa-0184-1000-8ccc-ee1f2ee77ca1] Got Group ID 
{"type":"record","name":"document","fields":[{"name":"doc_id","type":"string"},{"name":"doc_text","type":"string","default":""},{"name":"processing_timestamp","type":{"type":"string","logicalType":"timestamp-millis"}},{"name":"metadata_x_ocr_applied","type":"boolean"},{"name":"metadata_x_parsed_by","type":"string"},{"name":"metadata_content_type","type":["null","string"],"default":null},{"name":"metadata_page_count","type":["null","int"],"default":null},{"name":"metadata_creation_date","type":["null",{"type":"string","logicalType

Re: Expected mergerecord performance

[Mark Payne]

Hi Richard,

Can you try increasing the Maximum Number of Bins? I think there was an issue 
that was recently addressed in which the merge processors had an issue when Max 
Number of Bins = 1.

If you still see the same issue, please provide a copy of the flow that can be 
used to replicate the issue.

Thanks
-Mark


On Dec 3, 2022, at 5:21 AM, Richard Beare  wrote:

Hi,

Pretty much the same - I seem to end up with flowfiles containing about 7 
records, presumably always triggered by the timeout.

I had thought the timeout needed to be less than the run schedule, but it looks 
like it can be the same.

Here's a debug dump

10:13:43 UTC
DEBUG
99ae16aa-0184-1000-8ccc-ee1f2ee77ca1

MergeRecord[id=99ae16aa-0184-1000-8ccc-ee1f2ee77ca1] Migrating id=1066297 to 
RecordBin[size=4, full=false, isComplete=false, id=4021]

10:13:43 UTC
DEBUG
99ae16aa-0184-1000-8ccc-ee1f2ee77ca1

MergeRecord[id=99ae16aa-0184-1000-8ccc-ee1f2ee77ca1] Transferred id=1066297 to 
RecordBin[size=5, full=false, isComplete=false, id=4021]

10:13:44 UTC
DEBUG
99ae16aa-0184-1000-8ccc-ee1f2ee77ca1

MergeRecord[id=99ae16aa-0184-1000-8ccc-ee1f2ee77ca1] Got Group ID 
{"type":"record","name":"document","fields":[{"name":"doc_id","type":"string"},{"name":"doc_text","type":"string","default":""},{"name":"processing_timestamp","type":{"type":"string","logicalType":"timestamp-millis"}},{"name":"metadata_x_ocr_applied","type":"boolean"},{"name":"metadata_x_parsed_by","type":"string"},{"name":"metadata_content_type","type":["null","string"],"default":null},{"name":"metadata_page_count","type":["null","int"],"default":null},{"name":"metadata_creation_date","type":["null",{"type":"string","logicalType":"timestamp-millis"}],"default":null},{"name":"metadata_last_modified","type":["null",{"type":"string","logicalType":"timestamp-millis"}],"default":null}]}
 for FlowFile[filename=9e9908f6-b28e-4615-b6c8-4bd163a3bc00]

10:13:44 UTC
DEBUG
99ae16aa-0184-1000-8ccc-ee1f2ee77ca1

MergeRecord[id=99ae16aa-0184-1000-8ccc-ee1f2ee77ca1] Migrating id=1066372 to 
RecordBin[size=5, full=false, isComplete=false, id=4021]

10:13:44 UTC
DEBUG
99ae16aa-0184-1000-8ccc-ee1f2ee77ca1

MergeRecord[id=99ae16aa-0184-1000-8ccc-ee1f2ee77ca1] Transferred id=1066372 to 
RecordBin[size=6, full=false, isComplete=false, id=4021]

10:13:45 UTC
DEBUG
99ae16aa-0184-1000-8ccc-ee1f2ee77ca1

MergeRecord[id=99ae16aa-0184-1000-8ccc-ee1f2ee77ca1] Transferred id=1066575 to 
RecordBin[size=7, full=false, isComplete=true, id=4021]

10:13:46 UTC
DEBUG
99ae16aa-0184-1000-8ccc-ee1f2ee77ca1

MergeRecord[id=99ae16aa-0184-1000-8ccc-ee1f2ee77ca1] Got Group ID 
{"type":"record","name":"document","fields":[{"name":"doc_id","type":"string"},{"name":"doc_text","type":"string","default":""},{"name":"processing_timestamp","type":{"type":"string","logicalType":"timestamp-millis"}},{"name":"metadata_x_ocr_applied","type":"boolean"},{"name":"metadata_x_parsed_by","type":"string"},{"name":"metadata_content_type","type":["null","string"],"default":null},{"name":"metadata_page_count","type":["null","int"],"default":null},{"name":"metadata_creation_date","type":["null",{"type":"string","logicalType":"timestamp-millis"}],"default":null},{"name":"metadata_last_modified","type":["null",{"type":"string","logicalType":"timestamp-millis"}],"default":null}]}
 for FlowFile[filename=9e9908f6-b28e-4615-b6c8-4bd163a3bc00]

10:13:46 UTC
DEBUG
99ae16aa-0184-1000-8ccc-ee1f2ee77ca1

MergeRecord[id=99ae16aa-0184-1000-8ccc-ee1f2ee77ca1] Migrating id=1066612 to 
RecordBin[size=0, full=false, isComplete=false, id=4022]

10:13:46 UTC
DEBUG
99ae16aa-0184-1000-8ccc-ee1f2ee77ca1

MergeRecord[id=99ae16aa-0184-1000-8ccc-ee1f2ee77ca1] Created OutputStream using 
session StandardProcessSession[id=83204] for RecordBin[size=0, full=false, 
isComplete=false, id=4022]

10:13:46 UTC
DEBUG
99ae16aa-0184-1000-8ccc-ee1f2ee77ca1

MergeRecord[id=99ae16aa-0184-1000-8ccc-ee1f2ee77ca1] Transferred id=1066612 to 
RecordBin[size=1, full=false, isComplete=false, id=4022]

10:13:48 UTC
DEBUG
99ae16aa-0184-1000-8ccc-ee1f2ee77ca1

MergeRecord[id=99ae16aa-0184-1000-8ccc-ee1f2ee77ca1] Migrating id=1066896 to 
RecordBin[size=2, full=false, isComplete=false, id=4022]

10:13:48 UTC
DEBUG
99ae16aa-0184-1000-8ccc-ee1f2ee77ca1

MergeRecord[id=99ae16aa-0184-1000-8ccc-ee1f2ee77ca1] Transferred id=1066896 to 
RecordBin[size=3, full=false, isComplete=false, id=4022]

10:13:49 UTC
DEBUG
99ae16aa-0184-1000-8ccc-ee1f2ee77ca1

MergeRecord[id=99ae16aa-0184-1000-8ccc-ee1f2ee77ca1] Got Group ID 

Re: Empty Queue - but UI shows messages

[Mark Payne]

Joe,

The problem is that NiFi is not able to load balance the data. What errors do 
you see in the logs around load balancing?

Thanks
-Mark

On Nov 22, 2022, at 9:15 AM, Joe Obernberger 
mailto:joseph.obernber...@gmail.com>> wrote:


Hi Joe - this is happening only in queues that do round robin load balancing:



When I list queue on that queue it shows no flow files.  If I stop the 
processors, and change the queue to no longer load balance, then I can not only 
list the queue, but the 22k messages are then processed.

I did the dump as suggested - and it nifi-bootstrap.log file is attached.

-Joe


On 11/9/2022 5:09 PM, Joe Witt wrote:
Please get back to the bad state and then do a thread dump.  Please share

./bin/nifi.sh dump

Only if LB appears stuck.
Thanks

On Wed, Nov 9, 2022 at 3:05 PM Joe Obernberger 
mailto:joseph.obernber...@gmail.com>> wrote:

If I stop the RouteOnContent and InvokeHTTP processors, and then change the 
queue to no longer load balance, then the messages appear and the processor 
start to run.  I can also then list the messages in the queue.

-Joe

On 11/9/2022 12:04 PM, Joe Obernberger wrote:

The processor consuming from the queue runs on all nodes and is a InokeHTTP 
processor.  That processor is currently idle.

I can stop/start InvokeHTTP, but the queue size remains.  I can't list what is 
in the queue.  This seems like a bug?  What I'm confused about is - do I have 
data to process or not?  I can empty the queue, and something is removed, but I 
can't see what...so I don't know if I lost messages or not.

<6aueDc0O2l8gSHvp.png>

-Joe

On 11/9/2022 11:50 AM, Joe Witt wrote:
Hello

This likely means the processor consuming from this queue has the flowfiles 
held being processed.  MergeContent is a common processor that would do this 
but others certainly can.  What processor do you have there?

If you stop the target processor then delete it should always work though you 
generally should not need to do so but normally deleting queue content is a 
debug thing so you can.

Thanks

On Wed, Nov 9, 2022 at 9:47 AM Joe Obernberger 
mailto:joseph.obernber...@gmail.com>> wrote:

Hi - I'm using NiFi 1.18.0 in a three node cluster using internal zookeeper.  
Occasionally, I will see a queue showing queued messages, but when I list the 
queue, the UI says that there are no flow files.  Also the consumer of this 
queue seems to be idle.  I can then empty the queue and it reports that n 
messages were deleted.

Any idea what is happening here?






-Joe


[https://s-install.avcdn.net/ipm/preview/icons/icon-envelope-tick-green-avg-v1.png]
 
Virus-free.www.avg.com

Re: Nifi unable to list/empty queue.

[Mark Payne]

Hello,

For access to data, because it’s considered more sensitive than the flow 
definition, both the user and the nifi node accessing the data must be granted 
permissions to view and modify data. Did you give the nodes permissions to view 
and modify data?

Also moving this from security@ to users@ mailing list, as this is more of a 
use-based question. The security@ mailing list should be used for sensitive 
topics such as potential vulnerabilities, etc.

Thanks
Mark


Sent from my iPhone

On Nov 22, 2022, at 4:58 AM, lemontree <715733...@qq.com> wrote:



Hello

we have secured Nifi cluster in 1.1.7 with 3 nodes. When we click to list or 
empty queue on connection, there is error message



Insufficient Permissions

Node 192.168.106.5:9443 is unable to fulfill this request due to: Unable to 
modify the data for Processor with ID 682706ca-08e4-3d90-9b6a-5f845573299f. 
Contact the system administrator. Contact the system administrator.


the error request:

  1.
请求 URL:
https://192.168.106.5:9443/nifi-api/flowfile-queues/f7b68394-8e95-3e87-902f-90a74d3d8a42/listing-requests
  2.
请求方法:
POST
  3.
状态代码:
403 Forbidden



We grant user policy to view and modify data, but no success. Admin user got 
the same error.message


we use the managed-authorizer configed as this : 
nifi.security.user.authorizer=managed-authorizer;

very strangely, other action policy work normally, such as create 
processor...





Regards

Re: Empty Queue - but UI shows messages

[Mark Payne]

Joe,

Given that the icon shows the data is still being load balanced, I am guessing 
that the system is having trouble load balancing the data. Can you check the 
logs to see if there are errors related to load balancing?

Thanks
-Mark


On Nov 9, 2022, at 12:04 PM, Joe Obernberger 
mailto:joseph.obernber...@gmail.com>> wrote:


The processor consuming from the queue runs on all nodes and is a InokeHTTP 
processor.  That processor is currently idle.

I can stop/start InvokeHTTP, but the queue size remains.  I can't list what is 
in the queue.  This seems like a bug?  What I'm confused about is - do I have 
data to process or not?  I can empty the queue, and something is removed, but I 
can't see what...so I don't know if I lost messages or not.

<6aueDc0O2l8gSHvp.png>

-Joe

On 11/9/2022 11:50 AM, Joe Witt wrote:
Hello

This likely means the processor consuming from this queue has the flowfiles 
held being processed.  MergeContent is a common processor that would do this 
but others certainly can.  What processor do you have there?

If you stop the target processor then delete it should always work though you 
generally should not need to do so but normally deleting queue content is a 
debug thing so you can.

Thanks

On Wed, Nov 9, 2022 at 9:47 AM Joe Obernberger 
mailto:joseph.obernber...@gmail.com>> wrote:

Hi - I'm using NiFi 1.18.0 in a three node cluster using internal zookeeper.  
Occasionally, I will see a queue showing queued messages, but when I list the 
queue, the UI says that there are no flow files.  Also the consumer of this 
queue seems to be idle.  I can then empty the queue and it reports that n 
messages were deleted.

Any idea what is happening here?






-Joe


[https://s-install.avcdn.net/ipm/preview/icons/icon-envelope-tick-green-avg-v1.png]
 
Virus-free.www.avg.com

Re: Empty Queue - but UI shows messages

[Mark Payne]

Just to add to what Joe said, in the screenshot the queue that you’re listing 
shows that blue load balancing icon, meaning that it is actively load 
balancing. The listing will not include any FlowFiles that are queued up to 
move to another node. So on a connection like that where it’s actively load 
balancing you may not be able to view the contents.

Thanks
-Mark


On Nov 9, 2022, at 11:50 AM, Joe Witt 
mailto:joe.w...@gmail.com>> wrote:

Hello

This likely means the processor consuming from this queue has the flowfiles 
held being processed.  MergeContent is a common processor that would do this 
but others certainly can.  What processor do you have there?

If you stop the target processor then delete it should always work though you 
generally should not need to do so but normally deleting queue content is a 
debug thing so you can.

Thanks

On Wed, Nov 9, 2022 at 9:47 AM Joe Obernberger 
mailto:joseph.obernber...@gmail.com>> wrote:

Hi - I'm using NiFi 1.18.0 in a three node cluster using internal zookeeper.  
Occasionally, I will see a queue showing queued messages, but when I list the 
queue, the UI says that there are no flow files.  Also the consumer of this 
queue seems to be idle.  I can then empty the queue and it reports that n 
messages were deleted.

Any idea what is happening here?






-Joe


[https://s-install.avcdn.net/ipm/preview/icons/icon-envelope-tick-green-avg-v1.png]
 
Virus-free.www.avg.com

Re: DistributedMapCacheServer persistent directory - Cluster wide same values after primary node changes

[Mark Payne]

I wouldn’t say that it should never be used in production. In fact, it’s used 
quite heavily in production. But it does have some limitations. Where those 
limitations are acceptable, it’s still very reasonable to use. Redis offers a 
lot on top, but it comes with complexity also, having to manage another 
service, etc.

When DIstriibutedMapCacheServer is used in a cluster, it is run on all nodes. 
But you need to point to just a single node. It can be any node in the cluster. 
But the client on every node should point to the same node in your cluster 
(nifi-0, for instance), not localhost. In this way, Primary Node doesn’t 
matter. Primary Node can switch 1,000 times and you’ll still be pointing at the 
same node that has all of the data. But it does mean that the service has a 
very real limitation, in that it’s a single point of failure. If that 
particular node goes down, the DistributedMapCache clients won’t be able to 
communicate with it until that node comes back up. If that limitation is okay 
for you, then by all means you can use it in production. But if you need 
something that provides High Availability, most people turn to Redis.

Thanks
-Mark


> On Oct 15, 2022, at 11:19 AM, Jörg Hammerbacher  
> wrote:
> 
> Hi Chris,
> 
> thanks a lot for you information.
> 
> I was not aware that "DistributedMapCacheServer" should not be used in 
> production. Maybe a short hint in the Controller Service Documentation would 
> be helpful also for other users.
> 
> Pointing to RedisDistributedMapCacheClientService lead us to the decision 
> using Redis in the future for distributed caching data (used it he first time 
> now). What type of Redis persistence type to be used (RDB and/or AOF) would 
> be important to handle data loss vs. performance.
> 
> In general i would like to say thank you to all the people who constantly 
> develop the NiFi ecosystem! Well done.
> 
> regards,
> Jörg
> 
> 
> On 2022/10/14 16:22:14 Chris Sampson wrote:
> > The DistributedMapCacheServer is, I believe, meant as a reference
> > implementation of the service to be used as an example rather than in
> > production. The kind of scenario you describe is exactly the reason to not
> > use this in-memory (optionally locally persisted on disk) in a clustered
> > production environment.
> >
> > That said, it can be used if the use case of the Flow doesn't have problems
> > if a node goes offline, etc.
> >
> > The recommended approach is to use an external service such as Redis with
> > the RedisDistributedMapCacheClientService [1]. This can interface with your
> > external Redis cluster/instance using the same API. Other external services
> > can be used, see the selection of related Controller Services in the nifi
> > docs [2] (e.g. search for "cache").
> >
> > [1]:
> > https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-redis-nar/1.18.0/org.apache.nifi.redis.service.RedisDistributedMapCacheClientService/index.html
> >
> > [2]: https://nifi.apache.org/docs/nifi-docs
> >
> > On Fri, 14 Oct 2022, 17:13 Jörg Hammerbacher,  
> > wrote:
> >
> > > Hi,
> > >
> > >
> > > I have one thing where i am looking for a solution. Maybe someone can
> > > help me out or give me hint how to do.
> > >
> > >
> > > Problem:
> > >
> > > I often use a NiFi Clusters with "DistributedMapCacheClientService"
> > > which uses a "DistributedMapCacheServer" for cluster wide key/value
> > > storage. Per default the DMCS uses "in memory" and sockets for
> > > synchronization. We use a persistence directory to make the data
> > > persistent and to avoid that the data is gone after restarting the
> > > entire cluster. But in the case, if the primary node changes, i think
> > > the data will be outdated or used from a potential outdated other node.
> > > If this other Node takes the primary node role, old data will be used
> > > for next FecthDistrubutedMapCache. The latest updates over the old
> > > primar node are gone.
> > >
> > > Is there a service using e.g. zookeeper "int the backgroud" to get a
> > > real distributed persitent Cache - even after restarting the entire
> > > cluster / all nodes?
> > >
> > >
> > > I know, the standard cache is able to provide a hugh frequent
> > > read/update servise if the data is in memory. But if we need just one or
> > > max a few updates per minute ...
> > >
> > > Yes, using another system like a Database (as persistent singleton) can
> > > be a solution - a not really matching solution. Why is there no standard
> > > service in NiFi for this? Isn't it a good idea or i am the only one with
> > > this problem in the past?
> > >
> > >
> > > Thanks in advance for answers,
> > >
> > > Jörg (Hammerbacher)
> > >
> > >
> > >
> >
> 
> -- 
> mit freundlichen Grüßen,
> Jörg Hammerbacher
> http://www.hammerbacher-it.de
> j...@hammerbacher-it.de
> 

Re: NiFi 1.18.0 Sensitive Property broken after Upgrade

[Mark Payne]

Hey Josef,

I’m sorry about the trouble. It looks like this issue was reported here [1]. We 
are looking into a fix for it.

Fortunately, if you don’t want to wait for the fix there is a workaround 
available.

The work around is to follow these steps:
1. Instead of jumping straight to 1.18, update first to 1.16.4
2. Start NiFi and wait for it to start up. Ensure that all looks healthy.
3. Shutdown NiFi
4. Upgrade to 1.18, ensuring that you copy over the conf/flow.json.gz file from 
1.16.4

So essentially, you’d need to upgrade from 1.15 to 1.16, and then to 1.18.

The reason this works is that prior to 1.16, we stored the flow in 
conf/flow.xml.gz. But in 1.16 we updated that to flow.json.gz - and also kept 
around flow.xml.gz in order to make this change seemless.
But it looks like when Sensitive Dynamic Properties was added, there was a bug 
that caused us to not properly load things from flow.xml.gz, only from 
flow.json.gz.
So, if you upgrade first to 1.16.4, you’ll end up with a flow.json.gz that you 
can then copy over to your 1.18 instance.

I know this is not ideal, and I apologize for that. But if you’re looking to 
upgrade right away this will be quicker than waiting for a resolution of 
NIFI-10567.

Thanks!
-Mark

[1] https://issues.apache.org/jira/browse/NIFI-10567


On Oct 13, 2022, at 8:28 AM, 
josef.zahn...@swisscom.com wrote:

I just found this blog 
https://exceptionfactory.com/posts/2022/08/02/implementing-apache-nifi-support-for-sensitive-dynamic-properties/
 about the jira ticket 
https://issues.apache.org/jira/browse/NIFI-9957?jql=text%20~%20%22sensitive%20dynamical%22
 . What we found out it is that the controller DBCPConnectionPool is fine with 
the password as well as the invokeHTTP. So for the ones where sensitive dynamic 
properties has been enabled the migration of the password was successful, but 
not for the others…

Cheers Josef


From: "Zahner Josef, GSB-LR-TRW-LI" 
mailto:josef.zahn...@swisscom.com>>
Date: Thursday, 13 October 2022 at 13:59
To: "users@nifi.apache.org" 
mailto:users@nifi.apache.org>>
Subject: NiFi 1.18.0 Sensitive Property broken after Upgrade

Hi guys

We just upgraded from NiFi 1.15.3 to 1.18.0. We have somehow a migration issue, 
it seems that all our sensitive properties are broken with 1.18.0. Check my 
screenshot below, It’s related to controller services as well as to processors. 
All sensitive properties shows us an error. If we replace the password it’s 
fine, so it seems that the password got corrupt due to the upgrade. Any hints? 
It leads to a ton of work, we have hundreds of processors with a hardcoded 
password… I’ve seen that we can use external password providers, do we have to 
migrate somehow?










Cheers Josef

Re: Content Repository Performance

[Mark Payne]

Joe,

What kind of hardware are you running on?

Thanks
Mark

Sent from my iPhone

> On Oct 5, 2022, at 9:23 AM, Joe Obernberger  
> wrote:
> 
> Hi all - I'm using NiFi 1.16.3 in a 3 node cluster with content repository 
> settings:
> 
> # Content Repository
> nifi.content.repository.implementation=org.apache.nifi.controller.repository.FileSystemRepository
> nifi.content.claim.max.appendable.size=50 KB
> nifi.content.repository.directory.default=./content_repository
> nifi.content.repository.archive.max.retention.period=2 days
> nifi.content.repository.archive.max.usage.percentage=50%
> nifi.content.repository.archive.enabled=false
> nifi.content.repository.always.sync=false
> nifi.content.viewer.url=../nifi-content-viewer/
> 
> All 3 nodes are pegging the disk where the content repository is stored.  
> Each directory (on each node) is about 8GBytes, but I'm having poor 
> performance because nifi is pummeling the disk.  What can I check / change to 
> help?  The disk is only for NiFi.
> Thank you!
> 
> -Joe
> 
> 
> -- 
> This email has been checked for viruses by AVG antivirus software.
> www.avg.com

Re: Trouble configuring logging

[Mark Payne]

This is because of how NiFi is run. When you startup nifi (bin/nifi.sh start) 
it doesn’t directly start the NiFi process.
Instead, it starts a different processor, which is called RunNiFi. This RunNiFi 
process is responsible for doing a lot of things, including monitoring the nifi 
process and if it dies restarting it.
Anything written to NiFi’s Standard Out goes to this processor, which then logs 
it.
So you’d probably need to update the logging for the appender writing to the 
bootstrap file:
mailto:kdo...@apache.org>> wrote:

Dylan - I looked into this and am yet unable to offer an explaination. Perhaps 
others that are familiar with how org.apache.nifi.StdOut can shed some light, 
or else I will keep digging when I have a block of time. To help in my 
understanding: Which Docker image are you using? Is it the apace/nifi image or 
a custom one, and if custom, can you share the Dockerfile?

Thanks,
Kevin

On Sep 27, 2022 at 10:21:12, Dylan Klomparens 
mailto:dklompar...@foodallergy.org>> wrote:
I am attempting to configure logging for NiFi. I have NiFi running in a Docker 
container, which sends all console logs to AWS CloudWatch. Therefore, I am 
configuring NiFi to send all logs to the console.

The problem is, for some reason all log messages are coming from the 
org.apache.nifi.StdOut logger. I cannot figure out why, since I would like 
messages to be printed directly from the logger that is receiving them.

It seems like messages are "passing through" loggers, which are ultimately 
printed out from the org.apache.nifi.StdOut logger. Here is an example of one 
log message:
2022-09-27 10:08:01,849 INFO [NiFi logging handler] org.apache.nifi.StdOut 
2022-09-27 10:08:01,848 INFO [pool-6-thread-1] 
o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile 
Repository

Why would every single log message come from the StdOut logger? And how can I 
have logs delivered from the logger they're supposedly originally coming from?

My logback.xml configuration is below for reference.



  

  
true
  

  

  %date %level [%thread] %logger{40} %msg%n

  

  
  
  
  
  

  
  
  
  
  
  
  
  

  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  
  

  
  

  
  

  
  

  

  
  
  
  
  
  
  
  
  
  
  

  
  

  

  

Re: Need help to merge all records in cluster into one flowfile

[Mark Payne]

. But it's in fact the Details field which does not 
help me.
At 08:16:00 all 3 nodes generate a SiteToSiteStatusReport.
At 08:16:11.003 the MergeRecords have a JOIN event. Joining 2 files: "Records 
Merged due to: Bin has reached Max Bin Age"
At 08:16:11.008 the MergeRecords have another JOIN event. Joining 1 file: 
"Records Merged due to: Bin has reached Max Bin Age"

So one file is 0.005s younger than the other 2 files and therefore is not 
merged into the first bin of files. But how can we force all flowfiles to be 
merged into one flowfile?
If I set the minimum file size or records to be within range of the >2 files 
and <3 files, it will trigger a merge. But when we create more flows the 
records and filesize will increase and we will be back to the problem that not 
all files will be merged into one.

kind regards
Jens

Den tir. 30. aug. 2022 kl. 15.40 skrev Mark Payne 
mailto:marka...@hotmail.com>>:
Hey Jens,

My recommendation is to take a look at the data provenance for MergeRecord 
(i.e., right-click on the Processor and go to Data Provenance.) Click the 
little ‘i’ icon on the left for one of the JOIN events.
There, it will show a “Details” field, which will tell you why it merged the 
data in the bin.
Once you understand why it’s merging the data with only 2 FlowFiles, you should 
be to understand how to adjust your configuration to avoid doing that.

Thanks
-Mark


> On Aug 30, 2022, at 2:31 AM, Jens M. Kofoed 
> mailto:jmkofoed.ube%2bn...@gmail.com>> wrote:
>
> Hi all
>
> I'm running a 3 node cluster at version 1.16.2. I'm using the 
> SiteToSiteStatusReportingTask to monitor and check for any backpressures or 
> queues. I'm trying to merge all 3 reports into 1, but must of the times I 
> always get 2 flowfile after my MergeRecord.
>
> To be sure the nodes are creating the reports at the same time the 
> SiteToSiteStatusReportingTask is set to schedule via CRON driver every 5 mins.
> The connection from the input port to the next process is set with "Load 
> Balance Strategy" to Single node, to be sure all 3 reports are at the same 
> node.
> In my MergeRecord the "Correlation Attribute Name" is set to 
> "reporting.task.uuid" which is the same for all 3 flowfiles.
> "Minimum Number of Records" is set to 5000, which is much higher than the 
> total amounts of records.
> "Minimum Bin Size" is set to 5 MB, which is also much higher than the total 
> size. Maximum "Number of Bins" is at default: 10
> "Max Bin Age" is set to 10 s.
>
> With these setting I was hoping that all 3 reports, should be at the same 
> node within a few seconds. And that the mergeRecods will merge all 3 
> flowfiles into 1. But many time the mergeRecord outputs 2 flowfiles.
>
> Any ideas how to force all into one flowfile.
>
> Kind regards
> Jens M. Kofoed

Re: Need help to merge all records in cluster into one flowfile

[Mark Payne]

Hey Jens,

My recommendation is to take a look at the data provenance for MergeRecord 
(i.e., right-click on the Processor and go to Data Provenance.) Click the 
little ‘i’ icon on the left for one of the JOIN events.
There, it will show a “Details” field, which will tell you why it merged the 
data in the bin.
Once you understand why it’s merging the data with only 2 FlowFiles, you should 
be to understand how to adjust your configuration to avoid doing that.

Thanks
-Mark


> On Aug 30, 2022, at 2:31 AM, Jens M. Kofoed  
> wrote:
> 
> Hi all
> 
> I'm running a 3 node cluster at version 1.16.2. I'm using the 
> SiteToSiteStatusReportingTask to monitor and check for any backpressures or 
> queues. I'm trying to merge all 3 reports into 1, but must of the times I 
> always get 2 flowfile after my MergeRecord.
> 
> To be sure the nodes are creating the reports at the same time the 
> SiteToSiteStatusReportingTask is set to schedule via CRON driver every 5 mins.
> The connection from the input port to the next process is set with "Load 
> Balance Strategy" to Single node, to be sure all 3 reports are at the same 
> node.
> In my MergeRecord the "Correlation Attribute Name" is set to 
> "reporting.task.uuid" which is the same for all 3 flowfiles.
> "Minimum Number of Records" is set to 5000, which is much higher than the 
> total amounts of records.
> "Minimum Bin Size" is set to 5 MB, which is also much higher than the total 
> size. Maximum "Number of Bins" is at default: 10
> "Max Bin Age" is set to 10 s.
> 
> With these setting I was hoping that all 3 reports, should be at the same 
> node within a few seconds. And that the mergeRecods will merge all 3 
> flowfiles into 1. But many time the mergeRecord outputs 2 flowfiles.
> 
> Any ideas how to force all into one flowfile.
> 
> Kind regards
> Jens M. Kofoed

Re: Crash on startup due to Output port issue

[Mark Payne]

Benji,

Fantastic, thanks for following up! We’ve clearly got something to address but 
I’m very happy that we’ve found a way to get you unblocked for now.

Thanks
Mark

Sent from my iPhone

On Aug 1, 2022, at 12:46 PM, BeNJ  wrote:


Updates:
Mark: Deleting the json allows nifi to start up(!!) - note that restarting 
(with the newly generated json in place) causes the same issue on startup.
Joe: I'm happy to run custom debug nars to help figure out how to resolve this.

Thank you, I really appreciate everyone's help with this!
Benji



On Mon, Aug 1, 2022 at 8:00 AM Joe Witt 
mailto:joe.w...@gmail.com>> wrote:
Ben

I didn't see any log levels that would help narrow this down.  If Mark's 
suggestion does not address it it will likely require updating the flow 
configuration manually or a patched framework nar to do better logging.

I'm also wondering if the json form of the flow could be imported on a clean 
nifi (you'd have to re-enter sensitive values) or imported to a registry then 
imported to nifi.

Given you have hit this and someone else did (on stackoverflow) we clearly have 
an important case to figure out here as it is obviously disruptive.

Thanks

On Mon, Aug 1, 2022 at 5:41 AM Mark Payne 
mailto:marka...@hotmail.com>> wrote:
Benji,

I would recommend you try to remove (or rename) the flow.json.gz - but not the 
flow.xml.gz. See if that makes any difference.

Thanks
-Mark

Sent from my iPhone

On Jul 31, 2022, at 11:35 PM, BeNJ 
mailto:intothev...@gmail.com>> wrote:


Also please see the attached nifi.properties

Thanks,
Benji

On Sun, Jul 31, 2022 at 4:28 PM BeNJ 
mailto:intothev...@gmail.com>> wrote:
Hi Joe,
Stack with a couple of info logs from before and after, and the final exit 
shortly after.
--
2022-07-31 16:20:35,311 INFO [main] o.a.n.g.StandardProcessGroupSynchronizer 
Added Connection[ID=cfee198f-3d2b-1513-f741-e71ad122, Source 
ID=cfee198e-3d2b-1513-7a9c-f0c2a8cf0d43, Dest 
ID=cfee19cf-3d2b-1513-4a87-5f50a90fdabf] to 
StandardProcessGroup[identifier=cfee1961-3d2b-1513-c8a6-fdf1a8fe4ff5,name=Add 
Customer User]
2022-07-31 16:20:35,311 INFO [main] o.a.n.g.StandardProcessGroupSynchronizer 
Added Connection[ID=cfee1974-3d2b-1513-e4df-9dbba1241682, Source 
ID=cfee1971-3d2b-1513-555c-1aedf0f0801f, Dest 
ID=cfee1970-3d2b-1513-de1d-f5bee9ad679e] to 
StandardProcessGroup[identifier=cfee1961-3d2b-1513-c8a6-fdf1a8fe4ff5,name=Add 
Customer User]
2022-07-31 16:20:35,317 INFO [Timer-Driven Process Thread-9] 
o.a.n.c.s.StandardControllerServiceNode Successfully enabled 
StandardControllerServiceNode[service=StandardOauth2AccessTokenProvider[id=cfee1b5b-3d2b-1513-7124-85b028901ac8],
 name=customer user management idp, active=true]
2022-07-31 16:20:35,325 INFO [main] o.a.n.c.s.AffectedComponentSet Starting the 
following components: AffectedComponentSet[inputPorts=[], outputPorts=[], 
remoteInputPorts=[], remoteOutputPorts=[], processors=[], 
controllerServices=[], reportingTasks=[]]
2022-07-31 16:20:35,328 WARN [main] org.eclipse.jetty.webapp.WebAppContext 
Failed startup of context 
o.e.j.w.WebAppContext@ffaaaf0{nifi-api,/nifi-api,file:///opt/nifi/nifi-current/work/jetty/nifi-web-api-1.16.0.war/webapp/,UNAVAILABLE}{./work/nar/extensions/nifi-server-nar-1.16.0.nar-unpacked/NAR-INF/bundled-dependencies/nifi-web-api-1.16.0.war}
org.apache.nifi.controller.serialization.FlowSynchronizationException: 
java.lang.IllegalStateException: Cannot add Connection to Process Group because 
source is an Output Port that does not belong to a child Process Group
at 
org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.synchronizeFlow(VersionedFlowSynchronizer.java:362)
at 
org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.sync(VersionedFlowSynchronizer.java:185)
at 
org.apache.nifi.controller.serialization.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:43)
at 
org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1479)
at 
org.apache.nifi.persistence.StandardFlowConfigurationDAO.load(StandardFlowConfigurationDAO.java:104)
at 
org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:815)
at 
org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:538)
at 
org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:67)
at 
org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1073)
at 
org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:572)
at 
org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:1002)
at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:746)
at 
org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:379)
at org.eclipse.jetty.webapp.W

Re: Crash on startup due to Output port issue

[Mark Payne]

Benji,

I would recommend you try to remove (or rename) the flow.json.gz - but not the 
flow.xml.gz. See if that makes any difference.

Thanks
-Mark

Sent from my iPhone

On Jul 31, 2022, at 11:35 PM, BeNJ  wrote:


Also please see the attached nifi.properties

Thanks,
Benji

On Sun, Jul 31, 2022 at 4:28 PM BeNJ 
mailto:intothev...@gmail.com>> wrote:
Hi Joe,
Stack with a couple of info logs from before and after, and the final exit 
shortly after.
--
2022-07-31 16:20:35,311 INFO [main] o.a.n.g.StandardProcessGroupSynchronizer 
Added Connection[ID=cfee198f-3d2b-1513-f741-e71ad122, Source 
ID=cfee198e-3d2b-1513-7a9c-f0c2a8cf0d43, Dest 
ID=cfee19cf-3d2b-1513-4a87-5f50a90fdabf] to 
StandardProcessGroup[identifier=cfee1961-3d2b-1513-c8a6-fdf1a8fe4ff5,name=Add 
Customer User]
2022-07-31 16:20:35,311 INFO [main] o.a.n.g.StandardProcessGroupSynchronizer 
Added Connection[ID=cfee1974-3d2b-1513-e4df-9dbba1241682, Source 
ID=cfee1971-3d2b-1513-555c-1aedf0f0801f, Dest 
ID=cfee1970-3d2b-1513-de1d-f5bee9ad679e] to 
StandardProcessGroup[identifier=cfee1961-3d2b-1513-c8a6-fdf1a8fe4ff5,name=Add 
Customer User]
2022-07-31 16:20:35,317 INFO [Timer-Driven Process Thread-9] 
o.a.n.c.s.StandardControllerServiceNode Successfully enabled 
StandardControllerServiceNode[service=StandardOauth2AccessTokenProvider[id=cfee1b5b-3d2b-1513-7124-85b028901ac8],
 name=customer user management idp, active=true]
2022-07-31 16:20:35,325 INFO [main] o.a.n.c.s.AffectedComponentSet Starting the 
following components: AffectedComponentSet[inputPorts=[], outputPorts=[], 
remoteInputPorts=[], remoteOutputPorts=[], processors=[], 
controllerServices=[], reportingTasks=[]]
2022-07-31 16:20:35,328 WARN [main] org.eclipse.jetty.webapp.WebAppContext 
Failed startup of context 
o.e.j.w.WebAppContext@ffaaaf0{nifi-api,/nifi-api,file:///opt/nifi/nifi-current/work/jetty/nifi-web-api-1.16.0.war/webapp/,UNAVAILABLE}{./work/nar/extensions/nifi-server-nar-1.16.0.nar-unpacked/NAR-INF/bundled-dependencies/nifi-web-api-1.16.0.war}
org.apache.nifi.controller.serialization.FlowSynchronizationException: 
java.lang.IllegalStateException: Cannot add Connection to Process Group because 
source is an Output Port that does not belong to a child Process Group
at 
org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.synchronizeFlow(VersionedFlowSynchronizer.java:362)
at 
org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.sync(VersionedFlowSynchronizer.java:185)
at 
org.apache.nifi.controller.serialization.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:43)
at 
org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1479)
at 
org.apache.nifi.persistence.StandardFlowConfigurationDAO.load(StandardFlowConfigurationDAO.java:104)
at 
org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:815)
at 
org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:538)
at 
org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:67)
at 
org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1073)
at 
org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:572)
at 
org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:1002)
at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:746)
at 
org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:379)
at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1449)
at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1414)
at 
org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:916)
at 
org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:288)
at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524)
at 
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at 
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at 
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at 
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at 
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at 
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at 
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
at 
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:426)
at 
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at 

Re: retry issue

[Mark Payne]

Greg,

Whether the output goes to “nonzero status” or “output stream”, the original 
FlowFile should still be sent to the “original” relationship.

Now, if you set retries to say 3 for “nonzero status” and a FlowFile is routed 
to that relationship, what will happen is that the FlowFile is retried. Nothing 
is routed to “original” or to “nonzero status”.
It then tries again. This time, let’s say it goes to “nonzero status” again. 
Adds it back to the input queue.

It then tries a third time. If it goes to “nonzero status”, then the original 
FlowFile should still go to “original” and the result of the command goes to 
“nonzero status.”
If it instead goes to “output stream” then the original FlowFiles goes to 
“original” and the result of the command goes to “output stream.”
In either case, the contents going to the “original” relationship should be the 
incoming FlowFile, unmodified.

Am I misunderstanding what you’re trying to achieve?

Thanks
-Mark


> On Jul 20, 2022, at 5:37 PM, Gregory M. Foreman 
>  wrote:
> 
> Mark:
> 
> Thank you for the explanation, yes it helps.  I made the changes and 
> everything worked with one exception: preserving the original content.  The 
> non-zero status will replace the flowfile content with whatever came through 
> the output stream, so the original content is lost.  When the file is 
> processed correctly, two flowfiles are created for the original and the 
> output stream relationships.  Connecting the original to a routeonattribute 
> processor that inspects the execution.status code and traps original files 
> that are errors is an option.  Are there any options besides using the second 
> processor?
> 
> Thanks,
> Greg
> 
>> On Jul 20, 2022, at 9:22 AM, Mark Payne  wrote:
>> 
>> Greg,
>> 
>> You wouldn’t want to retry the “original” relationship. The processor has 3 
>> relationships: original, output stream, and nonzero status. It should always 
>> send the incoming FlowFile to original. So if you retry that relationship 
>> you’ll always retry the flowfile, regardless of whether it was successful or 
>> not.
>> 
>> Instead, you should retry the “nonzero status” relationship. If a FlowFIle 
>> is routed to this relationship, it will instead be re-queued and processed 
>> again. Nothing will go to original in this case, because the processing is 
>> atomic.
>> 
>> But if a FlowFile is routed to the “output stream” relationship (because 
>> it’s not retried) the FlowFile will continue on normally. And the original 
>> FlowFile will go to ‘original’.
>> 
>> Additionally, if all retries are completed and it still is routed to 
>> “nonzero status” then at that point, the FlowFile will go to “nonzero 
>> status” and the original will be transferred to whatever connection (if any) 
>> you have for the “original” relationship.
>> 
>> Does that help?
>> 
>> Thanks
>> -Mark
>> 
>>> On Jul 20, 2022, at 9:12 AM, Gregory M. Foreman 
>>>  wrote:
>>> 
>>> Hello:
>>> 
>>> The ExecuteStreamCommand processor has 3 relationships.  To trap processing 
>>> failures, I capture flowfiles from the original relationship, inspect the 
>>> cmd exit status, and reroute to a RetryFlowFile processor if the exit 
>>> status is not 0.  I wanted to see if this could be simplified with the new 
>>> retry feature in 1.16.  When I enable retry on the original relationship 
>>> and execute a failing cmd, the flowfiles remain penalized in the previous 
>>> success queue, but never exit to the original relationship.  Is this 
>>> scenario supposed to work as I have it setup?
>>> 
>>> Thanks,
>>> Greg
>> 
> 

Re: retry issue

[Mark Payne]

Greg,

You wouldn’t want to retry the “original” relationship. The processor has 3 
relationships: original, output stream, and nonzero status. It should always 
send the incoming FlowFile to original. So if you retry that relationship 
you’ll always retry the flowfile, regardless of whether it was successful or 
not.

Instead, you should retry the “nonzero status” relationship. If a FlowFIle is 
routed to this relationship, it will instead be re-queued and processed again. 
Nothing will go to original in this case, because the processing is atomic.

But if a FlowFile is routed to the “output stream” relationship (because it’s 
not retried) the FlowFile will continue on normally. And the original FlowFile 
will go to ‘original’.

Additionally, if all retries are completed and it still is routed to “nonzero 
status” then at that point, the FlowFile will go to “nonzero status” and the 
original will be transferred to whatever connection (if any) you have for the 
“original” relationship.

Does that help?

Thanks
-Mark

> On Jul 20, 2022, at 9:12 AM, Gregory M. Foreman 
>  wrote:
> 
> Hello:
> 
> The ExecuteStreamCommand processor has 3 relationships.  To trap processing 
> failures, I capture flowfiles from the original relationship, inspect the cmd 
> exit status, and reroute to a RetryFlowFile processor if the exit status is 
> not 0.  I wanted to see if this could be simplified with the new retry 
> feature in 1.16.  When I enable retry on the original relationship and 
> execute a failing cmd, the flowfiles remain penalized in the previous success 
> queue, but never exit to the original relationship.  Is this scenario 
> supposed to work as I have it setup?
> 
> Thanks,
> Greg

Re: Search only within current Process Group

[Mark Payne]

Jim,

Take a look at 
https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#filters

Thanks
-Mark


On Jul 13, 2022, at 8:55 AM, James McMahon 
mailto:jsmcmah...@gmail.com>> wrote:

Within the UI there is a Search frame in the upper right, and this seems to 
result in a search globally against all throughout the NiFi UI. Is there a way 
to initiate a search solely within the current Process Group?

Re: flow.xml.gz > flow.json.gz migration

[Mark Payne]

Hi Greg,

WIth 1.16, NiFi expects to write out both the xml version AND the json version.

While we are in the process of making the json version the canonical version of 
representing a full flow and a flow snippet, the change in 1.16 was very large 
and critical to get right. So, just in case there were any bugs introduced in 
the logic, we write out the flow in both the new and the old form. This way, if 
there’s an issue with reading/restoring from json or writing it, users have a 
way to work around it by simply deleting the .json file and loading the xml 
version. Eventually, we’ll remove support for the XML version, but for now we 
keep both around.

Thanks
-Mark


> On Jul 12, 2022, at 8:34 AM, Gregory M. Foreman 
>  wrote:
> 
> Hello:
> 
> My client upgraded to Nifi 1.16.1.  With the original flow.xml.gz and 
> autogenerated flow.json.gz in place everything works fine.  When 
> nifi.flow.configuration.json.file is set to the json file location and the 
> original nifi.flow.configuration.file is commented out and flow.xml.gz 
> removed, the server will not start.  Is there any migration guidance for 
> moving from flow.xml.gz to flow.json.gz?
> 
> Thanks,
> Greg

Re: How to configure wait process along with ListSFTP process

[Mark Payne]

Ben,

The Wait processor is not intended for this purpose. It’s intended to be used 
along with the Notify processor. So the processor will block FlowFiles going 
forward until the Notify processor indicates that the FlowFile should be 
released.

Take a look at ListSFTP’s “Minimum File Age” property. You can set that to say 
1 minute. That will avoid listing the file until it’s at least 1 minute old.

Or, alternatively, you can configure FetchSFTP to retry failures and set the 
Penalty Duration to 1 minute. That way, if it fails, it will wait 1 minute and 
try again, up to the configured number of retries.

Thanks
-Mark


> On Jun 26, 2022, at 6:31 AM, Ben .T.George  wrote:
> 
> Hello,
> 
> How to configure the wait process along with ListSFTP process because before 
> fetching the file i need to run a schedule to change the permission of the 
> file which runs every minute.
> 
> 
> 
> Regards,
> Ben

Re: Fork-Join Enrichment, Input Dependent Behaviour

[Mark Payne]

Hey Steve,

Thanks for reporting this. I filed a Jira [1] for the issue. I have a Pull 
Request up for it, so it should be fixed in the next release.

Thanks
-Mark


[1] https://issues.apache.org/jira/browse/NIFI-10169


On Jun 24, 2022, at 1:23 PM, 
stephen.hindma...@bt.com wrote:

HI,

I seem to have a situation similar to 
[NIFI-9903], this time in the 
JoinEnrichment processor.

I am performing some enrichments, and sometimes the enrichment look up fails as 
the item is not there. If the enrichment fails for the first item in an array 
of records then the merge fails to properly merge subsequent records in the 
same array. The join processor uses an “infer” JSON reader for the original 
leg, a schema based reader for the enriched leg, and a “inherit” JSON writer. I 
am using the “Insert” join strategy. Here is an example.

If the original record is

[{"transport_protocol_id":17,"Enrichment":{}},{"transport_protocol_id":6,"Enrichment":{}}]

Then both lookups succeed, and the enrichment record looks like this

[
  {"network_transport" : {"name" : "udp", "code" : 17, "alias" : "UDP", 
"comment" : "user datagram protocol"}},
  {"network_transport" : {"name" : "tcp", "code" : 6, "alias" : "TCP", 
"comment" : "transmission control protocol"}
]

And the joined record looks like this.

[
  {"transport_protocol_id" : 17,"Enrichment" : {
"network_transport" : {"name" : "udp","code" : 17,"alias" : "UDP","comment" 
: "user datagram protocol"}}},
  {"transport_protocol_id" : 6,"Enrichment" : {
"network_transport" : {"name" : "tcp","code" : 6,"alias" : "TCP","comment" 
: "transmission control protocol"}}}
]

However if the first record has a key value that is out of range, such as this

[{"transport_protocol_id":,"Enrichment":{}},{"transport_protocol_id":6,"Enrichment":{}}]

Then the first record in the enriched leg will be null, even if the rest of the 
records are correct. However the enrichment is still valid JSON once I have 
processed it in the enrichment leg.

[
  {"network_transport" : null},
  {"network_transport" : {"name" : "tcp", "code" : 6,"alias" : "TCP", "comment" 
: "transmission control protocol"}}
]

But the joined record does not properly process the subsequent records, and the 
content looks like this.

[
  {"transport_protocol_id" : ,"Enrichment" : {
"network_transport" : null}},
  {"transport_protocol_id" : 6,"Enrichment" : {
"network_transport" : "MapRecord[{name=tcp, alias=TCP, comment=transmission 
control protocol, code=6}]"}}
]

Is there any step I could use to ensure the join happens as expected? Or is 
this the same situation as the JIRA I mentioned above? I am not able to use a 
schema based writer as our real case has too many input record types and 
enrichment options that the number of combinations, and hence schemas, could 
not be managed.

Thanks

Steve Hindmarch

Re: Requesting Apache NiFi Help/Guidelines

[Mark Payne]

Ben,

If you run sftp from the commandline what do you see for the timestamps on 
those files?

I am wondering if either there’s a big discrepancy between the time on the SFTP 
server and the time on the server where NiFi is running; or if the SFTP server 
is setup in such a way that it does not update the Last Modified Time for files.

Thanks
-Mark


On Jun 20, 2022, at 12:54 PM, Ben .T.George 
mailto:bentech4...@gmail.com>> wrote:

HI,

I have not set any values to Min/Max age and size properties as I was not aware 
of it.



What should I set for this?


On Mon, Jun 20, 2022 at 7:42 PM Mark Payne 
mailto:marka...@hotmail.com>> wrote:
Ben,

So in the message there, you can see that it found 11 files in the /tmp 
directory, but none of those files matched the filter. So you’ll get no output.
What do you have set for the Minimum/Maximum File Age and for the 
Minimum/Maximum File Size properties? Those are used to filter the results.

Thanks
-Mark

On Jun 20, 2022, at 12:35 PM, Ben .T.George 
mailto:bentech4...@gmail.com>> wrote:

HI

Thanks for response,

i Still i am very new to it and not sure how to explain , i can attach some 
screenshots


 this connector is showing zero files



but when i test that process, file found



ListSFTP process




On Mon, Jun 20, 2022 at 7:21 PM Mark Payne 
mailto:marka...@hotmail.com>> wrote:
Ben,

ListSFTP -> FetchSFTP is how you’d want to get started. You’d then want to 
connect that to a PutSFTP to send to Server B and another PutSFTP to send to 
Server C.

You said that it is not working as expected. What does that mean? Are you 
seeing errors? Not seeing the data show up?

Thanks
-Mark


On Jun 20, 2022, at 10:20 AM, Ben .T.George 
mailto:bentech4...@gmail.com>> wrote:


Hello,

i am very new to NiFI and trying to get more details about NiFi,

My end goal is to achieve some kind of SFTP solution , were i need to transfer 
file from server A to Server B, Then Server C, which is public,

Can you please help me to achieve this, or explain the basics of it.

I was trying to use ListSFTP and FetchSFTP, which does not work as expected.

Your help will be highly appreciated.

Thanks & Regards,
Ben




--
Yours Sincerely
Ben.T.George
Phone : +965 - 50629829 / 94100799

" Live like you will die tomorrow, learn like you will live forever "



--
Yours Sincerely
Ben.T.George
Phone : +965 - 50629829 / 94100799

" Live like you will die tomorrow, learn like you will live forever "

Re: Nifi custom processor to consume two flow files at once.

[Mark Payne]

Vibhath,

That’s correct, all of the data is received as if through a single connection. 
There’s no notion of named inputs.

Unfortunately, that makes this a pattern that’s a bit more difficult to 
implement than I’d like.
Generally, the way this is handled would be to add an attribute to the FlowFile 
in your flow using UpdateAttribute.
Then your processor can use that to make sense of what the FlowFile is.
So you might have a flow like:

SourceA -> UpdateAttribute (add attribute ’type’ with value ’Type1’) —> 
YourProcessor
SourceB -> UpdateAttribute (add attribute ’type’ with value ’Type2’) —/

Then, in YourProcessor, you can get the FlowFiles using a FlowFileFilter so 
that you can grab one FlowFile of ’Type1’ and one FlowFile of ’Type2’.

OR, the alternative way, which may be easier, is to have your processor extend 
BinFiles instead of AbstractProcessor.
If you decide to go this route, you may want to take a look at JoinEnrichment. 
It uses this approach and does something similar where it needs two inputs, one 
of type ‘Original’ and one of type ‘Enrichment’.

Thanks
-Mark




On Jun 20, 2022, at 10:54 AM, Vibhath Ileperuma 
mailto:vibhatharunapr...@gmail.com>> wrote:

Hi All,

We are planning to develop a custom Nifi processor which consumes two files at 
once.
We have a set of files which contains two types of data; say 'A' type data and 
'B' type data. This processor receives these two type files from two upstream 
connections.
Processor needs to get one file from both the connections at once and do the 
processing.
As I understand, even though we connect multiple upstream connections as inputs 
to a processor, in the code, it treats all the data coming from a single 
upstream queue.
Is there a way to specify the number of input connections to the processor and 
take one file from each processor? If not, what is the flow file reading order 
if multiple input connections are available for a processor.?

Thank You.

Best regards,

Vibhath.

Re: Requesting Apache NiFi Help/Guidelines

[Mark Payne]

Ben,

ListSFTP -> FetchSFTP is how you’d want to get started. You’d then want to 
connect that to a PutSFTP to send to Server B and another PutSFTP to send to 
Server C.

You said that it is not working as expected. What does that mean? Are you 
seeing errors? Not seeing the data show up?

Thanks
-Mark


On Jun 20, 2022, at 10:20 AM, Ben .T.George 
mailto:bentech4...@gmail.com>> wrote:


Hello,

i am very new to NiFI and trying to get more details about NiFi,

My end goal is to achieve some kind of SFTP solution , were i need to transfer 
file from server A to Server B, Then Server C, which is public,

Can you please help me to achieve this, or explain the basics of it.

I was trying to use ListSFTP and FetchSFTP, which does not work as expected.

Your help will be highly appreciated.

Thanks & Regards,
Ben

Re: Issue with registry and Port error

[Mark Payne]

Agreed. Would definitely recommend 1.16 if possible. Otherwise, you might try 
stopping the process group, then change version, and then restart the process 
group.

Thanks
-Mark


On Jun 16, 2022, at 12:50 PM, Joe Witt 
mailto:joe.w...@gmail.com>> wrote:

Dave

The issue you flag might be related.  In the 1.15/1.16 lines we've improved a 
good bit of funky lifecycle state handling.

If you're able I recommend trying this on the 1.16.3 release for both NiFi and 
Registry.  If that still occurs we would like to see the nifi-app.log entries 
of nifi.

Thanks

On Thu, Jun 16, 2022 at 9:40 AM David Early 
mailto:david.ea...@grokstream.com>> wrote:
Hi,

I am using 1.15.1 NiFi and registry.

I have a process group with 72 processors and a bunch of services that is 
versioned.  I can install it new just fine, but  when I try and update from one 
version to the next (for a change as simple as moving a processor), I get the 
following error:




"Failed to perform update flow request due to Port cannot be disabled because 
it is not stopped"

The nearest thing I could find was this:
https://issues.apache.org/jira/browse/NIFI-8040

Does anyone know what is happening and if there is a fix?

Dave

Re: Setting an attribute in EvaluateXPath

[Mark Payne]

James,

XPath is namespace-specific. I.e., /root is not the same as 
/{urn:exo:/root}root or whatever syntax is appropriate (i haven’t used XPath in 
about a decade so i can’t remember the syntax off the top of my head).
But you should be able to use * to wildcard the namespace, I believe. So 
something like /*:root/*:set/*:rep/*:repId should work…. you may need /text() 
at the end?

> On Jun 17, 2022, at 10:18 AM, James McMahon  wrote:
> 
> I'm having a problem parsing values out of xml. In this example, I need to 
> get repId from this structure:
> 
> 
>   
>   
>   
>12345
> 
> In my EvaluateXPath I try to do this:
> Property:myId
> Value:/meeting/set/repSet/rep/repId
> 
> I try doing this to set an attribute named myId, but get back an empty value. 
> What do I need to change to get 12345 for myId?

Re: Load balancing just stopped working in NIFI 1.16.1

[Mark Payne]

Jens,

So that would tell us that the hostname or the port is wrong, or that NiFi is 
not running/listening for load balanced connections. I would recommend the 
following:

- Check nifi.properties on all nodes to make sure that the 
nifi.cluster.load.balance.host property is set to the node’s hostname.
- You could try deleting the state/ directory on the nodes and restarting. 
Perhaps somehow it’s got an old value cached and is trying to communicate with 
the wrong port?

Thanks
-Mark


> On May 19, 2022, at 7:42 AM, Jens M. Kofoed  wrote:
> 
> Hi
> 
> I have a 3 node test cluster running v. 1.16.1. Which has been working fine, 
> with no errors. But i doesn't do much, since it is my test cluster.
> But now I am struggling with load balance connection refuse between nodes.
> Both node 2 and 3 are refusing load balancing connections, even after 
> stopping the cluster. I have deleted all files at node 2 and 3, but they 
> still refuse connection.
> I have not made any change to the configuration, but I have created a new 
> flow which got node 1 to die.
> The new flow is using an ExecuteStreamCommand using stdin/stdout to a shell 
> command to manipulate with some data (1 GB files). I set the "Concurrent 
> Tasks" to high so node 1 ran out of memory and stopped. It continues to stop 
> if I tried to start it again. So I deleted the flow.gz file and the run, 
> state and work folders and started the node. Now node 1 was running again. 
> But after this node 2 and 3 are refusing load balancing connections.
> I can't see what this flow have to do with this, but I have now tried to 
> remove all files at node 2 and 3 to get clean nodes. But they still doesn't 
> work
> 
> Does any one have an idea how to debug further?
> 
> From the log:
> 2022-05-19 13:22:46,268 ERROR [Load-Balanced Client Thread-2] 
> o.a.n.c.q.c.c.a.n.NioAsyncLoadBalanceClient Unable to connect to 
> nifi-n03:8443 for load balancing
> java.net.ConnectException: Connection refused
> at sun.nio.ch.Net.connect0(Native Method)
> at sun.nio.ch.Net.connect(Net.java:482)
> at sun.nio.ch.Net.connect(Net.java:474)
> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:647)
> at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:107)
> at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:92)
> at 
> org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClient.createChannel(NioAsyncLoadBalanceClient.java:497)
> at 
> org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClient.establishConnection(NioAsyncLoadBalanceClient.java:440)
> at 
> org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClient.communicate(NioAsyncLoadBalanceClient.java:234)
> at 
> org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClientTask.run(NioAsyncLoadBalanceClientTask.java:81)
> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 
>  kind regards
> Jens

Re: Null Record in ConsumeKafkaRecord

[Mark Payne]

Hi Prasanth,

Take a look at the Record Writer that you’re using with ConsumeKafkaRecord. 
There’s a property name “Suppress Null Values.” You’ll want to set that to 
“Suppress Missing Values.” That should give you what you’re looking for.

Thanks
-Mark


On May 19, 2022, at 7:53 AM, Prasanth M Sasidharan 
mailto:prasha...@gmail.com>> wrote:

Hello Team,

I am using ConsumeKafkaRecord_2_0 1.15.3 processor in Nifi to consume JSON data 
from Kafka Topic.

My issue is that the consumeKafka output matches the schema of both records and 
adds the missing tags in the JSON with null value .

Eg:
[
{
"acknowledged": "0",
"internal_last": "2022-04-26 15:40:00",
"specific_probcause": "no associated text",
"poll": "0",
"type": "1",
"probable_cause": "no associated text",
"last_occurrence": "2022-04-26 15:40:00",
"service": "default",
"node": "re0",
"site_id": "no-siteid",
"device_class": "juniperjunosrtr",
"location": " ",
"network_first_time": "2022-04-26 15:40:00",
"agent": null,
"nodqqe": null
},
{
"acknowledged": "0",
"internal_last": "2022-04-26 15:40:00",
"specific_probcause": "no associated text",
"poll": "0",
"type": "1",
"probable_cause": " no associated text",
"node": null,
"last_occurrence": "2022-04-26 15:40:00",
"service": "default",
"site_id": "no-siteid",
"device_class": "juniperjunosrtr4",
"location": " ",
"network_first_time": "2022-04-26 15:40:00",
"agent": "test1",
"nodqqe": "ie-0"
}
]

In the above output the tags highlighted in RED are automatically inserted by 
the consumeKafkaRecord Processor. I assume that this is being done to match the 
schema of both the records. Is there a way to disable this? I would need the 
record as it is.
I am performing a header check after this step and due to the presence of this 
null value in the record, my header check isnt failing.

Any help would be much appreciated



--
Live every day as if it were your last, because one of these days, it will be.

Regards,

Prasanth M Sasidharan

Re: Referencing an attribute as the final ifelse/ifelse conditional value

[Mark Payne]

James,

If I’m reading it right, it looks like you’re using a comma between the )}  and 
the ${currentAttribute} ?

Thanks
-Mark



On May 6, 2022, at 12:47 PM, James McMahon 
mailto:jsmcmah...@gmail.com>> wrote:

UpdateAttribute is flagging the final } in my expression. How can I make the 
final value an attribute reference?

I need to set attribute currentAttribute to itself if neither ifElse case 
matches

${this.set:equals('ABCD'):and(${filename:find('.*daily.*)}):ifElse(
${this.set:equals('ABCD'):ifElse(
  'val_if_second', 'val_if_first'
 )}
${currentAttribute}
)}

I have used this syntax before, but with a value in that last position. How can 
I do this?

I would be happy to just leave it out, but I am given to understand it the 
ifElse / ifElse pair required it.

Re: Issue with version controlled inner PG and parameter contexts

[Mark Payne]

Hei Reinhard,

Thanks for reaching out. I believe you’re running into NIFI-9874 [1]. This has 
been addressed, but hasn’t been released yet. I think a 1.16.1 release will be 
coming soon, so you should be in good shape once that gets released.

Thanks
-Mark


[1] https://issues.apache.org/jira/browse/NIFI-9874


On Apr 25, 2022, at 11:14 AM, Reinhard Sell 
mailto:s...@dfn-cert.de>> wrote:

Hi,

we are using nested PGs in non-trivial Nifi flows. The flow of one of these 
inner PGs is itself version controlled in Nifi Registry. And this inner PG is 
used in several instances at different places within the outer PGs.

Depending on the use case, the instances of the inner PGs need different 
configuration settings.


So far (up to Nifi 1.14) we just assigned different parameter contexts to the 
different instances of the inner PG and it worked fine. All instances of the 
inner PG link back to the same version controlled flow in the registry. So it 
was relatively simple to maintain changes to the flow configuration of the 
inner PG and keep everything consistent.

But now (Nifi 1.16.0) it seems that the name of the parameter context is part 
of the data that is stored in the registry. At least we get warnings that the 
inner PGs have been modified (see attached image). And therefore it is now 
impossible to save the outer PGs to the registry - as there are uncommitted 
changes to the inner PGs.

And of course it's also impossible to commit the inner PGs - at least we could 
not keep them all linked to the same flow in the registry, as they are now all 
assumed to be different.


I'm aware that there have been changes to the parameter contexts and that they 
support inheritance now. Perhaps it's obvious, but I do not yet understand if 
and how we can use this to support our use case.

Any hint would be a great help and very much appreciated.


Just observed: While "Version | Show local changes" lists the modified 
parameter context as local change, the command "Version | Revert local changes" 
does not revert this alleged change. You have to manually re-assign the 
original parameter context (or remove the context, depending on what is stored 
in the registry).


Thanks a lot
Reinhard Sell


--
Dr. Reinhard Sell (Projekt- u. Entwicklungsteam)
Phone: +49 40 808077-714  Fax: +49 40 808077-556  Mail: 
s...@dfn-cert.de

DFN-CERT Services GmbH, https://www.dfn-cert.de/
Sitz / Register: Hamburg, AG Hamburg, HRB 88805, Ust-IdNr.: DE 232129737
Nagelsweg 41, 20097 Hamburg, Germany. CEO: Dr. Klaus-Peter 
Kossakowski

Re: NiFi web UI not responsive under load

[Mark Payne]

uot;INFO","thread":"pool-20-thread-1","message":"o.a.n.p.store.WriteAheadStorePartition
 The last Provenance Event indexed for partition default is 13072369,
but the last event written to partition has ID 13089683. Re-indexing up
to the last 17314 events for ./provenance_repository to ensure that the
Event Index is accurate and up-to-date"}

{"level":"INFO","thread":"pool-20-thread-1","message":"o.a.n.p.store.WriteAheadStorePartition
 Finished re-indexing 17315 events across 2 files for
./provenance_repository in 9.713 seconds"}

{"level":"INFO","thread":"main","message":"o.a.n.c.repository.FileSystemRepository
 Maximum Threshold for Container default set to 2858730232217 bytes; if
volume exceeds this size, archived data will be deleted until it no
longer exceeds this size"}

{"level":"INFO","thread":"main","message":"o.a.n.c.repository.FileSystemRepository
 Initializing FileSystemRepository with 'Always Sync' set to false"}

{"level":"INFO","thread":"Thread-1","message":"org.apache.nifi.NiFi Initiating 
shutdown of Jetty web server..."}

But there are no system error or warning messages.

Despite seeing logs that the webserver is listening, we always get "connection 
refused" when trying to communicate with it.

Thanks,
Eric


On Tue, Apr 19, 2022 at 12:57 PM Mark Payne 
mailto:marka...@hotmail.com>> wrote:
Eric,

I certainly agree with what Joe said. I would also recommend checking in 
nifi.properties if you have a value set for:


nifi.monitor.long.running.task.schedule

I recommend setting that to “ hours”

In 1.14.0, we introduced the notion of a Long-Running Task Monitor. It’s 
generally very fast. Typically runs in 10s of milliseconds on my macbook. But 
it relies on JVM-specific code, and we’ve seen in some environments that can 
cause the UI responsiveness to be very adversely affected. We disabled the task 
monitor by default in 1.15, I believe, because of this.

Thanks
-Mark


On Apr 19, 2022, at 3:44 PM, Joe Witt 
mailto:joe.w...@gmail.com>> wrote:

Eric

When the UI isn't responsive it would be great to have a snapshot of:
- CPU usage at that time
- GC behavior/logging at/around that time.
- IO Utilization around that time
- NiFi Thread dump precisely during it and ideally also one after it responds 
again

NiFi Restarting itself is very interesting of course.  There should be more in 
the app log and bootstrap that will help illuminate the issue then.

Thanks


On Tue, Apr 19, 2022 at 12:42 PM Eric Secules 
mailto:esecu...@gmail.com>> wrote:
By the way, I am running NiFi 1.14.0 and it looks like it keeps restarting 
itself. I am seeing this in the logs about once an hour.

{"level":"INFO","thread":"main","message":"org.apache.nifi.NiFi Controller 
initialization took 4737354582168 nanoseconds (4737 seconds)."}

On Tue, Apr 19, 2022 at 12:34 PM Eric Secules 
mailto:esecu...@gmail.com>> wrote:
Hello,

When my nifi system goes under high load the web UI becomes unresponsive until 
load comes down. Is there a way I can see what's going on (processor status 
summary, queued count, active thread count) when the UI is unresponsive?

The logs are not showing any errors and the various repositories are all 
mounted to separate volumes with elastic capacity so I am sure that none of 
them ran out of space. Our monitoring shows bursts of CPU usage and memory use 
lower than normal.

The logs show that the StandardProcessScheduler stops processors followed by 
starting them, but I never see logs related to the UI being ready to serve. It 
does this about once an hour. I see that the flow is slowly processing based on 
log activity and databases.

How can I see what's going on when the web UI is not responding?

Thanks,
Eric

Re: NiFi web UI not responsive under load

[Mark Payne]

Eric,

I certainly agree with what Joe said. I would also recommend checking in 
nifi.properties if you have a value set for:


nifi.monitor.long.running.task.schedule

I recommend setting that to “ hours”

In 1.14.0, we introduced the notion of a Long-Running Task Monitor. It’s 
generally very fast. Typically runs in 10s of milliseconds on my macbook. But 
it relies on JVM-specific code, and we’ve seen in some environments that can 
cause the UI responsiveness to be very adversely affected. We disabled the task 
monitor by default in 1.15, I believe, because of this.

Thanks
-Mark


On Apr 19, 2022, at 3:44 PM, Joe Witt 
mailto:joe.w...@gmail.com>> wrote:

Eric

When the UI isn't responsive it would be great to have a snapshot of:
- CPU usage at that time
- GC behavior/logging at/around that time.
- IO Utilization around that time
- NiFi Thread dump precisely during it and ideally also one after it responds 
again

NiFi Restarting itself is very interesting of course.  There should be more in 
the app log and bootstrap that will help illuminate the issue then.

Thanks


On Tue, Apr 19, 2022 at 12:42 PM Eric Secules 
mailto:esecu...@gmail.com>> wrote:
By the way, I am running NiFi 1.14.0 and it looks like it keeps restarting 
itself. I am seeing this in the logs about once an hour.

{"level":"INFO","thread":"main","message":"org.apache.nifi.NiFi Controller 
initialization took 4737354582168 nanoseconds (4737 seconds)."}

On Tue, Apr 19, 2022 at 12:34 PM Eric Secules 
mailto:esecu...@gmail.com>> wrote:
Hello,

When my nifi system goes under high load the web UI becomes unresponsive until 
load comes down. Is there a way I can see what's going on (processor status 
summary, queued count, active thread count) when the UI is unresponsive?

The logs are not showing any errors and the various repositories are all 
mounted to separate volumes with elastic capacity so I am sure that none of 
them ran out of space. Our monitoring shows bursts of CPU usage and memory use 
lower than normal.

The logs show that the StandardProcessScheduler stops processors followed by 
starting them, but I never see logs related to the UI being ready to serve. It 
does this about once an hour. I see that the flow is slowly processing based on 
log activity and databases.

How can I see what's going on when the web UI is not responding?

Thanks,
Eric

Re: Unexpected Behaviour In LookupRecord With "Route To success" Strategy

[Mark Payne]

Yeah, I just created one [1].

Thanks
-Mark


[1] https://issues.apache.org/jira/browse/NIFI-9903

On Apr 11, 2022, at 10:16 AM, 
stephen.hindma...@bt.com<mailto:stephen.hindma...@bt.com> wrote:

Thanks Mark,

Is there a JIRA open for this?

Regards
Steve


From: Mark Payne mailto:marka...@hotmail.com>>
Sent: 11 April 2022 14:34
To: users@nifi.apache.org<mailto:users@nifi.apache.org>
Subject: Re: Unexpected Behaviour In LookupRecord With "Route To success" 
Strategy

Steve,

Thanks for the note. Ironically, I ran into this issue just yesterday. 
Unfortunately, it’s a bug that will have to be addressed.

In the meantime, if you define the schema for your Record Writer explicitly, it 
should work as expected. The issue comes down to the fact that the first record 
is enriched. And then the schema is determined from the enriched data. Then the 
rest are enriched. But if the first one doesn’t have any enrichment data added, 
the result is that the schema is determined for the flowfile without any 
enrichment. So while the records do get enriched, the schema that is associated 
with the FlowFile is missing those fields. So explicitly defining the schema 
should work.

Thanks
-Mark



On Apr 11, 2022, at 9:08 AM, 
stephen.hindma...@bt.com<mailto:stephen.hindma...@bt.com> wrote:

Hi all,

I am trying to set up a simple enrichment pipeline where flow records get 
enriched from a Redis Distributed Map cache and I use a sequence of 
LookupRecord processors to gather the enrichment data. I am using the “Route to 
success” routing strategy because I would like to avoid fragmenting my flow 
files. However, the results are not what I expected and if the first record 
does not match an enrichment record then no records get enriched.

Here is a simple test case I have created.

1: Create a lookup record processor with the following parameters
Result RecordPath = /mood
Routing Strategy = Route To Success
key = concat('mood/',name)

2: Add these keys to my Redis index.
set mood/fred happy
set mood/charlie sad

3: Send in this flow file
[{"name":"fred"},{"name":"bill"},{"name":"charlie"}]

4: View the result
[{"name":"fred","mood":"happy"},{"name":"bill","mood":null},{"name":"charlie","mood":"sad"}]

That looks OK, every lookup has happened, and I can see that Bill was not 
matched as the enriched value is null.

5: Now try a different flow file, with Bill first.
[{"name":"bill"},{"name":"fred"},{"name":"charlie"}]

Result
[{"name":"bill"},{"name":"fred"},{"name":"charlie"}]

So because the first record did not match, no matches are made, and it looks as 
if the processing never happened.

6: Change the routing strategy to “Route to matched/unmatched”. The result is
Matched => [{"name":"fred","mood":"happy"},{"name":"charlie","mood":"sad"}]
Unmatched => [{"name":"bill"}]

So I have achieved all of my lookups, but the cost is I have fragmented my flow 
file. After 4 lookups my original flow file (which in production will have a. 
1000 records) will have been fragmented into 16 separate files, with a 
consequent impact on performance. Also the indication that the unmatched record 
was not matched is lost, which may be a feature I would like to use.

So my question is, does this look like expected behaviour or is this an issue?

Thanks
Steve Hindmarch,
BT’s Global Division
This email contains information from BT, that might be privileged or 
confidential. And it's only meant for the person above. If that's not you, 
we're sorry - we must have sent it to you by mistake. Please email us to let us 
know, and don't copy or forward it to anyone else. Thanks.
We monitor our email systems and may record all our emails.


British Telecommunications plc., 81 Newgate Street London EC1A 7AJ
Registered in England no: 180

Re: Unexpected Behaviour In LookupRecord With "Route To success" Strategy

[Mark Payne]

Steve,

Thanks for the note. Ironically, I ran into this issue just yesterday. 
Unfortunately, it’s a bug that will have to be addressed.

In the meantime, if you define the schema for your Record Writer explicitly, it 
should work as expected. The issue comes down to the fact that the first record 
is enriched. And then the schema is determined from the enriched data. Then the 
rest are enriched. But if the first one doesn’t have any enrichment data added, 
the result is that the schema is determined for the flowfile without any 
enrichment. So while the records do get enriched, the schema that is associated 
with the FlowFile is missing those fields. So explicitly defining the schema 
should work.

Thanks
-Mark


On Apr 11, 2022, at 9:08 AM, 
stephen.hindma...@bt.com wrote:

Hi all,

I am trying to set up a simple enrichment pipeline where flow records get 
enriched from a Redis Distributed Map cache and I use a sequence of 
LookupRecord processors to gather the enrichment data. I am using the “Route to 
success” routing strategy because I would like to avoid fragmenting my flow 
files. However, the results are not what I expected and if the first record 
does not match an enrichment record then no records get enriched.

Here is a simple test case I have created.

1: Create a lookup record processor with the following parameters
Result RecordPath = /mood
Routing Strategy = Route To Success
key = concat('mood/',name)

2: Add these keys to my Redis index.
set mood/fred happy
set mood/charlie sad

3: Send in this flow file
[{"name":"fred"},{"name":"bill"},{"name":"charlie"}]

4: View the result
[{"name":"fred","mood":"happy"},{"name":"bill","mood":null},{"name":"charlie","mood":"sad"}]

That looks OK, every lookup has happened, and I can see that Bill was not 
matched as the enriched value is null.

5: Now try a different flow file, with Bill first.
[{"name":"bill"},{"name":"fred"},{"name":"charlie"}]

Result
[{"name":"bill"},{"name":"fred"},{"name":"charlie"}]

So because the first record did not match, no matches are made, and it looks as 
if the processing never happened.

6: Change the routing strategy to “Route to matched/unmatched”. The result is
Matched => [{"name":"fred","mood":"happy"},{"name":"charlie","mood":"sad"}]
Unmatched => [{"name":"bill"}]

So I have achieved all of my lookups, but the cost is I have fragmented my flow 
file. After 4 lookups my original flow file (which in production will have a. 
1000 records) will have been fragmented into 16 separate files, with a 
consequent impact on performance. Also the indication that the unmatched record 
was not matched is lost, which may be a feature I would like to use.

So my question is, does this look like expected behaviour or is this an issue?

Thanks
Steve Hindmarch,
BT’s Global Division
This email contains information from BT, that might be privileged or 
confidential. And it's only meant for the person above. If that's not you, 
we're sorry - we must have sent it to you by mistake. Please email us to let us 
know, and don't copy or forward it to anyone else. Thanks.
We monitor our email systems and may record all our emails.

British Telecommunications plc., 81 Newgate Street London EC1A 7AJ
Registered in England no: 180

Re: Cannot Delete Process Group Because Source of Connection is "Running"

[Mark Payne]

Eric,

I hear you, this can be a bit confusing.

The idea is that “stopped” is the “scheduled state”. I.e., it’s scheduled to be 
stopped. It may still have active threads, though.
It’s one of those things where it made a lot of sense when it was done, but 
maybe wasn’t the best decision in the long run :)
But we try to remain pretty strict about breaking backward compatibility in our 
API’s, both REST API and the extension API.
So we’re a bit stuck with that oddity. At least until a 2.0 comes out.

Thanks
-Mark


On Apr 5, 2022, at 7:16 PM, Eric Secules 
mailto:esecu...@gmail.com>> wrote:

Side note, that's the behavior I'd expect from a "stopping" processor but not 
one that's in the "stopped" state.

On Tue., Apr. 5, 2022, 4:07 p.m. Eric Secules, 
mailto:esecu...@gmail.com>> wrote:
Hi Mark,

Is there an API for this that can filter by processors within a process group? 
Would it be the one that provides the data for this portion of the UI?

Thanks,
Eric

On Tue, Apr 5, 2022 at 3:29 PM Mark Payne 
mailto:marka...@hotmail.com>> wrote:
Eric,

Once a processor state transitions to stopped, it may still have active threads 
that haven’t completed yet. You’ll need to wait until the processor is stopped 
and the active threads on the processor reach 0.

Thanks
-Mark

On Apr 5, 2022, at 6:25 PM, Eric Secules 
mailto:esecu...@gmail.com>> wrote:

Hello,

I have this program which stops and deletes flows from NiFi, when we're done 
with them and once in a long while we fail this operation because of this:

2022-04-05 13:03:18,569 WARN [NiFi Web Server-281] 
o.a.n.w.a.c.IllegalStateExceptionMapper java.lang.IllegalStateException: 
Destination of Connection (ce5a7658-59db-360f-f978-f275e69c072e) is running. 
Returning Conflict response.
java.lang.IllegalStateException: Destination of Connection 
(ce5a7658-59db-360f-f978-f275e69c072e) is running
at 
org.apache.nifi.connectable.StandardConnection.verifyCanDelete(StandardConnection.java:508)
at 
org.apache.nifi.groups.StandardProcessGroup.removeConnection(StandardProcessGroup.java:1270)
at 
org.apache.nifi.groups.StandardProcessGroup.removeComponents(StandardProcessGroup.java:864)
at 
org.apache.nifi.groups.StandardProcessGroup.removeProcessGroup(StandardProcessGroup.java:851)
at 
org.apache.nifi.groups.StandardProcessGroup.removeComponents(StandardProcessGroup.java:897)
at 
org.apache.nifi.groups.StandardProcessGroup.removeProcessGroup(StandardProcessGroup.java:851)
at 
org.apache.nifi.web.dao.impl.StandardProcessGroupDAO.deleteProcessGroup(StandardProcessGroupDAO.java:528)
at 
org.apache.nifi.web.dao.impl.StandardProcessGroupDAO$$FastClassBySpringCGLIB$$10a99b47.invoke()

We run version 1.14.0 right now and I couldn't find any relevant bug reports in 
Nifi's Jira solved between this version and the current. The closest I got was 
https://issues.apache.org/jira/browse/NIFI-1777

When deactivating and removing flows from the canvas we follow this procedure:


  *   Stop the top level process group (which might contain 400-2000 
processors), using the equivalent API of right clicking on the process group 
and selecting stop
  *   Wait for stopped state on that process group (we assume this means that 
the stopped state is persisted on all contained processors and process groups)
  *   Traverse inner process groups, for each:
 *   deactivate controller services and wait for deactivated state
 *   delete templates associated with the process group
 *   delete parameter context
  *   Delete top level process group  < error happens here

Upon noticing the error I go into the NiFi canvas and see that all processors 
are either stopped or in error.

I can cause the same IllegalStateException through the canvas and get the same 
error message, but when I go search for Connection 
(ce5a7658-59db-360f-f978-f275e69c072e), all the processors and input ports are 
rightfully in the error state (missing parameters, deactivated controller 
services) I can work around and delete all the components individually, and 
then continue to remove the rest of the flow, it's almost as if there's some 
lingering state that the processor is running and that's what's preventing me 
from removing a connection that stems from it.


Thanks,
Eric

Re: Cannot Delete Process Group Because Source of Connection is "Running"

[Mark Payne]

Eric,

The endpoint you’d want is /nifi-api/flow/process-groups/{id}/status

So for the root group you’d use /nifi-api/flow/process-groups/root/status

That will give you a big JSON blurb. From there you’d want to look at the 
element at
$.processGroupStatus.aggregateSnapshot.activeThreadCount

Hope that helps!
-Mark




On Apr 5, 2022, at 7:07 PM, Eric Secules 
mailto:esecu...@gmail.com>> wrote:

Hi Mark,

Is there an API for this that can filter by processors within a process group? 
Would it be the one that provides the data for this portion of the UI?

Thanks,
Eric

On Tue, Apr 5, 2022 at 3:29 PM Mark Payne 
mailto:marka...@hotmail.com>> wrote:
Eric,

Once a processor state transitions to stopped, it may still have active threads 
that haven’t completed yet. You’ll need to wait until the processor is stopped 
and the active threads on the processor reach 0.

Thanks
-Mark

On Apr 5, 2022, at 6:25 PM, Eric Secules 
mailto:esecu...@gmail.com>> wrote:

Hello,

I have this program which stops and deletes flows from NiFi, when we're done 
with them and once in a long while we fail this operation because of this:

2022-04-05 13:03:18,569 WARN [NiFi Web Server-281] 
o.a.n.w.a.c.IllegalStateExceptionMapper java.lang.IllegalStateException: 
Destination of Connection (ce5a7658-59db-360f-f978-f275e69c072e) is running. 
Returning Conflict response.
java.lang.IllegalStateException: Destination of Connection 
(ce5a7658-59db-360f-f978-f275e69c072e) is running
at 
org.apache.nifi.connectable.StandardConnection.verifyCanDelete(StandardConnection.java:508)
at 
org.apache.nifi.groups.StandardProcessGroup.removeConnection(StandardProcessGroup.java:1270)
at 
org.apache.nifi.groups.StandardProcessGroup.removeComponents(StandardProcessGroup.java:864)
at 
org.apache.nifi.groups.StandardProcessGroup.removeProcessGroup(StandardProcessGroup.java:851)
at 
org.apache.nifi.groups.StandardProcessGroup.removeComponents(StandardProcessGroup.java:897)
at 
org.apache.nifi.groups.StandardProcessGroup.removeProcessGroup(StandardProcessGroup.java:851)
at 
org.apache.nifi.web.dao.impl.StandardProcessGroupDAO.deleteProcessGroup(StandardProcessGroupDAO.java:528)
at 
org.apache.nifi.web.dao.impl.StandardProcessGroupDAO$$FastClassBySpringCGLIB$$10a99b47.invoke()

We run version 1.14.0 right now and I couldn't find any relevant bug reports in 
Nifi's Jira solved between this version and the current. The closest I got was 
https://issues.apache.org/jira/browse/NIFI-1777

When deactivating and removing flows from the canvas we follow this procedure:


  *   Stop the top level process group (which might contain 400-2000 
processors), using the equivalent API of right clicking on the process group 
and selecting stop
  *   Wait for stopped state on that process group (we assume this means that 
the stopped state is persisted on all contained processors and process groups)
  *   Traverse inner process groups, for each:
 *   deactivate controller services and wait for deactivated state
 *   delete templates associated with the process group
 *   delete parameter context
  *   Delete top level process group  < error happens here

Upon noticing the error I go into the NiFi canvas and see that all processors 
are either stopped or in error.

I can cause the same IllegalStateException through the canvas and get the same 
error message, but when I go search for Connection 
(ce5a7658-59db-360f-f978-f275e69c072e), all the processors and input ports are 
rightfully in the error state (missing parameters, deactivated controller 
services) I can work around and delete all the components individually, and 
then continue to remove the rest of the flow, it's almost as if there's some 
lingering state that the processor is running and that's what's preventing me 
from removing a connection that stems from it.


Thanks,
Eric

Re: Cannot Delete Process Group Because Source of Connection is "Running"

[Mark Payne]

Eric,

Once a processor state transitions to stopped, it may still have active threads 
that haven’t completed yet. You’ll need to wait until the processor is stopped 
and the active threads on the processor reach 0.

Thanks
-Mark

On Apr 5, 2022, at 6:25 PM, Eric Secules 
mailto:esecu...@gmail.com>> wrote:

Hello,

I have this program which stops and deletes flows from NiFi, when we're done 
with them and once in a long while we fail this operation because of this:

2022-04-05 13:03:18,569 WARN [NiFi Web Server-281] 
o.a.n.w.a.c.IllegalStateExceptionMapper java.lang.IllegalStateException: 
Destination of Connection (ce5a7658-59db-360f-f978-f275e69c072e) is running. 
Returning Conflict response.
java.lang.IllegalStateException: Destination of Connection 
(ce5a7658-59db-360f-f978-f275e69c072e) is running
at 
org.apache.nifi.connectable.StandardConnection.verifyCanDelete(StandardConnection.java:508)
at 
org.apache.nifi.groups.StandardProcessGroup.removeConnection(StandardProcessGroup.java:1270)
at 
org.apache.nifi.groups.StandardProcessGroup.removeComponents(StandardProcessGroup.java:864)
at 
org.apache.nifi.groups.StandardProcessGroup.removeProcessGroup(StandardProcessGroup.java:851)
at 
org.apache.nifi.groups.StandardProcessGroup.removeComponents(StandardProcessGroup.java:897)
at 
org.apache.nifi.groups.StandardProcessGroup.removeProcessGroup(StandardProcessGroup.java:851)
at 
org.apache.nifi.web.dao.impl.StandardProcessGroupDAO.deleteProcessGroup(StandardProcessGroupDAO.java:528)
at 
org.apache.nifi.web.dao.impl.StandardProcessGroupDAO$$FastClassBySpringCGLIB$$10a99b47.invoke()

We run version 1.14.0 right now and I couldn't find any relevant bug reports in 
Nifi's Jira solved between this version and the current. The closest I got was 
https://issues.apache.org/jira/browse/NIFI-1777

When deactivating and removing flows from the canvas we follow this procedure:


  *   Stop the top level process group (which might contain 400-2000 
processors), using the equivalent API of right clicking on the process group 
and selecting stop
  *   Wait for stopped state on that process group (we assume this means that 
the stopped state is persisted on all contained processors and process groups)
  *   Traverse inner process groups, for each:
 *   deactivate controller services and wait for deactivated state
 *   delete templates associated with the process group
 *   delete parameter context
  *   Delete top level process group  < error happens here

Upon noticing the error I go into the NiFi canvas and see that all processors 
are either stopped or in error.

I can cause the same IllegalStateException through the canvas and get the same 
error message, but when I go search for Connection 
(ce5a7658-59db-360f-f978-f275e69c072e), all the processors and input ports are 
rightfully in the error state (missing parameters, deactivated controller 
services) I can work around and delete all the components individually, and 
then continue to remove the rest of the flow, it's almost as if there's some 
lingering state that the processor is running and that's what's preventing me 
from removing a connection that stems from it.


Thanks,
Eric

Re: VolatileContentRepository removal

[Mark Payne]

Hey Matthieu,

If using a RAM disk, I would recommend trying 1.16 and also setting 
“nifi.content.claim.max.appendable.size” in nifi.properties to “1 byte”. This 
will help to ensure that you’re eliminating data from the repository as quickly 
as possible. Additionally, I would recommend that you configure 
“nifi.flowfile.repository.checkpoint.interval” to a very small value, such as 
“5 secs” or “10 secs” as this will also help in eliminating data from the 
Content Repo as quickly as possible. And you’ll want to ensure that you have 
“nifi.content.repository.archive.enabled” set to “false”.

Thanks
-Mark


On Mar 31, 2022, at 8:46 AM, Matthieu Ré 
mailto:re.matth...@gmail.com>> wrote:

Hi Mike, David,

Thanks for your answers and your clarity !

To sum up our use case, our team has set up two cloud-based NiFi ("stateful") 
clusters. The second one deals with a big amount of record-based data, to drag 
data from sources (Kafka, files and databases) to other systems (OpenSearch and 
internal tools) performing a lot of costly transformations (in the last 5 
minutes for instance, my prod instance indicates 300G of read, 170G of write, 
within 400 processors). This cluster deals with data we can recover from a 
process, so in this use case we are data-loss tolerant. The 12 NiFi nodes of 
the cluster (as well as a 3-nodes ZK cluster, and 1 NiFi Registry) are running 
on private-cloud VM instances. Their operations on disks are not as performant 
as it could be on physical machines, and are restricted to a certain amount of 
IOps that we can easily reach with disk-based Repositories ; that is the main 
reason why we tried the VolatileContentRepository. Since 1.13.1, we use a 
version of NiFi that we build with our custom bundles and the fix that was 
linked in NIFI-8760 and for now it is working fine for us. We had some memory 
leaks on custom processors but after correction and limitation on queue sizes 
we never encountered OOM on the Volatile again.
Don't hesitate to ask for more information or precision on the use case, or for 
any advice !

About the OOM on RAM-disk approach, the last time we tried it was on 1.14.0. In 
the next few weeks we will try to migrate to 1.16.0, I'll be glad to 
investigate if we still experience OOMs and report it if we do. And if we 
don't, it could be a great solution for us to replace the 
VolatileContentRepository. I have to add that I just discovered the NiFi 
Stateless engine and especially the ExecuteStateless processor and the 
repository you mentioned, with some refacto we could reduce the number of 
queues in between ExecuteStateless processors that would induce FlowFiles to be 
written on disks. We will investigate if this is enough for our nodes to be 
below the threshold of authorized IOps per volume.

So thanks to you I understand that the Volatile has some weaknesses and could 
have great alternatives. If all these alternatives fail, I would be glad to 
investigate for 1.17 the possibility to promote the ByteArrayContentRepository 
to the main framework.

Thank you!
Matthieu

Le mer. 30 mars 2022 à 15:18, David Handermann 
mailto:exceptionfact...@apache.org>> a écrit :
Hi Matthieu,

Thanks for raising this question for discussion. Other maintainers may be able 
to provide additional background, but part of the reason for removing the 
VolatileContentRepository implementation was that there were some more 
fundamental problems with the implementation. Although various framework 
updates included patching the implementation along the way, the repository was 
not maintained on a regular basis, which resulted in it being broken for 
several releases.

As Mike said, it would be helpful to share more details about your use case, 
and also to hear more about whether you still experience memory issues with the 
file system repository in current releases.

On a related note, NiFi Stateless includes a new in-memory content repository 
named ByteArrayContentRepository [1]. It is currently packaged in the NiFi 
Stateless bundle, but it might be possible to consider promoting it to the 
framework level, if there is value in a non-persistent content repository going 
forward.

Regards,
David Handermann

[1] 
https://github.com/apache/nifi/blob/main/nifi-stateless/nifi-stateless-bundle/nifi-stateless-engine/src/main/java/org/apache/nifi/stateless/repository/ByteArrayContentRepository.java

On Wed, Mar 30, 2022 at 7:45 AM Mike Thomsen 
mailto:mikerthom...@gmail.com>> wrote:
We've been moving away from supporting it for a while, and I think it
comes down to a lot of both factors when you consider the time
involved in getting good patches and reviewing them. That said, until
1.17 is released, I think there's room for community members like you
and your team to work with us on fixing the gaps that made a strong
case for removing it.

I think I saw in your ticket that you provided patches through Jira.
My recommendation would be to do a feature branch that reverts the
removal, 

Re: Insufficient Permissions for Expression Language

[Mark Payne]

Hi Stanley,

That error message is not coming from NiFi. I would guess that you have some 
sort of load balancer, proxy, etc. between you and the NiFi instance? WOuld 
recommend looking at that to see if you can determine what’s happening there.

Thanks
-Mark


On Mar 30, 2022, at 3:32 PM, Martin, Stanley L 
mailto:stanley.mar...@hexagonusfederal.com>>
 wrote:



I have an instance of NiFi (v. 1.15.3) running in Cloud Foundry, and several 
weeks ago I started getting a message that I have Insufficient Permissions when 
I try to add or modify a processor property that contains Expression Language.  
The message I get is:

Request RejectedThe requested URL was 
rejected. Please consult with your administrator.Your support ID is: 
14975944297778607952[Go 
Back]

Does anyone have an idea what this means and how I can fix it?

Thanks,
Stanley

Re: Where are my processes?

[Mark Payne]

Jean-Sebastian,

I’d recommend grabbing a thread dump (bin/nifi.sh dump dump1.txt) and check the 
thread dump to see what the processors are doing.

Would also grep logs for “waiting for” to see if it shows anything.

Thanks
-Mark



Sent from my iPhone

On Mar 28, 2022, at 7:27 PM, Jean-Sebastien Vachon  
wrote:


Some additional information as I haven't found what's happening.

The average task time is about 15 seconds.. Is there any reason why the process 
wouldn't show up in top or ps?
I setup a watch updated every 1s and I can see at most a few instances of my 
process.
I ramped up to 35 concurrent processes and I saw a few more processes but with 
an average of 15 s, I was expecting to see way more than a few.

Any thoughts?

Jean-Sébastien Vachon
Co-Founder & Architect
Brizo Data, Inc.
www.brizodata.com

From: Jean-Sebastien Vachon 
Sent: Friday, March 25, 2022 9:40 AM
To: users@nifi.apache.org 
Subject: Where are my processes?

Hi all,

A strange thing seems to be happening on my server this morning I can't 
find the processes reported by Nifi.
Nifi shows 15 running processes for one of my processor (ExecuteStreamCommand 
with Python script) but when I look at the OS, I can see at most 2 of them.
I understand that very short-lived processes will be harder to spot at the OS 
level but the discrepancy seems too large to make any sense.
If I stop the processor, It takes a good minute for any activity to stop but I 
can't see anything in the list of processes.

My throughput is also 4-5 times slower than usual with that processor. There 
does not seem to be any issue with GC and the global setting for maximum thread 
count is set to 600 (I went from 500 to 600 with no effect). There is nothing 
else running in Nifi on this server at the moment.

Any idea? I'm using Nifi 1.13.2

Thanks

Re: QueryRecord with Union type

[Mark Payne]

Steve,

What version of nifi are you running? I’d tried that on the latest “main” 
branch and it worked as expected.

Thanks
-Mark

On Mar 18, 2022, at 5:49 AM, 
stephen.hindma...@bt.com<mailto:stephen.hindma...@bt.com> wrote:

Mark,

Thank you for your response. I thought that was probably the case, but I tried 
a cast and it did not work. I got this error.

Query:
select *
from flowfile
where cast(flag_s as boolean) = true

Error:
org.apache.calcite.sql.validate.SqlValidatorException: Cast function cannot 
convert value of type JavaType(class java.lang.Object) to type BOOLEAN

By taking the union out of the input schema I could get the query to work, but 
I did find myself getting tangled up in managing various schemas so I am trying 
to use infer/inherit read/write services instead. I have inherited a very 
complex flow from a team that have long departed and am looking to simplify it 
to improve performance and maintainability. I need to convert from CSV/TSV to 
JSON, normalise fields, filter unwanted records, enrich with more JSON and 
finally publish to a customer defined schema, so I do need a few steps along 
the way. I am exploring each step in order to validate my redesign so I take 
your point about minimising the number of processes and will look again at 
combining steps in the query process, although I am also a fan of the JOLT 
transform as I have used that often in previous projects.

Regards
Steve Hindmarch

From: Mark Payne mailto:marka...@hotmail.com>>
Sent: 17 March 2022 14:17
To: users mailto:users@nifi.apache.org>>
Subject: Re: QueryRecord with Union type

Steve,

Because your schema has a union, the SQL engine doesn’t really know how to 
interpret the data. So it interprets it as a “Java Object.” Essentially,
it could be anything. But you can’t compare just anything to true - you need to 
compare a boolean to true. So you need to tell the SQL engine that the
value you’re looking at is, in fact, a boolean.

You can do that with a simple CAST() function in your SQL:

SELECT *
FROM FLOWFILE
WHERE CAST(flag_s AS BOOLEAN) = true

That should give you what you’re looking for.

Also worth nothing - you mentioned that you’re using ConvertRecord and 
UpdateRecord before QueryRecord.
99% of the time, you should not be using ConvertRecord in conjunction with any 
other Record processor. Because the Record processors like UpdateRecord
allow you to use any Record Reader, it doesn’t make sense to convert the data 
first using ConvertRecord - it’s just extra overhead.
And, in fact, you may be able to eliminated the UpdateRecord, as well, as just 
use the SQL within QueryRecord to perform the transformation needed on the fly,
rather than having another step to update the data, which requires reading the 
data, parsing it, updating it, serializing the data, writing the data. This may 
not
be possible, depends on what you’re updating. But QueryRecord does support 
RecordPath expressions so it’s worth considering.

Thanks
-Mark




On Mar 15, 2022, at 8:35 AM, 
stephen.hindma...@bt.com<mailto:stephen.hindma...@bt.com> wrote:

I am having a play with QueryRecord to do some filtering but I have run across 
this problem. I have a schema for my records which includes a union type, so 
the relevant part of the schema is

{
  "type":"record",
  "namespace":"blah",
  "name":"SimpleTraffic",
  "fields":[
{"name":"src_address","type":"string"},
{"name":"flag_s","type":["int","boolean"]}
  ]
}

This is because I am processing CSV records that look this, where 1 is true and 
0 is false.

192.168.0.1,1

Into JSON that looks like this, using a ConvertRecord and an Update Record.

{"src_address":"192.168.0.1","flag_s":true}

Then I create a QueryRecord so I can filter out the cases where the flag is 
false. So I use this query.

select * from flowfile where flag_s = true

But I get this error

org.apache.calcite.sql.validate.SqlValidatorException: Cannot apply '=' to 
arguments of type ' = '

Is this because the type is a Union type and the Calcite processor cannot work 
out which subtype it should be? Can I do anything to persuade the query to use 
an operator or a function on this field to make it usable? I have tried casting 
to Boolean or Char but no success. Or do I need to use two separate “before” 
and “after” schemas to eliminate the union?

Regards

Steve Hindmarch

Re: QueryRecord with Union type

[Mark Payne]

Steve,

Because your schema has a union, the SQL engine doesn’t really know how to 
interpret the data. So it interprets it as a “Java Object.” Essentially,
it could be anything. But you can’t compare just anything to true - you need to 
compare a boolean to true. So you need to tell the SQL engine that the
value you’re looking at is, in fact, a boolean.

You can do that with a simple CAST() function in your SQL:

SELECT *
FROM FLOWFILE
WHERE CAST(flag_s AS BOOLEAN) = true

That should give you what you’re looking for.

Also worth nothing - you mentioned that you’re using ConvertRecord and 
UpdateRecord before QueryRecord.
99% of the time, you should not be using ConvertRecord in conjunction with any 
other Record processor. Because the Record processors like UpdateRecord
allow you to use any Record Reader, it doesn’t make sense to convert the data 
first using ConvertRecord - it’s just extra overhead.
And, in fact, you may be able to eliminated the UpdateRecord, as well, as just 
use the SQL within QueryRecord to perform the transformation needed on the fly,
rather than having another step to update the data, which requires reading the 
data, parsing it, updating it, serializing the data, writing the data. This may 
not
be possible, depends on what you’re updating. But QueryRecord does support 
RecordPath expressions so it’s worth considering.

Thanks
-Mark



On Mar 15, 2022, at 8:35 AM, 
stephen.hindma...@bt.com wrote:

I am having a play with QueryRecord to do some filtering but I have run across 
this problem. I have a schema for my records which includes a union type, so 
the relevant part of the schema is

{
  "type":"record",
  "namespace":"blah",
  "name":"SimpleTraffic",
  "fields":[
{"name":"src_address","type":"string"},
{"name":"flag_s","type":["int","boolean"]}
  ]
}

This is because I am processing CSV records that look this, where 1 is true and 
0 is false.

192.168.0.1,1

Into JSON that looks like this, using a ConvertRecord and an Update Record.

{"src_address":"192.168.0.1","flag_s":true}

Then I create a QueryRecord so I can filter out the cases where the flag is 
false. So I use this query.

select * from flowfile where flag_s = true

But I get this error

org.apache.calcite.sql.validate.SqlValidatorException: Cannot apply '=' to 
arguments of type ' = '

Is this because the type is a Union type and the Calcite processor cannot work 
out which subtype it should be? Can I do anything to persuade the query to use 
an operator or a function on this field to make it usable? I have tried casting 
to Boolean or Char but no success. Or do I need to use two separate “before” 
and “after” schemas to eliminate the union?

Regards

Steve Hindmarch

Re: Getting java.util.concurrent.RejectedExecutionException upon nifi restart

[Mark Payne]

Hi Shweta,

The exception that you see here is rather harmless - it’s indicating that there 
was an attempt to enable a Controller Service while NiFi
was in the middle of shutting down. Recommend you look higher up in the logs to 
understand why NiFi was shutting down, if this was
not intentional.

Thanks
-Mark


> On Mar 9, 2022, at 8:49 AM, shweta julur  wrote:
> 
> Hi,
> 
> Once I deployed nars for the custom processors and custom controller services 
> and restarted nifi I got the below exception-
> 
> 2022-03-09 06:38:19,759 INFO [Shutdown Cluster Coordinator] 
> o.a.n.c.c.node.NodeClusterCoordinator Successfully notified other nodes that 
> I am shutting down
> 2022-03-09 06:38:19,761 INFO [Curator-Framework-0] 
> o.a.c.f.imps.CuratorFrameworkImpl backgroundOperationsLoop exiting
> 2022-03-09 06:38:19,865 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager 
> CuratorLeaderElectionManager[stopped=true] stopped and closed
> 2022-03-09 06:38:19,865 INFO [main] o.a.n.c.c.h.AbstractHeartbeatMonitor 
> Heartbeat Monitor stopped
> 2022-03-09 06:38:19,866 INFO [main] o.apache.nifi.controller.FlowController 
> Initiated graceful shutdown of flow controller...waiting up to 10 seconds
> 2022-03-09 06:38:19,869 INFO [main] o.a.zookeeper.server.ZooKeeperServer 
> Shutting down
> 2022-03-09 06:38:19,870 INFO [main] o.a.zookeeper.server.ZooKeeperServer 
> shutting down
> 2022-03-09 06:38:19,871 INFO [main] o.a.z.server.FinalRequestProcessor 
> shutdown of request processor complete
> 2022-03-09 06:38:19,874 INFO [main] o.a.z.server.SyncRequestProcessor 
> Shutting down
> 2022-03-09 06:38:19,876 INFO [SyncThread:1] o.a.z.server.SyncRequestProcessor 
> SyncRequestProcessor exited!
> 2022-03-09 06:38:19,878 INFO [main] o.a.z.server.DatadirCleanupManager 
> Shutting down purge task.
> 2022-03-09 06:38:20,049 ERROR [Timer-Driven Process Thread-8] 
> org.apache.nifi.engine.FlowEngine Uncaught Exception in Runnable task
> java.util.concurrent.RejectedExecutionException: Task 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@8ad2a80 
> rejected from org.apache.nifi.engine.FlowEngine@66ad5e5[Shutting down, pool 
> size = 36, active threads = 1, queued tasks = 6, completed tasks = 75]
> at 
> java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
> at 
> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:326)
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:533)
> at org.apache.nifi.engine.FlowEngine.schedule(FlowEngine.java:87)
> at 
> org.apache.nifi.controller.service.StandardControllerServiceNode$2.run(StandardControllerServiceNode.java:431)
> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 
> Any help would be appreciated.
> 
> Thanks,
> 
> Shweta

Re: Records - Best Approach to Enrich Record From Cache

[Mark Payne]

Mike, Steven, Nick,

The DistributedMapCache Client does already have that. The method is called 
subMap. But I don’t think it performs an MGET at present, it just loops over 
the keys calling GET.
But I think the more appropriate approach here would be to provide a 
RedisRecordLookupService so that it could be used with LookupRecord.

However, in version 1.16 you may be able to do this much more efficiently 
without that.
In 1.16 we introduce a pair of processors: ForkEnrichment and JoinEnrichment.
This allows you to take your incoming data, transform the data in some way to 
gather enrichment data, gather the enrichment data, and then join together the 
enrichment data with the original data.

So you may be able to do something like:

ForkEnrichment — (original) —> JoinEnrichment
  — (enrichment) —> JoltTransform/QueryRecord/etc. —> 
InvokeHTTP —> JoinEnrichment

This would mean that you’d need to transform your data into a single 
REST-friendly web request and use InvokeHTTP to gather all of the data from 
Redis in a single call. You could then use JoinEnrichment with a SQL JOIN in 
order to join together your original data with the enrichment data. This 
wouldn’t be nearly as simple/straight-forward as just a single LookupRecord 
processor, but it should provide very good performance and should still be 
simpler than many enrichment processors.

Hope this helps!
-Mark


> On Mar 7, 2022, at 9:36 AM, Mike Thomsen  wrote:
> 
> I skimmed over the code in the Redis DMC client, and did not see any
> place where we could do a MGET there. Not sure if that's relevant to
> Nick's use case, but it would be relevant to that general pattern
> going forward. It wouldn't be hard to add a bulk get method to the DMC
> interface and provide a default interface that just loops and does
> multiple get operations and stacks them together. Then the Redis
> version could do a MGET and stack them together.
> 
> That said, AFAIK we'd need to create a new enrichment process or
> extend something like ScriptedTransformRecord to integrate with a DMC.
> 
> I have the time to work on this, but would like to hear from
> committers and users before I start banging out the code to make sure
> I'm not missing something.
> 
> On Mon, Mar 7, 2022 at 7:18 AM  wrote:
>> 
>> Redis does allow multiple gets in the one hit with MGET. If you search for 
>> all keys the response is an ordered list of matching values, with null in 
>> place where there is no match.
>> 
>> 
>> 
>> Steve Hindmarch
>> 
>> 
>> 
>> From: Nick Lange 
>> Sent: 07 March 2022 04:46
>> To: users@nifi.apache.org
>> Subject: Records - Best Approach to Enrich Record From Cache
>> 
>> 
>> 
>> HI all -
>> 
>> I have a record set of objects that each need enrichment of about 10/20 
>> fields of data from a Redis Cache. In a perfect world, I'd hit the cache 
>> once and return a json blob for further extraction  - ideally in a single 
>> hop.  I don't see an easy way to do this with the record language, but 
>> perhaps I've missed something.
>> 
>> 
>> 
>> Lacking any better sophistication, I'm currently doing this brute-force with 
>> 10-20 hits to the cache for each field. I'm hoping that the mailing list has 
>> better suggestions.
>> 
>> 
>> 
>> Thank you
>> 
>> Nick
>> 
>> 

Re: ListSFTP doesn't follow symlinks

[Mark Payne]

Guille,

Thanks for the extra details.

I just tried again. In my case, all worked as expected when I had a symlink to 
a directory. But when I had a symlink to a file, I got the same error and stack 
trace as you. So looks like we are handling the case properly for symlinked 
directories but not symlinked files.

Thanks
-Mark



On Feb 3, 2022, at 11:43 AM, Guillermo Muñoz 
mailto:guillermo.munoz.salg...@gmail.com>> 
wrote:

Hi, David.

Sorry for the misunderstanding, my fault. Firstly, we tried using ListSFTP and 
FetchSFTP, and when it didn't work, we tried another option (GetSFTP), and I 
pasted the wrong stack trace. So, i've done the following tests:

  *   ListSFTP + FetchSFTP: Error in ListSFTP [1]
  *   GetSFTP: Error [2]
  *   Generate flowfile + FetchSFTP with the name of the symlink in the Remote 
File property: OK, the file is downloaded.

So, it seems the issue is in ListSFTP and GetSFTP, but FetchSFTP  works fine.

Thanks. Regards

--
Guille

[1]
2022-02-03 17:36:45,466 ERROR [Timer-Driven Process Thread-8] 
o.a.nifi.processors.standard.ListSFTP 
ListSFTP[id=64443154-ac76-1736-9e49-f2ca388dfbdf] Unable to get listing from  
*.gz; skipping: java.io.FileNotFoundException: Could not perform listing on 
 *.gz because could not find the file on the remote server
java.io.FileNotFoundException: Could not perform listing on  *.gz because 
could not find the file on the remote server
at 
org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:350)
at 
org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:365)
at 
org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:262)
at 
org.apache.nifi.processors.standard.ListFileTransfer.performListing(ListFileTransfer.java:120)
at 
org.apache.nifi.processors.standard.ListSFTP.performListing(ListSFTP.java:150)
at 
org.apache.nifi.processors.standard.ListFileTransfer.performListing(ListFileTransfer.java:112)
at 
org.apache.nifi.processor.util.list.AbstractListProcessor.listByTrackingTimestamps(AbstractListProcessor.java:750)
at 
org.apache.nifi.processor.util.list.AbstractListProcessor.onTrigger(AbstractListProcessor.java:525)
at 
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at 
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1273)
at 
org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214)
at 
org.apache.nifi.controller.scheduling.AbstractTimeBasedSchedulingAgent.lambda$doScheduleOnce$0(AbstractTimeBasedSchedulingAgent.java:63)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)


[2]
2022-02-03 17:39:14,714 ERROR [Timer-Driven Process Thread-27] 
o.a.nifi.processors.standard.GetSFTP 
GetSFTP[id=c0300c77-017e-1000--fa9c1f31] Unable to get listing from 
*.gz; skipping: java.io.FileNotFoundException: Could not perform listing on 
*.gz because could not find the file on the remote server
java.io.FileNotFoundException: Could not perform listing on *.gz because 
could not find the file on the remote server
at 
org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:350)
at 
org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:365)
at 
org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:262)
at 
org.apache.nifi.processors.standard.GetFileTransfer.fetchListing(GetFileTransfer.java:299)
at 
org.apache.nifi.processors.standard.GetFileTransfer.onTrigger(GetFileTransfer.java:126)
at 
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at 
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1273)
at 
org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214)
at 
org.apache.nifi.controller.scheduling.AbstractTimeBasedSchedulingAgent.lambda$doScheduleOnce$0(AbstractTimeBasedSchedulingAgent.java:63)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at 

Re: ListSFTP doesn't follow symlinks

[Mark Payne]

Guille,

I did a quick test on my MacBook and things worked as expected following 
symlinks. If I set the property to “false” it didn’t get the files. If I set it 
to “true” it did retrieve the files. Either way it didn’t error, though - just 
didn’t follow the symlink.

Of course, that’s not to say that there’s not some issue, just that it’s not 
obviously always broken :)

One thing that I notice in the log message there, though:

"Could not perform listing on   testfile.gz…”

There are a couple of spaces there - it’s not looking for “testfile.gz” but 
rather “testfile.gz” - are there actually spaces in the filename? 
Or any other sort of character there, that is perhaps not being properly 
escaped?

Thanks
-Mark


On Feb 3, 2022, at 10:57 AM, Guillermo Muñoz Salgado 
mailto:mun...@gmail.com>> wrote:

Hi all,

We are developing a use case in which we have to get some files from a server. 
We have implemented it by the listSFTP + FetchSFTP way in a 3 nodes cluster 
running nifi 1.15.3. But we are having some issues when what we want to get are 
symlinks instead of files. We have set true the property "Follow symlink" but 
we get the same results. Are we doing something wrong? Or is it a bug or a 
known issue? We have found this issue [1] but it is old and resolved and this 
other one [2], that is older and unresolved.  We're not sure if they are 
related to this behaviour or not.

I paste our error log:

2022-02-03 16:27:41,002 ERROR [Timer-Driven Process Thread-18] 
o.a.nifi.processors.standard.GetSFTP 
GetSFTP[id=c0300c77-017e-1000--fff-ffa9c1f31] Unable to get listing from 
testfile.gz; skipping: java.io.FileNotFoundException: Could not perform listing 
on testfile.gz because could not find the file on the remote server
java.io.FileNotFoundException: Could not perform listing on   testfile.gz 
because could not find the file on the remote server
at 
org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:350)
at 
org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:365)
at 
org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:262)
at 
org.apache.nifi.processors.standard.GetFileTransfer.fetchListing(GetFileTransfer.java:299)
at 
org.apache.nifi.processors.standard.GetFileTransfer.onTrigger(GetFileTransfer.java:126)
at 
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at 
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1273)
at 
org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214)
at 
org.apache.nifi.controller.scheduling.AbstractTimeBasedSchedulingAgent.lambda$doScheduleOnce$0(AbstractTimeBasedSchedulingAgent.java:63)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

Thanks in advance
--
Guille

[1] https://issues.apache.org/jira/browse/NIFI-5560
[2] https://issues.apache.org/jira/browse/NIFI-6699

Re: NiFi V1.15.2 Conversion to Avro from JSON

[Mark Payne]

Thanks Nathan, the template is helpful. I put up a PR that I believe addresses 
the issues.

Thanks
-Mark


On Jan 25, 2022, at 7:21 AM, 
nathan.engl...@bt.com<mailto:nathan.engl...@bt.com> wrote:

Hey Mark,

Apologies if this is more of a dev mailing list issue. Happy to go across to it 
if needs be.

I've been testing the fix you've implemented for this and noticed a slight 
issue when the default value should be null. It took a little while to work out 
what was going on, but I think I've cracked it. I have attached a flow I've 
been using to test this.

So in AvroTypeUtil on line 630, it's creating the field as it doesn't exist. 
Instead of setting the default value to null, it is set to 
org.apache.avro.JsonProperties$Null, which throws an error when it tries to 
encode as Avro. For example, the Null String & Byte Type generator in the 
attached flow reproduces this.

In addition to this, I did a quick bit of testing with int and long types with 
default values in the Avro Schema set to 0. So, it looks like Long fields in 
the schema have the default values set as Integers? I have reproduced it with 
the attached flow's Int & Long Type generator.

Of course, this isn't an exhaustive list of the possible Avro types, but it 
does seem the changes made to the Avro Util class since v1.12.1 may be causing 
issues? Or at least it feels that way to me?

The only way I see forward is to restore the logic that was previously in place?

I have also created a bug NIFI-9629 [1].

Kind Regards,

Nathan

https://issues.apache.org/jira/browse/NIFI-9629
From: English,N,Nathan,VIR R
Sent: 20 January 2022 19:53
To: users@nifi.apache.org<mailto:users@nifi.apache.org>
Subject: Re: NiFi V1.15.2 Conversion to Avro from JSON

Hey Mark,

You beat me to the fix, Thanks for taking your time to look at it! I appreciate 
that one.

Cheers!

Nathan
________
From: Mark Payne mailto:marka...@hotmail.com>>
Sent: Wednesday, January 19, 2022 6:54:23 PM
To: users@nifi.apache.org<mailto:users@nifi.apache.org> 
mailto:users@nifi.apache.org>>
Subject: Re: NiFi V1.15.2 Conversion to Avro from JSON

Thanks Nathan. I created a Jira [1] for this. I was able to easily replicate 
with your template. Thanks for including that. Just put up a Pull Request for 
it, as well.

Thanks
-Mark



[1] 
https://issues.apache.org/jira/browse/NIFI-9594<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FNIFI-9594=04%7C01%7Cnathan.english%40bt.com%7C651a0a4448f246567e7008d9db642828%7Ca7f356889c004d5eba4129f146377ab0%7C0%7C0%7C637782045482450189%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000=PLRsmSVsDV5NTFaAvC9f1Q%2BXJrCmpK0ffcHyfZM7nr8%3D=0>



On Jan 18, 2022, at 4:49 AM, 
nathan.engl...@bt.com<mailto:nathan.engl...@bt.com> wrote:

Hi There,

We've recently upgraded to NiFi V1.15.2 from V1.12.1 and have noticed a 
difference between the JSON Tree Reader Controller Service, which is a slight 
annoyance.

When we transform our records, we have some fields that will exist or won't 
after the transformation, depending on whether there is a value for the field. 
Previously with NiFi v1.12.1, when we would convert these records to Avro using 
the Avro Writer Controller and JSON Tree Reader Services (Both with Schemas 
Set), these fields would be created and set with the default value in the Avro 
Schema. But in NiFi v1.15.2, the conversion fails with "null of boolean in 
field field_name of Avro.Schema.Name", it doesn't matter which field, type or 
default value. It all seems to fail if the field doesn't exist.

Interestingly, there isn't an issue when I set the JSON Tree Reader to Infer 
Schema instead of 'use Schema Name' with a Schema Registry.

Annoyingly before we convert to Avro from JSON, we go through a Validate Schema 
Processor, which uses the same Schema, and JSON Tree Reader but instead uses a 
JSON Writer and successfully validates the record against Schema!

It seems similar to the NIFI-9335 issue on Jira [1], but maybe with the JSON 
Tree Reader Controller service instead?

I have attached a sample flow to reproduce the issue.

I didn't know if it was worth me raising a bug ticket?

Kind Regards,

Nathan

[1] 
https://issues.apache.org/jira/browse/NIFI-9335<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FNIFI-9335=04%7C01%7Cnathan.english%40bt.com%7C651a0a4448f246567e7008d9db642828%7Ca7f356889c004d5eba4129f146377ab0%7C0%7C0%7C637782045482450189%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000=1xMp3ATXGA926%2BMqqlUHra2nvlzEfqmtc1MxFZ9rDPs%3D=0>

Re: NiFi V1.15.2 Conversion to Avro from JSON

[Mark Payne]

Thanks Nathan. I created a Jira [1] for this. I was able to easily replicate 
with your template. Thanks for including that. Just put up a Pull Request for 
it, as well.

Thanks
-Mark



[1] https://issues.apache.org/jira/browse/NIFI-9594


On Jan 18, 2022, at 4:49 AM, 
nathan.engl...@bt.com wrote:

Hi There,

We've recently upgraded to NiFi V1.15.2 from V1.12.1 and have noticed a 
difference between the JSON Tree Reader Controller Service, which is a slight 
annoyance.

When we transform our records, we have some fields that will exist or won't 
after the transformation, depending on whether there is a value for the field. 
Previously with NiFi v1.12.1, when we would convert these records to Avro using 
the Avro Writer Controller and JSON Tree Reader Services (Both with Schemas 
Set), these fields would be created and set with the default value in the Avro 
Schema. But in NiFi v1.15.2, the conversion fails with "null of boolean in 
field field_name of Avro.Schema.Name", it doesn't matter which field, type or 
default value. It all seems to fail if the field doesn't exist.

Interestingly, there isn't an issue when I set the JSON Tree Reader to Infer 
Schema instead of 'use Schema Name' with a Schema Registry.

Annoyingly before we convert to Avro from JSON, we go through a Validate Schema 
Processor, which uses the same Schema, and JSON Tree Reader but instead uses a 
JSON Writer and successfully validates the record against Schema!

It seems similar to the NIFI-9335 issue on Jira [1], but maybe with the JSON 
Tree Reader Controller service instead?

I have attached a sample flow to reproduce the issue.

I didn't know if it was worth me raising a bug ticket?

Kind Regards,

Nathan

[1] https://issues.apache.org/jira/browse/NIFI-9335

Re: DistributedMapCacheServer not included when making copy of ProcessGroup

[Mark Payne]

OK thanks. So that is not intended to happen. It’s a bug. Good news is that I 
was able to easily replicate by creating a new process group, just adding the 
distributed map cache server, and then copy & paste. Not sure why it would do 
that, but we can certainly look into it. You mind filing a Jira for it?

Thanks
-Mark


On Jan 19, 2022, at 9:52 AM, Isha Lamboo 
mailto:isha.lam...@virtualsciences.nl>> wrote:

Hi Mark,

It happens when using copy & paste to duplicate the Process Group on the canvas.

We’ve since committed the fixed process group to the registry from the dev 
server and the DistributedMapCacheServer is created successfully when changing 
version on the production server, so the component must be present in the 
flowdefinition.

Thanks,

Isha

Van: Mark Payne mailto:marka...@hotmail.com>>
Verzonden: woensdag 19 januari 2022 15:46
Aan: users@nifi.apache.org<mailto:users@nifi.apache.org>
Onderwerp: Re: DistributedMapCacheServer not included when making copy of 
ProcessGroup

Hi Isha,

When you say it was “copied” what precisely do you mean? Did you copy & paste 
the Process Group? Did you create a template and then instantiate it? Did you 
download the flow definition and then upload it again? There are a few 
different mechanism that you can use for duplicating process groups.

Thanks
-Mark



On Jan 19, 2022, at 7:25 AM, Isha Lamboo 
mailto:isha.lam...@virtualsciences.nl>> wrote:

Hi all,

I’ve encountered an issue today where a DistributedMapCacheServer controller 
service was missing from a process group with a flow.
It turns out that during development the entire process group was copied to 
test some changes and then the copy was used from there on.

A bit of testing shows that in NiFi 1.15.2 the DistributedMapCacheServer is 
silently excluded from the copy operation. I found no warnings or errors in the 
log concerning the operation.

I can imagine this is done on purpose since a copy using the same port could 
never work, but I would expect either a warning, or for the validation/enabling 
of the copy to fail.


Is this by design and are the DistributedMapCacheServer services supposed to be 
created at the root level, or can I create a bug/improvement ticket for this?

Kind regards,

Isha Lamboo

Re: DistributedMapCacheServer not included when making copy of ProcessGroup

[Mark Payne]

Hi Isha,

When you say it was “copied” what precisely do you mean? Did you copy & paste 
the Process Group? Did you create a template and then instantiate it? Did you 
download the flow definition and then upload it again? There are a few 
different mechanism that you can use for duplicating process groups.

Thanks
-Mark


On Jan 19, 2022, at 7:25 AM, Isha Lamboo 
mailto:isha.lam...@virtualsciences.nl>> wrote:

Hi all,

I’ve encountered an issue today where a DistributedMapCacheServer controller 
service was missing from a process group with a flow.
It turns out that during development the entire process group was copied to 
test some changes and then the copy was used from there on.

A bit of testing shows that in NiFi 1.15.2 the DistributedMapCacheServer is 
silently excluded from the copy operation. I found no warnings or errors in the 
log concerning the operation.

I can imagine this is done on purpose since a copy using the same port could 
never work, but I would expect either a warning, or for the validation/enabling 
of the copy to fail.

Is this by design and are the DistributedMapCacheServer services supposed to be 
created at the root level, or can I create a bug/improvement ticket for this?

Kind regards,

Isha Lamboo

Re: PutSQL in combination with ConvertJSONToSQL gives java.sql.SQLException: Invalid column type for Orcale DataType BINARY_DOUBLE

[Mark Payne]

Hi Sven,

That is interesting. Looking at the JDBC Types class, there is no BINARY_DOUBLE 
type - and no constant with a value of 101. So it appears to be a non-standard 
type.

So the script that you have in place is certainly one alternative.

It is worth noting, though, that ConvertJsonToSql and PutSQL are older 
implementations an PutDatabaseRecord should be preferred. This allows you to 
send your JSON directly to the database without that intermediate step of 
converting the JSON into SQL. Recommend you give that a try and see if it works 
for you.

Thanks
-Mark


On Jan 18, 2022, at 9:12 AM, Sven Ritter 
mailto:sven.rit...@sedapta.com>> wrote:

Hi all,

It’s my first post here and I hope I give all needed information to help me 

We are using PutSQL (1.15.2) in combination with ConvertJSONToSQL to perform 
inserts/updates into Oracle 19 DB.
It seems that Nifi has problems to handle the BINARY_DOUBLE data type.

The ConvertJSONToSQL processor produces the attributes sql.args.N.type = ‘101’ 
and sql.args.N.value = ‘0.32’.
The PutSQL processor fails with java.sql.SQLException: Invalid column type

When I pass the sql statement directly as SQL Statement in PutSQL it works fine.

As a workaround I change by groovy script all sql.args.N.type = ‘101’ 
(BINARY_DOUBLE) to sql.args.N.type = ‘6’ (FLOAT), which is working for now.

When I read the source code correctly, it seems that there is no case for 
BINARY_DOUBE and it jumps to the default (at 
org.apache.nifi.util.db.JdbcCommon.setParameter(JdbcCommon.java:863))

Did I forget something essential or is the data type BINARY_DOUBLE not 
supported?

Here the log entry:

2022-01-18 13:10:38,802 ERROR [Timer-Driven Process Thread-5] 
o.apache.nifi.processors.standard.PutSQL 
PutSQL[id=40bb12a1-cf65-3999-6d1d-d98c308f52fb] Failed to update database for 
StandardFlowFileRecord[uuid=7fd2db89-c559-44ce-b934-d47a8b6195a9,claim=StandardContentClaim
 [resourceClaim=StandardResourceClaim[id=1642429584073-1, container=default, 
section=1], offset=599401, 
length=761],offset=0,name=2beadf83-626f-4ec6-aa32-e1232db34a68,size=761] due to 
java.sql.SQLException: Invalid column type; it is possible that retrying the 
operation will succeed, so routing to retry: java.sql.SQLException: Invalid 
column type
java.sql.SQLException: Invalid column type
at 
oracle.jdbc.driver.OraclePreparedStatement.setObjectCritical(OraclePreparedStatement.java:8148)
at 
oracle.jdbc.driver.OraclePreparedStatement.setObjectInternal(OraclePreparedStatement.java:7639)
at 
oracle.jdbc.driver.OraclePreparedStatement.setObject(OraclePreparedStatement.java:8213)
at 
oracle.jdbc.driver.OraclePreparedStatementWrapper.setObject(OraclePreparedStatementWrapper.java:227)
at 
org.apache.commons.dbcp2.DelegatingPreparedStatement.setObject(DelegatingPreparedStatement.java:529)
at 
org.apache.commons.dbcp2.DelegatingPreparedStatement.setObject(DelegatingPreparedStatement.java:529)
at sun.reflect.GeneratedMethodAccessor919.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:254)
at 
org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.access$100(StandardControllerServiceInvocationHandler.java:38)
at 
org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler$ProxiedReturnObjectInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:240)
at com.sun.proxy.$Proxy192.setObject(Unknown Source)
at 
org.apache.nifi.util.db.JdbcCommon.setParameter(JdbcCommon.java:863)
at 
org.apache.nifi.util.db.JdbcCommon.setParameters(JdbcCommon.java:696)
at 
org.apache.nifi.processors.standard.PutSQL.lambda$null$4(PutSQL.java:344)
at 
org.apache.nifi.processor.util.pattern.ExceptionHandler.execute(ExceptionHandler.java:127)
at 
org.apache.nifi.processors.standard.PutSQL.lambda$new$5(PutSQL.java:342)
at 
org.apache.nifi.processors.standard.PutSQL.lambda$new$7(PutSQL.java:387)
at 
org.apache.nifi.processor.util.pattern.PutGroup.putFlowFiles(PutGroup.java:91)
at 
org.apache.nifi.processor.util.pattern.Put.onTrigger(Put.java:103)
at 
org.apache.nifi.processors.standard.PutSQL.lambda$onTrigger$19(PutSQL.java:635)
at 
org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:120)
at 
org.apache.nifi.processor.util.pattern.RollbackOnFailure.onTrigger(RollbackOnFailure.java:184)
at 

Re: Problem using DistributedMapCache

[Mark Payne]

Christian,

It looks like you are attempting to use a DistributedMapCacheClientService with 
a DistributedSetCacheServer. I.e., you’re using a MAP-based client with a 
SET-based server. You need to use a DistributedMapCacheServer, not a 
DistributedSetCacheServer.

Thanks
-Mark


On Jan 10, 2022, at 11:54 AM, Weiss, Christian 
mailto:christian.we...@sva.de>> wrote:

Hi guys,

we got a problem using DistributedMapCacheClientService with 
DistributedSetCacheServer in NiFi 1.15.1.

After setting up both services we got the following exception:

2022-01-10 16:00:15,519 ERROR [Timer-Driven Process Thread-7] 
org.apache.nifi.processors.standard.Wait 
Wait[id=4a040239-ff5d-357c-2b59-3c5417c2fae6] Failed to process session due to 
java.lang.UnsupportedOperationException: Remote cache server doesn't support 
protocol version 2; Processor Administratively Yielded for 1 sec: 
java.lang.UnsupportedOperationException: Remote cache server doesn't support 
protocol version 2
java.lang.UnsupportedOperationException: Remote cache server doesn't support 
protocol version 2
at 
org.apache.nifi.distributed.cache.client.CacheClientRequestHandler.invoke(CacheClientRequestHandler.java:94)
at 
org.apache.nifi.distributed.cache.client.DistributedCacheClient.invoke(DistributedCacheClient.java:69)
at 
org.apache.nifi.distributed.cache.client.NettyDistributedMapCacheClient.fetch(NettyDistributedMapCacheClient.java:268)
at 
org.apache.nifi.distributed.cache.client.DistributedMapCacheClientService.fetch(DistributedMapCacheClientService.java:209)
at sun.reflect.GeneratedMethodAccessor822.invoke(Unknown Source)
   at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:254)
at 
org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:105)
at com.sun.proxy.$Proxy403.fetch(Unknown Source)
at 
org.apache.nifi.processors.standard.WaitNotifyProtocol.getSignal(WaitNotifyProtocol.java:251)
at org.apache.nifi.processors.standard.Wait.onTrigger(Wait.java:420)
at 
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at 
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1273)
at 
org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214)
at 
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:103)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

Configuration:





Anyone else with similar problems out there?

Thanks and regards,
Christian

Das SVA Mail-System ist mit einem Mailverschluesselungs-Gateway ausgestattet. 
Wenn Sie moechten, dass an Sie gerichtete E-Mails verschluesselt werden, senden 
Sie einfach eine S/MIME-signierte E-Mail oder Ihren PGP Public Key an 
christian.we...@sva.de.

The SVA mail system is equipped with an email encryption gateway. If you want 
email sent to you to be encrypted please send a S/MIME signed email or your PGP 
public key to christian.we...@sva.de.

Re: javax.net.ssl.SSLPeerUnverifiedException

[Mark Payne]

Lior,

What do you have set for the “nifi.zookeeper.connect.string” property in 
nifi.properties?

Thanks
-Mark


On Dec 16, 2021, at 4:26 AM, Lior Halperin 
mailto:lior.halpe...@outseer.com>> wrote:

Hi,
We are using nifi 1.15 secured cluster with external zk 3.7.0 defined in the zk 
conf:
ssl.hostnameVerification=false
ssl.quorum.hostnameVerification=false
sslQuorum=false
also in the nifi nodes zookeeper properties we defined
ssl.hostnameVerification=false
ssl.quorum.hostnameVerification=false

on nifi start up nodes we get :
2021-12-15 21:57:43,440 ERROR [nioEventLoopGroup-2-1] 
o.apache.zookeeper.common.ZKTrustManager Failed to verify host address: 
127.0.0.1
javax.net.ssl.SSLPeerUnverifiedException: Certificate for 
<127.0.0.1> doesn't match common name of the certificate subject: APP SERVER KEY




what are definitions we miss that should eliminate the 
SSLPeerUnverifiedException?


Internal Use - Confidential

Re: NiFi flow.xml.gz corruption

[Mark Payne]

Hi Ravi,

Not sure why you would be seeing the flow get corrupted. When you say 
“corrupted” - do you mean truly corrupted? As in, the file cannot be 
read/parsed? Or do you mean that it’s out of sync, meaning that NiFi can read 
it but won’t join the cluster because its flow is different from the cluster’s 
flow?

In either case, though, removing the file and restarting is the easiest option 
traditionally. Within Kubernetes I can see how this would get complicated. In 
the upcoming release (1.16) we have made a lot of improvements around 
clustering that should resolve this. But you’d need to update to the latest to 
get these improvements.

I don’t know enough about kubernetes to make any recommendations about 
lifecycle hooks, etc. You could certainly reach out to the Kubernetes community 
about that.

As far as NiFi registry goes: I don’t think it’s going to help here. NiFi 
Registry makes it convenient to build a dataflow and then store it in the 
registry. You can then check out that flow in another NiFi instance (or 
checkout a second copy in the same NiFi instance) and then see when changes are 
made, update to the newest version of the flow, etc. Think version control for 
individual Process Groups.

Hope this helps!
-Mark


> On Dec 16, 2021, at 3:54 AM, Ravi Nallappan  wrote:
> 
> Hi,
> 
> A general question regarding flow.xml.gz. (Nifi version 1.10.0)
> 
> We have about 6-8 nifi nodes in a cluster (kubernetes environment) and we do 
> see the file get corrupted at times and causing the node to not come up on 
> restarts and eventually kubernetes gives up.
> 
> Based on my search on the issue, shows the easiest recovery is to remove the 
> corrupted flow.xml.gz and let the node come up, join the cluster and sync up 
> with golden copy from other nodes.
> 
> However, this will be a challenge to do in kubernetes environment, any 
> suggestion on this? (possibly can write a script to check and do the action, 
> but how do we add the hook to do just the pod gets restarted)
> 
> Is NiFi Registry a better recommendation in production environment? If yes, 
> will look at this as a longer term solution.
> 
> Thanks in advance.
> 
> regards,
> 
> Ravi Nallappan
> 

Re: CryptographicHashContent calculates 2 differents sha256 hashes on the same content

[Mark Payne]

 New Count: 11934098
>> Byte Value: histogram.44, Previous Count: 11936337, New Count: 11936346
>> Byte Value: histogram.45, Previous Count: 11935580, New Count: 11935582
>> Byte Value: histogram.46, Previous Count: 11929598, New Count: 11929599
>> Byte Value: histogram.47, Previous Count: 11934083, New Count: 11934085
>> Byte Value: histogram.48, Previous Count: 11928858, New Count: 11928860
>> Byte Value: histogram.49, Previous Count: 11931098, New Count: 11931113
>> Byte Value: histogram.50, Previous Count: 11930618, New Count: 11930614
>> Byte Value: histogram.51, Previous Count: 11925429, New Count: 11925435
>> Byte Value: histogram.52, Previous Count: 11929741, New Count: 11929733
>> Byte Value: histogram.53, Previous Count: 11934160, New Count: 11934155
>> Byte Value: histogram.54, Previous Count: 11931999, New Count: 11931980
>> Byte Value: histogram.55, Previous Count: 11930465, New Count: 11930477
>> Byte Value: histogram.56, Previous Count: 11926194, New Count: 11926190
>> Byte Value: histogram.57, Previous Count: 11926386, New Count: 11926381
>> Byte Value: histogram.58, Previous Count: 11924871, New Count: 11924865
>> Byte Value: histogram.59, Previous Count: 11929331, New Count: 11929326
>> Byte Value: histogram.60, Previous Count: 11926951, New Count: 11926943
>> Byte Value: histogram.61, Previous Count: 11928631, New Count: 11928619
>> Byte Value: histogram.62, Previous Count: 11927549, New Count: 11927553
>> Byte Value: histogram.63, Previous Count: 23856730, New Count: 23856718
>> Byte Value: histogram.64, Previous Count: 11930288, New Count: 11930293
>> Byte Value: histogram.65, Previous Count: 11931523, New Count: 11931527
>> Byte Value: histogram.66, Previous Count: 11932821, New Count: 11932818
>> Byte Value: histogram.67, Previous Count: 11932509, New Count: 11932510
>> Byte Value: histogram.68, Previous Count: 11929613, New Count: 11929614
>> Byte Value: histogram.69, Previous Count: 11928651, New Count: 11928654
>> Byte Value: histogram.70, Previous Count: 11929253, New Count: 11929247
>> Byte Value: histogram.71, Previous Count: 11931521, New Count: 11931512
>> Byte Value: histogram.72, Previous Count: 11925805, New Count: 11925808
>> Byte Value: histogram.73, Previous Count: 11934833, New Count: 11934826
>> Byte Value: histogram.74, Previous Count: 11928314, New Count: 11928312
>> Byte Value: histogram.75, Previous Count: 11923854, New Count: 11923863
>> Byte Value: histogram.76, Previous Count: 11930892, New Count: 11930898
>> Byte Value: histogram.77, Previous Count: 11927528, New Count: 11927525
>> Byte Value: histogram.78, Previous Count: 11932850, New Count: 11932857
>> Byte Value: histogram.79, Previous Count: 11934471, New Count: 11934461
>> Byte Value: histogram.80, Previous Count: 11925707, New Count: 11925714
>> Byte Value: histogram.81, Previous Count: 11929213, New Count: 11929206
>> Byte Value: histogram.82, Previous Count: 11931334, New Count: 11931323
>> Byte Value: histogram.83, Previous Count: 11936739, New Count: 11936732
>> Byte Value: histogram.84, Previous Count: 11927855, New Count: 11927832
>> Byte Value: histogram.85, Previous Count: 11931668, New Count: 11931665
>> Byte Value: histogram.86, Previous Count: 11928609, New Count: 11928604
>> Byte Value: histogram.87, Previous Count: 11931930, New Count: 11931933
>> Byte Value: histogram.88, Previous Count: 11934341, New Count: 11934345
>> Byte Value: histogram.89, Previous Count: 11927519, New Count: 11927518
>> Byte Value: histogram.9, Previous Count: 11928004, New Count: 11928001
>> Byte Value: histogram.90, Previous Count: 11933502, New Count: 11933517
>> Byte Value: histogram.94, Previous Count: 11932024, New Count: 11932035
>> Byte Value: histogram.95, Previous Count: 11932693, New Count: 11932679
>> Byte Value: histogram.97, Previous Count: 11928428, New Count: 11928424
>> Byte Value: histogram.98, Previous Count: 11933195, New Count: 11933196
>> Byte Value: histogram.99, Previous Count: 11924273, New Count: 11924282
>> 
>> Den tir. 2. nov. 2021 kl. 15.41 skrev Mark Payne :
>>> 
>>> Jens,
>>> 
>>> The histograms, in and of themselves, are not very interesting. The 
>>> interesting thing would be the difference in the histogram before & after 
>>> the hash. Can you provide the ERROR level logs generated by the 
>>> ExecuteScript? That’s what is of interest.
>>> 
>>> Thanks
>>> -Mark
>>> 
>>> 
>>> On Nov 2, 2021, at 1:35 AM, Jens M. Kofoed  wrote:
>>> 
>>> Hi Mark and Joe
>>> 
>>> Yesterday morning I implemented Mark's script in my

Re: Large amounts of data in cluster state

[Mark Payne]

Hi Isha,

ListHDFS does not store the listing of files that it’s found in its state. 
Doing so would cause a lot of problems. Instead, it only stores the timestamps 
of the latest file that it has found and the timestamp of the latest file that 
it has listed/sent out. If peak loads are triggering instability, it likely has 
more to do with either overutilization of the CPU or excessive garbage 
collection because you’re running out of heap space. Would recommend monitoring 
both the CPU Load and the Garbage Collection. Also, what is the Scheduling 
Period set to for your ListHDFS (in the Settings tab)? It probably is defaulted 
to 0 sec but should probably be set to something like 1 min or something to 
avoid constantly hitting both HDFS and ZooKeeper.

Also, there have been many improvements since 1.9 to improve cluster 
performance and stability. Probably worth looking into upgrading.

Thanks
-Mark



On Oct 27, 2021, at 12:56 PM, Isha Lamboo 
mailto:isha.lam...@virtualsciences.nl>> wrote:

Hi all,

I have a question that some of you must have tackled already. On a NiFi cluster 
(still 1.9 at the moment) that is normally very stable, the users sometimes 
trigger peak loads that cause disconnections or other issues either in NiFi 
itself or the external zookeeper cluster. The clear example I found is a 
ListHDFS processor that maintains large amount of state (many millions of 
files) being cleared and refilled, but I suspect it may just keep adding more 
and more to the state.

So far, we’ve increased Zookeeper initLimit and SyncLimit and did some NiFi 
timeout tuning, but it’s hard to figure out a sensible value when the reported 
times are normally nowhere near the limits. The users also keep finding bigger 
data loads which they repeatedly process through NiFi into some processing 
applications. Deleting the files is also not an option because of the 
re-processing It seems to me that increasing timeouts from several seconds to 
what’s going to be minutes must impact some other aspect of Zookeeper.

Is there a flow design or tuning strategy that avoids large changes to the 
state in a short time like this?

Are Zookeeper timeouts of 60+ secs actually usual?

Met vriendelijke groet,

Isha Lamboo
Data Engineer
+31 (0)6 20 50 15 91


isha.lam...@virtualsciences.nl

Edisonbaan 15
3439 MN Nieuwegein
www.virtualsciences.nl
www.conclusion.nl
Bekijk hier de algemene voorwaarden van 
Conclusion

Re: CryptographicHashContent calculates 2 differents sha256 hashes on the same content

[Mark Payne]

And the actual script:



import org.apache.nifi.flowfile.FlowFile

import java.util.stream.Collectors

Map getPreviousHistogram(final FlowFile flowFile) {
final Map histogram = 
flowFile.getAttributes().entrySet().stream()
.filter({ entry -> entry.getKey().startsWith("histogram.") })
.collect(Collectors.toMap({ entry -> entry.key}, { entry -> entry.value 
}))
return histogram;
}

Map createHistogram(final FlowFile flowFile, final InputStream 
inStream) {
final Map histogram = new HashMap<>();
final int[] distribution = new int[256];
Arrays.fill(distribution, 0);

long total = 0L;
final byte[] buffer = new byte[8192];
int len;
while ((len = inStream.read(buffer)) > 0) {
for (int i=0; i < len; i++) {
final int val = buffer[i];
distribution[val]++;
total++;
}
}

for (int i=0; i < 256; i++) {
histogram.put("histogram." + i, String.valueOf(distribution[i]));
}
histogram.put("histogram.totalBytes", String.valueOf(total));

return histogram;
}

void logHistogramDifferences(final Map previous, final 
Map updated) {
final StringBuilder sb = new StringBuilder("There are differences in the 
histogram\n");
final Map sorted = new TreeMap<>(previous)
for (final Map.Entry entry : sorted.entrySet()) {
final String key = entry.getKey();
final String previousValue = entry.getValue();
final String updatedValue = updated.get(entry.getKey())

if (!Objects.equals(previousValue, updatedValue)) {
sb.append("Byte Value: ").append(key).append(", Previous Count: 
").append(previousValue).append(", New Count: 
").append(updatedValue).append("\n");
}
}

log.error(sb.toString());
}


def flowFile = session.get()
if (flowFile == null) {
return
}

final Map previousHistogram = getPreviousHistogram(flowFile)
Map histogram = null;

final InputStream inStream = session.read(flowFile);
try {
histogram = createHistogram(flowFile, inStream);
} finally {
inStream.close()
}

if (!previousHistogram.isEmpty()) {
if (previousHistogram.equals(histogram)) {
log.info<http://log.info>("Histograms match")
} else {
logHistogramDifferences(previousHistogram, histogram)
session.transfer(flowFile, REL_FAILURE)
return;
}
}

flowFile = session.putAllAttributes(flowFile, histogram)
session.transfer(flowFile, REL_SUCCESS)





On Oct 27, 2021, at 9:43 AM, Mark Payne 
mailto:marka...@hotmail.com>> wrote:

Jens,

For a bit of background here, the reason that Joe and I have expressed interest 
in NFS file systems is that the way the protocol works, it is allowed to 
receive packets/chunks of the file out-of-order. So, what happens is let’s say 
a 1 MB file is being written. The first 500 KB are received. Then instead of 
the the 501st KB it receives the 503rd KB. What happens is that the size of the 
file on the file system becomes 503 KB. But what about 501 & 502? Well when you 
read the data, the file system just returns ASCII NUL characters (byte 0) for 
those bytes. Once the NFS server receives those bytes, it then goes back and 
fills in the proper bytes. So if you’re running on NFS, it is possible for the 
contents of the file on the underlying file system to change out from under 
you. It’s not clear to me what other types of file system might do something 
similar.

So, one thing that we can do is to find out whether or not the contents of the 
underlying file have changed in some way, or if there’s something else 
happening that could perhaps result in the hashes being wrong. I’ve put 
together a script that should help diagnose this.

Can you insert an ExecuteScript processor either just before or just after your 
CryptographicHashContent processor? Doesn’t really matter whether it’s run just 
before or just after. I’ll attach the script here. It’s a Groovy Script so you 
should be able to use ExecuteScript with Script Engine = Groovy and the 
following script as the Script Body. No other changes needed.

The way the script works, it reads in the contents of the FlowFile, and then it 
builds up a histogram of all byte values (0-255) that it sees in the contents, 
and then adds that as attributes. So it adds attributes such as:
histogram.0 = 280273
histogram.1 = 2820
histogram.2 = 48202
histogram.3 = 3820
…
histogram.totalBytes = 1780928732

It then checks if those attributes have already been added. If so, after 
calculating that histogram, it checks against the previous values (in the 
attributes). If they are the same, the FlowFile goes to ’success’. If they are 
different, it logs an error indicating the before/after value for any byte 
whose distribution was different, and it routes to failure.

So, if for example, the first time through it sees 280,273 b

Re: CryptographicHashContent calculates 2 differents sha256 hashes on the same content

[Mark Payne]

FlowFile-stream file and unpacked it back into NiFi I 
> > > calculate a sha256. 1 minutes later I recalculate the sha256 on the exact 
> > > same file. And got a new hash. That is what worry’s me.
> > > The fact that the same file can be recalculated and produce two different 
> > > hashes, is very strange, but it happens. Over the last 5 months it have 
> > > only happen 35-40 times.
> > >
> > > I can understand if the file is not completely loaded and saved into the 
> > > content repository before the hashing starts. But I believe that the 
> > > unpack process don’t forward the flow file to the next process before it 
> > > is 100% finish unpacking and saving the new content to the repository.
> > >
> > > I have a test flow, where a GenerateFlowfile has created 6x 1GB files (2 
> > > files per node) and next process was a hashcontent before it run into a 
> > > test loop. Where files are uploaded via PutSFTP to a test server, and 
> > > downloaded again and recalculated the hash. I have had one issue after 3 
> > > days of running.
> > > Now the test flow is running without the Put/Fetch sftp processors.
> > >
> > > Another problem is that I can’t find any correlation to other events. Not 
> > > within NIFI, nor the server itself or VMWare. If I just could find any 
> > > other event which happens at the same time, I might be able to force some 
> > > kind of event to trigger the issue.
> > > I have tried to force VMware to migrate a NiFi node to another host. 
> > > Forcing it to do a snapshot and deleting snapshots, but nothing can 
> > > trigger and error.
> > >
> > > I know it will be very very difficult to reproduce. But I will setup 
> > > multiple NiFi instances running different test flows to see if I can find 
> > > any reason why it behaves as it does.
> > >
> > > Kind Regards
> > > Jens M. Kofoed
> > >
> > > Den 20. okt. 2021 kl. 16.39 skrev Mark Payne 
> > > mailto:marka...@hotmail.com>>:
> > >
> > > Jens,
> > >
> > > Thanks for sharing the images.
> > >
> > > I tried to setup a test to reproduce the issue. I’ve had it running for 
> > > quite some time. Running through millions of iterations.
> > >
> > > I’ve used 5 KB files, 50 KB files, 50 MB files, and larger (to the tune 
> > > of hundreds of MB). I’ve been unable to reproduce an issue after millions 
> > > of iterations.
> > >
> > > So far I cannot replicate. And since you’re pulling the data via SFTP and 
> > > then unpacking, which preserves all original attributes from a different 
> > > system, this can easily become confusing.
> > >
> > > Recommend trying to reproduce with SFTP-related processors out of the 
> > > picture, as Joe is mentioning. Either using GetFile/FetchFile or 
> > > GenerateFlowFile. Then immediately use CryptographicHashContent to 
> > > generate an ‘initial hash’, copy that value to another attribute, and 
> > > then loop, generating the hash and comparing against the original one. 
> > > I’ll attach a flow that does this, but not sure if the email server will 
> > > strip out the attachment or not.
> > >
> > > This way we remove any possibility of actual corruption between the two 
> > > nifi instances. If we can still see corruption / different hashes within 
> > > a single nifi instance, then it certainly warrants further investigation 
> > > but i can’t see any issues so far.
> > >
> > > Thanks
> > > -Mark
> > >
> > >
> > >
> > >
> > >
> > > On Oct 20, 2021, at 10:21 AM, Joe Witt 
> > > mailto:joe.w...@gmail.com>> wrote:
> > >
> > > Jens
> > >
> > > Actually is this current loop test contained within a single nifi and 
> > > there you see corruption happen?
> > >
> > > Joe
> > >
> > > On Wed, Oct 20, 2021 at 7:14 AM Joe Witt 
> > > mailto:joe.w...@gmail.com>> wrote:
> > >
> > > Jens,
> > >
> > > You have a very involved setup including other systems (non NiFi).  Have 
> > > you removed those systems from the equation so you have more evidence to 
> > > support your expectation that NiFi is doing something other than you 
> > > expect?
> > >
> > > Joe
> > >
> > > On Wed, Oct 20, 2021 at 7:10 AM Jens M. Kofoed 
> > > mailto:jmkofoed@gmail.com>&

Re: Penalty feature of Processor (Disable)

[Mark Payne]

Bilal,

In the Settings tab, you can set the “Penalty Duration” to “0 secs”.

Thanks
-Mark


On Oct 25, 2021, at 10:20 AM, Bilal Bektas 
mailto:bilal.bek...@obase.com>> wrote:

Hi Community,

We use LookupAttribute processor in order to get lookup value from Teradata or 
Oracle DB. Processors work as follows:

LookupAttribute (Teradata)  ---(failure & unmatched) ---> LookupAttribute 
(Oracle)

This flows works well and LookupAttribute (Teradata) penalizes to flow files 
when Teradata DB is down. Therefore, the queue on upstream connection of 
LookupAttribute (Teradata) increases. But, we don't want to that 
LookupAttribute (Teradata) penalizes to flow files. We want to that 
LookupAttribute (Teradata) processor forwards flow files to failure downstream 
connection when all failure situation on LookupAttribute (Teradata). Thus, 
LookupAttribute (Oracle) can process flow files which cannot process on 
LookupAttribute (Teradata).

Is it possible to disable penalty feature of processor or is there any solution 
which you can suggest for this situation.

Thank you in advance,

--Bilal

obase
TEL: +90216 527 30 00
FAX: +90216 527 31 11
[http://www.obase.com/images/signature/home.png] 
[http://www.obase.com/images/signature/facebook.png] 
  
[http://www.obase.com/images/signature/twitter.png] 
  
[http://www.obase.com/images/signature/linkedin.png] 

[http://www.obase.com/images/signature/obaselogo.png]

Bu elektronik posta ve onunla iletilen bütün dosyalar sadece göndericisi 
tarafindan almasi amaclanan yetkili gercek ya da tüzel kisinin kullanimi 
icindir. Eger söz konusu yetkili alici degilseniz bu elektronik postanin 
icerigini aciklamaniz, kopyalamaniz, yönlendirmeniz ve kullanmaniz kesinlikle 
yasaktir ve bu elektronik postayi derhal silmeniz gerekmektedir. OBASE bu 
mesajin icerdigi bilgilerin doğruluğu veya eksiksiz oldugu konusunda herhangi 
bir garanti vermemektedir. Bu nedenle bu bilgilerin ne sekilde olursa olsun 
iceriginden, iletilmesinden, alinmasindan ve saklanmasindan sorumlu degildir. 
Bu mesajdaki görüsler yalnizca gönderen kisiye aittir ve OBASE görüslerini 
yansitmayabilir.

Bu e-posta bilinen bütün bilgisayar virüslerine karsi taranmistir.

This e-mail and any files transmitted with it are confidential and intended 
solely for the use of the individual or entity to whom they are addressed. If 
you are not the intended recipient you are hereby notified that any 
dissemination, forwarding, copying or use of any of the information is strictly 
prohibited, and the e-mail should immediately be deleted. OBASE makes no 
warranty as to the accuracy or completeness of any information contained in 
this message and hereby excludes any liability of any kind for the information 
contained therein or for the information transmission, recepxion, storage or 
use of such in any way whatsoever. The opinions expressed in this message 
belong to sender alone and may not necessarily reflect the opinions of OBASE.

This e-mail has been scanned for all known computer viruses.

Re: CryptographicHashContent calculates 2 differents sha256 hashes on the same content

[Mark Payne]

Jens,

Thanks for sharing the images.

I tried to setup a test to reproduce the issue. I’ve had it running for quite 
some time. Running through millions of iterations.

I’ve used 5 KB files, 50 KB files, 50 MB files, and larger (to the tune of 
hundreds of MB). I’ve been unable to reproduce an issue after millions of 
iterations.

So far I cannot replicate. And since you’re pulling the data via SFTP and then 
unpacking, which preserves all original attributes from a different system, 
this can easily become confusing.

Recommend trying to reproduce with SFTP-related processors out of the picture, 
as Joe is mentioning. Either using GetFile/FetchFile or GenerateFlowFile. Then 
immediately use CryptographicHashContent to generate an ‘initial hash’, copy 
that value to another attribute, and then loop, generating the hash and 
comparing against the original one. I’ll attach a flow that does this, but not 
sure if the email server will strip out the attachment or not.

This way we remove any possibility of actual corruption between the two nifi 
instances. If we can still see corruption / different hashes within a single 
nifi instance, then it certainly warrants further investigation but i can’t see 
any issues so far.

Thanks
-Mark





On Oct 20, 2021, at 10:21 AM, Joe Witt 
mailto:joe.w...@gmail.com>> wrote:

Jens

Actually is this current loop test contained within a single nifi and there you 
see corruption happen?

Joe

On Wed, Oct 20, 2021 at 7:14 AM Joe Witt 
mailto:joe.w...@gmail.com>> wrote:
Jens,

You have a very involved setup including other systems (non NiFi).  Have you 
removed those systems from the equation so you have more evidence to support 
your expectation that NiFi is doing something other than you expect?

Joe

On Wed, Oct 20, 2021 at 7:10 AM Jens M. Kofoed 
mailto:jmkofoed@gmail.com>> wrote:
Hi

Today I have another file which have been running through the retry loop one 
time. To test the processors and the algorithm I added the HashContent 
processor and also added hashing by SHA-1.
I file have been going through the system, and both the SHA-1 and SHA-256 are 
both different than expected. with a 1 minutes delay the file is going back 
into the hashing content flow and this time it calculates both hashes fine.

I don't believe that the hashing is buggy, but something is very very strange. 
What can influence the processors/algorithm to calculate a different hash???
All the input/output claim information is exactly the same. It is the same 
flow/content file going in a loop. It happens on all 3 nodes.

Any suggestions for where to dig ?

Regards
Jens M. Kofoed



Den ons. 20. okt. 2021 kl. 06.34 skrev Jens M. Kofoed 
mailto:jmkofoed@gmail.com>>:
Hi Mark

Thanks for replaying and the suggestion to look at the content Claim.
These 3 pictures is from the first attempt:
  

Yesterday I realized that the content was still in the archive, so I could 
Replay the file.

So here are the same pictures but for the replay and as you can see the 
Identifier, offset and Size are all the same.
  

In my flow if the hash does not match my original first calculated hash, it 
goes into a retry loop. Here are the pictures for the 4th time the file went 
through:
  
Here the content Claim is all the same.

It is very rare that we see these issues <1 : 1.000.000 files and only with 
large files. Only once have I seen the error with a 110MB file, the other times 
the files size are above 800MB.
This time it was a Nifi-Flowstream v3 file, which has been exported from one 
system and imported in another. But while the file has been imported it is the 
same file inside NIFI and it stays at the same node. Going through the same 
loop of processors multiple times and in the end the CryptographicHashContent 
calculate a different SHA256 than it did earlier. This should not be 
possible!!! And that is what concern my the most.
What can influence the same processor to calculate 2 different sha256 on the 
exact same content???

Regards
Jens M. Kofoed


Den tir. 19. okt. 2021 kl. 16.51 skrev Mark Payne 
mailto:marka...@hotmail.com>>:
Jens,

In the two provenance events - one showing a hash of dd4cc… and the other 
showing f6f0….
If you go to the Content tab, do they both show the same Content Claim? I.e., 
do the Input Claim / Output Claim show the same values for Container, Section, 
Identifier, Offset, and Size?

Thanks
-Mark

On Oct 19, 2021, at 1:22 AM, Jens M. Kofoed 
mailto:jmkofoed@gmail.com>> wrote:

Dear NIFI Users

I have posted this mail in the developers mailing list and just want to inform 
all of our about a very odd behavior we are facing.
The background:
We have data going between 2 different NIFI systems which has no direct network 
access to each other. Therefore we calculate a SHA256 hash value of the content 
at system 1, before the flowfile and data are combined and saved as a 
"flowfile-stream-v3" pkg file. The file is 

Re: CryptographicHashContent calculates 2 differents sha256 hashes on the same content

[Mark Payne]

Jens,

In the two provenance events - one showing a hash of dd4cc… and the other 
showing f6f0….
If you go to the Content tab, do they both show the same Content Claim? I.e., 
do the Input Claim / Output Claim show the same values for Container, Section, 
Identifier, Offset, and Size?

Thanks
-Mark

On Oct 19, 2021, at 1:22 AM, Jens M. Kofoed 
mailto:jmkofoed@gmail.com>> wrote:

Dear NIFI Users

I have posted this mail in the developers mailing list and just want to inform 
all of our about a very odd behavior we are facing.
The background:
We have data going between 2 different NIFI systems which has no direct network 
access to each other. Therefore we calculate a SHA256 hash value of the content 
at system 1, before the flowfile and data are combined and saved as a 
"flowfile-stream-v3" pkg file. The file is then transported to system 2, where 
the pkg file is unpacked and the flow can continue. To be sure about file 
integrity we calculate a new sha256 at system 2. But sometimes we see that the 
sha256 gets another value, which might suggest the file was corrupted. But 
recalculating the sha256 again gives a new hash value.



Tonight I had yet another file which didn't match the expected sha256 hash 
value. The content is a 1.7GB file and the Event Duration was "00:00:17.539" to 
calculate the hash.
I have created a Retry loop, where the file will go to a Wait process for 
delaying the file 1 minute and going back to the CryptographicHashContent for a 
new calculation. After 3 retries the file goes to the retries_exceeded and goes 
to a disabled process just to be in a queue so I manually can look at it. This 
morning I rerouted the file from my retries_exceeded queue back to the 
CryptographicHashContent for a new calculation and this time it calculated the 
correct hash value.

THIS CAN'T BE TRUE :-( :-( But it is. - Something very very strange is 
happening.


We are running NiFi 1.13.2 in a 3 node cluster at Ubuntu 20.04.02 with openjdk 
version "1.8.0_292", OpenJDK Runtime Environment (build 
1.8.0_292-8u292-b10-0ubuntu1~20.04-b10), OpenJDK 64-Bit Server VM (build 
25.292-b10, mixed mode). Each server is a VM with 4 CPU, 8GB Ram on VMware 
ESXi, 7.0.2. Each NIFI node is running at different vm physical hosts.
I have inspected different logs to see if I can find any correlation what 
happened at the same time as the file is going through my loop, but there are 
no event/task at that exact time.

System 1:
At 10/19/2021 00:15:11.247 CEST my file is going through a 
CryptographicHashContent: SHA256 value: 
dd4cc7ef8dbc8d70528e8aa788581f0ab88d297c9c9f39b6b542df68952efd20
The file is exported as a "FlowFile Stream, v3" to System 2

SYSTEM 2:
At 10/19/2021 00:18:10.528 CEST the file is going through a 
CryptographicHashContent: SHA256 value: 
f6f0909aacae4952f10f6fa7704f3e55d0481ec211d495993550aedbb3fe0819

At 10/19/2021 00:19:08.996 CEST the file is going through the same 
CryptographicHashContent at system 2: SHA256 value: 
f6f0909aacae4952f10f6fa7704f3e55d0481ec211d495993550aedbb3fe0819
At 10/19/2021 00:20:04.376 CEST the file is going through the same a 
CryptographicHashContent at system 2: SHA256 value: 
f6f0909aacae4952f10f6fa7704f3e55d0481ec211d495993550aedbb3fe0819
At 10/19/2021 00:21:01.711 CEST the file is going through the same a 
CryptographicHashContent at system 2: SHA256 value: 
f6f0909aacae4952f10f6fa7704f3e55d0481ec211d495993550aedbb3fe0819

At 10/19/2021 06:07:43.376 CEST the file is going through the same a 
CryptographicHashContent at system 2: SHA256 value: 
dd4cc7ef8dbc8d70528e8aa788581f0ab88d297c9c9f39b6b542df68952efd20


How on earth can this happen???

Kind Regards
Jens M. Kofoed