Hello.

It's me again regarding my mergerecord question.

I still don't manage to have what I want, I may have understand how bin based 
processor works, it's for clarification and a question regarding performance.

I want to merge a huge number of 300 octets flowfiles in 128 MB parquet file. 

My understanding is, for mergerecord to be able to create a bin with 128MB of 
data, these data must be in queue. We can't feed the bin "one flow at a time", 
so working with small flowfiles, I have to set the backpressure parameter to 
something really high, or remove completely the number of flowfile backpressure 
limit.

I understood by reading 
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.6.0/org.apache.nifi.processors.standard.MergeRecord/additionalDetails.html
 that it's not the "good" way to do, but I should cascade multiple merge to 
"slowly" make the flowfile bigger?

I've made some test with a single level but I hit the "provenance recording 
rate". Will multiple level help?

Thanks for any help.

Aurélien.

This electronic transmission (and any attachments thereto) is intended solely 
for the use of the addressee(s). It may contain confidential or legally 
privileged information. If you are not the intended recipient of this message, 
you must delete it immediately and notify the sender. Any unauthorized use or 
disclosure of this message is strictly prohibited.  Faurecia does not guarantee 
the integrity of this transmission and shall therefore never be liable if the 
message is altered or falsified nor for any virus, interception or damage to 
your system.

Reply via email to