That was an awesome answer/clarification Raman! Thanks
On Thu, Dec 21, 2017 at 10:21 AM, Ramaninder Singh Jhajj <[email protected]> wrote: > Hi Sally, > > You are facing issue with 10K files because that is the default value for > backpressure in Queue. The queue will be full with 10K files and will > initiate the backpressure and your MergeContent processor never receive the > number of files you are expecting. You will need to change a couple of > things in here: > > Rightclick on the queue just before the MergeContent and select "Configure > -> Settings" and set the "Back Pressure Object Threshold" to a number more > then what you want to merge (default is 10K) if you want to merge 14,000 > then maybe set the number here to 15,000 (You might also want to consider > changing the "Back Pressure Data Size Threshold" if flow files will exceed 1 > GB because backpressure will trigger if any one of those conditions are met) > In your screenshot, I can see that you have not set the "MaxBinAge" > property. Set a time for that property in mergeContent processor. This time > will depend on the velocity of your data arrival, it can be 5 minutes, 20 > minutes etc depending on your needs. > In your screenshot, the max. number of bins is set to 32000 which is same as > the maximum number of files you want to merge. This is not correct. Keep the > bins to maybe 5 or 10. > > > This should work for you, but as James already mentioned, you might want to > try with multiple MergeContent processors to achieve the best performance. > > > Regards, > Raman > > On Thu, Dec 21, 2017 at 3:59 PM, James Wing <[email protected]> wrote: >> >> You can use MergeContent to merge more than 10,000 flowfiles, but you may >> experience slower performance with very large numbers of files in a single >> merge. The recommended configuration is to use two MergeContent processors >> in sequence. The first MergeContent would merging groups of individual >> flowfiles into bundles of perhaps 1,000, and the seconds merging the bundles >> of thousands together. >> >> On Thu, Dec 21, 2017 at 3:00 AM, sally <[email protected]> >> wrote: >>> >>> I have 1400 flowFile with same name and i want to enroll them into one >>> file, >>> i have read that i can use mergeContent processor for this purpose but >>> when >>> the amount of flowfile is more than 10000 it can't combine them in one >>> file, >>> what should i change to make this processor work properly? here is my >>> configuration image example: >>> >>> >>> <http://apache-nifi-users-list.2361937.n4.nabble.com/file/t330/mergeExmple.png> >>> >>> >>> >>> >>> -- >>> Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/ >> >> >
