Hi Sally, You are facing issue with 10K files because that is the default value for backpressure in Queue. The queue will be full with 10K files and will initiate the backpressure and your MergeContent processor never receive the number of files you are expecting. You will need to change a couple of things in here:
1. Rightclick on the queue just before the MergeContent and select "Configure -> Settings" and set the "Back Pressure Object Threshold" to a number more then what you want to merge (default is 10K) if you want to merge 14,000 then maybe set the number here to 15,000 (You might also want to consider changing the "Back Pressure Data Size Threshold" if flow files will exceed 1 GB because backpressure will trigger if any one of those conditions are met) 2. In your screenshot, I can see that you have not set the "MaxBinAge" property. Set a time for that property in mergeContent processor. This time will depend on the velocity of your data arrival, it can be 5 minutes, 20 minutes etc depending on your needs. 3. In your screenshot, the max. number of bins is set to 32000 which is same as the maximum number of files you want to merge. This is not correct. Keep the bins to maybe 5 or 10. This should work for you, but as James already mentioned, you might want to try with multiple MergeContent processors to achieve the best performance. Regards, Raman On Thu, Dec 21, 2017 at 3:59 PM, James Wing <[email protected]> wrote: > You can use MergeContent to merge more than 10,000 flowfiles, but you may > experience slower performance with very large numbers of files in a single > merge. The recommended configuration is to use two MergeContent processors > in sequence. The first MergeContent would merging groups of individual > flowfiles into bundles of perhaps 1,000, and the seconds merging the > bundles of thousands together. > > On Thu, Dec 21, 2017 at 3:00 AM, sally <[email protected]> > wrote: > >> I have 1400 flowFile with same name and i want to enroll them into one >> file, >> i have read that i can use mergeContent processor for this purpose but >> when >> the amount of flowfile is more than 10000 it can't combine them in one >> file, >> what should i change to make this processor work properly? here is my >> configuration image example: >> >> <http://apache-nifi-users-list.2361937.n4.nabble.com/file/ >> t330/mergeExmple.png> >> >> >> >> >> -- >> Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/ >> > >
