Hi Mark, I'm using Next Available, and the Destination Queues are set with Zero (0) for Back Pressure and Size threshold, so the destinations should not fill up.
I did switch to using RoundRobin and set it to a yield of 0. That got me up to about 300,000 ff's / 5 minutes. I was hoping for something around 1,000,000 ff / 5 minutes. The overall flow looks a bit like this: Large amount of flow files -> Distribute Load -> PutElasticsearcHttp. Ryan On Tue, Sep 15, 2020 at 4:55 PM Mark Payne <[email protected]> wrote: > Ryan, > > I presume you’re using the Round Robin strategy? Looks like that strategy > will yield the processor if any destination is full. And it sounds like > that will be very common in your case. Would recommend configuring the > Processor and in the Settings tab, set the Yield Duration to “0 secs”. I > suspect that will give you dramatically better performance. > > Thanks > -Mark > > > > On Sep 15, 2020, at 4:41 PM, Ryan Hendrickson < > [email protected]> wrote: > > > > Hello, > > I've got 1 million plus FlowFiles (nothing I can do about the count), > that goto a DistributeLoad. The DistributeLoad with 2 threads, a run > duration of 1 sec can only sustain ~200,000 FlowFiles / five minutes. > > > > Is there a better design pattern or a processor that takes a Batch > Size to split a Relationship into two or more? > > > > Thanks, > > Ryan > >
