Hi Ryan I would merge the files into larger files before distribute load and use PutElasticsearchHttpRecord
On Tue, Sep 15, 2020, 5:43 PM Ryan Hendrickson < [email protected]> wrote: > Hi Mark, > I'm using Next Available, and the Destination Queues are set with Zero > (0) for Back Pressure and Size threshold, so the destinations should not > fill up. > > I did switch to using RoundRobin and set it to a yield of 0. That got > me up to about 300,000 ff's / 5 minutes. I was hoping for something around > 1,000,000 ff / 5 minutes. > > The overall flow looks a bit like this: Large amount of flow files -> > Distribute Load -> PutElasticsearcHttp. > > Ryan > > On Tue, Sep 15, 2020 at 4:55 PM Mark Payne <[email protected]> wrote: > >> Ryan, >> >> I presume you’re using the Round Robin strategy? Looks like that strategy >> will yield the processor if any destination is full. And it sounds like >> that will be very common in your case. Would recommend configuring the >> Processor and in the Settings tab, set the Yield Duration to “0 secs”. I >> suspect that will give you dramatically better performance. >> >> Thanks >> -Mark >> >> >> > On Sep 15, 2020, at 4:41 PM, Ryan Hendrickson < >> [email protected]> wrote: >> > >> > Hello, >> > I've got 1 million plus FlowFiles (nothing I can do about the >> count), that goto a DistributeLoad. The DistributeLoad with 2 threads, a >> run duration of 1 sec can only sustain ~200,000 FlowFiles / five minutes. >> > >> > Is there a better design pattern or a processor that takes a Batch >> Size to split a Relationship into two or more? >> > >> > Thanks, >> > Ryan >> >>
