That's probably a lot of the issue, especially if it's an evented service like a Node service running with one thread. I just did a simple test by posting 50k flowfiles with 4k of JSON in them to an Express hello world app and it was able to respond to 50k flowfiles in under 10s using InvokeHttp with only 3 threads. All on my MacBook Pro.
On Tue, Mar 17, 2020 at 4:54 PM Midhun Mohan <[email protected]> wrote: > Endpoint does execute db query in a transaction, yes it is in the same > setup but different cluster > > On Wed, 18 Mar 2020, 2:20 am Mike Thomsen, <[email protected]> wrote: > > > How is the endpoint implemented and what does it do? Also, is it located > in > > the same data center as the EC2 instance running NiFi? > > > > On Tue, Mar 17, 2020 at 3:55 PM Midhun Mohan <[email protected]> > > wrote: > > > > > Yeah the endpoint which am sending right now has plenty of resources > > > available. Only thing is I need to send more records > > > > > > On Wed, 18 Mar 2020, 1:23 am Chad Zobrisky, <[email protected]> > wrote: > > > > > > > I have not test throughput of InvokeHTTP so am not sure what the > > maximum > > > > is, but can give some general guidance. > > > > > > > > 1kb isn't bad. For bottleneck I'd use top, iotop, etc. to figure out > > > system > > > > resources usage while your flow is running. > > > > > > > > You should be able to increase both your nifi count by more and > adjust > > > your > > > > processors until you are limited by your system resources. > > > > > > > > Have you verified the endpoint you are sending to is not the bottle > > neck? > > > > > > > > Chad > > > > > > > > > > > > On Tue, Mar 17, 2020 at 3:47 PM Midhun Mohan <[email protected] > > > > > > wrote: > > > > > > > > > Size of file around kb not more than that, > > > > > How can I find the bottle neck, > > > > > > > > > > Yes I adjust the count to 11 then all other processors stopped but > > > > > processing improved. > > > > > Totally at present 6 is showing not more than that. > > > > > > > > > > I just need a way to post more records to endpoint to make it > > realtime > > > > > > > > > > Hope i was able to give more details > > > > > > > > > > On Wed, 18 Mar 2020, 1:05 am Chad Zobrisky, <[email protected]> > > > wrote: > > > > > > > > > > > Midhun, > > > > > > A little more information would help. > > > > > > > > > > > > What size files are you sending? > > > > > > Have you looked at resource usage to see what the bottleneck is? > > > > > > Did you adjust your nifi system thread count from the hamburger > > menu? > > > > > > How many threads are running total for nifi? It's the number in > the > > > top > > > > > > left of the screen. > > > > > > > > > > > > Thanks, > > > > > > Chad > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 17, 2020 at 3:25 PM Midhun Mohan < > > [email protected] > > > > > > > > > > wrote: > > > > > > > > > > > > > Do anyone have better idea on this > > > > > > > > > > > > > > On Tue, 17 Mar 2020, 6:32 pm Midhun Mohan, < > > [email protected] > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > I have a queue which will be filling up with realtime records > > of > > > > > around > > > > > > > > 20k records. Which is posting to an endpoint using Invokehttp > > > > > > processor. > > > > > > > > > > > > > > > > currently the average throughput is for 10k records it takes > > > around > > > > > 20 > > > > > > > > minutes to complete the invokehttp processor posting. > > > > > > > > I Increased the concurrent thread to larger number which is > > > > > increasing > > > > > > > > the throughput, but it is causing other processor to > > terminate. I > > > > > think > > > > > > > > that can be solved by running on larger instances. > > > > > > > > > > > > > > > > What all things I need to take care to make sure I get a > real > > > time > > > > > > > > posting on endpoint. > > > > > > > > > > > > > > > > I need 20k records to be processed as they receive it > > > > > > > > > > > > > > > > > > > > > > > > thanks in advance > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > Midhun Mohan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
