Did you change the size of the overall thread pool at NiFi level? If yes, to which number? How many cores do you have on NiFi?
Thanks Le mar. 17 mars 2020 à 20:55, Midhun Mohan <[email protected]> a écrit : > Yeah the endpoint which am sending right now has plenty of resources > available. Only thing is I need to send more records > > On Wed, 18 Mar 2020, 1:23 am Chad Zobrisky, <[email protected]> wrote: > > > I have not test throughput of InvokeHTTP so am not sure what the maximum > > is, but can give some general guidance. > > > > 1kb isn't bad. For bottleneck I'd use top, iotop, etc. to figure out > system > > resources usage while your flow is running. > > > > You should be able to increase both your nifi count by more and adjust > your > > processors until you are limited by your system resources. > > > > Have you verified the endpoint you are sending to is not the bottle neck? > > > > Chad > > > > > > On Tue, Mar 17, 2020 at 3:47 PM Midhun Mohan <[email protected]> > > wrote: > > > > > Size of file around kb not more than that, > > > How can I find the bottle neck, > > > > > > Yes I adjust the count to 11 then all other processors stopped but > > > processing improved. > > > Totally at present 6 is showing not more than that. > > > > > > I just need a way to post more records to endpoint to make it realtime > > > > > > Hope i was able to give more details > > > > > > On Wed, 18 Mar 2020, 1:05 am Chad Zobrisky, <[email protected]> > wrote: > > > > > > > Midhun, > > > > A little more information would help. > > > > > > > > What size files are you sending? > > > > Have you looked at resource usage to see what the bottleneck is? > > > > Did you adjust your nifi system thread count from the hamburger menu? > > > > How many threads are running total for nifi? It's the number in the > top > > > > left of the screen. > > > > > > > > Thanks, > > > > Chad > > > > > > > > > > > > > > > > On Tue, Mar 17, 2020 at 3:25 PM Midhun Mohan <[email protected] > > > > > > wrote: > > > > > > > > > Do anyone have better idea on this > > > > > > > > > > On Tue, 17 Mar 2020, 6:32 pm Midhun Mohan, <[email protected] > > > > > > wrote: > > > > > > > > > > > Hi all, > > > > > > > > > > > > I have a queue which will be filling up with realtime records of > > > around > > > > > > 20k records. Which is posting to an endpoint using Invokehttp > > > > processor. > > > > > > > > > > > > currently the average throughput is for 10k records it takes > around > > > 20 > > > > > > minutes to complete the invokehttp processor posting. > > > > > > I Increased the concurrent thread to larger number which is > > > increasing > > > > > > the throughput, but it is causing other processor to terminate. I > > > think > > > > > > that can be solved by running on larger instances. > > > > > > > > > > > > What all things I need to take care to make sure I get a real > time > > > > > > posting on endpoint. > > > > > > > > > > > > I need 20k records to be processed as they receive it > > > > > > > > > > > > > > > > > > thanks in advance > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > Regards, > > > > > > Midhun Mohan > > > > > > > > > > > > > > > > > > > > >
