I new question then ideally nifi should run on which instance, based on aws
On Wed, 18 Mar 2020, 1:58 am Chad Zobrisky, <[email protected]> wrote: > Probably can't answer how much it would increase processing speed, but > processor thread count is limited by the nifi system thread count. > > Normally you can increase the nifi system thread count a decent amount, > 100+, depending on the system. > > On Tue, Mar 17, 2020 at 4:19 PM Midhun Mohan <[email protected]> > wrote: > > > Yeah got it thank you. > > How will Changing that values affect the processing speed > > > > On Wed, 18 Mar 2020, 1:45 am Chad Zobrisky, <[email protected]> wrote: > > > > > Hamburger menu in the top right -> Controller Settings -> under the > > general > > > tab -> Maximum Timer Driven Thread Count. > > > > > > On Tue, Mar 17, 2020 at 4:11 PM Midhun Mohan <[email protected]> > > > wrote: > > > > > > > No overall thread pool I haven't changed. How can I change that? > > > > > > > > Nifi is running on a standalone aws instance. T2.medium I remember > so > > > that > > > > leaves us with 2 cores I guess > > > > > > > > On Wed, 18 Mar 2020, 1:29 am Pierre Villard, < > > > [email protected]> > > > > wrote: > > > > > > > > > Did you change the size of the overall thread pool at NiFi level? > If > > > yes, > > > > > to which number? How many cores do you have on NiFi? > > > > > > > > > > Thanks > > > > > > > > > > Le mar. 17 mars 2020 à 20:55, Midhun Mohan <[email protected] > > > > a > > > > > écrit : > > > > > > > > > > > Yeah the endpoint which am sending right now has plenty of > > resources > > > > > > available. Only thing is I need to send more records > > > > > > > > > > > > On Wed, 18 Mar 2020, 1:23 am Chad Zobrisky, <[email protected] > > > > > > wrote: > > > > > > > > > > > > > I have not test throughput of InvokeHTTP so am not sure what > the > > > > > maximum > > > > > > > is, but can give some general guidance. > > > > > > > > > > > > > > 1kb isn't bad. For bottleneck I'd use top, iotop, etc. to > figure > > > out > > > > > > system > > > > > > > resources usage while your flow is running. > > > > > > > > > > > > > > You should be able to increase both your nifi count by more and > > > > adjust > > > > > > your > > > > > > > processors until you are limited by your system resources. > > > > > > > > > > > > > > Have you verified the endpoint you are sending to is not the > > bottle > > > > > neck? > > > > > > > > > > > > > > Chad > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 17, 2020 at 3:47 PM Midhun Mohan < > > > [email protected] > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Size of file around kb not more than that, > > > > > > > > How can I find the bottle neck, > > > > > > > > > > > > > > > > Yes I adjust the count to 11 then all other processors > stopped > > > but > > > > > > > > processing improved. > > > > > > > > Totally at present 6 is showing not more than that. > > > > > > > > > > > > > > > > I just need a way to post more records to endpoint to make it > > > > > realtime > > > > > > > > > > > > > > > > Hope i was able to give more details > > > > > > > > > > > > > > > > On Wed, 18 Mar 2020, 1:05 am Chad Zobrisky, < > > [email protected] > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Midhun, > > > > > > > > > A little more information would help. > > > > > > > > > > > > > > > > > > What size files are you sending? > > > > > > > > > Have you looked at resource usage to see what the > bottleneck > > > is? > > > > > > > > > Did you adjust your nifi system thread count from the > > hamburger > > > > > menu? > > > > > > > > > How many threads are running total for nifi? It's the > number > > in > > > > the > > > > > > top > > > > > > > > > left of the screen. > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > Chad > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 17, 2020 at 3:25 PM Midhun Mohan < > > > > > [email protected] > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Do anyone have better idea on this > > > > > > > > > > > > > > > > > > > > On Tue, 17 Mar 2020, 6:32 pm Midhun Mohan, < > > > > > [email protected] > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > > > > > I have a queue which will be filling up with realtime > > > records > > > > > of > > > > > > > > around > > > > > > > > > > > 20k records. Which is posting to an endpoint using > > > Invokehttp > > > > > > > > > processor. > > > > > > > > > > > > > > > > > > > > > > currently the average throughput is for 10k records it > > > takes > > > > > > around > > > > > > > > 20 > > > > > > > > > > > minutes to complete the invokehttp processor posting. > > > > > > > > > > > I Increased the concurrent thread to larger number > which > > > is > > > > > > > > increasing > > > > > > > > > > > the throughput, but it is causing other processor to > > > > > terminate. I > > > > > > > > think > > > > > > > > > > > that can be solved by running on larger instances. > > > > > > > > > > > > > > > > > > > > > > What all things I need to take care to make sure I get > a > > > > real > > > > > > time > > > > > > > > > > > posting on endpoint. > > > > > > > > > > > > > > > > > > > > > > I need 20k records to be processed as they receive it > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > thanks in advance > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > > Midhun Mohan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
