Hi there, The issue I have isn’t my processor, it’s the fact I can’t get Nifi to run more threads. I’ve noticed that if I bump the max thread counts in the controller settings to total over 256, then nifi as a whole locks up and I have to clear the state and restart it.
Thanks, Phil On Mon, 21 Sep 2020 at 11:01, Mark Payne <[email protected]> wrote: > Phil, > > > > It sounds like you’re dealing with a large amount of lock contention. > Either using explicit java.util.concurrency.Lock or synchronized blocks. > Would review the code for lock contention or run a profiler against nifi > and see what that shows. > > > > Sent from my iPhone > > > > > On Sep 20, 2020, at 8:30 PM, Phil H <[email protected]> wrote: > > > > > > Various numbers - max of 96 so far. > > > > > >> On Sat, 19 Sep 2020 at 21:44, Pierre Villard < > [email protected]> > > >> wrote: > > >> > > >> How many concurrent tasks did you set on the two custom processors? > > >> > > >> > > >> > > >> Le sam. 19 sept. 2020 à 02:16, Phil H <[email protected]> a écrit : > > >> > > >> > > >> > > >>> Basically I have two custom processors that are slow (data intensive) > > >> that > > >> > > >>> create backlogs. All other aspects of the flow work well - I am running > > >> SSD > > >> > > >>> disks so that sort of I/O is not biting me. The process itself is > highly > > >> > > >>> ‘parallelisable’ but I can’t get NiFi to use more than about 10% of the > > >> CPU > > >> > > >>> load. This is the only purpose this machine has - I want it to work > much > > >> > > >>> harder! > > >> > > >>> > > >> > > >>> On Fri, 18 Sep 2020 at 20:56, Pierre Villard < > > >> [email protected]> > > >> > > >>> wrote: > > >> > > >>> > > >> > > >>>> Hi, > > >> > > >>>> > > >> > > >>>> > > >> > > >>>> > > >> > > >>>> Do you have backpressure with flow files accumulating somewhere in > your > > >> > > >>>> > > >> > > >>>> flow? If not, then you probably don't need more threads. > > >> > > >>>> > > >> > > >>>> If you do find bottlenecks in your flow design, then you can look at > > >> > > >>>> > > >> > > >>>> increasing the concurrent tasks on specific processors to have > > >> > > >>>> > > >> > > >>>> multi-threading. Note that it's not always the solution. > > >> > > >>>> > > >> > > >>>> Changing the pool size does not mean NiFi will use more resources. > > >> > > >>>> > > >> > > >>>> Actually, I usually recommend changing the pool size only when you see > > >> > > >>> that > > >> > > >>>> > > >> > > >>>> all the threads available are used. > > >> > > >>>> > > >> > > >>>> Overall we'd need more details about your flow and observations. > > >> > > >>>> > > >> > > >>>> > > >> > > >>>> > > >> > > >>>> Thanks, > > >> > > >>>> > > >> > > >>>> Pierre > > >> > > >>>> > > >> > > >>>> > > >> > > >>>> > > >> > > >>>> Le ven. 18 sept. 2020 à 03:49, Phil H <[email protected]> a écrit : > > >> > > >>>> > > >> > > >>>> > > >> > > >>>> > > >> > > >>>>> Hi there, > > >> > > >>>> > > >> > > >>>>> > > >> > > >>>> > > >> > > >>>>> Regardless of what I set in the maximum thread count settings, I > > >> never > > >> > > >>>> see > > >> > > >>>> > > >> > > >>>>> more than 128 active threads. The CPU load of the system is only 5-8% > > >> > > >>>> when > > >> > > >>>> > > >> > > >>>>> NiFi is running at this rate. It’s a Solaris system with 64 cores. > > >> > > >>>> > > >> > > >>>>> > > >> > > >>>> > > >> > > >>>>> Any help getting more performance would be appreciated! > > >> > > >>>> > > >> > > >>>>> > > >> > > >>>> > > >> > > >>>>> Regards, > > >> > > >>>> > > >> > > >>>>> Phil > > >> > > >>>> > > >> > > >>>>> > > >> > > >>>> > > >> > > >>>> > > >> > > >>> > > >> > > >> > >
