Mike, You can adjust the processor properties for the HandleHTTPRequest processor in the scheduling tab.
“Concurrent tasks” limits the number of threads this processor will use (default is 1) “Run schedule” determines the frequency that this processor will be run (default is ‘0 sec’ which means continuously) If you are only getting requests on a much slower schedule, you could reduce the run schedule to ~10 seconds and see if this is better for you. I have not encountered NiFi running at such high CPU percentage with that little data. As for the high number of tasks, that is definitely an anomaly. Configuration best practices [1] currently recommend increasing the limit to the 10k range, but 2.5M for a single processor is unusual. Can you inspect the logs (located in $NIFI_HOME/logs) to see if there are errors or more insight there? [1] https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#configuration-best-practices Andy LoPresto [email protected] [email protected] PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Jul 14, 2016, at 10:36 AM, Mike Harding <[email protected]> wrote: > > ps - also noticed it seems to generate a lot of tasks, currently 2.5M > compared to other processes in the pipeline which reports 10s of tasks. > > Mike > > On 14 July 2016 at 15:34, Mike Harding <[email protected] > <mailto:[email protected]>> wrote: > > Hi All, > > The node in my cluster running nifi crashed due to a CPU overload event. > After restarting I analysed the CPU consumption and found that nifi was the > issue. As you can see below it was running at 133% CPU: > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > > 2031 nifi 20 0 3392960 990.7m 34124 S 113.4 12.4 1179:47 java] > > I have only one data pipe line setup that is receiving data through a > HandleHTTPRequest processor and after playing around and turning other > processors off in the pipe it was only when I stopped this process that the > CPU dropped significantly to around 10% CPU. > > Its receiving around 67KB of data every 5 minutes from multiple requests from > a up stream web app. > > Has any one else seen this behaviour and or know whether there are ways of > managing the CPU usage of HandleHTTPRequest ? > > Thanks, > > Mike > >
signature.asc
Description: Message signed with OpenPGP using GPGMail
