Mike, To add some context, while NiFi will intelligently schedule processors to execute, given HandleHTTPRequest's function as a listener, it is constantly scheduled to run, checking for a request to handle. I assume by number of tasks, you mean the rolling count over the last 5 minutes. As mentioned by Andy, you can tamper this rate by increasing the run scheduld if the handling of the HTTP requests with a slight latency is acceptable to you and your needs.
On Thu, Jul 14, 2016 at 11:05 AM Andy LoPresto <[email protected]> wrote: > Mike, > > You can adjust the processor properties for the HandleHTTPRequest > processor in the scheduling tab. > > “Concurrent tasks” limits the number of threads this processor will use > (default is 1) > “Run schedule” determines the frequency that this processor will be run > (default is ‘0 sec’ which means continuously) > > If you are only getting requests on a much slower schedule, you could > reduce the run schedule to ~10 seconds and see if this is better for you. I > have not encountered NiFi running at such high CPU percentage with that > little data. > > As for the high number of tasks, that is definitely an anomaly. > Configuration best practices [1] currently recommend increasing the limit > to the 10k range, but 2.5M for a single processor is unusual. Can you > inspect the logs (located in $NIFI_HOME/logs) to see if there are errors or > more insight there? > > [1] > https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#configuration-best-practices > > > > Andy LoPresto > [email protected] > *[email protected] <[email protected]>* > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > On Jul 14, 2016, at 10:36 AM, Mike Harding <[email protected]> wrote: > > ps - also noticed it seems to generate a lot of tasks, currently 2.5M > compared to other processes in the pipeline which reports 10s of tasks. > > Mike > > On 14 July 2016 at 15:34, Mike Harding <[email protected]> wrote: > >> >> Hi All, >> >> The node in my cluster running nifi crashed due to a CPU overload event. >> After restarting I analysed the CPU consumption and found that nifi was the >> issue. As you can see below it was running at 133% CPU: >> >> PID USER PR NI VIRT RES SHR S *%CPU* %MEM TIME+ >> COMMAND >> >> 2031 nifi 20 0 3392960 990.7m 34124 S *113.4* 12.4 1179:47 >> java] >> >> I have only one data pipe line setup that is receiving data through a >> HandleHTTPRequest processor and after playing around and turning other >> processors off in the pipe it was only when I stopped this process that the >> CPU dropped significantly to around 10% CPU. >> >> Its receiving around 67KB of data every 5 minutes from multiple requests >> from a up stream web app. >> >> Has any one else seen this behaviour and or know whether there are ways >> of managing the CPU usage of HandleHTTPRequest ? >> >> Thanks, >> >> Mike >> > > >
