Thanks for the suggestions Joe. Actually the issue is persistent even after reverting to the 'older-regular-incremental-load' of the data flow* (which used to work fine since months on similarly-configured hardware a few days back by utilising just ~50% of resources)*.
These days, one of the 2-node cluster gets out of NiFi every now and then as the CPU peaks 100% for that particular machine. And subsequently the other node reaches 100% CPU too. When I restart NiFi on a particular node, CPU tanks to 0 and then spikes to 100% within few minutes - the data flowing through the pipeline is *just too less* to throttle my CPU ideally. The machine config and NiFi config remains untouched - this has left me confused where the problem might be. Something which had been running smoothly since months, has become a challenge now. On Fri, Jun 7, 2019 at 8:16 PM Joe Witt <[email protected]> wrote: > Shanker > > It sounds like you've gone through some changes in general and have worked > through those. Now you have a flow running with a high volume of data > (history load) and want to know which parts of the flow are most > expensive/consuming the CPU. > > You should be able to look at the statistics provided on the processors to > see where the majority of CPU time is spent. You can usually very easily > reason over this if it is doing compression/encryption/etc.. and determine > if you want to give it more threads/less threads/batch data together > better, etc.. > > The configuration of the VMs, the NiFi instance itself, the flow, and the > nature of the data are all important to see/understand to be of much help > here. > > THanks > > On Fri, Jun 7, 2019 at 7:07 AM Shanker Sneh <[email protected]> > wrote: > >> Hello all, >> >> I am facing strange issue with NiFi 1.8.0 (2 nodes) >> My flows had been running fine since months. >> >> Yesterday I had to do some history load which filled up my both disks (I >> have FlowFile repository as separate disk). >> >> I increased the size of the root & flowflile disk both. And 'grow' the >> disk partition and 'extended' the file system (it's an EC2 linux). >> But post that my CPU has been spiking to complete 100% - even at regular >> load (earlier it used to be somewhere around 50%) >> Also I did no change to the config values or thread count etc. >> >> I upgraded the 2 nodes to see if that solves the problem - from 16 Gb box >> (4 core) to 64 Gb (16 core). >> But even the larger box is throttling on the CPU at 100%. >> >> I tried clearing all repositories and restarted NiFi application and the >> EC2 - but no improvement. >> >> Kindly point me in the right direction. I am unable to pinpoint anything. >> >> -- >> Best, >> Sneh >> > -- Best, Sneh
