Hi All, Could someone please give me thoughts on the trailed mail issue, so i can do my further analysis.
Regards, Sanjeet On Wed, 6 Jan 2021, 7:40 pm sanjeet rath, <rath.sanj...@gmail.com> wrote: > Hi All, > > Happy New Year :) > > I have upgraded our cluster from 1.8 to 1.12.1, few days ago and everything > is working fine. I observed that Nifi was like hanged after running for few > days (I have observed its nearly after 15 days of nifi service start) issue > is after login the browser keep on loading , When I saw the bootstrap.log I > saw this message "*Apache nifi is running at PID () but not responding to > ping requests*”. > This happened to only one node from a 3 node cluster. > > This issue happened *3 times on different cluster on different nodes.* > > *Everytime issue got fixed by restarting NiFi service.* > > During the hanged state I tried see the resource utilisation > > -> top -n 1 -H -p 943785 (nifi processid ) > > > top - 08:26:36 up 40 days, 3:48, 2 users, load average: 5.28, 5.38, 5.43 > Threads: 239 total, 4 running, 235 sleeping, 0 stopped, 0 zombie %Cpu(s): > 98.7 us, 1.3 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st MiB Mem : > 15829.5 total, 610.8 free, 10823.7 used, 4395.0 buff/cache MiB Swap: 0.0 > total, 0.0 free, 0.0 used. 4456.1 avail Mem > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > > *943806* root 20 0 12.5g 9.4g 18692 R *88.9* 60.7 12698:50 *GC Thread#1 * > > 943807 root 20 0 12.5g 9.4g 18692 R 88.9 60.7 12698:48 GC Thread#2 > > 943808 root 20 0 12.5g 9.4g 18692 R 88.9 60.7 12698:58 GC Thread#3 > > 943787 root 20 0 12.5g 9.4g 18692 R 83.3 60.7 12698:51 GC Thread#0 > > 943785 root 20 0 12.5g 9.4g 18692 S 0.0 60.7 0:00.00 java > > > We have 4 core cpu, all *4 GC threads* are keep on this state and > consuming more CPU.*cluster is hung state for 2 days,* Then after 2 days > I saw these threads are moved and nifi comes out of the hung state for this > node , but saw another node from the same cluster moved to the hung state > with similar fashion means , 4 threads busy in GC and consuming more CPU. > > > Could you please help me to identify what could be the possible reason. > > Details: > > Nifi 1.12.1 > > Jdk 11 > > Zookeeper 3.5.8 > > 16g memory > > > > Thanks, > -- > Sanjeet Kumar Rath, > mob- +91 8777577470 > > >
