Hi all,
First thing first... sorry for the long post but my Nifi instance (1.9.1)
started to behave weird (according to my experience at least) this afternoon
and I'm struggling to determine the cause of the problem (and fix it)
All processors are stopped and I have about 205k flows queued in a single queue
as illustrated below.
[cid:1c8868f8-4527-4d5c-86f7-c35a16116385]
The process "SaveToDB" seems to be causing some issues. It actually does a few
more things than saving to the DB so this module was implemented as an
ExecuteStreamCommand processor using Python3.
This module hasn't changed in quite some time and this is the first time I'm
having issues around it. Whenever I start it (even with only 1 thread), Nifi
start to show the following message...
[cid:e107a064-c536-42c7-9ae9-4b88fbce3e93]
Another weird issue is that if I try to open the "View Status History" on this
processor, Nifi says: "Insufficient history, please try again later".
I tried many things, like restarting Nifi, and noticed that the CPU runs at
150% on startup with no indication that it will ever stop. I read about these
warning/error messages and tuned a few settings to increase the number of
threads to index and restarted Nifi but
the CPU is still over 100% when no actual process is running in Nifi. Looking
at /nifi-api/system-diagnostics shows the following:
{"systemDiagnostics": {"aggregateSnapshot": {"totalNonHeap": "235.4
MB","totalNonHeapBytes": 246833152,"usedNonHeap": "222.25
MB","usedNonHeapBytes": 233046120,"freeNonHeap": "13.15 MB","freeNonHeapBytes":
13787032,"maxNonHeap": "-1 bytes","maxNonHeapBytes": -1,"totalHeap": "12
GB","totalHeapBytes": 12884901888,"usedHeap": "7.14 GB","usedHeapBytes":
7670910608,"freeHeap": "4.86 GB","freeHeapBytes": 5213991280,"maxHeap": "12
GB","maxHeapBytes": 12884901888,"heapUtilization":
"60.0%","availableProcessors": 72,"processorLoadAverage": 1.06,"totalThreads":
916,"daemonThreads": 53,"uptime": "00:58:48.996",
I was wondering if the high number of threads is expected or not... 916 threads
seems a lot to me. Especially since my max threads count is set to 400 right
now.
The machine has 72 CPUs and is quite capable but the CPU has been running at
over 100% for hours now.
Do you have any recommendation?
thanks
