I've been getting the error:
2017-11-15 09:22:24,959 ERROR [NiFi Web Server-566674] org.apache.nifi.NiFi
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:714)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.startThreads(QueuedThreadPool.java:476)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$200(QueuedThreadPool.java:49)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:657)
at java.lang.Thread.run(Thread.java:745)
My research online says this isn't an out of memory error, but an out of
resources error. The system can't support making new threads.
This really sucks, because I can't su to my service account, because that
requires a new thread... and if I have a bash session still open I can't kill
NiFi because that requires creating a new thread... last time this happened my
Unix admin had to restart my server...
The first time this happened we changed the limits for our service account,
here is my ulimit statement.
# ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 96297
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 50000
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 50000
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
It's been about two weeks, and now the issue has come up again. Is this an
actual hardware limitation?
If I run "ps huH p <PID>| wc -l" I can get a thread count, right now it's about
9900 threads.
Thanks,
Peter
