Hi,
We have activemq running behind an LB. we have close to 55K
connections to activemq. during cases of more than 1000 connections gets
terminated concurrently, we observe a huge spike in the number of
threads created.following which the jvm crashes. on further
investigation, these threads are pertained to executors
SelectorManager - defaultExecutor - labelled with "ActiveMQ NIO Worker"
- the executor has maximum thread pool size of Integer.MAX_VALUE
TaskRunnerFactory - defaultExecutor - labelled as "ActiveMQ Task" - this
executor as well has Integer.MAX_VALUE as the max pool size.
BrokerService.getTaskRunnerFactory - labelled as "ActiveMQ
BrokerService[%s] Task-" - this executor is also created with
Integer.MAX_VALUE
the configured persistence store is leveldb and dedicatedtaskrunner is
disabled. The JVM is running with 8GB RAM with the max heap at 5G.
following snippet from the hs_pid crash file
Java Threads: ( => current thread )
0x00007f1e42863800 JavaThread "ActiveMQ NIO Worker 9053" daemon
[_thread_blocked, id=10690, stack(0x00007f1d6fa53000,0x00007f1d6fa8c000)]
0x00007f1ea40dc000 JavaThread "ActiveMQ NIO Worker 9047" daemon
[_thread_blocked, id=10689, stack(0x00007f1d6fa8c000,0x00007f1d6fac5000)]
0x00007f1eac116800 JavaThread "ActiveMQ NIO Worker 9054" daemon
[_thread_blocked, id=10688, stack(0x00007f1d6fac5000,0x00007f1d6fafe000)]
0x00007f1df43f5800 JavaThread "ActiveMQ NIO Worker 9046" daemon
[_thread_blocked, id=10687, stack(0x00007f1d6fafe000,0x00007f1d6fb37000)]
0x00007f1e42870000 JavaThread "ActiveMQ NIO Worker 9056" daemon
[_thread_blocked, id=10686, stack(0x00007f1d6fb37000,0x00007f1d6fb70000)]
0x00007f1ef6054800 JavaThread "ActiveMQ NIO Worker 9107" daemon
[_thread_blocked, id=10685, stack(0x00007f1d6fb70000,0x00007f1d6fba9000)]
0x00007f1ef46af800 JavaThread "ActiveMQ NIO Worker 9059" daemon
[_thread_blocked, id=10684, stack(0x00007f1d6fba9000,0x00007f1d6fbe2000)]
0x00007f1e240cd000 JavaThread "ActiveMQ NIO Worker 9041" daemon
[_thread_blocked, id=10683, stack(0x00007f1d6fbe2000,0x00007f1d6fc1b000)]
0x00007f1e0016a800 JavaThread "ActiveMQ NIO Worker 9060" daemon
[_thread_blocked, id=10682, stack(0x00007f1d6fc1b000,0x00007f1d6fc54000)]
0x00007f1e5019d800 JavaThread "ActiveMQ NIO Worker 9061" daemon
[_thread_blocked, id=10681, stack(0x00007f1d6fc54000,0x00007f1d6fc8d000)]
0x00007f1e8006f000 JavaThread "ActiveMQ NIO Worker 9039" daemon
[_thread_blocked, id=10680, stack(0x00007f1d6fc8d000,0x00007f1d6fcc6000)]
0x00007f1e4c7df800 JavaThread "ActiveMQ NIO Worker 9101" daemon
[_thread_blocked, id=10679, stack(0x00007f1d6fcc6000,0x00007f1d6fcff000)]
0x00007f1da6794800 JavaThread "ActiveMQ BrokerService[MQMaster]
Task-1781" daemon [_thread_blocked, id=8302,
stack(0x00007f1db5a51000,0x00007f1db5a8a000)]
0x00007f1da6791000 JavaThread "ActiveMQ BrokerService[MQMaster]
Task-1780" daemon [_thread_blocked, id=8298,
stack(0x00007f1dc6d4c000,0x00007f1dc6d85000)]
0x00007f1da6793000 JavaThread "ActiveMQ BrokerService[MQMaster]
Task-1779" daemon [_thread_blocked, id=8296,
stack(0x00007f1dc6e30000,0x00007f1dc6e69000)]
0x00007f1da6790000 JavaThread "ActiveMQ BrokerService[MQMaster]
Task-1778" daemon [_thread_blocked, id=8293,
stack(0x00007f1db6903000,0x00007f1db693c000)]
0x00007f1da678f000 JavaThread "ActiveMQ BrokerService[MQMaster]
Task-1777" daemon [_thread_blocked, id=8292,
stack(0x00007f1db9dc8000,0x00007f1db9e01000)]
Please, can someone provide inputs on why this is happening.
Thanks
nl
