ActiveMQ 5.3.2
java version "1.6.0_23"
Java(TM) SE Runtime Environment (build 1.6.0_23-b05)
Java HotSpot(TM) Server VM (build 19.0-b09, mixed mode)
Description: CentOS release 5.5 (Final)
The problem is this
We see FDs being consumed by the activemq java process:
/usr/sbin/lsof -p 6700 | grep " identify" | wc -l
798
/usr/sbin/lsof -p 6700 | wc -l
1025
Most "leaked" FDs look like this (they don't show up with netstat -a):
...
java 6700 lexo-ext 904u sock 0,5 1741176200
can't identify protocol
java 6700 lexo-ext 905u sock 0,5 1741241481
can't identify protocol
...
Now, I also noted the following errors in activemq logs:
WARN | Failed to register MBean:
org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=openwire,ViewType=address,Name=/192.168.235.8_62568
WARN | Failed to register MBean:
org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=openwire,ViewType=address,Name=/192.168.235.7_58352
WARN | Failed to register MBean:
org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=openwire,ViewType=address,Name=/192.168.235.8_9091
and finally, I now have hundreds of threads waiting on something:
~/activemq/apache-activemq-5.3.2$ jstack 6700 | grep "ActiveMQ Connection
Dispatcher" | wc -l
819
which seem to be waiting:
"ActiveMQ Connection Dispatcher: /192.168.235.8:22405" daemon prio=10
tid=0x0ab16800 nid=0x7196 in Object.wait() [0xbcf360
00]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xdd5a0cf8> (a java.lang.Object)
at java.lang.Object.wait(Object.java:485)
at
org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
- locked <0xdd5a0cf8> (a java.lang.Object)
at
org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
and only 15 "transport" threads:
~/activemq/apache-activemq-5.3.2$ jstack 6700 | grep "ActiveMQ Transport" |
wc -l
15
Now, is there any relationship between the "Failed to register MBean"
message, the number of FDs increasing (up to the per-process limit after a
few days) and the number of ActiveMQ Connection Dispatcher threads that seem
to remain in the system for nothing?
I saw https://issues.apache.org/jira/browse/AMQ-3286 that was similar to
what I can see and there are also a few bugs related to leaking FPs/Threads,
but could not see something that would match all these conditions (ex:
https://issues.apache.org/jira/browse/AMQ-1791).
Thanks for any help.
--
View this message in context:
http://activemq.2283324.n4.nabble.com/Thread-FD-Leak-in-Active-MQ-tp3734161p3734161.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
