On Mar 4, 2008, at 11:23 AM, Dhawan, Vikram (LNG-DAY) wrote:
Hi,
Today I got the following message on my AMQ Server logs, since then
the
server was all stuck and there was no client was able to connect to
the
server. It broke my whole production system. Can some one please help
me understand what happened here? There were not too many connections
open at that time. I have Java openwire and C++ STOMP clients. My C++
clients connect and disconnect every 15 seconds to look for more work;
my Java OpenWire clients stay connected and create consumers every
Minute to poll for the new messages.
Exception in thread "ActiveMQ Transport Server: stomp://localhost:
61619"
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:574)
at
org.apache.activemq.broker.TransportConnector
$1.onAccept(TransportConnec
tor.java:160)
at
org
.apache.activemq.transport.tcp.TcpTransportServer.run(TcpTransportSer
ver.java:213)
at java.lang.Thread.run(Thread.java:595)
What's the maximum heap size for your server process?
Sounds like you have a Thread-leak... Once you encounter this problem,
can you create a thread dump for all of the java threads in your
server process? On a unix system, this would be 'kill -3 <server-
process-id>'
Each java Thread is going to allocate some of your process memory so
that the OS can manage the thread. Your process is unable to allocate
enough memory to manage a new Thread. It's possible that specifying a
smaller max heapsize will give your process more memory for Threads (a
strange fix for an OutOfMemoryError). However, if the number of
Threads is growing in an unbounded manner, then it will simply delay
the inevitable OOME.
I would guess that your STOMP clients are causing the problem. Each
connect/disconnect is probably burning a thread. Recommend you ask on
the ActiveMQ user list to see if this is a known problem. Or, if you
can alter the behavior of your STOMP clients (i.e. long-lived
connections), I expect this problem will go away... However, I suspect
this isn't what you want... ;-)
--kevan
