ActiveMQ 5.1.0 runs out of file descriptors with lots of 'CLOSE_WAIT' sockets
-----------------------------------------------------------------------------
Key: AMQ-1739
URL: https://issues.apache.org/activemq/browse/AMQ-1739
Project: ActiveMQ
Issue Type: Bug
Components: Broker
Affects Versions: 5.1.0
Environment: We have a single broker with no special network-stuff.
Our broker-system has two single core Opterons, 8GB of memory, plenty of I/O
and runs a recent 64bit debian with 2.6.21 kernel.
Java(TM) SE Runtime Environment (build 1.6.0_06-b02)
Java HotSpot(TM) 64-Bit Server VM (build 10.0-b22, mixed mode)
We left most of the activemq.xml-configuration as-is and adjusted the start-up
script to run with 2GB heap size and parallel garbage collector, which was more
or less needed for 5.0 and left for 5.1 in the start-up script.
Reporter: Arjen van der Meijden
Priority: Blocker
We have no idea why or when, but within a few days after start-up, ActiveMQ
suddenly runs out of file descriptors (we've raised the limit to 10240).
According to lsof it has lots of sockets which are in CLOSE_WAIT when that
happens. As soon as that happened once, it would re-occur within a few hours.
This behavior did not happen with ActiveMQ 5.0.
We have five queues, all with only one consumer. All consumption and production
is via the Stomp-interface using PHP-clients. Three of those queues get up to
50-100 messages/second in peak moments, while the consumers adjust their own
consumption rate to the systems load (normally its maxed to about 50-150/sec).
So in high-load moments on the consumers, the queues can grow to a few thousand
messages, normally the queues are emptied as soon as a message occurs. Those
five consumers stay connected indefinitely.
The messages are all quite small (at most 1 KB or so) and come from 5 web
servers. For each web page-request (about 2-3M/day) a connection is made to
ActiveMQ via Stomp and at least one message is sent to ActiveMQ, for most
requests two are sent to the two most active queues. For all request a new
connection is made and at most 4 stomp-messages are sent to ActiveMQ (connect,
two messages, disconnect), since apache+php does not allow useful reuse of
sockets and similar resources.So
So the connection-rate is about the same as the highest message rate on any of
the queues (so 50-100connects/second).
When the high amount of sockets in CLOSE_WAIT occurs, we manually disable the
producers and the sockets disappear gradually. After that the amount of sockets
stays around 180-190 (mostly opened jars), but seams to re-increase more easily
than when ActiveMQ is restarted. I have not checked if anything special happens
on the web servers or databases, since their producer and consumer
implementation has not changed since 5.0.
What I did notice is that the memory-consumption increases heavily prior to
running out of descriptors, and the consumption re-increases way to fast
compared to before 11:45u:
http://achelois.tweakers.net/~acm/tnet/activemq-5.1-memory-consumption.png
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
