ActiveMQ broker stops functioning
---------------------------------
Key: AMQ-3739
URL: https://issues.apache.org/jira/browse/AMQ-3739
Project: ActiveMQ
Issue Type: Bug
Components: Broker, Performance Test
Affects Versions: 5.5.0, 5.4.2, 5.6.0
Environment: Windows server 2008
12GB Ram
12 cores
spring integration
embedded activemq network of brokers
Reporter: Matan Zruya
Priority: Critical
i'm encountering critical problems with activemq in our dev enviroment,
activemq brokers seem to hang and not respond to other brokers or jms
consumers/producers.
my example setup is an activemq network of embedded brokers (2 to be precise),
one process with an embedded broker produces work,
a worker process with an embedded broker as well that does the work.
2 queues, a running-queue and a stopped-queue, each msg has a counter which is
decremented every time work is done on it until 0 and then the msg is passed to
the stopped-queue, where it is consumed by the work producer.
the logic is implemented with basic spring integration.
activemq is configured with plenty of memory to spare, and no persistency, jmx
is exposed to allow monitoring of the queues,
what happens is that occasionally msg's get stuck inflight, meaning they do not
get an ack, sometimes the inflight is between brokers, and sometimes between an
embedded broker and it's internal jvm consumer.
the worker broker seems to hang, the jmx statistics seem to be frozen, although
they do refresh, the broker responds to force restarts of the network
connector, and sometime the restart frees msg's until it gets stuck again.
if i leave the worker on after it's stuck and relaunch the producer, both
brokers don't seem to communicate, the produced msg's on one side, don't reach
the other, same goes for all advisory msg's related stats (such as
producer/consumer counts)
i tried changing every bit of configuration, with the broker, and with spring
itself, but i have a feeling the problem doesn't have anything to do with
spring (because of inner broker communication issues), the problem is easily
reproducible on a powerful machine where the msg flow is fast, adding
persistency or doing it on a normal desktop slows everything down and makes it
more difficult to reproduce,
i tried all recent activemq versions up to the latest 5.6 snapshot.
i attached the relevent junit tests, one needs to launch the agentTest, and
then the producerTest,
my recommendation is leaving agentTest on and relaunching producerTest over and
over again, producerTest should be finished when all it's messages are done (do
not launch 2 producer processes at the same time otherwise they will never be
finished)
relevant maven test cases :
http://db.tt/diVWPNnJ
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
