Sinu Sekhar created AMQ-4208:
--------------------------------
Summary: Stuck Message Issue with ActiveMQ 5.6
Key: AMQ-4208
URL: https://issues.apache.org/jira/browse/AMQ-4208
Project: ActiveMQ
Issue Type: Bug
Components: activemq-pool, Broker
Affects Versions: 5.6.0
Environment: Production
Reporter: Sinu Sekhar
Priority: Critical
We have been facing a stuck message issue with 5.6 for a while now. This week I
got a chance to reproduce this in a non-production environment every time I ran
a high load.
We use Spring DMLC with the following configuration.
<bean id="asyncServiceContainer2"
class="org.springframework.jms.listener.DefaultMessageListenerContainer">
<property name="connectionFactory" ref="asyncServiceConnectionFactory2"
/>
<property name="destination" ref="asyncServiceDestination" />
<property name="messageListener" ref="asyncService" />
<!-- Cache JMS connection -->
<property name="cacheLevelName" value="CACHE_CONNECTION" />
<!-- The maximum number of concurrent consumers to create -->
<property name="maxConcurrentConsumers" value="30" />
<property name="receiveTimeout" value="10000" />
<property name="maxMessagesPerTask" value="20" />
<property name="idleTaskExecutionLimit" value="5" />
</bean>
The broker uses the default settings with producerFlowControl on.
If I send a jmeter load with 100 concurrent users, with 1000 requests from each
user...it generates a throughput of around 60TPS/node. This is for the producer
side.
I see no issues from a producer standpoint.
But everytime I run this test, it reaches a point where the deque count just
freezes up and the enqueue keeps incrementing because the producer is not
affected. This freeze affects all consumers. The only way to unlock it is by
restarting the broker. I have seen it being released by restarting the app as
well.
When this free happens, there is usually a large number of *.log files in the
kahadb folder and the size of the b-tree file db.data also keeps incrementing
indicating the backlog.
I have taken jstack traces for the broker as well as the application, which I
will attach with this request.
Please note that this happens everytime I run with the setup shown.
One way I can prevent the issue from happening so frequently, is by turning off
dynamic scaling by using concurrentConsumers to a high number and setting
maxMessagesPerTask to the default (-1 or infinity) for Spring DMLC. This keeps
the number of consumers constant and increased the consumption rate, preventing
the backlog from happening. In addition to these changes I also had to switch
from CACHE_CONNECTION cache type to CACHE_CONSUMER to prevent this from
happening.
I have seen a similar issue reported for an earlier version at
http://mail-archives.apache.org/mod_mbox/activemq-users/201005.mbox/%[email protected]%3E
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
