Using ActiveMQ 5.9.1
Our setup: A Spring/JMS producer places many (sometimes quite large)
messages on one queue (hf.request), which are being picked up by a PHP
Stomp client. PHP processes the data and sends a message to a second
queue (hf.response) to let it know how things went.
Occasionally, we're seeing the PHP producer being blocked.
Starting with both IDs known ...
php (stomp): ID:pan-51744-1418069977147-2:2
tcp://127.0.0.1:34916
java (openwire): ID:pan-36709-1418086070781-1:1
tcp://127.0.0.1:46827
2014-12-08 20:17:26,535 | INFO |
Usage(default:memory:queue://hf.response:memory) percentUsage=0%,
usage=0, limit=720791142,
percentUsageMinDelta=1%;Parent:Usage(default:memory) percentUsage=104%,
usage=752535801, limit=720791142, percentUsageMinDelta=1%: Usage Manager
Memory Limit reached. Producer (ID:pan-51744-1418069977147-2:2:-1:1)
stopped to prevent flooding queue://hf.response. See
http://activemq.apache.org/producer-flow-control.html for more info
(blocking for: 1352s) | org.apache.activemq.broker.region.Queue |
ActiveMQ Transport: tcp:///127.0.0.1:34916@61613
The log message is confusing as it appears to be telling me that the
response queue is being flooded. However, the amount of data sent to
response is trivial:
2014-12-08 19:55:01 Info: Array
(
[synchronizationId] => 6676ab36-f6cf-402d-9045-b58e210fcd3e
[type] => INVENTORY_ITEM
[command] => CREATE
[status] => OK
[statusMessage] =>
)
... compared to request, which can amount to several MB per msg. And it
can't possibly be that PHP is hitting the queue too fast as it has a
fair bit of data to process before it sends to hf.response.
The browser client shows that all messages enqueued to response have
been successfully dequeued, but there are several hundred pending in
hf.request.
Can anyone tell me what might be going on here? If anything, i'd expect
the java producer to be blocked to prevent hf.request from being
flooded. That is, i would have thought that the place where we'd run
into trouble would be constantPendingMessageLimitStrategy. Here's the
config for that:
<pendingMessageLimitStrategy>
<constantPendingMessageLimitStrategy limit="1000"/>
</pendingMessageLimitStrategy>
System usage is:
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage percentOfJvmHeap="70" />
</memoryUsage>
<storeUsage>
<storeUsage limit="100 gb"/>
</storeUsage>
<tempUsage>
<tempUsage limit="50 gb"/>
</tempUsage>
</systemUsage>
</systemUsage>
The only thing i can think of is that the log message, "stopped to
prevent flooding queue://hahnfarms.response" is getting its queue name
from the hf.request messages' reply-to headers, but that the real
problem is with hf.request. But then, i'd presume that it would be the
java client that would be blocked.
