[
https://issues.apache.org/jira/browse/AMQ-3460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13815049#comment-13815049
]
Sree Panchajanyam D edited comment on AMQ-3460 at 11/6/13 5:29 PM:
-------------------------------------------------------------------
************************************************************
Root Cause Analysis (Saving the devil for the very last)
************************************************************
This issue falls in no mans land between spring code and activemq code .
In my example above the main thread of spring tries to connect for first 60
seconds (first timeout), when it fails to connect to ActiveMQ as it is not up,
leaves it to listener threads ( if there are 20 concurrentConsumers we would
have 20 DMLC threads) to recover. The main thread of spring does not clear the
connection object of spring and hands over a stale connection reference for the
listeners to work with.
If ActiveMQ starts after the first time out and before the second timeout one
of the 20 listener threads creates sessions on an unstarted connection. When
ActiveMQ delivers a message to the consumer, the message flow into the consumer
JVM/process but DMLC threads cannot receive them and hence the issue.When this
issue occurs you can notice the following warning in the logs :
LOG EXTRACT:
"trying to receive messages on an unstarted connection" and subsequent messages
are not delivered".
This warning is logged for the first message alone and not for the subsequent
messages.
After this messages are pushed into the dispatch queue but not acked by the
consumer as he never received them. The messages are pushed into the dispatch
queue till the cursor memory is full. Once it is full no more messages are
pushed into dispatch queue and every consumer of the queue is affected.
We found ActiveMQConnection.java easiest and logical of all places to fix the
issue.
was (Author: sreepanchajanyam):
************************************************************
Root Cause Analysis (Saving the devil for the very last)
************************************************************
This issue falls in no mans land between spring code and activemq code .
In my example above the main thread of spring tries to connect for first 60
seconds (first timeout), when it fails to connect to ActiveMQ as it is not up,
leaves it to listener threads ( if there are 20 concurrentConsumers we would
have 20 DMLC threads) to recover. The main thread of spring does not clear the
connection object of spring and hands over a stale connection reference for the
listeners to work with.
If ActiveMQ starts after the first time out and before the second timeout one
of the 20 listener threads creates sessions on an unstarted connection. When
ActiveMQ delivers a message to the consumer, the message flow into the consumer
JVM/process but DMLC threads cannot receive them and hence the issue.When this
issue occurs you can notice the following warning in the logs :
LOG EXTRACT:
"trying to receive messages on an unstarted connection" and subsequent messages
are not delivered".
This warning is logged for the first message alone and not for the subsequent
messages.
We found ActiveMQConnection.java easiest and logical of all places to fix the
issue.
> Messages get stuck in the queue for some client sessions, "Dispatched Queue"
> grows but "Dequeues" doesn't move
> --------------------------------------------------------------------------------------------------------------
>
> Key: AMQ-3460
> URL: https://issues.apache.org/jira/browse/AMQ-3460
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.3.1
> Environment: java version "1.5.0_15"
> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_15-b04)
> Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_15-b04, mixed mode)
> Reporter: Timur Evdokimov
> Priority: Critical
> Fix For: AGING_TO_DIE
>
> Attachments: acmq01.png, acmq02.png
>
>
> We experience the following problem.
> There's 1 sender thread and 20 listener threads operating on the same queue.
> No other queues are involved.
> Message traffic goes generally OK via the queue, however Number of pending
> messages goes up one by one.
> Looking at the active customers list, the following appears (see attached
> screenshot).
> For all sessions but two, the messages are dispatched properly.
> For these two affected sessions, the "Dispatched Queue" value gots stuck at
> certain moment and doesn't deliver anymore.
> Eventually, more sessions got affected, and maybe after some hours, the stuck
> messages got delivered.
> What we tried - adding useCache="false" into <policyEntry queue=">" ... >
> tag. Didn't have any effect.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
