subhajit maiti created AMQ-7066:
-----------------------------------
Summary: Using dynamic scaling and message grouping feature
together makes messages stuck
Key: AMQ-7066
URL: https://issues.apache.org/jira/browse/AMQ-7066
Project: ActiveMQ
Issue Type: Bug
Components: activemq-camel
Affects Versions: 5.15.5
Reporter: subhajit maiti
Recently we observed a strange problem in one of our flows. We could see some
messages being stuck in the queue when we are using the message grouping and
dynamic scaling feature together.
*Usecase:*
We want to process the messages belonging to a user (application user) in a
sequence. Multiple users can send messages to the same queue, but messages
originating from transactions made by a user should be processed sequentially.
In order to do so we are using the "JMSXGroupID" header to group the messages
and the "username" serves as the group key.
*Our configuration and versions:*
# We are using ActiveMQ 5.15.5
# We are using camel routing to connect to ActiveMQ, both on the producer and
consumer side. Camel version : 2.20.1
# We are using the following properties in the consumer URI to enable dynamic
scaling : maxConcurrentConsumers,
idleTaskExecutionLimit (defaulted to 10)
maxMessagesPerTask (defaulted to 250)
# JDK version : 1.8.0_91
*Observations:*
# The messages are stuck for some of the groups. Messages belonging to other
groups are processing fine. There is no identifiable pattern for the groups
that are stuck. During each execution some random groups got stuck.
# The messages get processed when the consumer container is restarted and
works fine for sometime before the issue recurs.
# When we verified from the admin console, we could see that the number of
consumers are going beyond the max value set for it.
# We verified from the thread dumps that all the consumers are healthy and are
in TIMED_WAITING state. Even though there are some messages stuck in the queue,
they are not being delivered to the waiting consumers.
# We have entry-exit logs for the producers and consumers. We could see the
messages are being sent successfully by the producer and are never delivered to
the consumer.
Expected behaviour : All messages should be delivered to the consumers.
Actual behaviour : Some of the messages belonging to random groups get stuck in
the queue until the consumer container is restarted.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
