[
https://issues.apache.org/activemq/browse/AMQ-1918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=45226#action_45226
]
Richard Yarger commented on AMQ-1918:
-------------------------------------
I have a reproducible scenario for this issue.
I cannot guarantee that this is how it always happens, but
AbstractStoreCursor.size does go negative everytime I follow this procedure.
I am sorry I could not make it into a nice little test case. I tried to break
it down as best I could.
I wrapped up my eclipse project. It has 2 consumers, and 2 producers.
producer_flood sends a larger number of messages to test.queue.1 every 3 seconds
producer_steady sends a small number of messages to test.queue.1 every 3 seconds
consumer1 consumes messages off test.queue.1 and forwards them to test.queue.2
consumer2 consumes messages off test.queue.2 and prints text to system.out
My broker config file is included in the amq_broker_config dir. It sets the
memory limits low so that they can be reached faster.
The queues are setup for 1MB each and the broker is setup for 3MB.
Here is the script:
1) Start a clean broker
2) Start consumer1 and consumer2
3) Start producer_flood - this will push messages in faster than consumer1 can
handle.
Let the queue size build up to 1000 messages. This should be 100% of the 1MB
allowed for that queue.
4) Stop consumer2 - this will allow messages to build up in queue2.
If you watch this queue in a jconsole, you will notice that the percent memory
used does not rise, even after you pass the memory limit.
Let the queue2 size grow to around 2000.
This will put you near the memory limit for the broker.
5) Start consumer2 - if you look at jconsole now, the percent memory used is
updated and > 100%.
6) Wait for queue2 to fall below 1000 messages and stop the producer. Let the
messages drain and one or both of the queues should now have negative counts.
7) If you start the producer_steady now you'll notice that messages do not
reach consumer2 at the rate that they go in.
If you debug the broker now and look at AbstractStoreCursor.size, it
will be negative.
Please let me know if you need more info.
Thanks.
> AbstractStoreCursor.size gets out of synch with Store size and blocks
> consumers
> -------------------------------------------------------------------------------
>
> Key: AMQ-1918
> URL: https://issues.apache.org/activemq/browse/AMQ-1918
> Project: ActiveMQ
> Issue Type: Bug
> Components: Message Store
> Affects Versions: 5.1.0
> Reporter: Richard Yarger
> Priority: Critical
> Attachments: activemq.xml
>
>
> In version 5.1.0, we are seeing our queue consumers stop consuming for no
> reason.
> We have a staged queue environment and we occasionally see one queue display
> negative pending message counts that hang around -x, rise to -x+n gradually
> and then fall back to -x abruptly. The messages are building up and being
> processed in bunches but its not easy to see because the counts are negative.
> We see this behavior in the messages coming out of the system. Outbound
> messages come out in bunches and are synchronized with the queue pending
> count dropping to -x.
> This issue does not happen ALL of the time. It happens about once a week and
> the only way to fix it is to bounce the broker. It doesn't happen to the same
> queue everytime, so it is not our consuming code.
> Although we don't have a reproducible scenario, we have been able to debug
> the issue in our test environment.
> We traced the problem to the cached store size in the AbstractStoreCursor.
> This value becomes 0 or negative and prevents the AbstractStoreCursor from
> retrieving more messages from the store. (see AbstractStoreCursor.fillBatch()
> )
> We have seen size value go lower than -1000.
> We have also forced it to fix itself by sending in n+1 messages. Once the
> size goes above zero, the cached value is refreshed and things work ok again.
> Unfortunately, during low volume times, it could be hours before n+1 messages
> are received, so our message latency can rise during low volume times.... :(
> I have attached our broker config.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
