[
https://issues.apache.org/jira/browse/AMQ-5249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Chernyakhovsky updated AMQ-5249:
-------------------------------------
Comment: was deleted
(was: Gary,
I will check the unit test module but I am afraid it might prove hard to
replicate our environment.
Maybe it will be more productive if I work with you or someone assigned to this
issue to understand what exactly is happening and then proceed with the test.
Here is some info:
1) Our production system consists of about half a dozen of servers running
clients producing messages, then another set of about 10 consuming the load
while producing some messages for consumption by themselves and another set of
about 5 servers running apache storm. (The numbers change as we scale). 8
queues are created across the board.
2) We've ran into some reliability issues with the network of brokers while
using AMQ 5.8.0 (transient) so we resorted to a single beefed up server.
3) Now we are on 5.10.0 and it is able to keep up with the current load but the
load tends to grow and we would want to have some failover capability too.
4) I am running a load test for the same configuration as in production with 5,
7, 6 servers and again a beefy AMQ 5.10.0 server works fine.
5) When I add another AMQ server using static connector everything seems to
work fine under lighter load. As load grows, even before the message
consumption starts falling behind I am seeing a flood of the "duplicate"
messages. They of course are not a problem by themselves but the DLQ fills up
really quickly and AMQs die horrible out of memory death. Exactly the same
behavior under the same setup and load is observed for the 5 or .11-SNAPSHOT (I
basically swap the activemq link in the /opt folder)
6) Keeping the whole system the same, when I switch to 5.9.0 the issue
disappears - no "duplicate" in the log, DLQ does not get populated.
So the question is what gives? The consumers seem to process all the messages
at least it shouldn't be different from 5.9 to 5.10 to 5.11. Do you believe
something has changed in 5.10 and it might be a legit behavior?
Let me know. I can provide additional details and assist with debugging if
needed
Thank you,
Alex
)
> "cursor got duplicate" error after upgrade
> ------------------------------------------
>
> Key: AMQ-5249
> URL: https://issues.apache.org/jira/browse/AMQ-5249
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.9.1, 5.10.0
> Reporter: Rural Hunter
>
> I was using 5.9.0 and meet one problem so I tried to upgrade activemq. I
> tried both 5.9.1 and 5.10.0 and encouterred a same problem. I saw messages
> filled DLQ very quickly. I checked the clients both producer and consumer but
> there was no error. I checked activemq log and found the log is full of these
> warnings:
> 2014-06-27 23:22:09,337 | WARN |
> org.apache.activemq.broker.region.cursors.QueueStorePrefetch@19117501:com.cyyun.webmon.spider.update,batchResetNeeded=false,storeHasMessages=true,size=211,cacheEnabled=true,maxBatchSize:200,hasSpace:true
> - cursor got duplicate: ID:211.com-52399-1400732399425-1:1:235992:1:1, 4 |
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor | ActiveMQ
> Broker[localhost] Scheduler
> 2014-06-27 23:22:09,337 | WARN |
> org.apache.activemq.broker.region.cursors.QueueStorePrefetch@19117501:com.xxxx.update,batchResetNeeded=false,storeHasMessages=true,size=211,cacheEnabled=true,maxBatchSize:200,hasSpace:true
> - cursor got duplicate: ID:nbzjjf22805-34129-1403880308671-1:1:28:1:1, 4 |
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor | ActiveMQ
> Broker[localhost] Scheduler
> 2014-06-27 23:22:09,338 | WARN |
> org.apache.activemq.broker.region.cursors.QueueStorePrefetch@19117501:com.xxxxx.update,batchResetNeeded=false,storeHasMessages=true,size=211,cacheEnabled=true,maxBatchSize:200,hasSpace:true
> - cursor got duplicate: ID:jxncxnj2-48598-1403856107346-1:1:6007:1:1, 4 |
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor | ActiveMQ
> Broker[localhost] Scheduler
> 2014-06-27 23:22:09,338 | WARN |
> org.apache.activemq.broker.region.cursors.QueueStorePrefetch@19117501:com.xxxx.update,batchResetNeeded=false,storeHasMessages=true,size=211,cacheEnabled=true,maxBatchSize:200,hasSpace:true
> - cursor got duplicate: ID:jxnc17-60227-1400730816361-1:1:149072:1:1, 4 |
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor | ActiveMQ
> Broker[localhost] Scheduler
> 2014-06-27 23:22:09,339 | WARN |
> org.apache.activemq.broker.region.cursors.QueueStorePrefetch@19117501:com.xxxx.update,batchResetNeeded=false,storeHasMessages=true,size=211,cacheEnabled=true,maxBatchSize:200,hasSpace:true
> - cursor got duplicate: ID:cyyun-46954-1403800808565-1:1:9765:1:1, 4 |
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor | ActiveMQ
> Broker[localhost] Scheduler
> 2014-06-27 23:22:09,339 | WARN |
> org.apache.activemq.broker.region.cursors.QueueStorePrefetch@19117501:com.xxxx.update,batchResetNeeded=false,storeHasMessages=true,size=211,cacheEnabled=true,maxBatchSize:200,hasSpace:true
> - cursor got duplicate: ID:ubuntu-55495-1403497638437-1:1:53086:1:1, 4 |
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor | ActiveMQ
> Broker[localhost] Scheduler
> 2014-06-27 23:22:09,340 | WARN |
> org.apache.activemq.broker.region.cursors.QueueStorePrefetch@19117501:com.xxxx.update,batchResetNeeded=false,storeHasMessages=true,size=211,cacheEnabled=true,maxBatchSize:200,hasSpace:true
> - cursor got duplicate: ID:cyyun-39030-1403880008363-1:1:70:1:1, 4 |
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor | ActiveMQ
> Broker[localhost] Scheduler
> The problem mostly happens right after activemq starts and sometimes happened
> after activemq worked normally for a while.
> For now I have to roll back to 5.9.0 and the problem doesn't occure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
