[
https://issues.apache.org/activemq/browse/AMQ-2102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=50230#action_50230
]
ying commented on AMQ-2102:
---------------------------
hi, Gary
unfortunately the patch I did yesterday does not fix this issue completely. i
did more load test and it failed for the same exception.
i then used your patch and get the following with 5000 msg 16 consumers. I will
try to troubleshoot this but i will appreciate your help. Thanks
ERROR MasterBroker - Slave Failed
java.lang.ClassCastException: org.apache.activemq.command.ActiveMQTextMessage
cannot be cast to org.apache.activemq.broker.region.QueueMessageReference
at
org.apache.activemq.broker.region.QueueSubscription.acknowledge(QueueSubscription.java:49)
at
org.apache.activemq.broker.region.PrefetchSubscription.acknowledge(PrefetchSubscription.java:242)
at
org.apache.activemq.broker.region.AbstractRegion.acknowledge(AbstractRegion.java:373)
at
org.apache.activemq.broker.region.RegionBroker.acknowledge(RegionBroker.java:462)
at
org.apache.activemq.broker.TransactionBroker.acknowledge(TransactionBroker.java:194)
at
org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:74)
at
org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:74)
at
org.apache.activemq.broker.MutableBrokerFilter.acknowledge(MutableBrokerFilter.java:85)
at
org.apache.activemq.broker.TransportConnection.processMessageAck(TransportConnection.java:456)
at org.apache.activemq.command.MessageAck.visit(MessageAck.java:205)
at
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:305)
at
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:179)
at
org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:104)
at
org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
at
org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:205)
at
org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
at
org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
ERROR MasterBroker - Slave Failed
javax.jms.JMSException: Slave broker out of sync with master - Message:
ID:yhe-1337-1236273055924-0:16:1:1:738 on queue://MasterSlaveBug does not exist
among pending(200) for subscription: ID:yhe-3603-1236271682179-0:13:1:1
at
org.apache.activemq.broker.region.Queue.getMatchingMessage(Queue.java:1411)
at
org.apache.activemq.broker.region.Queue.processDispatchNotification(Queue.java:1357)
at
org.apache.activemq.broker.region.AbstractRegion.processDispatchNotificationViaDestination(AbstractRegion.java:433)
at
org.apache.activemq.broker.region.QueueRegion.processDispatchNotification(QueueRegion.java:77)
at
org.apache.activemq.broker.region.RegionBroker.processDispatchNotification(RegionBroker.java:585)
at
org.apache.activemq.broker.BrokerFilter.processDispatchNotification(BrokerFilter.java:202)
at
org.apache.activemq.broker.BrokerFilter.processDispatchNotification(BrokerFilter.java:202)
at
org.apache.activemq.broker.BrokerFilter.processDispatchNotification(BrokerFilter.java:202)
at
org.apache.activemq.broker.MutableBrokerFilter.processDispatchNotification(MutableBrokerFilter.java:209)
at
org.apache.activemq.broker.TransportConnection.processMessageDispatchNotification(TransportConnection.java:466)
at
org.apache.activemq.command.MessageDispatchNotification.visit(MessageDispatchNotification.java:77)
at
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:305)
at
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:179)
at
org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:104)
at
org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
at
org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:205)
at
org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
at
org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
> Master/slave out of sync with multiple consumers
> ------------------------------------------------
>
> Key: AMQ-2102
> URL: https://issues.apache.org/activemq/browse/AMQ-2102
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.2.0
> Reporter: Dan James
> Assignee: Gary Tully
> Attachments: master.xml, MasterSlaveBug.java, MasterSlavePatch.patch,
> slave.xml, slaveDispatchOnNotification.patch
>
>
> I'm seeing exceptions like this in a simple master/slave setup:
> ERROR Service - Async error occurred:
> javax.jms.JMSException: Slave broker out of sync with master: Dispatched
> message (ID:DUL1SJAMES-L2-1231-1233929569359-0:4:1:1:207) was not in the
> pending list for MasterSlaveBug
> javax.jms.JMSException: Slave broker out of sync with master: Dispatched
> message (ID:DUL1SJAMES-L2-1231-1233929569359-0:4:1:1:207) was not in the
> pending list for MasterSlaveBug
> The problem only happens when there are multiple consumers listening to the
> queue, and is more likely to occur as there are more consumers listening.
> I've written a test program that demonstrates the problem.
> I start the master and slave with an empty data directory and let them both
> startup and settle. Then start the test program. The test program creates a
> specified number of consumers, and then starts queuing 256 messages. The
> consumers process the message by sending a reply. The producer counts the
> replies. Both consumers and the producer see all the messages, but with
> multiple consumers it is very likely that the error above will occur and
> several of the messages will still be queued on the slave.
> While debugging through the activemq code, I noticed that both the master and
> the slave dispatch the message to a consumer's pending list independently.
> In other words, it is possible that the master will add the message to
> consumer A's pending list and the slave will add the message to consumer B's
> pending list. Once the message has been processed by consumer A, the master
> sends a message to the slaving which specifies consumer A so that the slave
> can remove the message. The slave looks on its copy of consumer A's pending
> list and cannot find the message. As a result, it throws this exception and
> the message stays stuck on consumer B's pending list on the slave.
> Master and slave configurations along with MasterSlaveBug.java are attached
> to this issue.
> Start master and slave brokers:
> activemq xbean:master.xml
> activemq xbean:slave.xml
> Run with (only one consumer, the bug does not appear):
> java -classpath .:activemq-all-5.2.0.jar MasterSlaveBug 1
> Run with (sixteen consumers, the bug does appear):
> java -classpath .:activemq-all-5.2.0.jar MasterSlaveBug 16
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
