[
https://issues.apache.org/activemq/browse/AMQ-1976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=46482#action_46482
]
Gary Tully commented on AMQ-1976:
---------------------------------
One possibility is that the consuming broker is hovering about its memory
limit. Which causes out of order message delivery to the forwarded queue. There
are advisories on full that would help verify this. I can produce a memory full
test that can also verify this suspicion.
This being the case, the assertAckMatches.. needs to be changed.
Could you the apply the following patch (that just removes the failing check)
to a trunk checkout, build and re run your test?
{code}
Index:
activemq-core/src/main/java/org/apache/activemq/broker/region/PrefetchSubscription.java
===================================================================
---
activemq-core/src/main/java/org/apache/activemq/broker/region/PrefetchSubscription.java
(revision 704890)
+++
activemq-core/src/main/java/org/apache/activemq/broker/region/PrefetchSubscription.java
(working copy)
@@ -440,10 +440,10 @@
throw new JMSException("Unmatched acknowledege: Could
not find Message-ID "+firstAckedMsg+" in dispatched-list (start of ack)");
if (!checkFoundEnd && lastAckedMsg != null)
throw new JMSException("Unmatched acknowledege: Could
not find Message-ID "+lastAckedMsg+" in dispatched-list (end of ack)");
- if (ack.getMessageCount() != checkCount) {
- throw new JMSException("Unmatched acknowledege:
Expected message count ("+ack.getMessageCount()+
- ") differs from count in
dispatched-list ("+checkCount+")");
- }
+// if (ack.getMessageCount() != checkCount) {
+// throw new JMSException("Unmatched acknowledege:
Expected message count ("+ack.getMessageCount()+
+// ") differs from count in
dispatched-list ("+checkCount+")");
+// }
}
/**
{code}
> Network of brokers gets out of sync for dispatched messages causing a
> shutdown of the bridge connections and stalling the brokers
> ---------------------------------------------------------------------------------------------------------------------------------
>
> Key: AMQ-1976
> URL: https://issues.apache.org/activemq/browse/AMQ-1976
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.2.0
> Reporter: Jayson Minard
> Assignee: Gary Tully
>
> Running 5.2.0 RC2 + patch AMQ-1971
> I have a network of 5 brokers, running around 50,000 messages per minute per
> broker within each broker and sending across a total of around 20,000
> messages per minute to one of the 5 brokers via a bridge connection. The
> configuration is the 5.2.0 default with broker names changed on each config
> and with a network connector set for autodiscovery.
> The system runs for a while then eventually on one broker (the receiving
> broker that is the only one with the consumers for this queue) these errors
> appear:
> ERROR RecoveryListenerAdapter - Message id
> ID:ingest1.mag.ss.com-54179-1223898804441-0:6:1:1:6134 could not be recovered
> from the data store - already dispatched
> ERROR RecoveryListenerAdapter - Message id
> ID:query1.mag.ss.com-38822-1223898893819-0:58:16388:1:48 could not be
> recovered from the data store - already dispatched
> They appear in really high counts.
> On the sending side brokers (publishers but no consumers on these brokers for
> this queue), these errors appear:
> INFO DemandForwardingBridge - query2.mag.ss.com bridge to
> query1.mag.ss.com stopped
> INFO DiscoveryNetworkConnector - Establishing network connection
> between from vm://query2.mag.ss.com to tcp://query1.mag.ss.com:61616
> INFO DemandForwardingBridge - Network connection between
> vm://query2.mag.ss.com#30 and
> tcp://query1.mag.ss.com/10.100.0.101:61616(query1.mag.ss.com) has been
> established.
> ERROR Service - Async error occurred:
> javax.jms.JMSException: Unmatched acknowledege: Expected message count (1)
> differs from count in dispatched-list (732)
> javax.jms.JMSException: Unmatched acknowledege: Expected message count (1)
> differs from count in dispatched-list (732)
> at
> org.apache.activemq.broker.region.PrefetchSubscription.assertAckMatchesDispatched(PrefetchSubscription.java:445)
> at
> org.apache.activemq.broker.region.PrefetchSubscription.acknowledge(PrefetchSubscription.java:187)
> at
> org.apache.activemq.broker.region.AbstractRegion.acknowledge(AbstractRegion.java:373)
> at
> org.apache.activemq.broker.region.RegionBroker.acknowledge(RegionBroker.java:462)
> at
> org.apache.activemq.broker.TransactionBroker.acknowledge(TransactionBroker.java:194)
> at
> org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:74)
> at
> org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:74)
> at
> org.apache.activemq.broker.MutableBrokerFilter.acknowledge(MutableBrokerFilter.java:85)
> at
> org.apache.activemq.broker.TransportConnection.processMessageAck(TransportConnection.java:456)
> at org.apache.activemq.command.MessageAck.visit(MessageAck.java:205)
> at
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:305)
> at
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:179)
> at
> org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:104)
> at
> org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
> at
> org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:113)
> at
> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:40)
> at
> org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
> at
> org.apache.activemq.network.DemandForwardingBridgeSupport$8.onCompletion(DemandForwardingBridgeSupport.java:643)
> at
> org.apache.activemq.transport.FutureResponse.set(FutureResponse.java:61)
> at
> org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:97)
> at
> org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
> at
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:143)
> at
> org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:206)
> at
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
> at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
> at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:184)
> at java.lang.Thread.run(Thread.java:619)
> INFO DemandForwardingBridge - Network connection between
> vm://query2.mag.ss.com#30 and tcp://query1.mag.ss.com/10.100.0.101:61616
> shutdown due to a local error: javax.jms.JMSException: Unmatched
> acknowledege: Expected message count (1) differs from count in
> dispatched-list (732)
> INFO DemandForwardingBridge - query2.mag.ss.com bridge to
> query1.mag.ss.com stopped
> INFO DiscoveryNetworkConnector - Establishing network connection
> between from vm://query2.mag.ss.com to tcp://query1.mag.ss.com:61616
> INFO DemandForwardingBridge - Network connection between
> vm://query2.mag.ss.com#32 and
> tcp://query1.mag.ss.com/10.100.0.101:61616(query1.mag.ss.com) has been
> established.
> ERROR Service - Async error occurred:
> javax.jms.JMSException: Unmatched acknowledege: Expected message count (1)
> differs from count in dispatched-list (730)
> javax.jms.JMSException: Unmatched acknowledege: Expected message count (1)
> differs from count in dispatched-list (730)
> at
> org.apache.activemq.broker.region.PrefetchSubscription.assertAckMatchesDispatched(PrefetchSubscription.java:445)
> at
> org.apache.activemq.broker.region.PrefetchSubscription.acknowledge(PrefetchSubscription.java:187)
> at
> org.apache.activemq.broker.region.AbstractRegion.acknowledge(AbstractRegion.java:373)
> at
> org.apache.activemq.broker.region.RegionBroker.acknowledge(RegionBroker.java:462)
> at
> org.apache.activemq.broker.TransactionBroker.acknowledge(TransactionBroker.java:194)
> at
> org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:74)
> at
> org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:74)
> at
> org.apache.activemq.broker.MutableBrokerFilter.acknowledge(MutableBrokerFilter.java:85)
> at
> org.apache.activemq.broker.TransportConnection.processMessageAck(TransportConnection.java:456)
> at org.apache.activemq.command.MessageAck.visit(MessageAck.java:205)
> at
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:305)
> at
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:179)
> at
> org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:104)
> at
> org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
> at
> org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:113)
> at
> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:40)
> at
> org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
> at
> org.apache.activemq.network.DemandForwardingBridgeSupport$8.onCompletion(DemandForwardingBridgeSupport.java:643)
> at
> org.apache.activemq.transport.FutureResponse.set(FutureResponse.java:61)
> at
> org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:97)
> at
> org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
> at
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:143)
> at
> org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:206)
> at
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
> at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
> at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:184)
> at java.lang.Thread.run(Thread.java:619)
> INFO DemandForwardingBridge - Network connection between
> vm://query2.mag.ss.com#32 and tcp://query1.mag.ss.com/10.100.0.101:61616
> shutdown due to a local error: javax.jms.JMSException: Unmatched
> acknowledege: Expected message count (1) differs from count in
> dispatched-list (730)
> ERROR Service - Async error occurred:
> javax.jms.JMSException: Unmatched acknowledege: Expected message count (1)
> differs from count in dispatched-list (729)
> javax.jms.JMSException: Unmatched acknowledege: Expected message count (1)
> differs from count in dispatched-list (729)
> at
> org.apache.activemq.broker.region.PrefetchSubscription.assertAckMatchesDispatched(PrefetchSubscription.java:445)
> at
> org.apache.activemq.broker.region.PrefetchSubscription.acknowledge(PrefetchSubscription.java:187)
> at
> org.apache.activemq.broker.region.AbstractRegion.acknowledge(AbstractRegion.java:373)
> at
> org.apache.activemq.broker.region.RegionBroker.acknowledge(RegionBroker.java:462)
> at
> org.apache.activemq.broker.TransactionBroker.acknowledge(TransactionBroker.java:194)
> ...
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
