Ranjit Nethi created AMQ-6457:
---------------------------------
Summary: Durable Subscribers going offline - Over Network Of
Brokers - Duplex Connection
Key: AMQ-6457
URL: https://issues.apache.org/jira/browse/AMQ-6457
Project: ActiveMQ
Issue Type: Bug
Components: activemq-camel, networkbridge
Affects Versions: 5.14.0, 5.13.3
Environment: Active MQ Version 5.13.3
Karaf 4.0.6
Camel 2.16.3
-Active MQ brokers are deployed in Karaf container.
-Cluster of Active MQ Brokers (Network of brokers, configured as HUB and
SPOKES).
-Broker(s) at Hub (Master/Slave configuration using Shared File System)
- Broker(s) at spoke(s)- Connected to HUB using Duplex Network Connector (
Openwire/NIO)
Reporter: Ranjit Nethi
-Active MQ brokers are deployed in Karaf container.
-Cluster of Active MQ Brokers (Network of brokers, configured as HUB and
SPOKES).
-Broker(s) at Hub (Master/Slave configuration using Shared File System)
- Broker(s) at spoke(s)- Connected to HUB using Duplex Network Connector (
Openwire/NIO), one for topic and one for queue
Issue: We have durable subscriptions to topics on HUB and producers on Spokes.
We are aware of network outages between the HUB and SPOKE brokers, sometimes we
are experiencing multiple outages and as a result we see the topic subscribers
end up as offline and continue to store messages as the spoke broker. We are
need a restart of the broker to reestablish the network bridge.
Relevant Logs:
2016-09-22 14:21:40,105 | WARN | tyMonitor Worker |
DemandForwardingBridgeSupport | 20 - org.apache.activemq.activemq-osgi -
5.13.3 | Network connection between vm://central-amq-broker#44 and
tcp:///xx.xxx.xx.xx:53103@61616 shutdown due to a remote error:
org.apache.activemq.transport.InactivityIOException: Channel was inactive for
too (>30000) long: tcp:///xx.xxx.xx.xx:53103
2016-09-22 14:21:40,117 | INFO | -broker] Task-21 |
DemandForwardingBridgeSupport | 20 - org.apache.activemq.activemq-osgi -
5.13.3 | central-amq-broker bridge to T1-amq-broker stopped
2016-09-22 14:23:44,246 | INFO | eMQ NIO Worker 4 | TransportConnection
| 20 - org.apache.activemq.activemq-osgi - 5.13.3 | Started responder end
of duplex bridge T:broker1->broker2@ID:T1-35496-1474555572634-0:1
2016-09-22 14:23:44,255 | INFO | al-amq-broker#52 |
DemandForwardingBridgeSupport | 20 - org.apache.activemq.activemq-osgi -
5.13.3 | Network connection between vm://central-amq-broker#52 and
tcp:///xx.xxx.xx.xx:53162@61616 (T1-amq-broker) has been established.
Network Connector on Spoke Broker:
<networkConnectors>
<networkConnector
name="T:broker1->broker2"
userName="karaf"
password="karaf"
uri="masterslave:(tcp://${ACTIVEMQ_HEAD_1_IP_ADDRESS}:61616,tcp://${ACTIVEMQ_HEAD_2_IP_ADDRESS}:61616)"
duplex="true"
decreaseNetworkConsumerPriority="true"
networkTTL="2"
dynamicOnly="true">
<excludedDestinations>
<queue physicalName=">" />
<topic physicalName="Shop.Pos.Api.Outbound" />
</excludedDestinations>
</networkConnector>
<networkConnector
name="Q:broker1->broker2"
userName="karaf"
password="karaf"
uri="masterslave:(tcp://${ACTIVEMQ_HEAD_1_IP_ADDRESS}:61616,tcp://${ACTIVEMQ_HEAD_2_IP_ADDRESS}:61616)"
duplex="true"
decreaseNetworkConsumerPriority="true"
networkTTL="2"
dynamicOnly="true">
<excludedDestinations>
<topic physicalName=">" />
</excludedDestinations>
</networkConnector>
</networkConnectors>
Test to Recreate:
To recreate this issue, we can block the network between the two hosts while
simulating some load
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
