Hey Folks,
Can you chime in with any thoughts on how you'd configure ActiveMQ for this
architecture...?
I have about 10 instances of a web application that require the services of an
external component. The external component is written in C/C++ and uses the
CMS library to connect to ActiveMQ 5.2. The web applications run in Jetty 6 on
top of a Java 6 JVM.
Every web server session is assigned to an instance of the external component.
Every web server instance creates an exclusive topic on which messages destined
for their end users are to be placed. A correlation ID consisting of username
and sessionid are attached to every message.
Every C/C++ service subscribes to a common Queue for connection requests. The
Queue has a roundRobinDispatchPolicy and a noSubscriptionRecoveryPolicy
configuration. When the service receives a connect request, it replies with a
unique topic that further commands for that correlationID should be sent to.
This topic is unique to an instance of the service.
High availability is required for the architecture such that in the event an
ActiveMQ server dies, the application can continue to run. We can tolerate
very occasional message loss especially if it buys us a performance benefit.
We need to support about 15k msg/s across the system and need to be able to
scale beyond that.
Given these parameters, I thought it would make sense to configure two ActiveMQ
brokers in a network-of-brokers configuration. This is the part of the
activemq.xml file that I have for that:
<!-- by default just auto discover the other brokers -->
<networkConnector uri="multicast://default?group=${activemq.ENV}"
name="default-nc"
networkTTL="2"
dynamicOnly="true"
conduitSubscriptions="true"
decreaseNetworkConsumerPriority="false">
</networkConnector>
Does this look right? Everything works well until we get up to about 10k
msgs/s across the system. After this point, the connections between the two
brokers aborted and clients across the system had to reconnect causing large
CPU spikes.
The webserver connections were pretty evenly distributed across the two JMS
servers. The C/C++ components were almost all connected to only one of the JMS
servers. It's this server that was showing increased CPU utilization before
the two brokers stopped talking to each other momentarily. Could that be an
issue?
Here's the exception that I saw on server2 (the one with all of the C/C++
connections):
2009-03-31 12:18:49,426 [rt: vm://null#6] WARN DemandForwardingBridge
- Caught an exception processing local command
java.net.SocketException: Socket closed
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:99)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at
org.apache.activemq.transport.tcp.TcpBufferedOutputStream.flush(TcpBufferedOutputStream.java:115)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at
org.apache.activemq.transport.tcp.TcpTransport.oneway(TcpTransport.java:167)
at
org.apache.activemq.transport.InactivityMonitor.oneway(InactivityMonitor.java:233)
at
org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:83)
at
org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:100)
at
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:40)
at
org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
at
org.apache.activemq.network.DemandForwardingBridgeSupport.serviceLocalCommand(DemandForwardingBridgeSupport.java:647)
at
org.apache.activemq.network.DemandForwardingBridgeSupport$1.onCommand(DemandForwardingBridgeSupport.java:147)
at
org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:104)
at
org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
at
org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:205)
at
org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
at
org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
2009-03-31 12:18:49,430 [ Agent Notifier] INFO DemandForwardingBridge
- amqbroker_jms2 bridge to amqbroker_jms1 stopped
2009-03-31 12:18:49,750 [ Agent Notifier] INFO DiscoveryNetworkConnector
- Establishing network connection between from vm://null to
tcp://jms1.radiusim.com:61516
2009-03-31 12:18:49,750 [ Agent Notifier] WARN BrokerRegistry
- Broker localhost not started so using amqbroker_jms2 instead
2009-03-31 12:18:49,750 [ Agent Notifier] WARN BrokerRegistry
- Broker localhost not started so using amqbroker_jms2 instead
2009-03-31 12:18:49,750 [ Agent Notifier] INFO TransportConnector
- Connector vm://null Started
2009-03-31 12:19:02,545 [ker=vm://null#8] INFO DemandForwardingBridge
- Network connection between vm://null#8 and
tcp://jms1.radiusim.com/10.2.20.1:61516(amqbroker_jms1) has
been established.
2009-03-31 12:19:24,049 [ Agent Notifier] INFO DemandForwardingBridge
- amqbroker_jms2 bridge to amqbroker_jms1 stopped
2009-03-31 12:19:24,049 [rt: vm://null#8] WARN DemandForwardingBridge
- Caught an exception processing local command
java.net.SocketException: Socket closed
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:99)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at
org.apache.activemq.transport.tcp.TcpBufferedOutputStream.flush(TcpBufferedOutputStream.java:115)
Here's what I saw on server1:
2009-03-31 12:19:12,637 [ Agent Notifier] INFO DemandForwardingBridge
- amqbroker_jms1 bridge to amqbroker_jms2 stopped
2009-03-31 12:19:15,914 [ Agent Notifier] INFO DiscoveryNetworkConnector
- Establishing network connection between from vm://null to
tcp://jms2.mycompany.com:61516
2009-03-31 12:19:15,916 [ Agent Notifier] WARN BrokerRegistry
- Broker localhost not started so using amqbroker_jms1 instead
2009-03-31 12:19:15,916 [ Agent Notifier] WARN BrokerRegistry
- Broker localhost not started so using amqbroker_jms1 instead
2009-03-31 12:19:15,916 [ Agent Notifier] INFO TransportConnector
- Connector vm://null Started
2009-03-31 12:19:22,118 [ Agent Notifier] INFO DemandForwardingBridge
- amqbroker_jms1 bridge to Unknown stopped
2009-03-31 12:19:23,621 [ Agent Notifier] INFO DiscoveryNetworkConnector
- Establishing network connection between from vm://null to
tcp://jms2.mycompany.com:61516
2009-03-31 12:19:23,623 [ Agent Notifier] WARN BrokerRegistry
- Broker localhost not started so using amqbroker_jms1 instead
2009-03-31 12:19:23,623 [ Agent Notifier] WARN BrokerRegistry
- Broker localhost not started so using amqbroker_jms1 instead
2009-03-31 12:19:23,623 [ Agent Notifier] INFO TransportConnector
- Connector vm://null Started
2009-03-31 12:19:33,740 [ Agent Notifier] INFO DemandForwardingBridge
- amqbroker_jms1 bridge to Unknown stopped
2009-03-31 12:19:33,740 [ Agent Notifier] INFO DiscoveryNetworkConnector
- Establishing network connection between from vm://null to
tcp://jms2.mycompany.com:61516
2009-03-31 12:19:33,741 [ Agent Notifier] WARN BrokerRegistry
- Broker localhost not started so using amqbroker_jms1 instead
2009-03-31 12:19:33,741 [ Agent Notifier] WARN BrokerRegistry
- Broker localhost not started so using amqbroker_jms1 instead
2009-03-31 12:19:33,741 [ Agent Notifier] INFO TransportConnector
- Connector vm://null Started
2009-03-31 12:19:44,031 [er=vm://null#22] WARN DemandForwardingBridge
- Network connection between vm://null#22 and
tcp://jms2.mycompany.com/192.168.1.4:61516 shutdown due to a remo
te error: java.io.IOException: Wire format negotiation timeout: peer did not
send his wire format.
2009-03-31 12:19:44,166 [NetworkBridge ] INFO DemandForwardingBridge
- amqbroker_jms1 bridge to Unknown stopped
2009-03-31 12:19:49,169 [ Agent Notifier] INFO DiscoveryNetworkConnector
- Establishing network connection between from vm://null to
tcp://jms2.mycompany.com:61516
2009-03-31 12:19:49,170 [ Agent Notifier] WARN BrokerRegistry
- Broker localhost not started so using amqbroker_jms1 instead
2009-03-31 12:19:49,170 [ Agent Notifier] WARN BrokerRegistry
- Broker localhost not started so using amqbroker_jms1 instead
2009-03-31 12:19:49,170 [ Agent Notifier] INFO TransportConnector
- Connector vm://null Started
2009-03-31 12:19:59,398 [er=vm://null#24] WARN DemandForwardingBridge
- Network connection between vm://null#24 and
tcp://jms2.mycompany.com/192.168.1.4:61516 shutdown due to a remo
te error: java.io.IOException: Wire format negotiation timeout: peer did not
send his wire format.
Other configuration I have:
<destinationPolicy>
<policyMap>
<policyEntries>
<!-- Configure the Interop connection dispatch queue -->
<policyEntry
queue="Queue.com.mycompany.connectrequests" memoryLimit="5mb">
<dispatchPolicy>
<roundRobinDispatchPolicy/>
</dispatchPolicy>
<subscriptionRecoveryPolicy>
<noSubscriptionRecoveryPolicy/>
</subscriptionRecoveryPolicy>
</policyEntry>
<policyEntry queue=">" memoryLimit="8mb"/>
<policyEntry topic=">" memoryLimit="8mb"/>
</policyEntries>
</policyMap>
</destinationPolicy>
<!-- Use the following to configure how ActiveMQ is exposed in JMX -->
<managementContext>
<managementContext createConnector="false"/>
</managementContext>
<persistenceAdapter>
<amqPersistenceAdapter syncOnWrite="false"
directory="${activemq.base}/data" maxFileLength="20 mb"/>
</persistenceAdapter>
<sslContext>
<sslContext keyStore="file:${activemq.base}/conf/broker.ks"
keyStorePassword="password" trustStore="file:${activemq.base}/conf/broker.ts"
trustStorePassword="password"/>
</sslContext>
<!-- The maximum about of space the broker will use before slowing
down producers -->
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage limit="128 mb"/>
</memoryUsage>
<storeUsage>
<storeUsage limit="1 gb" name="foo"/>
</storeUsage>
<tempUsage>
<tempUsage limit="128 mb"/>
</tempUsage>
</systemUsage>
</systemUsage>
<!-- The transport connectors ActiveMQ will listen to -->
<transportConnectors>
<transportConnector name="openwire" uri="tcp://localhost:61516"
discoveryUri="multicast://default?group=${activemq.ENV}"/>
</transportConnectors>
