Opps, I meant ActiveMQ v4.1.1.
On Mon, 18 Jun 2007, [EMAIL PROTECTED] wrote:
I'm trying to run ActiveMQ v1.1.1 in a pure master/slave setup on a pair of
RHEL servers using Java(TM) 2 Runtime Environment, Standard Edition (build
1.5.0_11-b03) and it doesn't seem to reliably handle failover.
The first issue I am seeing is that the master server keeps generating the
following message in the log file:
2007-06-18 10:04:22,961 [127.0.0.1:52327] ERROR MasterBroker - Slave Failed
java.lang.AssertionError: Unsupported Method
at
org.apache.activemq.transport.TransportSupport.request(TransportSupport.java:71)
at
org.apache.activemq.transport.TransportFilter.request(TransportFilter.java:88)
at
org.apache.activemq.transport.TransportFilter.request(TransportFilter.java:88)
at
org.apache.activemq.transport.MutexTransport.request(MutexTransport.java:54)
at
org.apache.activemq.broker.ft.MasterBroker.sendSyncToSlave(MasterBroker.java:363)
at
org.apache.activemq.broker.ft.MasterBroker.sendToSlave(MasterBroker.java:345)
at
org.apache.activemq.broker.ft.MasterBroker.acknowledge(MasterBroker.java:320)
at
org.apache.activemq.broker.MutableBrokerFilter.acknowledge(MutableBrokerFilter.java:88)
at
org.apache.activemq.broker.TransportConnection.processMessageAck(TransportConnection.java:506)
at org.apache.activemq.command.MessageAck.visit(MessageAck.java:179)
at
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:294)
at
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:185)
at
org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:65)
at
org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:133)
at
org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:122)
at
org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:137)
at java.lang.Thread.run(Thread.java:595)
I'm not sure if this is a problem, but it doesn't give my any warm fuzzies.
Can this be ignored or is this a real problem?
After the slave starts up the connection between the master and the slave
times out (from the perspective of the slave), the slave tries to start up
it's connectors and fails. Here's what I'm seeing in the slave's log file:
2007-06-18 10:04:11,735 [main ] INFO BrokerService - ActiveMQ
4.1.1 JMS Message Broker (shakira-slave) is starting
2007-06-18 10:04:11,845 [JMX connector ] INFO ManagementContext - JMX
consoles can connect to service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi
2007-06-18 10:04:12,721 [main ] INFO JDBCPersistenceAdapter -
Database driver recognized: [apache_derby_embedded_jdbc_driver]
2007-06-18 10:04:13,943 [main ] INFO DefaultDatabaseLocker -
Attempting to acquire the exclusive lock to become the Master broker
2007-06-18 10:04:13,944 [main ] INFO DefaultDatabaseLocker -
Becoming the master on dataSource:
[EMAIL PROTECTED]
2007-06-18 10:04:14,012 [main ] INFO JournalPersistenceAdapter -
Journal Recovery Started from: Active Journal: using 5 x 20.0 Megs at:
/data/run/activemq-data/journal
2007-06-18 10:04:14,079 [main ] INFO JournalPersistenceAdapter -
Journal Recovered: 0 message(s) in transactions recovered.
2007-06-18 10:04:14,221 [main ] INFO TransportServerThreadSupport
- Listening for connections at: tcp://slave.cac.washington.edu:61616
2007-06-18 10:04:14,235 [main ] INFO TransportConnector -
Connector openwire Started
2007-06-18 10:04:14,244 [main ] INFO TransportServerThreadSupport
- Listening for connections at: stomp://slave.cac.washington.edu:61613
2007-06-18 10:04:14,244 [main ] INFO TransportConnector -
Connector stomp Started
2007-06-18 10:04:14,247 [main ] INFO NetworkConnector - Network
Connector default-nc Started
2007-06-18 10:04:14,253 [main ] INFO TransportConnector -
Connector vm://shakira-slave Started
2007-06-18 10:04:14,281 [main ] INFO MasterConnector - Starting a
network connection between vm://shakira-slave#0 and tcp://null:0 has been
established.
2007-06-18 10:04:14,287 [main ] INFO BrokerService - ActiveMQ JMS
Message Broker (shakira-slave,
ID:slave.cac.washington.edu-38591-1182186251386-2:0) started
2007-06-18 10:04:14,333 [Thread-4 ] INFO MasterConnector - Slave
connection between vm://shakira-slave#0 and
tcp://master.cac.washington.edu/10.10.10.1:61616 has been established.
2007-06-18 10:04:14,740 [ Agent Notifier] INFO NetworkConnector -
Establishing network connection between from vm://shakira-slave?network=true
to tcp://master.cac.washington.edu:61616
2007-06-18 10:04:14,762 [Thread-7 ] INFO DemandForwardingBridge -
Network connection between vm://shakira-slave#2 and
tcp://master.cac.washington.edu/10.10.10.1:61616(shakira-master) has been
established.
2007-06-18 12:52:44,648 [iveMQ Scheduler] ERROR MasterConnector - Network
connection between vm://shakira-slave#0 and
tcp://master.cac.washington.edu/10.10.10.1:61616 shutdown: Channel was
inactive for too long.
2007-06-18 12:52:44,771 [iveMQ Scheduler] WARN BrokerService - Master Failed
- starting all connectors
2007-06-18 12:52:44,773 [iveMQ Scheduler] ERROR BrokerService - Failed to
startAllConnectors
The slave keeps timing out and then fails to startup the connectors. I have
no idea why the channel is inactive since there are thousands of events per
minute being sent into ActiveMQ topics using the a failover URL of:
failover://(tcp://master.cac.washington.edu:61616,tcp://slave.cac.washington.edu:61616)?randomize=false
If I kill the master server (kill -9) the slave doesn't take over at this
point. The slave doesn't seem to log any messages once startAllConnectors
fails, although the activeMQ process is still running.
I've also tried restarting the slave once the slave's connection to the
master broker times out and the master server seems to think the slave is
still connected and won't let the slave reconnect.
My master config is:
<beans>
<bean
class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer"/>
<broker brokerName="shakira-master" useJmx="true"
xmlns="http://activemq.org/config/1.0";>
<memoryManager>
<usageManager id="memory-manager" limit="1 GB"/>
</memoryManager>
<persistenceAdapter>
<journaledJDBC journalLogFiles="5"
dataDirectory="/data/run/activemq-data"/>
</persistenceAdapter>
<transportConnectors>
<transportConnector name="openwire"
uri="tcp://master.cac.washington.edu:61616"
discoveryUri="multicast://default"/>
<transportConnector name="stomp"
uri="stomp://master.cac.washington.edu:61613"/>
</transportConnectors>
<networkConnectors>
<networkConnector name="default-nc" uri="multicast://default"/>
</networkConnectors>
</broker>
</beans>
And my slave config is:
<beans>
<bean
class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer"/>
<broker brokerName="shakira-slave" useJmx="true"
masterConnectorURI="tcp://master.cac.washington.edu:61616"
shutdownOnMasterFailure="false" xmlns="http://activemq.org/config/1.0";>
<memoryManager>
<usageManager id="memory-manager" limit="1 GB"/>
</memoryManager>
<persistenceAdapter>
<journaledJDBC journalLogFiles="5"
dataDirectory="/data/run/activemq-data"/>
</persistenceAdapter>
<transportConnectors>
<transportConnector name="openwire"
uri="tcp://slave.cac.washington.edu:61616"
discoveryUri="multicast://default"/>
<transportConnector name="stomp"
uri="stomp://slave.cac.washington.edu:61613"/>
</transportConnectors>
<networkConnectors>
<networkConnector name="default-nc" uri="multicast://default"/>
</networkConnectors>
</broker>
</beans>
Has anyone had the same problem? Our product is getting ready to be released
and the reliablility problems with the pure master/slave setup is becoming a
show stopper for us. Any help would be greatly appreciated.
Thanks,
Todd
