Wei Wei created AMQ-4150:
----------------------------
Summary: in pure master/slave, when the network is not stable,
neither master nor slave can provide service to client
Key: AMQ-4150
URL: https://issues.apache.org/jira/browse/AMQ-4150
Project: ActiveMQ
Issue Type: Bug
Components: Broker, Transport
Affects Versions: 5.7.0
Environment: CentOS release 5.6 (Final) java version "1.6.0_30"
Reporter: Wei Wei
We use pure master/slave. When the network between master and slave is not
stable, the connection between master and slave shutdown, then the slave try to
startAllConnectors because of the follow exception:
2012-10-25 18:48:12,083 | ERROR | Network connection between
vm://172_17_7_92_61619#0 and tcp:///172.17.7.91:61618 shutdown: Connection
reset | org.apache.activemq.broker.ft.MasterConnector | ActiveMQ Transport:
tcp:///172.17.7.91:61618
java.net.SocketException: Connection reset
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at
org.apache.activemq.transport.tcp.TcpBufferedOutputStream.flush(TcpBufferedOutputStream.java:115)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at
org.apache.activemq.transport.tcp.TcpTransport.oneway(TcpTransport.java:184)
at
org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:262)
at
org.apache.activemq.transport.AbstractInactivityMonitor.oneway(AbstractInactivityMonitor.java:244)
at
org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:85)
at
org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:104)
at
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
at
org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
at
org.apache.activemq.broker.ft.MasterConnector.serviceRemoteCommand(MasterConnector.java:268)
at
org.apache.activemq.broker.ft.MasterConnector$2.onCommand(MasterConnector.java:138)
at
org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
at
org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
at
org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
at
org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:229)
at
org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:222)
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:204)
at java.lang.Thread.run(Thread.java:662)
After the exception the follow log was printed:
2012-10-25 18:48:12,084 | WARN | Master Failed - starting all connectors |
org.apache.activemq.broker.BrokerService | ActiveMQ Transport:
tcp:///172.17.7.91:61618
it seems that the connector started properly, but it can not provide service to
client.
After a while, another exception occured:
2012-10-25 18:49:01,212 | ERROR | Network connection between
vm://172_17_7_92_61619#0 and tcp:///172.17.7.91:61618 shutdown: Broken pipe |
org.apache.activemq.broker.ft.MasterConnector | InactivityMonitor Async Task:
java.util.concurrent.ThreadPoolExecutor$Worker@351dd9b3
java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at
org.apache.activemq.transport.tcp.TcpBufferedOutputStream.flush(TcpBufferedOutputStream.java:115)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at
org.apache.activemq.transport.tcp.TcpTransport.oneway(TcpTransport.java:184)
at
org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:262)
at
org.apache.activemq.transport.AbstractInactivityMonitor.access$500(AbstractInactivityMonitor.java:40)
at
org.apache.activemq.transport.AbstractInactivityMonitor$3.run(AbstractInactivityMonitor.java:151)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
this exception lead to the slave trying to startAllConnectors again, this time
the follow exception occured:
2012-10-25 18:49:26,233 | ERROR | Failed to startAllConnectors |
org.apache.activemq.broker.BrokerService | InactivityMonitor Async Task:
java.util.concurrent.ThreadPoolExecutor$Worker@351dd9b3
java.io.IOException: Transport Connector could not be registered in JMX: Failed
to bind to server socket: nio://0.0.0.0:61619 due to: java.net.BindException:
Address already in use
at
org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:27)
at
org.apache.activemq.broker.BrokerService.registerConnectorMBean(BrokerService.java:1836)
at
org.apache.activemq.broker.BrokerService.startTransportConnector(BrokerService.java:2327)
at
org.apache.activemq.broker.BrokerService.startAllConnectors(BrokerService.java:2243)
at
org.apache.activemq.broker.BrokerService.masterFailed(BrokerService.java:446)
at
org.apache.activemq.broker.ft.MasterConnector.shutDown(MasterConnector.java:345)
at
org.apache.activemq.broker.ft.MasterConnector.serviceRemoteException(MasterConnector.java:250)
at
org.apache.activemq.broker.ft.MasterConnector$2.onException(MasterConnector.java:144)
at
org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:101)
at
org.apache.activemq.transport.ResponseCorrelator.onException(ResponseCorrelator.java:126)
at
org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:101)
at
org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:101)
at
org.apache.activemq.transport.WireFormatNegotiator.onException(WireFormatNegotiator.java:160)
at
org.apache.activemq.transport.AbstractInactivityMonitor.onException(AbstractInactivityMonitor.java:268)
at
org.apache.activemq.transport.AbstractInactivityMonitor$3.run(AbstractInactivityMonitor.java:154)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Failed to bind to server socket:
nio://0.0.0.0:61619 due to: java.net.BindException: Address already in use
at
org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:33)
at
org.apache.activemq.transport.tcp.TcpTransportServer.bind(TcpTransportServer.java:138)
at
org.apache.activemq.transport.tcp.TcpTransportFactory.doBind(TcpTransportFactory.java:60)
at
org.apache.activemq.transport.TransportFactory.bind(TransportFactory.java:131)
at
org.apache.activemq.broker.TransportConnector.createTransportServer(TransportConnector.java:329)
at
org.apache.activemq.broker.TransportConnector.getServer(TransportConnector.java:155)
at
org.apache.activemq.broker.TransportConnector.asManagedConnector(TransportConnector.java:110)
at
org.apache.activemq.broker.BrokerService.registerConnectorMBean(BrokerService.java:1831)
... 16 more
Caused by: java.net.BindException: Address already in use
at sun.nio.ch.Net.bind(Native Method)
at
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:126)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
at
org.apache.activemq.transport.nio.NIOTransportFactory$2.createServerSocket(NIOTransportFactory.java:69)
at
org.apache.activemq.transport.tcp.TcpTransportServer.bind(TcpTransportServer.java:134)
... 22 more
At last, neither the master nor the slave can provide service to client.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
