We are running ActiveMQ in Master/Slave configuration. Currently mqarms02p is the Master. Earlier today we ran into a situation where we lost access to ActiveMQ and on reviewing the client side logs we see the information below cycle continuously.

Is there anything we can enable on the server side to better log what happened from the server perspective? Unfortunately in this instance the logs had already rolled so we could not see anything, however from previously experience we know almost nothing shows up.

The server was online and other clients were using it without incident

Any ideas as to what happened and how we can prevent this occurring in future?

08-31 11:46:02 DEBUG - Attempting 0th connect to: tcp://mqarms01p:61616
08-31 11:46:02 TRACE - Starting connection check task for: tcp://mqarms01p:61616 08-31 11:46:02 DEBUG - Connect fail to: tcp://mqarms01p:61616, reason: java.net.ConnectException: Connection refused 08-31 11:46:02 TRACE - Stopping connection check task for: tcp://mqarms01p:61616
08-31 11:46:02 DEBUG - Stopping transport tcp://mqarms01p:61616
08-31 11:46:02 DEBUG - Initialized TaskRunnerFactory[ActiveMQ Task] using ExecutorService: java.util.concurrent.ThreadPoolExecutor@1f0f564[Running, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0] 08-31 11:46:02 TRACE - Execute[ActiveMQ Task] runnable: org.apache.activemq.transport.tcp.TcpTransport$1@727e793a 08-31 11:46:02 TRACE - Created thread[ActiveMQ Task-1]: Thread[ActiveMQ Task-1,5,main]
08-31 11:46:02 TRACE - Closing socket Socket[unconnected]
08-31 11:46:02 DEBUG - Closed socket Socket[unconnected]
08-31 11:46:02 DEBUG - Forcing shutdown of ExecutorService: java.util.concurrent.ThreadPoolExecutor@1f0f564[Running, pool size = 1, active threads = 0, queued tasks = 0, completed tasks = 1] 08-31 11:46:02 TRACE - Shutdown of ExecutorService: java.util.concurrent.ThreadPoolExecutor@1f0f564[Shutting down, pool size = 1, active threads = 0, queued tasks = 0, completed tasks = 1] is shutdown: true and terminated: false.
08-31 11:46:02 DEBUG - Attempting 0th connect to: tcp://mqarms02p:61616
08-31 11:46:02 TRACE - Starting connection check task for: tcp://mqarms02p:61616 08-31 11:46:02 DEBUG - Sending: WireFormatInfo { version=12, properties={MaxFrameSize=9223372036854775807, CacheSize=1024, ProviderVersion=5.14.3, CacheEnabled=true, ProviderName=ActiveMQ, SizePrefixDisabled=false, PlatformDetails=JVM: 1.7.0_80, 24.80-b11, Oracle Corporation, OS: Linux, 2.6.32-358.6.2.el6.x86_64, amd64, MaxInactivityDurationInitalDelay=10000, TcpNoDelayEnabled=true, MaxInactivityDuration=30000, TightEncodingEnabled=true, StackTraceEnabled=true}, magic=[A,c,t,i,v,e,M,Q]} 08-31 11:46:02 TRACE - Starting connection check task for: tcp://mqarms02p/172.16.101.116:61616@47626 08-31 11:46:02 TRACE - TCP consumer thread for tcp://mqarms02p/172.16.101.116:61616@47626 starting 08-31 11:46:02 TRACE - Stopping connection check task for: tcp://mqarms02p/172.16.101.116:61616@47626 08-31 11:46:02 DEBUG - Using min of local: WireFormatInfo { version=12, properties={MaxFrameSize=9223372036854775807, CacheSize=1024, ProviderVersion=5.14.3, CacheEnabled=true, ProviderName=ActiveMQ, SizePrefixDisabled=false, PlatformDetails=JVM: 1.7.0_80, 24.80-b11, Oracle Corporation, OS: Linux, 2.6.32-358.6.2.el6.x86_64, amd64, MaxInactivityDurationInitalDelay=10000, TcpNoDelayEnabled=true, MaxInactivityDuration=30000, TightEncodingEnabled=true, StackTraceEnabled=true}, magic=[A,c,t,i,v,e,M,Q]} and remote: WireFormatInfo { version=12, properties={MaxFrameSize=104857600, CacheSize=1024, ProviderVersion=5.14.3, CacheEnabled=true, ProviderName=ActiveMQ, SizePrefixDisabled=false, MaxInactivityDurationInitalDelay=10000, PlatformDetails=JVM: 1.8.0_111, 25.111-b15, Oracle Corporation, OS: Linux, 3.8.13-68.3.3.el6uek.x86_64, amd64, TcpNoDelayEnabled=true, MaxInactivityDuration=30000, TightEncodingEnabled=true, StackTraceEnabled=true}, magic=[A,c,t,i,v,e,M,Q]} 08-31 11:46:02 DEBUG - Received WireFormat: WireFormatInfo { version=12, properties={MaxFrameSize=104857600, CacheSize=1024, ProviderVersion=5.14.3, CacheEnabled=true, ProviderName=ActiveMQ, SizePrefixDisabled=false, MaxInactivityDurationInitalDelay=10000, PlatformDetails=JVM: 1.8.0_111, 25.111-b15, Oracle Corporation, OS: Linux, 3.8.13-68.3.3.el6uek.x86_64, amd64, TcpNoDelayEnabled=true, MaxInactivityDuration=30000, TightEncodingEnabled=true, StackTraceEnabled=true}, magic=[A,c,t,i,v,e,M,Q]} 08-31 11:46:02 DEBUG - tcp://mqarms02p/172.16.101.116:61616@47626 before negotiation: OpenWireFormat{version=12, cacheEnabled=false, stackTraceEnabled=false, tightEncodingEnabled=false, sizePrefixDisabled=false, maxFrameSize=9223372036854775807} 08-31 11:46:02 DEBUG - tcp://mqarms02p/172.16.101.116:61616@47626 after negotiation: OpenWireFormat{version=12, cacheEnabled=true, stackTraceEnabled=true, tightEncodingEnabled=true, sizePrefixDisabled=false, maxFrameSize=104857600}
08-31 11:46:02 DEBUG - conn: ID:tnsags01p.xxxx.com-53927-1504137796960-5:1
08-31 11:46:02 DEBUG - session: ID:tnsags01p.xxxx.com-53927-1504137796960-5:1:1441358 08-31 11:46:02 DEBUG - restore consumer: ID:tnsags01p.xxxx.com-53927-1504137796960-5:1:1441358:1 in pull mode pending recovery, overriding prefetch: 1000 08-31 11:46:02 DEBUG - consumer: ID:tnsags01p.xxxx.com-53927-1504137796960-5:1:1441358:1 08-31 11:46:02 DEBUG - session: ID:tnsags01p.xxxx.com-53927-1504137796960-5:1:-1 08-31 11:46:02 DEBUG - restore consumer: ID:tnsags01p.xxxx.com-53927-1504137796960-5:1:-1:1 in pull mode pending recovery, overriding prefetch: 1000 08-31 11:46:02 DEBUG - consumer: ID:tnsags01p.xxxx.com-53927-1504137796960-5:1:-1:1 08-31 11:46:02 DEBUG - session: ID:tnsags01p.xxxx.com-53927-1504137796960-5:1:1442703 08-31 11:46:02 DEBUG - producer: ID:tnsags01p.xxxx.com-53927-1504137796960-5:1:1442703:1
08-31 11:46:02 DEBUG - Connection established
08-31 11:46:02 INFO  - Successfully reconnected to tcp://mqarms02p:61616
08-31 11:46:02 TRACE - Running task iteration 0 - org.apache.activemq.transport.failover.FailoverTransport$2@112605fd 08-31 11:46:02 TRACE - Run task done: org.apache.activemq.transport.failover.FailoverTransport$2@112605fd 08-31 11:46:02 TRACE - Run task done: org.apache.activemq.transport.failover.FailoverTransport$2@112605fd 08-31 11:46:02 TRACE - Received ConnectionControl: ConnectionControl {commandId = 0, responseRequired = false, suspend = false, resume = false, close = false, exit = false, faultTolerant = false, connectedBrokers = , reconnectTo = , token = null, rebalanceConnection = false} 08-31 11:46:02 DEBUG - Send oneway attempt: 17752 failed for command: ActiveMQObjectMessage {commandId = 7234566, responseRequired = true, messageId = ID:tnsags01p.xxxx.com-53927-1504137796960-5:1:1442703:1:1, originalDestination = null, originalTransactionId = null, producerId = ID:tnsags01p.xxxx.com-53927-1504137796960-5:1:1442703:1, destination = queue://routing_mia_client, transactionId = null, expiration = 0, timestamp = 1504192175689, arrival = 0, brokerInTime = 0, brokerOutTime = 0, correlationId = null, replyTo = null, persistent = true, type = null, priority = 4, groupID = null, groupSequence = 0, targetConsumerId = null, compressed = false, userID = null, content = org.apache.activemq.util.ByteSequence@1558c374, marshalledProperties = null, dataStructure = null, redeliveryCounter = 0, size = 0, properties = null, readOnlyProperties = true, readOnlyBody = true, droppable = false, jmsXGroupFirstForConsumer = false} 08-31 11:46:02 TRACE - tcp://mqarms02p:61616 handleTransportFailure: java.net.SocketException: Broken pipe
java.net.SocketException: Broken pipe
        at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
        at java.net.SocketOutputStream.write(SocketOutputStream.java:159)
at org.apache.activemq.transport.tcp.TcpBufferedOutputStream.write(TcpBufferedOutputStream.java:96)
        at java.io.DataOutputStream.write(DataOutputStream.java:107)
at org.apache.activemq.openwire.v12.BaseDataStreamMarshaller.tightMarshalByteSequence2(BaseDataStreamMarshaller.java:431) at org.apache.activemq.openwire.v12.MessageMarshaller.tightMarshal2(MessageMarshaller.java:182) at org.apache.activemq.openwire.v12.ActiveMQMessageMarshaller.tightMarshal2(ActiveMQMessageMarshaller.java:89) at org.apache.activemq.openwire.v12.ActiveMQObjectMessageMarshaller.tightMarshal2(ActiveMQObjectMessageMarshaller.java:89) at org.apache.activemq.openwire.OpenWireFormat.marshal(OpenWireFormat.java:235) at org.apache.activemq.transport.tcp.TcpTransport.oneway(TcpTransport.java:193) at org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:335) at org.apache.activemq.transport.AbstractInactivityMonitor.oneway(AbstractInactivityMonitor.java:317) at org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:94) at org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:116) at org.apache.activemq.transport.failover.FailoverTransport.oneway(FailoverTransport.java:668)

Reply via email to