Hi All, We experienced a major outage yesterday where ActveMQ 5.5 broker crashed and stopped accepting any connections from the client. Please note that broker process was still running. Below is the error that we got from the client end.
"Connection refused at /usr/local/pkgs/site_perl-5.10/lib/Net/Stomp.pm line 19" >From the broker logs, I can see that it started spitting out below errors >around that time: 6 | ERROR | Could not accept connection : java.lang.IllegalStateException: Timer already cancelled. | org.apache.activemq.broker.TransportConnector | ActiveMQ Task-241791 2011-11-24 16:51:39,582 | ERROR | Could not accept connection : java.lang.IllegalStateException: Timer already cancelled. | org.apache.activemq.broker.TransportConnector | ActiveMQ Task-241876 2011-11-24 16:51:46,365 | ERROR | Could not accept connection : java.lang.IllegalStateException: Timer already cancelled. | org.apache.activemq.broker.TransportConnector | ActiveMQ Task-241889 2011-11-24 16:52:10,114 | ERROR | Could not accept connection : java.lang.IllegalStateException: Timer already cancelled. | org.apache.activemq.broker.TransportConnector | ActiveMQ Task-241914 Before these errors came up I can see that there are info level messages but these we have been getting almost daily: 011-11-24 00:11:29,492 | INFO | Transport failed: org.apache.activemq.transport.InactivityIOException: Channel was inactive for too (>30000) long: /10.77.73.176:54410 | org.apache.activemq.broker.TransportConnection.Transport | InactivityMonitor Async Task: java.util.concurrent.ThreadPoolExecutor$Worker@10b8b87 011-11-24 00:11:29,492 | INFO | Transport failed: org.apache.activemq.transport.InactivityIOException: Channel was inactive for too (>30000) long: /10.77.73.176:54410 | org.apache.activemq.broker.TransportConnection.Transport | InactivityMonitor Async Task: java.util.concurrent.ThreadPoolExecutor$Worker@10b8b87 >From the jconsole only suspicious thing that we could find that is the PS Survivor memory space had a spike of 80 MB during that time. Could someone please help us here figure what could have been the root cause. Any help would be appreciated. Thanks
