I keep seing this behavior time to time. I have to cycle instance. broker logs are filled with info logs:
INFO | The connection to 'tcp://10.190.133.15:37168' is taking a long time to shutdown. | org.apache.activemq.broker.TransportConnection | ActiveMQ Transport: tcp:///10.190.133.15:55356@61616 any suggestions. On Wed, Apr 16, 2014 at 10:14 AM, Chirag Pujara <chiragpuj...@gmail.com>wrote: > I checked logs. there are bunch of Info messages about conection taking > too long to shutdown. > > 2014-04-15 10:51:00,668 | INFO | The connection to 'tcp:// > 10.190.133.15:37168' is taking a long time to shutdown. | > org.apache.activemq.broker.TransportConnection | ActiveMQ Transport: > tcp:///10.190.133.15:55356@61616 > 2014-04-15 10:51:00,669 | INFO | The connection to 'tcp:// > 10.190.133.14:58333' is taking a long time to shutdown. | > org.apache.activemq.broker.TransportConnection | ActiveMQ Transport: > tcp:///10.190.133.14:48314@61616 > 2014-04-15 10:51:00,740 | INFO | The connection to 'tcp:// > 10.190.133.15:54436' is taking a long time to shutdown. | > org.apache.activemq.broker.TransportConnection | ActiveMQ Transport: > tcp:///10.190.133.15:55568@61616 > 2014-04-15 10:51:00,741 | INFO | The connection to 'tcp:// > 10.190.133.15:37172' is taking a long time to shutdown. | > org.apache.activemq.broker.TransportConnection | ActiveMQ Transport: > tcp:///10.190.133.15:55575@61616 > 2014-04-15 10:51:00,989 | INFO | The connection to 'tcp:// > 10.190.133.14:58360' is taking a long time to shutdown. | > org.apache.activemq.broker.TransportConnection | ActiveMQ Transport: > tcp:///10.190.133.14:48072@61616 > 2014-04-15 10:51:00,989 | INFO | The connection to 'tcp:// > 10.190.133.14:58370' is taking a long time to shutdown. | > org.apache.activemq.broker.TransportConnection | ActiveMQ Transport: > tcp:///10.190.133.14:48290@61616 > 2014-04-15 10:51:00,994 | INFO | The connection to 'tcp:// > 10.190.133.14:47258' is taking a long time to shutdown. | > org.apache.activemq.broker.TransportConnection | ActiveMQ Transport: > tcp:///10.190.133.14:48296@61616 > 2014-04-15 10:51:01,755 | INFO | The connection to 'tcp:// > 10.190.133.14:47191' is taking a long time to shutdown. | > org.apache.activemq.broker.TransportConnection | ActiveMQ Transport: > tcp:///10.190.133.14:47985@61616 > 2014-04-15 10:51:01,935 | INFO | The connection to 'tcp:// > 10.190.133.14:46963' is taking a long time to shutdown. | > org.apache.activemq.broker.TransportConnection | ActiveMQ Transport: > tcp:///10.190.133.14:48308@61616 > 2014-04-15 10:51:02,362 | INFO | The connection to 'tcp:// > 10.190.133.15:54359' is taking a long time to shutdown. | > org.apache.activemq.broker.TransportConnection | ActiveMQ Transport: > tcp:///10.190.133.15:55261@61616 > > I also checked logs around time when stop processing messages. I saw some > warning messages. > > 2014-04-14 18:08:35,782 | WARN | Transport Connection to: > tcp://10.184.132.34:56916 failed: java.net.SocketException: Connection reset > | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ > Transport: tcp:///10.184.132.34:56916@61616 > > 2014-04-14 18:52:54,576 | WARN | Transport Connection to: > tcp://10.190.225.39:48556 failed: java.io.EOFException | > org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ > Transport: tcp:///10.190.225.39:48556@61616 > > 2014-04-14 19:43:35,398 | WARN | Transport Connection to: > tcp://10.190.225.39:49153 failed: java.io.EOFException | > org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ > Transport: tcp:///10.190.225.39:49153@61616 > > 2014-04-14 19:43:46,813 | WARN | Transport Connection to: > tcp://10.190.133.15:37178 failed: java.net.SocketException: Broken pipe | > org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ > InactivityMonitor Worker > > 2014-04-14 19:51:36,954 | WARN | Transport Connection to: > tcp://10.190.225.56:51240 failed: java.io.EOFException | > org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ > Transport: tcp:///10.190.225.56:51240@61616 > > 2014-04-14 20:00:27,651 | WARN | Transport Connection to: > tcp://10.190.225.72:53818 failed: java.io.EOFException | > org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ > Transport: tcp:///10.190.225.72:53818@61616 > > 2014-04-14 20:00:28,096 | WARN | Transport Connection to: > tcp://10.190.225.72:53676 failed: java.io.EOFException | > org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ > Transport: tcp:///10.190.225.72:53676@61616 > > 2014-04-14 20:00:28,097 | WARN | Transport Connection to: > tcp://10.190.225.72:53780 failed: java.io.EOFException | > org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ > Transport: tcp:///10.190.225.72:53780@61616 > > 2014-04-14 20:00:28,097 | WARN | Transport Connection to: > tcp://10.190.225.72:53718 failed: java.io.EOFException | > org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ > Transport: tcp:///10.190.225.72:53718@61616 > > 2014-04-14 20:00:28,098 | WARN | Transport Connection to: > tcp://10.190.225.72:53766 failed: java.io.EOFException | > org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ > Transport: tcp:///10.190.225.72:53766@61616 > > > We have zabbix monitor for activemq. I didnt see anything alarming for memory > usage or store usage. > > Enqueue and dequeue count were almost same most of time. Total message count > per minute was almost zero all the time. > > > I found this similar post and there was bug related to issue. but it was > fixed in 5.6 release. Does it mean it was not fixed or reintroduced due to > other code change. > > Post: > http://activemq.2283324.n4.nabble.com/Connection-taking-long-time-to-shutdown-td3717825.html > > Bug: https://issues.apache.org/jira/browse/AMQ-3435 > > > any suggestions. > > > thanks, > > chirag > > > > On Tue, Apr 15, 2014 at 9:06 PM, artnaseef <a...@artnaseef.com> wrote: > >> The most common cause of message flow stopping is slow consumption >> leading to >> exhaustion of the broker's resources. >> >> Things to check: >> - Broker memory usage >> - Broker store usage >> - ActiveMQ log file (check for producer flow control or resource usage >> messages) >> >> BTW, it is critical with any ActiveMQ installation to ensure consumption >> >= >> production always. Short bursts do happen and can be handled, but >> ultimately, you really never want to be in a situation in which >> consumption >> is less than production for a long time. And, when it is known that a >> burst >> of production will outpace consumption, it's important to understand the >> numbers and sizes of messages that will be in the backlog to ensure the >> broker's memory and store settings can handle it. >> >> >> >> >> -- >> View this message in context: >> http://activemq.2283324.n4.nabble.com/Activemq-froze-and-msgs-dont-get-processed-tp4680313p4680318.html >> Sent from the ActiveMQ - User mailing list archive at Nabble.com. >> > >
