[
https://issues.apache.org/jira/browse/STRATOS-1308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14483048#comment-14483048
]
Mariangela Hills commented on STRATOS-1308:
-------------------------------------------
I have labelled this JIRA as critical as this issue has been happening more
frequently when the connect is left idle. Once this issue occurs it is a must
to shut down the Stratos server (Ctrl+C) and start it again (./stratos.sh).
> Reconnection logic is missing the connection cleaning up part
> -------------------------------------------------------------
>
> Key: STRATOS-1308
> URL: https://issues.apache.org/jira/browse/STRATOS-1308
> Project: Stratos
> Issue Type: Bug
> Affects Versions: 4.1.0 Alpha
> Reporter: Mariangela Hills
> Priority: Critical
>
> I got a message saying "could not connect to the message broker" even when
> the message broker connection was already in place (started). There seems to
> be some reconnection logic, which finally leads to a too many files opened
> error (part of the log can be seen in [1]). When this happens the web browser
> on my machine (Chrome) gets stuck and I have to restart Stratos to solve the
> issue. In addition, I would like to add that my max file limit was 8000 when
> this issue occurred (ulimit -Sn 8000). Therefore, it looks like the
> reconnection logic is missing the connection cleaning up part.
> ---------------------
> [1]
> [2015-04-02 13:40:16,319] INFO
> {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} -
> Topic publisher will try to reconnect in 5 seconds: [topic-name]
> tenant.CompleteTenantEvent
> [2015-04-02 13:40:16,319] WARN
> {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} -
> Connection to the message broker failed
> [2015-04-02 13:40:16,319] INFO
> {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} -
> Topic publisher will try to reconnect in 2 seconds: [topic-name]
> tenant.CompleteTenantEvent
> Exception in thread "ActiveMQ Transport:
> tcp://localhost/127.0.0.1:61616@50628" java.lang.OutOfMemoryError: unable to
> create new native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Thread.java:673)
> at
> java.util.concurrent.ThreadPoolExecutor.addThread(ThreadPoolExecutor.java:681)
> at
> java.util.concurrent.ThreadPoolExecutor.addIfUnderCorePoolSize(ThreadPoolExecutor.java:706)
> at
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:650)
> at
> org.apache.activemq.ActiveMQConnection.onAsyncException(ActiveMQConnection.java:2002)
> at
> org.apache.activemq.ActiveMQConnection.onException(ActiveMQConnection.java:2017)[2015-04-02
> 13:40:16,423] ERROR
> {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} -
> Could not connect to message broker
> javax.jms.JMSException: Cannot send, channel has already failed:
> tcp://127.0.0.1:61616
> at
> org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:72)
> at
> org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1435)
> at
> org.apache.activemq.ActiveMQConnection.ensureConnectionInfoSent(ActiveMQConnection.java:1522)
> at
> org.apache.activemq.ActiveMQConnection.start(ActiveMQConnection.java:527)
> at
> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.connect(AmqpTopicConnector.java:78)
> at
> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:115)
> at
> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75)
> at
> org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> at java.lang.Thread.run(Thread.java:695)
> Caused by: org.apache.activemq.transport.InactivityIOException: Cannot send,
> channel has already failed: tcp://127.0.0.1:61616
> at
> org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:297)
> at
> org.apache.activemq.transport.AbstractInactivityMonitor.oneway(AbstractInactivityMonitor.java:286)
> at
> org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:85)
> at
> org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:104)
> at
> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
> at
> org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:81)
> at org.apache.activemq.transport.ResponseCorrelator.
> request(ResponseCorrelator.java:86)
> at
> org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1406)
> ... 9 more
> at
> org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:101)
> at
> org.apache.activemq.transport.ResponseCorrelator.onException(ResponseCorrelator.java:126)
> at
> org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:101)
> at
> org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:101)
> at
> org.apache.activemq.transport.WireFormatNegotiator.onException(WireFormatNegotiator.java:160)
> at
> org.apache.activemq.transport.AbstractInactivityMonitor.onException(AbstractInactivityMonitor.java:314)
> at
> org.apache.activemq.transport.TransportSupport.onException(TransportSupport.java:96)
> at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:200)
> at java.lang.Thread.run(Thread.java:695)
> [2015-04-02 13:40:16,424] WARN
> {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} -
> Could not reconnect to message broker
> org.apache.stratos.messaging.domain.exception.MessagingException: Could not
> connect to message broker
> at
> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.connect(AmqpTopicConnector.java:82)
> at
> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:115)
> at
> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75)
> at
> org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> at java.lang.Thread.run(Thread.java:695)
> Caused by: javax.jms.JMSException: Cannot send, channel has already failed:
> tcp://127.0.0.1:61616
> at
> org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:72)
> at
> org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1435)
> at
> org.apache.activemq.ActiveMQConnection.ensureConnectionInfoSent(ActiveMQConnection.java:1522)
> at
> org.apache.activemq.ActiveMQConnection.start(ActiveMQConnection.java:527)
> at
> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.connect(AmqpTopicConnector.java:78)
> ... 6 more
> Caused by: org.apache.activemq.transport.InactivityIOException: Cannot send,
> channel has already failed: tcp://127.0.0.1:61616
> at
> org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:297)
> at
> org.apache.activemq.transport.AbstractInactivityMonitor.oneway(AbstractInactivityMonitor.java:286)
> at
> org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:85)
> at
> org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:104)
> at
> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
> at
> org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:81)
> at
> org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:86)
> at
> org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1406)
> ... 9 more
> [2015-04-02 13:40:16,424] ERROR
> {org.apache.stratos.messaging.util.MessagingUtil} - Failed to load
> properties from file:
> /Users/Admin16/Documents/Stratos_4.1.0/master-pack/apache-stratos-4.1.0-SNAPSHOT/repository/conf/jndi.properties
> java.io.FileNotFoundException:
> /Users/Admin16/Documents/Stratos_4.1.0/master-pack/apache-stratos-4.1.0-SNAPSHOT/repository/conf/jndi.properties
> (Too many open files in system)
> at java.io.FileInputStream.open(Native Method)
> at java.io.FileInputStream.<init>(FileInputStream.java:120)
> at
> org.apache.stratos.messaging.util.MessagingUtil.getProperties(MessagingUtil.java:108)
> at
> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.create(AmqpTopicConnector.java:49)
> at
> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:114)
> at
> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75)
> at
> org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> at java.lang.Thread.run(Thread.java:695)
> [2015-04-02 13:40:16,425] INFO
> {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} -
> Topic publisher will try to reconnect in 20 seconds: [topic-name]
> application.signup.CompleteApplicationSignUpsEvent
> [2015-04-02 13:40:16,427] ERROR
> {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} -
> Could not create topic connector
> java.lang.NullPointerException
> at
> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.create(AmqpTopicConnector.java:53)
> at
> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:114)
> at
> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75)
> at
> org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> at java.lang.Thread.run(Thread.java:695)
> [2015-04-02 13:40:16,427] WARN
> {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} -
> Could not reconnect to message broker
> org.apache.stratos.messaging.domain.exception.MessagingException: Could not
> create topic connector
> at
> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.create(AmqpTopicConnector.java:58)
> at
> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:114)
> at
> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75)
> at
> org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> at java.lang.Thread.run(Thread.java:695)
> Caused by: java.lang.NullPointerException
> at
> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.create(AmqpTopicConnector.java:53)
> ... 6 more
> [2015-04-02 13:40:16,428] INFO
> {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} -
> Topic publisher will try to reconnect in 20 seconds: [topic-name]
> application.signup.CompleteApplicationSignUpsEvent
> [2015-04-02 13:40:16,445] ERROR
> {org.apache.stratos.messaging.util.MessagingUtil} - Failed to load
> properties from file:
> /Users/Admin16/Documents/Stratos_4.1.0/master-pack/apache-stratos-4.1.0-SNAPSHOT/repository/conf/jndi.properties
> java.io.FileNotFoundException:
> /Users/Admin16/Documents/Stratos_4.1.0/master-pack/apache-stratos-4.1.0-SNAPSHOT/repository/conf/jndi.properties
> (Too many open files in system)
> at java.io.FileInputStream.open(Native Method)
> at java.io.FileInputStream.<init>(FileInputStream.java:120)
> at
> org.apache.stratos.messaging.util.MessagingUtil.getProperties(MessagingUtil.java:108)
> at
> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.create(AmqpTopicConnector.java:49)
> at
> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:114)
> at
> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75)
> at
> org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> at java.lang.Thread.run(Thread.java:695)
> [2015-04-02 13:40:16,448] ERROR
> {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} -
> Could not create topic connector
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
