Hi, Quick solution is to increase max file limit from the OS.
However if this happens after a long idle time, it seems that there is a connection leak which needs to be investigated. On Thu, Apr 2, 2015 at 3:13 PM, Mariangela Hills <[email protected]> wrote: > Hi Devs, > > I got a message saying "could not connect to the message broker" even when > the message broker connection was already in place (started). There seems > to be some reconnection logic, which finally leads to a too many files > opened error (part of the log can be seen in [1]). When this happens the > web browser on my machine (Chrome) gets stuck and I have to restart Stratos > to solve the issue. Any idea how I can over come this issue? I have a > feeling that this issue crops up when Stratos has been started and left > idle for a considerable duration (i.e., 1 hr 30min). > > > -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > > [1] > [2015-04-02 13:40:16,319] INFO > {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - > Topic publisher will try to reconnect in 5 seconds: [topic-name] > tenant.CompleteTenantEvent > [2015-04-02 13:40:16,319] WARN > {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - > Connection to the message broker failed > [2015-04-02 13:40:16,319] INFO > {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - > Topic publisher will try to reconnect in 2 seconds: [topic-name] > tenant.CompleteTenantEvent > Exception in thread "ActiveMQ Transport: > tcp://localhost/127.0.0.1:61616@50628" > java.lang.OutOfMemoryError: unable to create new native thread > at java.lang.Thread.start0(Native Method) > at java.lang.Thread.start(Thread.java:673) > at > java.util.concurrent.ThreadPoolExecutor.addThread(ThreadPoolExecutor.java:681) > at > java.util.concurrent.ThreadPoolExecutor.addIfUnderCorePoolSize(ThreadPoolExecutor.java:706) > at > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:650) > at > org.apache.activemq.ActiveMQConnection.onAsyncException(ActiveMQConnection.java:2002) > at > org.apache.activemq.ActiveMQConnection.onException(ActiveMQConnection.java:2017)[2015-04-02 > 13:40:16,423] ERROR > {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - > Could not connect to message broker > javax.jms.JMSException: Cannot send, channel has already failed: tcp:// > 127.0.0.1:61616 > at > org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:72) > at > org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1435) > at > org.apache.activemq.ActiveMQConnection.ensureConnectionInfoSent(ActiveMQConnection.java:1522) > at > org.apache.activemq.ActiveMQConnection.start(ActiveMQConnection.java:527) > at > org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.connect(AmqpTopicConnector.java:78) > at > org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:115) > at > org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75) > at > org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) > at java.lang.Thread.run(Thread.java:695) > Caused by: org.apache.activemq.transport.InactivityIOException: Cannot > send, channel has already failed: tcp://127.0.0.1:61616 > at > org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:297) > at > org.apache.activemq.transport.AbstractInactivityMonitor.oneway(AbstractInactivityMonitor.java:286) > at > org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:85) > at > org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:104) > at > org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68) > at > org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:81) > at org.apache.activemq.transport.ResponseCorrelator. > request(ResponseCorrelator.java:86) > at > org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1406) > ... 9 more > at > org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:101) > at > org.apache.activemq.transport.ResponseCorrelator.onException(ResponseCorrelator.java:126) > at > org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:101) > at > org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:101) > at > org.apache.activemq.transport.WireFormatNegotiator.onException(WireFormatNegotiator.java:160) > at > org.apache.activemq.transport.AbstractInactivityMonitor.onException(AbstractInactivityMonitor.java:314) > at > org.apache.activemq.transport.TransportSupport.onException(TransportSupport.java:96) > at > org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:200) > at java.lang.Thread.run(Thread.java:695) > [2015-04-02 13:40:16,424] WARN > {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - > Could not reconnect to message broker > org.apache.stratos.messaging.domain.exception.MessagingException: Could > not connect to message broker > at > org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.connect(AmqpTopicConnector.java:82) > at > org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:115) > at > org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75) > at > org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) > at java.lang.Thread.run(Thread.java:695) > Caused by: javax.jms.JMSException: Cannot send, channel has already > failed: tcp://127.0.0.1:61616 > at > org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:72) > at > org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1435) > at > org.apache.activemq.ActiveMQConnection.ensureConnectionInfoSent(ActiveMQConnection.java:1522) > at > org.apache.activemq.ActiveMQConnection.start(ActiveMQConnection.java:527) > at > org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.connect(AmqpTopicConnector.java:78) > ... 6 more > Caused by: org.apache.activemq.transport.InactivityIOException: Cannot > send, channel has already failed: tcp://127.0.0.1:61616 > at > org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:297) > at > org.apache.activemq.transport.AbstractInactivityMonitor.oneway(AbstractInactivityMonitor.java:286) > at > org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:85) > at > org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:104) > at > org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68) > at > org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:81) > at > org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:86) > at > org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1406) > ... 9 more > [2015-04-02 13:40:16,424] ERROR > {org.apache.stratos.messaging.util.MessagingUtil} - Failed to load > properties from file: > /Users/Admin16/Documents/Stratos_4.1.0/master-pack/apache-stratos-4.1.0-SNAPSHOT/repository/conf/jndi.properties > java.io.FileNotFoundException: > /Users/Admin16/Documents/Stratos_4.1.0/master-pack/apache-stratos-4.1.0-SNAPSHOT/repository/conf/jndi.properties > (Too many open files in system) > at java.io.FileInputStream.open(Native Method) > at java.io.FileInputStream.<init>(FileInputStream.java:120) > at > org.apache.stratos.messaging.util.MessagingUtil.getProperties(MessagingUtil.java:108) > at > org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.create(AmqpTopicConnector.java:49) > at > org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:114) > at > org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75) > at > org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) > at java.lang.Thread.run(Thread.java:695) > [2015-04-02 13:40:16,425] INFO > {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - > Topic publisher will try to reconnect in 20 seconds: [topic-name] > application.signup.CompleteApplicationSignUpsEvent > [2015-04-02 13:40:16,427] ERROR > {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - > Could not create topic connector > java.lang.NullPointerException > at > org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.create(AmqpTopicConnector.java:53) > at > org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:114) > at > org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75) > at > org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) > at java.lang.Thread.run(Thread.java:695) > [2015-04-02 13:40:16,427] WARN > {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - > Could not reconnect to message broker > org.apache.stratos.messaging.domain.exception.MessagingException: Could > not create topic connector > at > org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.create(AmqpTopicConnector.java:58) > at > org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:114) > at > org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75) > at > org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) > at java.lang.Thread.run(Thread.java:695) > Caused by: java.lang.NullPointerException > at > org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.create(AmqpTopicConnector.java:53) > ... 6 more > [2015-04-02 13:40:16,428] INFO > {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - > Topic publisher will try to reconnect in 20 seconds: [topic-name] > application.signup.CompleteApplicationSignUpsEvent > [2015-04-02 13:40:16,445] ERROR > {org.apache.stratos.messaging.util.MessagingUtil} - Failed to load > properties from file: > /Users/Admin16/Documents/Stratos_4.1.0/master-pack/apache-stratos-4.1.0-SNAPSHOT/repository/conf/jndi.properties > java.io.FileNotFoundException: > /Users/Admin16/Documents/Stratos_4.1.0/master-pack/apache-stratos-4.1.0-SNAPSHOT/repository/conf/jndi.properties > (Too many open files in system) > at java.io.FileInputStream.open(Native Method) > at java.io.FileInputStream.<init>(FileInputStream.java:120) > at > org.apache.stratos.messaging.util.MessagingUtil.getProperties(MessagingUtil.java:108) > at > org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.create(AmqpTopicConnector.java:49) > at > org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:114) > at > org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75) > at > org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) > at java.lang.Thread.run(Thread.java:695) > [2015-04-02 13:40:16,448] ERROR > {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - > Could not create topic connector > > > Regards, > Mariangela > > > > > *--* > Mariangela Hills > Senior Technical Writer > > *WSO2, Inc.*lean.enterprise.middleware. > > -- Udara Liyanage Software Engineer WSO2, Inc.: http://wso2.com lean. enterprise. middleware web: http://udaraliyanage.wordpress.com phone: +94 71 443 6897
