Hi Mariangela, As an FYI, this was possibly caused by https://issues.apache.org/jira/browse/STRATOS-723.
A fix was just checked in for this issue to 4.1.0, see commit 8985d96eb811aa8e9ce2c114f1856b4c4e20517b. Regards Martin From: Mariangela Hills [mailto:[email protected]] Sent: Thursday, April 02, 2015 3:22 AM To: dev Cc: Udara Liyanage; Isuru Haththotuwa Subject: Re: [Error] Too many files open error when running Stratos I have created a JIRA [1] to track this issue! [1] https://issues.apache.org/jira/browse/STRATOS-1308 Regards, Mariangela -- Mariangela Hills Senior Technical Writer WSO2, Inc. lean.enterprise.middleware. m: +94 773 500185 w: http://wso2.com<http://wso2.com/> [http://c.content.wso2.com/signatures/general.png]<http://wso2.com/events/> On Thu, Apr 2, 2015 at 3:43 PM, Mariangela Hills <[email protected]<mailto:[email protected]>> wrote: Thanks, Udara and Isuru for your quick response! @Udara - My file limit was 4000 and even after increasing it to 8000 I face the same issue: ulimit -Sn 8000 Regards, Mariangela -- Mariangela Hills Senior Technical Writer WSO2, Inc. lean.enterprise.middleware. On Thu, Apr 2, 2015 at 3:29 PM, Isuru Haththotuwa <[email protected]<mailto:[email protected]>> wrote: Hi Mari, Good catch. Yes, the reconnection logic seems to be missing the connection cleaning up part, in case an error occurs in connecting to the broker. Will create a Jira to track this. On Thu, Apr 2, 2015 at 3:13 PM, Mariangela Hills <[email protected]<mailto:[email protected]>> wrote: Hi Devs, I got a message saying "could not connect to the message broker" even when the message broker connection was already in place (started). There seems to be some reconnection logic, which finally leads to a too many files opened error (part of the log can be seen in [1]). When this happens the web browser on my machine (Chrome) gets stuck and I have to restart Stratos to solve the issue. Any idea how I can over come this issue? I have a feeling that this issue crops up when Stratos has been started and left idle for a considerable duration (i.e., 1 hr 30min). -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [1] [2015-04-02 13:40:16,319] INFO {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - Topic publisher will try to reconnect in 5 seconds: [topic-name] tenant.CompleteTenantEvent [2015-04-02 13:40:16,319] WARN {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - Connection to the message broker failed [2015-04-02 13:40:16,319] INFO {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - Topic publisher will try to reconnect in 2 seconds: [topic-name] tenant.CompleteTenantEvent Exception in thread "ActiveMQ Transport: tcp://localhost/127.0.0.1:61616@50628" java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:673) at java.util.concurrent.ThreadPoolExecutor.addThread(ThreadPoolExecutor.java:681) at java.util.concurrent.ThreadPoolExecutor.addIfUnderCorePoolSize(ThreadPoolExecutor.java:706) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:650) at org.apache.activemq.ActiveMQConnection.onAsyncException(ActiveMQConnection.java:2002) at org.apache.activemq.ActiveMQConnection.onException(ActiveMQConnection.java:2017)[2015-04-02 13:40:16,423] ERROR {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - Could not connect to message broker javax.jms.JMSException: Cannot send, channel has already failed: tcp://127.0.0.1:61616<http://127.0.0.1:61616> at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:72) at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1435) at org.apache.activemq.ActiveMQConnection.ensureConnectionInfoSent(ActiveMQConnection.java:1522) at org.apache.activemq.ActiveMQConnection.start(ActiveMQConnection.java:527) at org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.connect(AmqpTopicConnector.java:78) at org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:115) at org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75) at org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:695) Caused by: org.apache.activemq.transport.InactivityIOException: Cannot send, channel has already failed: tcp://127.0.0.1:61616<http://127.0.0.1:61616> at org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:297) at org.apache.activemq.transport.AbstractInactivityMonitor.oneway(AbstractInactivityMonitor.java:286) at org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:85) at org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:104) at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68) at org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:81) at org.apache.activemq.transport.ResponseCorrelator. request(ResponseCorrelator.java:86) at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1406) ... 9 more at org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:101) at org.apache.activemq.transport.ResponseCorrelator.onException(ResponseCorrelator.java:126) at org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:101) at org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:101) at org.apache.activemq.transport.WireFormatNegotiator.onException(WireFormatNegotiator.java:160) at org.apache.activemq.transport.AbstractInactivityMonitor.onException(AbstractInactivityMonitor.java:314) at org.apache.activemq.transport.TransportSupport.onException(TransportSupport.java:96) at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:200) at java.lang.Thread.run(Thread.java:695) [2015-04-02 13:40:16,424] WARN {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - Could not reconnect to message broker org.apache.stratos.messaging.domain.exception.MessagingException: Could not connect to message broker at org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.connect(AmqpTopicConnector.java:82) at org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:115) at org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75) at org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:695) Caused by: javax.jms.JMSException: Cannot send, channel has already failed: tcp://127.0.0.1:61616<http://127.0.0.1:61616> at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:72) at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1435) at org.apache.activemq.ActiveMQConnection.ensureConnectionInfoSent(ActiveMQConnection.java:1522) at org.apache.activemq.ActiveMQConnection.start(ActiveMQConnection.java:527) at org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.connect(AmqpTopicConnector.java:78) ... 6 more Caused by: org.apache.activemq.transport.InactivityIOException: Cannot send, channel has already failed: tcp://127.0.0.1:61616<http://127.0.0.1:61616> at org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:297) at org.apache.activemq.transport.AbstractInactivityMonitor.oneway(AbstractInactivityMonitor.java:286) at org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:85) at org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:104) at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68) at org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:81) at org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:86) at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1406) ... 9 more [2015-04-02 13:40:16,424] ERROR {org.apache.stratos.messaging.util.MessagingUtil} - Failed to load properties from file: /Users/Admin16/Documents/Stratos_4.1.0/master-pack/apache-stratos-4.1.0-SNAPSHOT/repository/conf/jndi.properties java.io.FileNotFoundException: /Users/Admin16/Documents/Stratos_4.1.0/master-pack/apache-stratos-4.1.0-SNAPSHOT/repository/conf/jndi.properties (Too many open files in system) at java.io.FileInputStream.open(Native Method) at java.io.FileInputStream.<init>(FileInputStream.java:120) at org.apache.stratos.messaging.util.MessagingUtil.getProperties(MessagingUtil.java:108) at org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.create(AmqpTopicConnector.java:49) at org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:114) at org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75) at org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:695) [2015-04-02 13:40:16,425] INFO {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - Topic publisher will try to reconnect in 20 seconds: [topic-name] application.signup.CompleteApplicationSignUpsEvent [2015-04-02 13:40:16,427] ERROR {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - Could not create topic connector java.lang.NullPointerException at org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.create(AmqpTopicConnector.java:53) at org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:114) at org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75) at org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:695) [2015-04-02 13:40:16,427] WARN {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - Could not reconnect to message broker org.apache.stratos.messaging.domain.exception.MessagingException: Could not create topic connector at org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.create(AmqpTopicConnector.java:58) at org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:114) at org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75) at org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:695) Caused by: java.lang.NullPointerException at org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.create(AmqpTopicConnector.java:53) ... 6 more [2015-04-02 13:40:16,428] INFO {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - Topic publisher will try to reconnect in 20 seconds: [topic-name] application.signup.CompleteApplicationSignUpsEvent [2015-04-02 13:40:16,445] ERROR {org.apache.stratos.messaging.util.MessagingUtil} - Failed to load properties from file: /Users/Admin16/Documents/Stratos_4.1.0/master-pack/apache-stratos-4.1.0-SNAPSHOT/repository/conf/jndi.properties java.io.FileNotFoundException: /Users/Admin16/Documents/Stratos_4.1.0/master-pack/apache-stratos-4.1.0-SNAPSHOT/repository/conf/jndi.properties (Too many open files in system) at java.io.FileInputStream.open(Native Method) at java.io.FileInputStream.<init>(FileInputStream.java:120) at org.apache.stratos.messaging.util.MessagingUtil.getProperties(MessagingUtil.java:108) at org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.create(AmqpTopicConnector.java:49) at org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:114) at org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75) at org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:695) [2015-04-02 13:40:16,448] ERROR {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - Could not create topic connector Regards, Mariangela -- Mariangela Hills Senior Technical Writer WSO2, Inc. lean.enterprise.middleware. -- Thanks and Regards, Isuru H. +94 716 358 048<tel:%2B94%20716%20358%20048>
