Thanks, Udara and Isuru for your quick response! @Udara -
My file limit was 4000 and even after increasing it to 8000 I face the same issue: ulimit -Sn 8000 Regards, Mariangela *--* Mariangela Hills Senior Technical Writer *WSO2, Inc.*lean.enterprise.middleware. On Thu, Apr 2, 2015 at 3:29 PM, Isuru Haththotuwa <[email protected]> wrote: > Hi Mari, > > Good catch. Yes, the reconnection logic seems to be missing the connection > cleaning up part, in case an error occurs in connecting to the broker. Will > create a Jira to track this. > > On Thu, Apr 2, 2015 at 3:13 PM, Mariangela Hills <[email protected]> > wrote: > >> Hi Devs, >> >> I got a message saying "could not connect to the message broker" even >> when the message broker connection was already in place (started). There >> seems to be some reconnection logic, which finally leads to a too many >> files opened error (part of the log can be seen in [1]). When this happens >> the web browser on my machine (Chrome) gets stuck and I have to restart >> Stratos to solve the issue. Any idea how I can over come this issue? I have >> a feeling that this issue crops up when Stratos has been started and left >> idle for a considerable duration (i.e., 1 hr 30min). >> >> >> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- >> >> [1] >> [2015-04-02 13:40:16,319] INFO >> {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - >> Topic publisher will try to reconnect in 5 seconds: [topic-name] >> tenant.CompleteTenantEvent >> [2015-04-02 13:40:16,319] WARN >> {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - >> Connection to the message broker failed >> [2015-04-02 13:40:16,319] INFO >> {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - >> Topic publisher will try to reconnect in 2 seconds: [topic-name] >> tenant.CompleteTenantEvent >> Exception in thread "ActiveMQ Transport: >> tcp://localhost/127.0.0.1:61616@50628" >> java.lang.OutOfMemoryError: unable to create new native thread >> at java.lang.Thread.start0(Native Method) >> at java.lang.Thread.start(Thread.java:673) >> at >> java.util.concurrent.ThreadPoolExecutor.addThread(ThreadPoolExecutor.java:681) >> at >> java.util.concurrent.ThreadPoolExecutor.addIfUnderCorePoolSize(ThreadPoolExecutor.java:706) >> at >> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:650) >> at >> org.apache.activemq.ActiveMQConnection.onAsyncException(ActiveMQConnection.java:2002) >> at >> org.apache.activemq.ActiveMQConnection.onException(ActiveMQConnection.java:2017)[2015-04-02 >> 13:40:16,423] ERROR >> {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - >> Could not connect to message broker >> javax.jms.JMSException: Cannot send, channel has already failed: tcp:// >> 127.0.0.1:61616 >> at >> org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:72) >> at >> org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1435) >> at >> org.apache.activemq.ActiveMQConnection.ensureConnectionInfoSent(ActiveMQConnection.java:1522) >> at >> org.apache.activemq.ActiveMQConnection.start(ActiveMQConnection.java:527) >> at >> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.connect(AmqpTopicConnector.java:78) >> at >> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:115) >> at >> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75) >> at >> org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) >> at java.lang.Thread.run(Thread.java:695) >> Caused by: org.apache.activemq.transport.InactivityIOException: Cannot >> send, channel has already failed: tcp://127.0.0.1:61616 >> at >> org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:297) >> at >> org.apache.activemq.transport.AbstractInactivityMonitor.oneway(AbstractInactivityMonitor.java:286) >> at >> org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:85) >> at >> org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:104) >> at >> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68) >> at >> org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:81) >> at org.apache.activemq.transport.ResponseCorrelator. >> request(ResponseCorrelator.java:86) >> at >> org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1406) >> ... 9 more >> at >> org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:101) >> at >> org.apache.activemq.transport.ResponseCorrelator.onException(ResponseCorrelator.java:126) >> at >> org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:101) >> at >> org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:101) >> at >> org.apache.activemq.transport.WireFormatNegotiator.onException(WireFormatNegotiator.java:160) >> at >> org.apache.activemq.transport.AbstractInactivityMonitor.onException(AbstractInactivityMonitor.java:314) >> at >> org.apache.activemq.transport.TransportSupport.onException(TransportSupport.java:96) >> at >> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:200) >> at java.lang.Thread.run(Thread.java:695) >> [2015-04-02 13:40:16,424] WARN >> {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - >> Could not reconnect to message broker >> org.apache.stratos.messaging.domain.exception.MessagingException: Could >> not connect to message broker >> at >> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.connect(AmqpTopicConnector.java:82) >> at >> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:115) >> at >> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75) >> at >> org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) >> at java.lang.Thread.run(Thread.java:695) >> Caused by: javax.jms.JMSException: Cannot send, channel has already >> failed: tcp://127.0.0.1:61616 >> at >> org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:72) >> at >> org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1435) >> at >> org.apache.activemq.ActiveMQConnection.ensureConnectionInfoSent(ActiveMQConnection.java:1522) >> at >> org.apache.activemq.ActiveMQConnection.start(ActiveMQConnection.java:527) >> at >> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.connect(AmqpTopicConnector.java:78) >> ... 6 more >> Caused by: org.apache.activemq.transport.InactivityIOException: Cannot >> send, channel has already failed: tcp://127.0.0.1:61616 >> at >> org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:297) >> at >> org.apache.activemq.transport.AbstractInactivityMonitor.oneway(AbstractInactivityMonitor.java:286) >> at >> org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:85) >> at >> org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:104) >> at >> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68) >> at >> org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:81) >> at >> org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:86) >> at >> org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1406) >> ... 9 more >> [2015-04-02 13:40:16,424] ERROR >> {org.apache.stratos.messaging.util.MessagingUtil} - Failed to load >> properties from file: >> /Users/Admin16/Documents/Stratos_4.1.0/master-pack/apache-stratos-4.1.0-SNAPSHOT/repository/conf/jndi.properties >> java.io.FileNotFoundException: >> /Users/Admin16/Documents/Stratos_4.1.0/master-pack/apache-stratos-4.1.0-SNAPSHOT/repository/conf/jndi.properties >> (Too many open files in system) >> at java.io.FileInputStream.open(Native Method) >> at java.io.FileInputStream.<init>(FileInputStream.java:120) >> at >> org.apache.stratos.messaging.util.MessagingUtil.getProperties(MessagingUtil.java:108) >> at >> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.create(AmqpTopicConnector.java:49) >> at >> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:114) >> at >> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75) >> at >> org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) >> at java.lang.Thread.run(Thread.java:695) >> [2015-04-02 13:40:16,425] INFO >> {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - >> Topic publisher will try to reconnect in 20 seconds: [topic-name] >> application.signup.CompleteApplicationSignUpsEvent >> [2015-04-02 13:40:16,427] ERROR >> {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - >> Could not create topic connector >> java.lang.NullPointerException >> at >> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.create(AmqpTopicConnector.java:53) >> at >> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:114) >> at >> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75) >> at >> org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) >> at java.lang.Thread.run(Thread.java:695) >> [2015-04-02 13:40:16,427] WARN >> {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - >> Could not reconnect to message broker >> org.apache.stratos.messaging.domain.exception.MessagingException: Could >> not create topic connector >> at >> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.create(AmqpTopicConnector.java:58) >> at >> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:114) >> at >> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75) >> at >> org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) >> at java.lang.Thread.run(Thread.java:695) >> Caused by: java.lang.NullPointerException >> at >> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.create(AmqpTopicConnector.java:53) >> ... 6 more >> [2015-04-02 13:40:16,428] INFO >> {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - >> Topic publisher will try to reconnect in 20 seconds: [topic-name] >> application.signup.CompleteApplicationSignUpsEvent >> [2015-04-02 13:40:16,445] ERROR >> {org.apache.stratos.messaging.util.MessagingUtil} - Failed to load >> properties from file: >> /Users/Admin16/Documents/Stratos_4.1.0/master-pack/apache-stratos-4.1.0-SNAPSHOT/repository/conf/jndi.properties >> java.io.FileNotFoundException: >> /Users/Admin16/Documents/Stratos_4.1.0/master-pack/apache-stratos-4.1.0-SNAPSHOT/repository/conf/jndi.properties >> (Too many open files in system) >> at java.io.FileInputStream.open(Native Method) >> at java.io.FileInputStream.<init>(FileInputStream.java:120) >> at >> org.apache.stratos.messaging.util.MessagingUtil.getProperties(MessagingUtil.java:108) >> at >> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.create(AmqpTopicConnector.java:49) >> at >> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:114) >> at >> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75) >> at >> org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) >> at java.lang.Thread.run(Thread.java:695) >> [2015-04-02 13:40:16,448] ERROR >> {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - >> Could not create topic connector >> >> >> Regards, >> Mariangela >> >> >> >> >> *--* >> Mariangela Hills >> Senior Technical Writer >> >> *WSO2, Inc.*lean.enterprise.middleware. >> >> -- >> Thanks and Regards, >> >> Isuru H. >> +94 716 358 048* <http://wso2.com/>* >> >> >> * <http://wso2.com/>* >> >> >>
