I have created a JIRA [1] to track this issue! [1] https://issues.apache.org/jira/browse/STRATOS-1308
Regards, Mariangela *--* Mariangela Hills Senior Technical Writer *WSO2, Inc.*lean.enterprise.middleware. m: +94 773 500185 w: http://wso2.com <http://wso2.com/events/> On Thu, Apr 2, 2015 at 3:43 PM, Mariangela Hills <[email protected]> wrote: > Thanks, Udara and Isuru for your quick response! > > @Udara - > > My file limit was 4000 and even after increasing it to 8000 I face the > same issue: > > ulimit -Sn 8000 > > Regards, > Mariangela > > > > > *--* > Mariangela Hills > Senior Technical Writer > > *WSO2, Inc.*lean.enterprise.middleware. > > > On Thu, Apr 2, 2015 at 3:29 PM, Isuru Haththotuwa <[email protected]> > wrote: > >> Hi Mari, >> >> Good catch. Yes, the reconnection logic seems to be missing the >> connection cleaning up part, in case an error occurs in connecting to the >> broker. Will create a Jira to track this. >> >> On Thu, Apr 2, 2015 at 3:13 PM, Mariangela Hills <[email protected]> >> wrote: >> >>> Hi Devs, >>> >>> I got a message saying "could not connect to the message broker" even >>> when the message broker connection was already in place (started). There >>> seems to be some reconnection logic, which finally leads to a too many >>> files opened error (part of the log can be seen in [1]). When this happens >>> the web browser on my machine (Chrome) gets stuck and I have to restart >>> Stratos to solve the issue. Any idea how I can over come this issue? I have >>> a feeling that this issue crops up when Stratos has been started and left >>> idle for a considerable duration (i.e., 1 hr 30min). >>> >>> >>> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- >>> >>> [1] >>> [2015-04-02 13:40:16,319] INFO >>> {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - >>> Topic publisher will try to reconnect in 5 seconds: [topic-name] >>> tenant.CompleteTenantEvent >>> [2015-04-02 13:40:16,319] WARN >>> {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - >>> Connection to the message broker failed >>> [2015-04-02 13:40:16,319] INFO >>> {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - >>> Topic publisher will try to reconnect in 2 seconds: [topic-name] >>> tenant.CompleteTenantEvent >>> Exception in thread "ActiveMQ Transport: >>> tcp://localhost/127.0.0.1:61616@50628" >>> java.lang.OutOfMemoryError: unable to create new native thread >>> at java.lang.Thread.start0(Native Method) >>> at java.lang.Thread.start(Thread.java:673) >>> at >>> java.util.concurrent.ThreadPoolExecutor.addThread(ThreadPoolExecutor.java:681) >>> at >>> java.util.concurrent.ThreadPoolExecutor.addIfUnderCorePoolSize(ThreadPoolExecutor.java:706) >>> at >>> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:650) >>> at >>> org.apache.activemq.ActiveMQConnection.onAsyncException(ActiveMQConnection.java:2002) >>> at >>> org.apache.activemq.ActiveMQConnection.onException(ActiveMQConnection.java:2017)[2015-04-02 >>> 13:40:16,423] ERROR >>> {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - >>> Could not connect to message broker >>> javax.jms.JMSException: Cannot send, channel has already failed: tcp:// >>> 127.0.0.1:61616 >>> at >>> org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:72) >>> at >>> org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1435) >>> at >>> org.apache.activemq.ActiveMQConnection.ensureConnectionInfoSent(ActiveMQConnection.java:1522) >>> at >>> org.apache.activemq.ActiveMQConnection.start(ActiveMQConnection.java:527) >>> at >>> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.connect(AmqpTopicConnector.java:78) >>> at >>> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:115) >>> at >>> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75) >>> at >>> org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) >>> at java.lang.Thread.run(Thread.java:695) >>> Caused by: org.apache.activemq.transport.InactivityIOException: Cannot >>> send, channel has already failed: tcp://127.0.0.1:61616 >>> at >>> org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:297) >>> at >>> org.apache.activemq.transport.AbstractInactivityMonitor.oneway(AbstractInactivityMonitor.java:286) >>> at >>> org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:85) >>> at >>> org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:104) >>> at >>> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68) >>> at >>> org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:81) >>> at org.apache.activemq.transport.ResponseCorrelator. >>> request(ResponseCorrelator.java:86) >>> at >>> org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1406) >>> ... 9 more >>> at >>> org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:101) >>> at >>> org.apache.activemq.transport.ResponseCorrelator.onException(ResponseCorrelator.java:126) >>> at >>> org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:101) >>> at >>> org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:101) >>> at >>> org.apache.activemq.transport.WireFormatNegotiator.onException(WireFormatNegotiator.java:160) >>> at >>> org.apache.activemq.transport.AbstractInactivityMonitor.onException(AbstractInactivityMonitor.java:314) >>> at >>> org.apache.activemq.transport.TransportSupport.onException(TransportSupport.java:96) >>> at >>> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:200) >>> at java.lang.Thread.run(Thread.java:695) >>> [2015-04-02 13:40:16,424] WARN >>> {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - >>> Could not reconnect to message broker >>> org.apache.stratos.messaging.domain.exception.MessagingException: Could >>> not connect to message broker >>> at >>> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.connect(AmqpTopicConnector.java:82) >>> at >>> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:115) >>> at >>> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75) >>> at >>> org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) >>> at java.lang.Thread.run(Thread.java:695) >>> Caused by: javax.jms.JMSException: Cannot send, channel has already >>> failed: tcp://127.0.0.1:61616 >>> at >>> org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:72) >>> at >>> org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1435) >>> at >>> org.apache.activemq.ActiveMQConnection.ensureConnectionInfoSent(ActiveMQConnection.java:1522) >>> at >>> org.apache.activemq.ActiveMQConnection.start(ActiveMQConnection.java:527) >>> at >>> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.connect(AmqpTopicConnector.java:78) >>> ... 6 more >>> Caused by: org.apache.activemq.transport.InactivityIOException: Cannot >>> send, channel has already failed: tcp://127.0.0.1:61616 >>> at >>> org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:297) >>> at >>> org.apache.activemq.transport.AbstractInactivityMonitor.oneway(AbstractInactivityMonitor.java:286) >>> at >>> org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:85) >>> at >>> org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:104) >>> at >>> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68) >>> at >>> org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:81) >>> at >>> org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:86) >>> at >>> org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1406) >>> ... 9 more >>> [2015-04-02 13:40:16,424] ERROR >>> {org.apache.stratos.messaging.util.MessagingUtil} - Failed to load >>> properties from file: >>> /Users/Admin16/Documents/Stratos_4.1.0/master-pack/apache-stratos-4.1.0-SNAPSHOT/repository/conf/jndi.properties >>> java.io.FileNotFoundException: >>> /Users/Admin16/Documents/Stratos_4.1.0/master-pack/apache-stratos-4.1.0-SNAPSHOT/repository/conf/jndi.properties >>> (Too many open files in system) >>> at java.io.FileInputStream.open(Native Method) >>> at java.io.FileInputStream.<init>(FileInputStream.java:120) >>> at >>> org.apache.stratos.messaging.util.MessagingUtil.getProperties(MessagingUtil.java:108) >>> at >>> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.create(AmqpTopicConnector.java:49) >>> at >>> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:114) >>> at >>> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75) >>> at >>> org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) >>> at java.lang.Thread.run(Thread.java:695) >>> [2015-04-02 13:40:16,425] INFO >>> {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - >>> Topic publisher will try to reconnect in 20 seconds: [topic-name] >>> application.signup.CompleteApplicationSignUpsEvent >>> [2015-04-02 13:40:16,427] ERROR >>> {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - >>> Could not create topic connector >>> java.lang.NullPointerException >>> at >>> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.create(AmqpTopicConnector.java:53) >>> at >>> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:114) >>> at >>> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75) >>> at >>> org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) >>> at java.lang.Thread.run(Thread.java:695) >>> [2015-04-02 13:40:16,427] WARN >>> {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - >>> Could not reconnect to message broker >>> org.apache.stratos.messaging.domain.exception.MessagingException: Could >>> not create topic connector >>> at >>> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.create(AmqpTopicConnector.java:58) >>> at >>> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:114) >>> at >>> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75) >>> at >>> org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) >>> at java.lang.Thread.run(Thread.java:695) >>> Caused by: java.lang.NullPointerException >>> at >>> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.create(AmqpTopicConnector.java:53) >>> ... 6 more >>> [2015-04-02 13:40:16,428] INFO >>> {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - >>> Topic publisher will try to reconnect in 20 seconds: [topic-name] >>> application.signup.CompleteApplicationSignUpsEvent >>> [2015-04-02 13:40:16,445] ERROR >>> {org.apache.stratos.messaging.util.MessagingUtil} - Failed to load >>> properties from file: >>> /Users/Admin16/Documents/Stratos_4.1.0/master-pack/apache-stratos-4.1.0-SNAPSHOT/repository/conf/jndi.properties >>> java.io.FileNotFoundException: >>> /Users/Admin16/Documents/Stratos_4.1.0/master-pack/apache-stratos-4.1.0-SNAPSHOT/repository/conf/jndi.properties >>> (Too many open files in system) >>> at java.io.FileInputStream.open(Native Method) >>> at java.io.FileInputStream.<init>(FileInputStream.java:120) >>> at >>> org.apache.stratos.messaging.util.MessagingUtil.getProperties(MessagingUtil.java:108) >>> at >>> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.create(AmqpTopicConnector.java:49) >>> at >>> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:114) >>> at >>> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75) >>> at >>> org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) >>> at java.lang.Thread.run(Thread.java:695) >>> [2015-04-02 13:40:16,448] ERROR >>> {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - >>> Could not create topic connector >>> >>> >>> Regards, >>> Mariangela >>> >>> >>> >>> >>> *--* >>> Mariangela Hills >>> Senior Technical Writer >>> >>> *WSO2, Inc.*lean.enterprise.middleware. >>> >>> -- >>> Thanks and Regards, >>> >>> Isuru H. >>> +94 716 358 048* <http://wso2.com/>* >>> >>> >>> * <http://wso2.com/>* >>> >>> >>> >
