I will look in to this Martin. On Wed, Apr 15, 2015 at 4:22 AM, Martin Eppel (meppel) <[email protected]> wrote:
> I have been out for a few days so I am not sure on the status of this > issue but I rerun the test (leaving stratos running idle) for a few hours > with the latest commits from today, April 14 > (commit 7b09bc1534c5708a47a7b61ae0a83f96957e7eeb, Author: Chamila de Alwis > [email protected], Date: Tue Apr 14 21:11:22 2015 +0530) > > and I still see the same issue happening, > > > > Thanks > > > > Martin > > > > Exceptions are (after letting it run for a few hours) : > > > > TID: [0] [STRATOS] [2015-04-14 15:37:21,331] WARN > {org.apache.thrift.server.TThreadPoolServer} - Transport error occurred > during acceptance of message. > > org.apache.thrift.transport.TTransportException: java.net.SocketException: > Too many open files > > at > org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:118) > > at > org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:35) > > at > org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31) > > at > org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:106) > > at > org.wso2.carbon.databridge.receiver.thrift.internal.ThriftDataReceiver$ServerThread.run(ThriftDataReceiver.java:199) > > at java.lang.Thread.run(Thread.java:745) > > Caused by: java.net.SocketException: Too many open files > > at java.net.PlainSocketImpl.socketAccept(Native Method) > > at > java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) > > at java.net.ServerSocket.implAccept(ServerSocket.java:530) > > at java.net.ServerSocket.accept(ServerSocket.java:498) > > at > org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:113) > > > > Btw, this is easily reproducible by just starting up stratos and let it > run idle for a few hours, > > > > Regards > > > > Martin > > > > > > *From:* Martin Eppel (meppel) > *Sent:* Thursday, April 02, 2015 5:45 PM > *To:* [email protected] > *Cc:* Udara Liyanage; Isuru Haththotuwa > *Subject:* RE: [Error] Too many files open error when running Stratos > > > > Hi Mariangela, > > > > Here is an update – seems this is a different issue as STRATOS-723. > > After letting the system run for a few hours with the below mentioned > commit (8985d96eb811aa8e9ce2c114f1856b4c4e20517b) included. > > I still see the same issue as you reported (with exception as below). The > stratos process is still running, no OOM yet but logging has stopped. > > > > I think this is an issue which needs further investigation, > > > > Regards > > > > Martin > > > > WARN {org.apache.thrift.server.TThreadPoolServer} - Transport error > occurred during acceptance of message. > > org.apache.thrift.transport.TTransportException: java.net.SocketException: > Too many open files > > at > org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:118) > > at > org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:35) > > at > org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31) > > at > org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:106) > > at > org.wso2.carbon.databridge.receiver.thrift.internal.ThriftDataReceiver$ServerThread.run(ThriftDataReceiver.java:199) > > > > *From:* Martin Eppel (meppel) > *Sent:* Thursday, April 02, 2015 1:36 PM > *To:* [email protected] > *Cc:* Udara Liyanage; Isuru Haththotuwa > *Subject:* RE: [Error] Too many files open error when running Stratos > > > > Hi Mariangela, > > > > As an FYI, this was possibly caused by > https://issues.apache.org/jira/browse/STRATOS-723. > > > > A fix was just checked in for this issue to 4.1.0, see commit > 8985d96eb811aa8e9ce2c114f1856b4c4e20517b. > > > > Regards > > > > Martin > > > > > > > > *From:* Mariangela Hills [mailto:[email protected] <[email protected]>] > > *Sent:* Thursday, April 02, 2015 3:22 AM > *To:* dev > *Cc:* Udara Liyanage; Isuru Haththotuwa > *Subject:* Re: [Error] Too many files open error when running Stratos > > > > I have created a JIRA [1] to track this issue! > > > > [1] https://issues.apache.org/jira/browse/STRATOS-1308 > > > Regards, > Mariangela > > *--* > > Mariangela Hills > > Senior Technical Writer > > > *WSO2, Inc. *lean.enterprise.middleware. > > m: +94 773 500185 > w: http://wso2.com > > [image: http://c.content.wso2.com/signatures/general.png] > <http://wso2.com/events/> > > > > On Thu, Apr 2, 2015 at 3:43 PM, Mariangela Hills <[email protected]> > wrote: > > Thanks, Udara and Isuru for your quick response! > > > > @Udara - > > > > My file limit was 4000 and even after increasing it to 8000 I face the > same issue: > > > > ulimit -Sn 8000 > > > Regards, > Mariangela > > *--* > > Mariangela Hills > > Senior Technical Writer > > > *WSO2, Inc. *lean.enterprise.middleware. > > > > > > On Thu, Apr 2, 2015 at 3:29 PM, Isuru Haththotuwa <[email protected]> > wrote: > > Hi Mari, > > Good catch. Yes, the reconnection logic seems to be missing the connection > cleaning up part, in case an error occurs in connecting to the broker. Will > create a Jira to track this. > > > > On Thu, Apr 2, 2015 at 3:13 PM, Mariangela Hills <[email protected]> > wrote: > > Hi Devs, > > > > I got a message saying "could not connect to the message broker" even when > the message broker connection was already in place (started). There seems > to be some reconnection logic, which finally leads to a too many files > opened error (part of the log can be seen in [1]). When this happens the > web browser on my machine (Chrome) gets stuck and I have to restart Stratos > to solve the issue. Any idea how I can over come this issue? I have a > feeling that this issue crops up when Stratos has been started and left > idle for a considerable duration (i.e., 1 hr 30min). > > > > > -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > > > > [1] > > [2015-04-02 13:40:16,319] INFO > {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - > Topic publisher will try to reconnect in 5 seconds: [topic-name] > tenant.CompleteTenantEvent > > [2015-04-02 13:40:16,319] WARN > {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - > Connection to the message broker failed > > [2015-04-02 13:40:16,319] INFO > {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - > Topic publisher will try to reconnect in 2 seconds: [topic-name] > tenant.CompleteTenantEvent > > Exception in thread "ActiveMQ Transport: > tcp://localhost/127.0.0.1:61616@50628" > java.lang.OutOfMemoryError: unable to create new native thread > > at java.lang.Thread.start0(Native Method) > > at java.lang.Thread.start(Thread.java:673) > > at > java.util.concurrent.ThreadPoolExecutor.addThread(ThreadPoolExecutor.java:681) > > at > java.util.concurrent.ThreadPoolExecutor.addIfUnderCorePoolSize(ThreadPoolExecutor.java:706) > > at > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:650) > > at > org.apache.activemq.ActiveMQConnection.onAsyncException(ActiveMQConnection.java:2002) > > at > org.apache.activemq.ActiveMQConnection.onException(ActiveMQConnection.java:2017)[2015-04-02 > 13:40:16,423] ERROR > {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - > Could not connect to message broker > > javax.jms.JMSException: Cannot send, channel has already failed: tcp:// > 127.0.0.1:61616 > > at > org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:72) > > at > org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1435) > > at > org.apache.activemq.ActiveMQConnection.ensureConnectionInfoSent(ActiveMQConnection.java:1522) > > at > org.apache.activemq.ActiveMQConnection.start(ActiveMQConnection.java:527) > > at > org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.connect(AmqpTopicConnector.java:78) > > at > org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:115) > > at > org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75) > > at > org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) > > at java.lang.Thread.run(Thread.java:695) > > Caused by: org.apache.activemq.transport.InactivityIOException: Cannot > send, channel has already failed: tcp://127.0.0.1:61616 > > at > org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:297) > > at > org.apache.activemq.transport.AbstractInactivityMonitor.oneway(AbstractInactivityMonitor.java:286) > > at > org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:85) > > at > org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:104) > > at > org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68) > > at > org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:81) > > at org.apache.activemq.transport.ResponseCorrelator. > > request(ResponseCorrelator.java:86) > > at > org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1406) > > ... 9 more > > at > org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:101) > > at > org.apache.activemq.transport.ResponseCorrelator.onException(ResponseCorrelator.java:126) > > at > org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:101) > > at > org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:101) > > at > org.apache.activemq.transport.WireFormatNegotiator.onException(WireFormatNegotiator.java:160) > > at > org.apache.activemq.transport.AbstractInactivityMonitor.onException(AbstractInactivityMonitor.java:314) > > at > org.apache.activemq.transport.TransportSupport.onException(TransportSupport.java:96) > > at > org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:200) > > at java.lang.Thread.run(Thread.java:695) > > [2015-04-02 13:40:16,424] WARN > {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - > Could not reconnect to message broker > > org.apache.stratos.messaging.domain.exception.MessagingException: Could > not connect to message broker > > at > org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.connect(AmqpTopicConnector.java:82) > > at > org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:115) > > at > org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75) > > at > org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) > > at java.lang.Thread.run(Thread.java:695) > > Caused by: javax.jms.JMSException: Cannot send, channel has already > failed: tcp://127.0.0.1:61616 > > at > org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:72) > > at > org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1435) > > at > org.apache.activemq.ActiveMQConnection.ensureConnectionInfoSent(ActiveMQConnection.java:1522) > > at > org.apache.activemq.ActiveMQConnection.start(ActiveMQConnection.java:527) > > at > org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.connect(AmqpTopicConnector.java:78) > > ... 6 more > > Caused by: org.apache.activemq.transport.InactivityIOException: Cannot > send, channel has already failed: tcp://127.0.0.1:61616 > > at > org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:297) > > at > org.apache.activemq.transport.AbstractInactivityMonitor.oneway(AbstractInactivityMonitor.java:286) > > at > org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:85) > > at > org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:104) > > at > org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68) > > at > org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:81) > > at > org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:86) > > at > org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1406) > > ... 9 more > > [2015-04-02 13:40:16,424] ERROR > {org.apache.stratos.messaging.util.MessagingUtil} - Failed to load > properties from file: > /Users/Admin16/Documents/Stratos_4.1.0/master-pack/apache-stratos-4.1.0-SNAPSHOT/repository/conf/jndi.properties > > java.io.FileNotFoundException: > /Users/Admin16/Documents/Stratos_4.1.0/master-pack/apache-stratos-4.1.0-SNAPSHOT/repository/conf/jndi.properties > (Too many open files in system) > > at java.io.FileInputStream.open(Native Method) > > at java.io.FileInputStream.<init>(FileInputStream.java:120) > > at > org.apache.stratos.messaging.util.MessagingUtil.getProperties(MessagingUtil.java:108) > > at > org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.create(AmqpTopicConnector.java:49) > > at > org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:114) > > at > org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75) > > at > org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) > > at java.lang.Thread.run(Thread.java:695) > > [2015-04-02 13:40:16,425] INFO > {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - > Topic publisher will try to reconnect in 20 seconds: [topic-name] > application.signup.CompleteApplicationSignUpsEvent > > [2015-04-02 13:40:16,427] ERROR > {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - > Could not create topic connector > > java.lang.NullPointerException > > at > org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.create(AmqpTopicConnector.java:53) > > at > org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:114) > > at > org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75) > > at > org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) > > at java.lang.Thread.run(Thread.java:695) > > [2015-04-02 13:40:16,427] WARN > {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - > Could not reconnect to message broker > > org.apache.stratos.messaging.domain.exception.MessagingException: Could > not create topic connector > > at > org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.create(AmqpTopicConnector.java:58) > > at > org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:114) > > at > org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75) > > at > org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) > > at java.lang.Thread.run(Thread.java:695) > > Caused by: java.lang.NullPointerException > > at > org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.create(AmqpTopicConnector.java:53) > > ... 6 more > > [2015-04-02 13:40:16,428] INFO > {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - > Topic publisher will try to reconnect in 20 seconds: [topic-name] > application.signup.CompleteApplicationSignUpsEvent > > [2015-04-02 13:40:16,445] ERROR > {org.apache.stratos.messaging.util.MessagingUtil} - Failed to load > properties from file: > /Users/Admin16/Documents/Stratos_4.1.0/master-pack/apache-stratos-4.1.0-SNAPSHOT/repository/conf/jndi.properties > > java.io.FileNotFoundException: > /Users/Admin16/Documents/Stratos_4.1.0/master-pack/apache-stratos-4.1.0-SNAPSHOT/repository/conf/jndi.properties > (Too many open files in system) > > at java.io.FileInputStream.open(Native Method) > > at java.io.FileInputStream.<init>(FileInputStream.java:120) > > at > org.apache.stratos.messaging.util.MessagingUtil.getProperties(MessagingUtil.java:108) > > at > org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.create(AmqpTopicConnector.java:49) > > at > org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:114) > > at > org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75) > > at > org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) > > at java.lang.Thread.run(Thread.java:695) > > [2015-04-02 13:40:16,448] ERROR > {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - > Could not create topic connector > > > > > Regards, > Mariangela > > *--* > > Mariangela Hills > > Senior Technical Writer > > > *WSO2, Inc. *lean.enterprise.middleware. > > -- > > Thanks and Regards, > > Isuru H. > > +94 716 358 048 > > -- > Thanks and Regards, > > Isuru H. > +94 716 358 048* <http://wso2.com/>* > > > * <http://wso2.com/>* > > >
