Hi, Just as a side note, do we need to read the jndi.properties file every time the connection is made? Is there a particular requirement that needs to load changes to the jndi.properties file while Stratos is running? If not we can use the initial read to the properties and re-use it in subsequent connections. This does not however solve the original issue.
Regards, Chamila de Alwis Software Engineer | WSO2 | +94772207163 Blog: code.chamiladealwis.com On Wed, Apr 15, 2015 at 9:06 AM, Isuru Haththotuwa <[email protected]> wrote: > I will look in to this Martin. > > On Wed, Apr 15, 2015 at 4:22 AM, Martin Eppel (meppel) <[email protected]> > wrote: > >> I have been out for a few days so I am not sure on the status of this >> issue but I rerun the test (leaving stratos running idle) for a few hours >> with the latest commits from today, April 14 >> (commit 7b09bc1534c5708a47a7b61ae0a83f96957e7eeb, Author: Chamila de >> Alwis [email protected], Date: Tue Apr 14 21:11:22 2015 +0530) >> >> and I still see the same issue happening, >> >> >> >> Thanks >> >> >> >> Martin >> >> >> >> Exceptions are (after letting it run for a few hours) : >> >> >> >> TID: [0] [STRATOS] [2015-04-14 15:37:21,331] WARN >> {org.apache.thrift.server.TThreadPoolServer} - Transport error occurred >> during acceptance of message. >> >> org.apache.thrift.transport.TTransportException: >> java.net.SocketException: Too many open files >> >> at >> org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:118) >> >> at >> org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:35) >> >> at >> org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31) >> >> at >> org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:106) >> >> at >> org.wso2.carbon.databridge.receiver.thrift.internal.ThriftDataReceiver$ServerThread.run(ThriftDataReceiver.java:199) >> >> at java.lang.Thread.run(Thread.java:745) >> >> Caused by: java.net.SocketException: Too many open files >> >> at java.net.PlainSocketImpl.socketAccept(Native Method) >> >> at >> java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) >> >> at java.net.ServerSocket.implAccept(ServerSocket.java:530) >> >> at java.net.ServerSocket.accept(ServerSocket.java:498) >> >> at >> org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:113) >> >> >> >> Btw, this is easily reproducible by just starting up stratos and let it >> run idle for a few hours, >> >> >> >> Regards >> >> >> >> Martin >> >> >> >> >> >> *From:* Martin Eppel (meppel) >> *Sent:* Thursday, April 02, 2015 5:45 PM >> *To:* [email protected] >> *Cc:* Udara Liyanage; Isuru Haththotuwa >> *Subject:* RE: [Error] Too many files open error when running Stratos >> >> >> >> Hi Mariangela, >> >> >> >> Here is an update – seems this is a different issue as STRATOS-723. >> >> After letting the system run for a few hours with the below mentioned >> commit (8985d96eb811aa8e9ce2c114f1856b4c4e20517b) included. >> >> I still see the same issue as you reported (with exception as below). The >> stratos process is still running, no OOM yet but logging has stopped. >> >> >> >> I think this is an issue which needs further investigation, >> >> >> >> Regards >> >> >> >> Martin >> >> >> >> WARN {org.apache.thrift.server.TThreadPoolServer} - Transport error >> occurred during acceptance of message. >> >> org.apache.thrift.transport.TTransportException: >> java.net.SocketException: Too many open files >> >> at >> org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:118) >> >> at >> org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:35) >> >> at >> org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31) >> >> at >> org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:106) >> >> at >> org.wso2.carbon.databridge.receiver.thrift.internal.ThriftDataReceiver$ServerThread.run(ThriftDataReceiver.java:199) >> >> >> >> *From:* Martin Eppel (meppel) >> *Sent:* Thursday, April 02, 2015 1:36 PM >> *To:* [email protected] >> *Cc:* Udara Liyanage; Isuru Haththotuwa >> *Subject:* RE: [Error] Too many files open error when running Stratos >> >> >> >> Hi Mariangela, >> >> >> >> As an FYI, this was possibly caused by >> https://issues.apache.org/jira/browse/STRATOS-723. >> >> >> >> A fix was just checked in for this issue to 4.1.0, see commit >> 8985d96eb811aa8e9ce2c114f1856b4c4e20517b. >> >> >> >> Regards >> >> >> >> Martin >> >> >> >> >> >> >> >> *From:* Mariangela Hills [mailto:[email protected] >> <[email protected]>] >> *Sent:* Thursday, April 02, 2015 3:22 AM >> *To:* dev >> *Cc:* Udara Liyanage; Isuru Haththotuwa >> *Subject:* Re: [Error] Too many files open error when running Stratos >> >> >> >> I have created a JIRA [1] to track this issue! >> >> >> >> [1] https://issues.apache.org/jira/browse/STRATOS-1308 >> >> >> Regards, >> Mariangela >> >> *--* >> >> Mariangela Hills >> >> Senior Technical Writer >> >> >> *WSO2, Inc. *lean.enterprise.middleware. >> >> m: +94 773 500185 >> w: http://wso2.com >> >> [image: http://c.content.wso2.com/signatures/general.png] >> <http://wso2.com/events/> >> >> >> >> On Thu, Apr 2, 2015 at 3:43 PM, Mariangela Hills <[email protected]> >> wrote: >> >> Thanks, Udara and Isuru for your quick response! >> >> >> >> @Udara - >> >> >> >> My file limit was 4000 and even after increasing it to 8000 I face the >> same issue: >> >> >> >> ulimit -Sn 8000 >> >> >> Regards, >> Mariangela >> >> *--* >> >> Mariangela Hills >> >> Senior Technical Writer >> >> >> *WSO2, Inc. *lean.enterprise.middleware. >> >> >> >> >> >> On Thu, Apr 2, 2015 at 3:29 PM, Isuru Haththotuwa <[email protected]> >> wrote: >> >> Hi Mari, >> >> Good catch. Yes, the reconnection logic seems to be missing the >> connection cleaning up part, in case an error occurs in connecting to the >> broker. Will create a Jira to track this. >> >> >> >> On Thu, Apr 2, 2015 at 3:13 PM, Mariangela Hills <[email protected]> >> wrote: >> >> Hi Devs, >> >> >> >> I got a message saying "could not connect to the message broker" even >> when the message broker connection was already in place (started). There >> seems to be some reconnection logic, which finally leads to a too many >> files opened error (part of the log can be seen in [1]). When this happens >> the web browser on my machine (Chrome) gets stuck and I have to restart >> Stratos to solve the issue. Any idea how I can over come this issue? I have >> a feeling that this issue crops up when Stratos has been started and left >> idle for a considerable duration (i.e., 1 hr 30min). >> >> >> >> >> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- >> >> >> >> [1] >> >> [2015-04-02 13:40:16,319] INFO >> {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - >> Topic publisher will try to reconnect in 5 seconds: [topic-name] >> tenant.CompleteTenantEvent >> >> [2015-04-02 13:40:16,319] WARN >> {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - >> Connection to the message broker failed >> >> [2015-04-02 13:40:16,319] INFO >> {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - >> Topic publisher will try to reconnect in 2 seconds: [topic-name] >> tenant.CompleteTenantEvent >> >> Exception in thread "ActiveMQ Transport: >> tcp://localhost/127.0.0.1:61616@50628" >> java.lang.OutOfMemoryError: unable to create new native thread >> >> at java.lang.Thread.start0(Native Method) >> >> at java.lang.Thread.start(Thread.java:673) >> >> at >> java.util.concurrent.ThreadPoolExecutor.addThread(ThreadPoolExecutor.java:681) >> >> at >> java.util.concurrent.ThreadPoolExecutor.addIfUnderCorePoolSize(ThreadPoolExecutor.java:706) >> >> at >> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:650) >> >> at >> org.apache.activemq.ActiveMQConnection.onAsyncException(ActiveMQConnection.java:2002) >> >> at >> org.apache.activemq.ActiveMQConnection.onException(ActiveMQConnection.java:2017)[2015-04-02 >> 13:40:16,423] ERROR >> {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - >> Could not connect to message broker >> >> javax.jms.JMSException: Cannot send, channel has already failed: tcp:// >> 127.0.0.1:61616 >> >> at >> org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:72) >> >> at >> org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1435) >> >> at >> org.apache.activemq.ActiveMQConnection.ensureConnectionInfoSent(ActiveMQConnection.java:1522) >> >> at >> org.apache.activemq.ActiveMQConnection.start(ActiveMQConnection.java:527) >> >> at >> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.connect(AmqpTopicConnector.java:78) >> >> at >> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:115) >> >> at >> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75) >> >> at >> org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005) >> >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) >> >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) >> >> at java.lang.Thread.run(Thread.java:695) >> >> Caused by: org.apache.activemq.transport.InactivityIOException: Cannot >> send, channel has already failed: tcp://127.0.0.1:61616 >> >> at >> org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:297) >> >> at >> org.apache.activemq.transport.AbstractInactivityMonitor.oneway(AbstractInactivityMonitor.java:286) >> >> at >> org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:85) >> >> at >> org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:104) >> >> at >> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68) >> >> at >> org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:81) >> >> at org.apache.activemq.transport.ResponseCorrelator. >> >> request(ResponseCorrelator.java:86) >> >> at >> org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1406) >> >> ... 9 more >> >> at >> org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:101) >> >> at >> org.apache.activemq.transport.ResponseCorrelator.onException(ResponseCorrelator.java:126) >> >> at >> org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:101) >> >> at >> org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:101) >> >> at >> org.apache.activemq.transport.WireFormatNegotiator.onException(WireFormatNegotiator.java:160) >> >> at >> org.apache.activemq.transport.AbstractInactivityMonitor.onException(AbstractInactivityMonitor.java:314) >> >> at >> org.apache.activemq.transport.TransportSupport.onException(TransportSupport.java:96) >> >> at >> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:200) >> >> at java.lang.Thread.run(Thread.java:695) >> >> [2015-04-02 13:40:16,424] WARN >> {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - >> Could not reconnect to message broker >> >> org.apache.stratos.messaging.domain.exception.MessagingException: Could >> not connect to message broker >> >> at >> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.connect(AmqpTopicConnector.java:82) >> >> at >> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:115) >> >> at >> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75) >> >> at >> org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005) >> >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) >> >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) >> >> at java.lang.Thread.run(Thread.java:695) >> >> Caused by: javax.jms.JMSException: Cannot send, channel has already >> failed: tcp://127.0.0.1:61616 >> >> at >> org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:72) >> >> at >> org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1435) >> >> at >> org.apache.activemq.ActiveMQConnection.ensureConnectionInfoSent(ActiveMQConnection.java:1522) >> >> at >> org.apache.activemq.ActiveMQConnection.start(ActiveMQConnection.java:527) >> >> at >> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.connect(AmqpTopicConnector.java:78) >> >> ... 6 more >> >> Caused by: org.apache.activemq.transport.InactivityIOException: Cannot >> send, channel has already failed: tcp://127.0.0.1:61616 >> >> at >> org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:297) >> >> at >> org.apache.activemq.transport.AbstractInactivityMonitor.oneway(AbstractInactivityMonitor.java:286) >> >> at >> org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:85) >> >> at >> org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:104) >> >> at >> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68) >> >> at >> org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:81) >> >> at >> org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:86) >> >> at >> org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1406) >> >> ... 9 more >> >> [2015-04-02 13:40:16,424] ERROR >> {org.apache.stratos.messaging.util.MessagingUtil} - Failed to load >> properties from file: >> /Users/Admin16/Documents/Stratos_4.1.0/master-pack/apache-stratos-4.1.0-SNAPSHOT/repository/conf/jndi.properties >> >> java.io.FileNotFoundException: >> /Users/Admin16/Documents/Stratos_4.1.0/master-pack/apache-stratos-4.1.0-SNAPSHOT/repository/conf/jndi.properties >> (Too many open files in system) >> >> at java.io.FileInputStream.open(Native Method) >> >> at java.io.FileInputStream.<init>(FileInputStream.java:120) >> >> at >> org.apache.stratos.messaging.util.MessagingUtil.getProperties(MessagingUtil.java:108) >> >> at >> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.create(AmqpTopicConnector.java:49) >> >> at >> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:114) >> >> at >> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75) >> >> at >> org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005) >> >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) >> >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) >> >> at java.lang.Thread.run(Thread.java:695) >> >> [2015-04-02 13:40:16,425] INFO >> {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - >> Topic publisher will try to reconnect in 20 seconds: [topic-name] >> application.signup.CompleteApplicationSignUpsEvent >> >> [2015-04-02 13:40:16,427] ERROR >> {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - >> Could not create topic connector >> >> java.lang.NullPointerException >> >> at >> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.create(AmqpTopicConnector.java:53) >> >> at >> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:114) >> >> at >> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75) >> >> at >> org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005) >> >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) >> >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) >> >> at java.lang.Thread.run(Thread.java:695) >> >> [2015-04-02 13:40:16,427] WARN >> {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - >> Could not reconnect to message broker >> >> org.apache.stratos.messaging.domain.exception.MessagingException: Could >> not create topic connector >> >> at >> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.create(AmqpTopicConnector.java:58) >> >> at >> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:114) >> >> at >> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75) >> >> at >> org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005) >> >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) >> >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) >> >> at java.lang.Thread.run(Thread.java:695) >> >> Caused by: java.lang.NullPointerException >> >> at >> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.create(AmqpTopicConnector.java:53) >> >> ... 6 more >> >> [2015-04-02 13:40:16,428] INFO >> {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - >> Topic publisher will try to reconnect in 20 seconds: [topic-name] >> application.signup.CompleteApplicationSignUpsEvent >> >> [2015-04-02 13:40:16,445] ERROR >> {org.apache.stratos.messaging.util.MessagingUtil} - Failed to load >> properties from file: >> /Users/Admin16/Documents/Stratos_4.1.0/master-pack/apache-stratos-4.1.0-SNAPSHOT/repository/conf/jndi.properties >> >> java.io.FileNotFoundException: >> /Users/Admin16/Documents/Stratos_4.1.0/master-pack/apache-stratos-4.1.0-SNAPSHOT/repository/conf/jndi.properties >> (Too many open files in system) >> >> at java.io.FileInputStream.open(Native Method) >> >> at java.io.FileInputStream.<init>(FileInputStream.java:120) >> >> at >> org.apache.stratos.messaging.util.MessagingUtil.getProperties(MessagingUtil.java:108) >> >> at >> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.create(AmqpTopicConnector.java:49) >> >> at >> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:114) >> >> at >> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75) >> >> at >> org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005) >> >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) >> >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) >> >> at java.lang.Thread.run(Thread.java:695) >> >> [2015-04-02 13:40:16,448] ERROR >> {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - >> Could not create topic connector >> >> >> >> >> Regards, >> Mariangela >> >> *--* >> >> Mariangela Hills >> >> Senior Technical Writer >> >> >> *WSO2, Inc. *lean.enterprise.middleware. >> >> -- >> >> Thanks and Regards, >> >> Isuru H. >> >> +94 716 358 048 >> >> -- >> <%2B94%20716%20358%20048> >> Thanks and Regards, >> >> Isuru H. >> <%2B94%20716%20358%20048> >> +94 716 358 048* <http://wso2.com/>* >> >> >> * <http://wso2.com/>* >> >> >>
