I have been out for a few days so I am not sure on the status of this issue but
I rerun the test (leaving stratos running idle) for a few hours with the latest
commits from today, April 14
(commit 7b09bc1534c5708a47a7b61ae0a83f96957e7eeb, Author: Chamila de Alwis
[email protected]<mailto:[email protected]>, Date: Tue Apr 14 21:11:22 2015
+0530)
and I still see the same issue happening,
Thanks
Martin
Exceptions are (after letting it run for a few hours) :
TID: [0] [STRATOS] [2015-04-14 15:37:21,331] WARN
{org.apache.thrift.server.TThreadPoolServer} - Transport error occurred during
acceptance of message.
org.apache.thrift.transport.TTransportException: java.net.SocketException: Too
many open files
at
org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:118)
at
org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:35)
at
org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31)
at
org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:106)
at
org.wso2.carbon.databridge.receiver.thrift.internal.ThriftDataReceiver$ServerThread.run(ThriftDataReceiver.java:199)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketException: Too many open files
at java.net.PlainSocketImpl.socketAccept(Native Method)
at
java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:530)
at java.net.ServerSocket.accept(ServerSocket.java:498)
at
org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:113)
Btw, this is easily reproducible by just starting up stratos and let it run
idle for a few hours,
Regards
Martin
From: Martin Eppel (meppel)
Sent: Thursday, April 02, 2015 5:45 PM
To: [email protected]<mailto:[email protected]>
Cc: Udara Liyanage; Isuru Haththotuwa
Subject: RE: [Error] Too many files open error when running Stratos
Hi Mariangela,
Here is an update – seems this is a different issue as STRATOS-723.
After letting the system run for a few hours with the below mentioned commit
(8985d96eb811aa8e9ce2c114f1856b4c4e20517b) included.
I still see the same issue as you reported (with exception as below). The
stratos process is still running, no OOM yet but logging has stopped.
I think this is an issue which needs further investigation,
Regards
Martin
WARN {org.apache.thrift.server.TThreadPoolServer} - Transport error occurred
during acceptance of message.
org.apache.thrift.transport.TTransportException: java.net.SocketException: Too
many open files
at
org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:118)
at
org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:35)
at
org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31)
at
org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:106)
at
org.wso2.carbon.databridge.receiver.thrift.internal.ThriftDataReceiver$ServerThread.run(ThriftDataReceiver.java:199)
From: Martin Eppel (meppel)
Sent: Thursday, April 02, 2015 1:36 PM
To: [email protected]<mailto:[email protected]>
Cc: Udara Liyanage; Isuru Haththotuwa
Subject: RE: [Error] Too many files open error when running Stratos
Hi Mariangela,
As an FYI, this was possibly caused by
https://issues.apache.org/jira/browse/STRATOS-723.
A fix was just checked in for this issue to 4.1.0, see commit
8985d96eb811aa8e9ce2c114f1856b4c4e20517b.
Regards
Martin
From: Mariangela Hills [mailto:[email protected]]
Sent: Thursday, April 02, 2015 3:22 AM
To: dev
Cc: Udara Liyanage; Isuru Haththotuwa
Subject: Re: [Error] Too many files open error when running Stratos
I have created a JIRA [1] to track this issue!
[1] https://issues.apache.org/jira/browse/STRATOS-1308
Regards,
Mariangela
--
Mariangela Hills
Senior Technical Writer
WSO2, Inc.
lean.enterprise.middleware.
m: +94 773 500185
w: http://wso2.com<http://wso2.com/>
[http://c.content.wso2.com/signatures/general.png]<http://wso2.com/events/>
On Thu, Apr 2, 2015 at 3:43 PM, Mariangela Hills
<[email protected]<mailto:[email protected]>> wrote:
Thanks, Udara and Isuru for your quick response!
@Udara -
My file limit was 4000 and even after increasing it to 8000 I face the same
issue:
ulimit -Sn 8000
Regards,
Mariangela
--
Mariangela Hills
Senior Technical Writer
WSO2, Inc.
lean.enterprise.middleware.
On Thu, Apr 2, 2015 at 3:29 PM, Isuru Haththotuwa
<[email protected]<mailto:[email protected]>> wrote:
Hi Mari,
Good catch. Yes, the reconnection logic seems to be missing the connection
cleaning up part, in case an error occurs in connecting to the broker. Will
create a Jira to track this.
On Thu, Apr 2, 2015 at 3:13 PM, Mariangela Hills
<[email protected]<mailto:[email protected]>> wrote:
Hi Devs,
I got a message saying "could not connect to the message broker" even when the
message broker connection was already in place (started). There seems to be
some reconnection logic, which finally leads to a too many files opened error
(part of the log can be seen in [1]). When this happens the web browser on my
machine (Chrome) gets stuck and I have to restart Stratos to solve the issue.
Any idea how I can over come this issue? I have a feeling that this issue crops
up when Stratos has been started and left idle for a considerable duration
(i.e., 1 hr 30min).
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[1]
[2015-04-02 13:40:16,319] INFO
{org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - Topic
publisher will try to reconnect in 5 seconds: [topic-name]
tenant.CompleteTenantEvent
[2015-04-02 13:40:16,319] WARN
{org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} -
Connection to the message broker failed
[2015-04-02 13:40:16,319] INFO
{org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - Topic
publisher will try to reconnect in 2 seconds: [topic-name]
tenant.CompleteTenantEvent
Exception in thread "ActiveMQ Transport: tcp://localhost/127.0.0.1:61616@50628"
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:673)
at
java.util.concurrent.ThreadPoolExecutor.addThread(ThreadPoolExecutor.java:681)
at
java.util.concurrent.ThreadPoolExecutor.addIfUnderCorePoolSize(ThreadPoolExecutor.java:706)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:650)
at
org.apache.activemq.ActiveMQConnection.onAsyncException(ActiveMQConnection.java:2002)
at
org.apache.activemq.ActiveMQConnection.onException(ActiveMQConnection.java:2017)[2015-04-02
13:40:16,423] ERROR
{org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - Could
not connect to message broker
javax.jms.JMSException: Cannot send, channel has already failed:
tcp://127.0.0.1:61616<http://127.0.0.1:61616>
at
org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:72)
at
org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1435)
at
org.apache.activemq.ActiveMQConnection.ensureConnectionInfoSent(ActiveMQConnection.java:1522)
at org.apache.activemq.ActiveMQConnection.start(ActiveMQConnection.java:527)
at
org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.connect(AmqpTopicConnector.java:78)
at
org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:115)
at
org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75)
at org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:695)
Caused by: org.apache.activemq.transport.InactivityIOException: Cannot send,
channel has already failed: tcp://127.0.0.1:61616<http://127.0.0.1:61616>
at
org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:297)
at
org.apache.activemq.transport.AbstractInactivityMonitor.oneway(AbstractInactivityMonitor.java:286)
at org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:85)
at
org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:104)
at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
at
org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:81)
at org.apache.activemq.transport.ResponseCorrelator.
request(ResponseCorrelator.java:86)
at
org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1406)
... 9 more
at
org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:101)
at
org.apache.activemq.transport.ResponseCorrelator.onException(ResponseCorrelator.java:126)
at
org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:101)
at
org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:101)
at
org.apache.activemq.transport.WireFormatNegotiator.onException(WireFormatNegotiator.java:160)
at
org.apache.activemq.transport.AbstractInactivityMonitor.onException(AbstractInactivityMonitor.java:314)
at
org.apache.activemq.transport.TransportSupport.onException(TransportSupport.java:96)
at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:200)
at java.lang.Thread.run(Thread.java:695)
[2015-04-02 13:40:16,424] WARN
{org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - Could
not reconnect to message broker
org.apache.stratos.messaging.domain.exception.MessagingException: Could not
connect to message broker
at
org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.connect(AmqpTopicConnector.java:82)
at
org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:115)
at
org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75)
at org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:695)
Caused by: javax.jms.JMSException: Cannot send, channel has already failed:
tcp://127.0.0.1:61616<http://127.0.0.1:61616>
at
org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:72)
at
org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1435)
at
org.apache.activemq.ActiveMQConnection.ensureConnectionInfoSent(ActiveMQConnection.java:1522)
at org.apache.activemq.ActiveMQConnection.start(ActiveMQConnection.java:527)
at
org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.connect(AmqpTopicConnector.java:78)
... 6 more
Caused by: org.apache.activemq.transport.InactivityIOException: Cannot send,
channel has already failed: tcp://127.0.0.1:61616<http://127.0.0.1:61616>
at
org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:297)
at
org.apache.activemq.transport.AbstractInactivityMonitor.oneway(AbstractInactivityMonitor.java:286)
at org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:85)
at
org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:104)
at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
at
org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:81)
at
org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:86)
at
org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1406)
... 9 more
[2015-04-02 13:40:16,424] ERROR
{org.apache.stratos.messaging.util.MessagingUtil} - Failed to load properties
from file:
/Users/Admin16/Documents/Stratos_4.1.0/master-pack/apache-stratos-4.1.0-SNAPSHOT/repository/conf/jndi.properties
java.io.FileNotFoundException:
/Users/Admin16/Documents/Stratos_4.1.0/master-pack/apache-stratos-4.1.0-SNAPSHOT/repository/conf/jndi.properties
(Too many open files in system)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:120)
at
org.apache.stratos.messaging.util.MessagingUtil.getProperties(MessagingUtil.java:108)
at
org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.create(AmqpTopicConnector.java:49)
at
org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:114)
at
org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75)
at org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:695)
[2015-04-02 13:40:16,425] INFO
{org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - Topic
publisher will try to reconnect in 20 seconds: [topic-name]
application.signup.CompleteApplicationSignUpsEvent
[2015-04-02 13:40:16,427] ERROR
{org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - Could
not create topic connector
java.lang.NullPointerException
at
org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.create(AmqpTopicConnector.java:53)
at
org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:114)
at
org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75)
at org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:695)
[2015-04-02 13:40:16,427] WARN
{org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - Could
not reconnect to message broker
org.apache.stratos.messaging.domain.exception.MessagingException: Could not
create topic connector
at
org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.create(AmqpTopicConnector.java:58)
at
org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:114)
at
org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75)
at org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:695)
Caused by: java.lang.NullPointerException
at
org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.create(AmqpTopicConnector.java:53)
... 6 more
[2015-04-02 13:40:16,428] INFO
{org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - Topic
publisher will try to reconnect in 20 seconds: [topic-name]
application.signup.CompleteApplicationSignUpsEvent
[2015-04-02 13:40:16,445] ERROR
{org.apache.stratos.messaging.util.MessagingUtil} - Failed to load properties
from file:
/Users/Admin16/Documents/Stratos_4.1.0/master-pack/apache-stratos-4.1.0-SNAPSHOT/repository/conf/jndi.properties
java.io.FileNotFoundException:
/Users/Admin16/Documents/Stratos_4.1.0/master-pack/apache-stratos-4.1.0-SNAPSHOT/repository/conf/jndi.properties
(Too many open files in system)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:120)
at
org.apache.stratos.messaging.util.MessagingUtil.getProperties(MessagingUtil.java:108)
at
org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.create(AmqpTopicConnector.java:49)
at
org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:114)
at
org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75)
at org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:695)
[2015-04-02 13:40:16,448] ERROR
{org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - Could
not create topic connector
Regards,
Mariangela
--
Mariangela Hills
Senior Technical Writer
WSO2, Inc.
lean.enterprise.middleware.
--
Thanks and Regards,
Isuru H.
+94 716 358 048<tel:%2B94%20716%20358%20048>
