Mariangela Hills created STRATOS-1308:
-----------------------------------------
Summary: Reconnection logic is missing the connection cleaning up
part
Key: STRATOS-1308
URL: https://issues.apache.org/jira/browse/STRATOS-1308
Project: Stratos
Issue Type: Bug
Affects Versions: 4.1.0 Alpha
Reporter: Mariangela Hills
I got a message saying "could not connect to the message broker" even when the
message broker connection was already in place (started). There seems to be
some reconnection logic, which finally leads to a too many files opened error
(part of the log can be seen in [1]). When this happens the web browser on my
machine (Chrome) gets stuck and I have to restart Stratos to solve the issue.
In addition, I would like to add that my max file limit was 8000 when this
issue occurred (ulimit -Sn 8000). Therefore, it looks like the reconnection
logic is missing the connection cleaning up part.
---------------------
[1]
[2015-04-02 13:40:16,319] INFO
{org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - Topic
publisher will try to reconnect in 5 seconds: [topic-name]
tenant.CompleteTenantEvent
[2015-04-02 13:40:16,319] WARN
{org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} -
Connection to the message broker failed
[2015-04-02 13:40:16,319] INFO
{org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - Topic
publisher will try to reconnect in 2 seconds: [topic-name]
tenant.CompleteTenantEvent
Exception in thread "ActiveMQ Transport: tcp://localhost/127.0.0.1:61616@50628"
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:673)
at
java.util.concurrent.ThreadPoolExecutor.addThread(ThreadPoolExecutor.java:681)
at
java.util.concurrent.ThreadPoolExecutor.addIfUnderCorePoolSize(ThreadPoolExecutor.java:706)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:650)
at
org.apache.activemq.ActiveMQConnection.onAsyncException(ActiveMQConnection.java:2002)
at
org.apache.activemq.ActiveMQConnection.onException(ActiveMQConnection.java:2017)[2015-04-02
13:40:16,423] ERROR
{org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - Could
not connect to message broker
javax.jms.JMSException: Cannot send, channel has already failed:
tcp://127.0.0.1:61616
at
org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:72)
at
org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1435)
at
org.apache.activemq.ActiveMQConnection.ensureConnectionInfoSent(ActiveMQConnection.java:1522)
at
org.apache.activemq.ActiveMQConnection.start(ActiveMQConnection.java:527)
at
org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.connect(AmqpTopicConnector.java:78)
at
org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:115)
at
org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75)
at
org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:695)
Caused by: org.apache.activemq.transport.InactivityIOException: Cannot send,
channel has already failed: tcp://127.0.0.1:61616
at
org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:297)
at
org.apache.activemq.transport.AbstractInactivityMonitor.oneway(AbstractInactivityMonitor.java:286)
at
org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:85)
at
org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:104)
at
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
at
org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:81)
at org.apache.activemq.transport.ResponseCorrelator.
request(ResponseCorrelator.java:86)
at
org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1406)
... 9 more
at
org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:101)
at
org.apache.activemq.transport.ResponseCorrelator.onException(ResponseCorrelator.java:126)
at
org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:101)
at
org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:101)
at
org.apache.activemq.transport.WireFormatNegotiator.onException(WireFormatNegotiator.java:160)
at
org.apache.activemq.transport.AbstractInactivityMonitor.onException(AbstractInactivityMonitor.java:314)
at
org.apache.activemq.transport.TransportSupport.onException(TransportSupport.java:96)
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:200)
at java.lang.Thread.run(Thread.java:695)
[2015-04-02 13:40:16,424] WARN
{org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - Could
not reconnect to message broker
org.apache.stratos.messaging.domain.exception.MessagingException: Could not
connect to message broker
at
org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.connect(AmqpTopicConnector.java:82)
at
org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:115)
at
org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75)
at
org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:695)
Caused by: javax.jms.JMSException: Cannot send, channel has already failed:
tcp://127.0.0.1:61616
at
org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:72)
at
org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1435)
at
org.apache.activemq.ActiveMQConnection.ensureConnectionInfoSent(ActiveMQConnection.java:1522)
at
org.apache.activemq.ActiveMQConnection.start(ActiveMQConnection.java:527)
at
org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.connect(AmqpTopicConnector.java:78)
... 6 more
Caused by: org.apache.activemq.transport.InactivityIOException: Cannot send,
channel has already failed: tcp://127.0.0.1:61616
at
org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:297)
at
org.apache.activemq.transport.AbstractInactivityMonitor.oneway(AbstractInactivityMonitor.java:286)
at
org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:85)
at
org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:104)
at
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
at
org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:81)
at
org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:86)
at
org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1406)
... 9 more
[2015-04-02 13:40:16,424] ERROR
{org.apache.stratos.messaging.util.MessagingUtil} - Failed to load properties
from file:
/Users/Admin16/Documents/Stratos_4.1.0/master-pack/apache-stratos-4.1.0-SNAPSHOT/repository/conf/jndi.properties
java.io.FileNotFoundException:
/Users/Admin16/Documents/Stratos_4.1.0/master-pack/apache-stratos-4.1.0-SNAPSHOT/repository/conf/jndi.properties
(Too many open files in system)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:120)
at
org.apache.stratos.messaging.util.MessagingUtil.getProperties(MessagingUtil.java:108)
at
org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.create(AmqpTopicConnector.java:49)
at
org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:114)
at
org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75)
at
org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:695)
[2015-04-02 13:40:16,425] INFO
{org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - Topic
publisher will try to reconnect in 20 seconds: [topic-name]
application.signup.CompleteApplicationSignUpsEvent
[2015-04-02 13:40:16,427] ERROR
{org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - Could
not create topic connector
java.lang.NullPointerException
at
org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.create(AmqpTopicConnector.java:53)
at
org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:114)
at
org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75)
at
org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:695)
[2015-04-02 13:40:16,427] WARN
{org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - Could
not reconnect to message broker
org.apache.stratos.messaging.domain.exception.MessagingException: Could not
create topic connector
at
org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.create(AmqpTopicConnector.java:58)
at
org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:114)
at
org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75)
at
org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:695)
Caused by: java.lang.NullPointerException
at
org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.create(AmqpTopicConnector.java:53)
... 6 more
[2015-04-02 13:40:16,428] INFO
{org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - Topic
publisher will try to reconnect in 20 seconds: [topic-name]
application.signup.CompleteApplicationSignUpsEvent
[2015-04-02 13:40:16,445] ERROR
{org.apache.stratos.messaging.util.MessagingUtil} - Failed to load properties
from file:
/Users/Admin16/Documents/Stratos_4.1.0/master-pack/apache-stratos-4.1.0-SNAPSHOT/repository/conf/jndi.properties
java.io.FileNotFoundException:
/Users/Admin16/Documents/Stratos_4.1.0/master-pack/apache-stratos-4.1.0-SNAPSHOT/repository/conf/jndi.properties
(Too many open files in system)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:120)
at
org.apache.stratos.messaging.util.MessagingUtil.getProperties(MessagingUtil.java:108)
at
org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.create(AmqpTopicConnector.java:49)
at
org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:114)
at
org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75)
at
org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:695)
[2015-04-02 13:40:16,448] ERROR
{org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} - Could
not create topic connector
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
