Hi Devs,
I got a message saying "could not connect to the message broker" even when
the message broker connection was already in place (started). There seems
to be some reconnection logic, which finally leads to a too many files
opened error (part of the log can be seen in [1]). When this happens the
web browser on my machine (Chrome) gets stuck and I have to restart Stratos
to solve the issue. Any idea how I can over come this issue? I have a
feeling that this issue crops up when Stratos has been started and left
idle for a considerable duration (i.e., 1 hr 30min).
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[1]
[2015-04-02 13:40:16,319] INFO
{org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} -
Topic publisher will try to reconnect in 5 seconds: [topic-name]
tenant.CompleteTenantEvent
[2015-04-02 13:40:16,319] WARN
{org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} -
Connection to the message broker failed
[2015-04-02 13:40:16,319] INFO
{org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} -
Topic publisher will try to reconnect in 2 seconds: [topic-name]
tenant.CompleteTenantEvent
Exception in thread "ActiveMQ Transport: tcp://localhost/127.0.0.1:61616@50628"
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:673)
at
java.util.concurrent.ThreadPoolExecutor.addThread(ThreadPoolExecutor.java:681)
at
java.util.concurrent.ThreadPoolExecutor.addIfUnderCorePoolSize(ThreadPoolExecutor.java:706)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:650)
at
org.apache.activemq.ActiveMQConnection.onAsyncException(ActiveMQConnection.java:2002)
at
org.apache.activemq.ActiveMQConnection.onException(ActiveMQConnection.java:2017)[2015-04-02
13:40:16,423] ERROR
{org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} -
Could not connect to message broker
javax.jms.JMSException: Cannot send, channel has already failed: tcp://
127.0.0.1:61616
at
org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:72)
at
org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1435)
at
org.apache.activemq.ActiveMQConnection.ensureConnectionInfoSent(ActiveMQConnection.java:1522)
at org.apache.activemq.ActiveMQConnection.start(ActiveMQConnection.java:527)
at
org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.connect(AmqpTopicConnector.java:78)
at
org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:115)
at
org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75)
at
org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:695)
Caused by: org.apache.activemq.transport.InactivityIOException: Cannot
send, channel has already failed: tcp://127.0.0.1:61616
at
org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:297)
at
org.apache.activemq.transport.AbstractInactivityMonitor.oneway(AbstractInactivityMonitor.java:286)
at
org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:85)
at
org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:104)
at
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
at
org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:81)
at org.apache.activemq.transport.ResponseCorrelator.
request(ResponseCorrelator.java:86)
at
org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1406)
... 9 more
at
org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:101)
at
org.apache.activemq.transport.ResponseCorrelator.onException(ResponseCorrelator.java:126)
at
org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:101)
at
org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:101)
at
org.apache.activemq.transport.WireFormatNegotiator.onException(WireFormatNegotiator.java:160)
at
org.apache.activemq.transport.AbstractInactivityMonitor.onException(AbstractInactivityMonitor.java:314)
at
org.apache.activemq.transport.TransportSupport.onException(TransportSupport.java:96)
at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:200)
at java.lang.Thread.run(Thread.java:695)
[2015-04-02 13:40:16,424] WARN
{org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} -
Could not reconnect to message broker
org.apache.stratos.messaging.domain.exception.MessagingException: Could not
connect to message broker
at
org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.connect(AmqpTopicConnector.java:82)
at
org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:115)
at
org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75)
at
org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:695)
Caused by: javax.jms.JMSException: Cannot send, channel has already failed:
tcp://127.0.0.1:61616
at
org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:72)
at
org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1435)
at
org.apache.activemq.ActiveMQConnection.ensureConnectionInfoSent(ActiveMQConnection.java:1522)
at org.apache.activemq.ActiveMQConnection.start(ActiveMQConnection.java:527)
at
org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.connect(AmqpTopicConnector.java:78)
... 6 more
Caused by: org.apache.activemq.transport.InactivityIOException: Cannot
send, channel has already failed: tcp://127.0.0.1:61616
at
org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:297)
at
org.apache.activemq.transport.AbstractInactivityMonitor.oneway(AbstractInactivityMonitor.java:286)
at
org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:85)
at
org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:104)
at
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
at
org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:81)
at
org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:86)
at
org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1406)
... 9 more
[2015-04-02 13:40:16,424] ERROR
{org.apache.stratos.messaging.util.MessagingUtil} - Failed to load
properties from file:
/Users/Admin16/Documents/Stratos_4.1.0/master-pack/apache-stratos-4.1.0-SNAPSHOT/repository/conf/jndi.properties
java.io.FileNotFoundException:
/Users/Admin16/Documents/Stratos_4.1.0/master-pack/apache-stratos-4.1.0-SNAPSHOT/repository/conf/jndi.properties
(Too many open files in system)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:120)
at
org.apache.stratos.messaging.util.MessagingUtil.getProperties(MessagingUtil.java:108)
at
org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.create(AmqpTopicConnector.java:49)
at
org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:114)
at
org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75)
at
org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:695)
[2015-04-02 13:40:16,425] INFO
{org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} -
Topic publisher will try to reconnect in 20 seconds: [topic-name]
application.signup.CompleteApplicationSignUpsEvent
[2015-04-02 13:40:16,427] ERROR
{org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} -
Could not create topic connector
java.lang.NullPointerException
at
org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.create(AmqpTopicConnector.java:53)
at
org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:114)
at
org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75)
at
org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:695)
[2015-04-02 13:40:16,427] WARN
{org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} -
Could not reconnect to message broker
org.apache.stratos.messaging.domain.exception.MessagingException: Could not
create topic connector
at
org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.create(AmqpTopicConnector.java:58)
at
org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:114)
at
org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75)
at
org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:695)
Caused by: java.lang.NullPointerException
at
org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.create(AmqpTopicConnector.java:53)
... 6 more
[2015-04-02 13:40:16,428] INFO
{org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} -
Topic publisher will try to reconnect in 20 seconds: [topic-name]
application.signup.CompleteApplicationSignUpsEvent
[2015-04-02 13:40:16,445] ERROR
{org.apache.stratos.messaging.util.MessagingUtil} - Failed to load
properties from file:
/Users/Admin16/Documents/Stratos_4.1.0/master-pack/apache-stratos-4.1.0-SNAPSHOT/repository/conf/jndi.properties
java.io.FileNotFoundException:
/Users/Admin16/Documents/Stratos_4.1.0/master-pack/apache-stratos-4.1.0-SNAPSHOT/repository/conf/jndi.properties
(Too many open files in system)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:120)
at
org.apache.stratos.messaging.util.MessagingUtil.getProperties(MessagingUtil.java:108)
at
org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.create(AmqpTopicConnector.java:49)
at
org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:114)
at
org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75)
at
org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:695)
[2015-04-02 13:40:16,448] ERROR
{org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} -
Could not create topic connector
Regards,
Mariangela
*--*
Mariangela Hills
Senior Technical Writer
*WSO2, Inc.*lean.enterprise.middleware.
