Hi Martin, You have reported shutdown thing when tenant creation also. It seems another process kill/send SIGINT to Stratos. May be it tries to kill when Stratos produces error messages.
Touched, not typed. Erroneous words are a feature, not a typo. On Oct 31, 2014 11:42 PM, "Nirmal Fernando" <[email protected]> wrote: > Hi Martin, > > Is this a puppet based deployment? If so please check whether it's puppet > causing this. > > On Fri, Oct 31, 2014 at 6:31 PM, Martin Eppel (meppel) <[email protected]> > wrote: > >> Yes, I tried telnet (telnet octl.qmog.cisco.com 61616) after the >> exception happened and it is accessible. >> >> >> >> According to the stratos log it will retry to connect in 30 seconds but >> then the shutdown hook is triggered. My question why / what will trigger >> the shutdown (see logs below) ? >> >> >> >> TID: [0] [STRATOS] [2014-10-31 17:22:27,196] ERROR >> {org.apache.stratos.messaging.broker.subscribe.TopicSubscriber} - Error >> while subscribing to the topic: application-status >> {org.apache.stratos.messaging.broker.subscribe.TopicSubscriber} >> >> javax.jms.JMSException: Could not connect to broker URL: tcp:// >> octl.qmog.cisco.com:61616. Reason: java.net.ConnectException: Connection >> refused >> >> at >> org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:36) >> at >> org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:358) >> >> at >> org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:303) >> >> at >> org.apache.activemq.ActiveMQConnectionFactory.createTopicConnection(ActiveMQConnectionFactory.java:277) >> >> at >> org.apache.stratos.messaging.broker.connect.TopicConnector.init(TopicConnector.java:68) >> >> at >> org.apache.stratos.messaging.broker.subscribe.TopicSubscriber.doSubscribe(TopicSubscriber.java:64) >> >> at >> org.apache.stratos.messaging.broker.subscribe.TopicSubscriber.run(TopicSubscriber.java:117) >> at java.lang.Thread.run(Thread.java:745) >> >> Caused by: java.net.ConnectException: Connection refused >> >> at java.net.PlainSocketImpl.socketConnect(Native Method) >> >> at >> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) >> >> at >> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) >> at >> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) >> >> at >> java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at >> java.net.Socket.connect(Socket.java:579) >> >> at >> org.apache.activemq.transport.tcp.TcpTransport.connect(TcpTransport.java:501) >> at >> org.apache.activemq.transport.tcp.TcpTransport.doStart(TcpTransport.java:464) >> >> at >> org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55) >> >> at >> org.apache.activemq.transport.AbstractInactivityMonitor.start(AbstractInactivityMonitor.java:138) >> >> at >> org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:58) >> >> at >> org.apache.activemq.transport.WireFormatNegotiator.start(WireFormatNegotiator.java:72) >> at >> org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:58) >> >> at >> org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:58) >> at >> org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:338) >> >> ... 6 moreTID: [0] [STRATOS] [2014-10-31 17:22:27,199] INFO >> {org.apache.stratos.messaging.broker.subscribe.TopicSubscriber} - Will try >> to subscribe again in 30 sec >> {org.apache.stratos.messaging.broker.subscribe.TopicSubscriber}TID: [0] >> [STRATOS] [2014-10-31 17:22:27,520] INFO >> {org.wso2.carbon.core.init.CarbonServerManager} - Shutdown hook >> triggered.... {org.wso2.carbon.core.init.CarbonServerManager}TID: [0] >> [STRATOS] [2014-10-31 17:22:27,524] INFO >> {org.wso2.carbon.core.init.CarbonServerManager} - Gracefully shutting down >> Apache Stratos... {org.wso2.carbon.core.init.CarbonServerManager}TID: [0] >> [STRATOS] [2014-10-31 17:22:27,531] INFO >> {org.wso2.carbon.core.ServerManagement} - Starting to switch to >> maintenance mode... {org.wso2.carbon.core.ServerManagement}TID: [0] >> [STRATOS] [2014-10-31 17:22:27,532] INFO >> {org.wso2.carbon.core.ServerManagement} - Stopped all transport listeners >> {org.wso2.carbon.core.ServerManagement}TID: [0] [STRATOS] [2014-10-31 >> 17:22:27,532] INFO {org.wso2.carbon.core.ServerManagement} - Waiting for >> request service completion... {org.wso2.carbon.core.ServerManagement} >> >> >> >> *From:* Martin Eppel (meppel) >> *Sent:* Friday, October 31, 2014 10:02 AM >> *To:* [email protected] >> *Subject:* RE: [grouping] [question] stratos error during startup >> >> >> >> Ok, I try it. >> >> >> >> One questions, does stratos limits the numbers of retries or does it >> eventually shut down (this is what it seems to happen that stratos shuts >> down after it fails to connect to the activemq) >> >> >> >> Thanks >> >> >> >> Martin >> >> >> >> *From:* Imesh Gunaratne [mailto:[email protected] <[email protected]>] >> *Sent:* Friday, October 31, 2014 9:57 AM >> *To:* dev >> *Subject:* Re: [grouping] [question] stratos error during startup >> >> >> >> Hi Martin, >> >> >> >> According to the error log it seems like Stratos could not reach the >> message broker at the given tcp URL. However it should try to reconnect >> afterwards. >> >> >> >> Will us be able to check whether it is possible to telnet to this tcp >> socket fromt he Stratos host? >> >> >> >> Thanks >> >> >> >> On Fri, Oct 31, 2014 at 7:49 PM, Martin Eppel (meppel) <[email protected]> >> wrote: >> >> Yes it is running when stratos starts up >> >> >> >> *From:* Udara Liyanage [mailto:[email protected]] >> *Sent:* Thursday, October 30, 2014 8:09 PM >> *To:* dev >> *Subject:* Re: [grouping] [question] stratos error during startup >> >> >> >> Hi Martin, >> >> Is ActiveMQ available at the time of Stratos startup time? >> >> Touched, not typed. Erroneous words are a feature, not a typo. >> >> On Oct 31, 2014 2:10 AM, "Martin Eppel (meppel)" <[email protected]> >> wrote: >> >> I am trying to migrate the grouping stratos binaries to our orchestration >> environment but consistently run into the issue that when stratos starts up >> the first time an (activemq) eror is thrown when trying to subscribe to >> application-status (Error while subscribing to the topic: >> application-status). ActiveMq seems to be up and running and after >> restarting stratos, stratos comes up without the exception (without having >> activemq to restart). >> >> >> >> Any hints / ideas what could cause this or what I should look for ? >> >> >> >> Thanks >> >> >> >> Martin >> >> >> >> (I attached the complete log) >> >> >> >> >> >> TID: [0] [STRATOS] [2014-10-30 19:34:52,071] INFO >> {org.wso2.carbon.ui.internal.CarbonUIServiceComponent} - Mgt Console URL >> : https://172.16.2.207:9443/carbon/ >> {org.wso2.carbon.ui.internal.CarbonUIServiceComponent} >> >> TID: [0] [STRATOS] [2014-10-30 19:34:58,071] INFO >> {org.apache.stratos.cep.extension.CEPTopologyEventReceiver} - CEP topology >> receiver thread started >> {org.apache.stratos.cep.extension.CEPTopologyEventReceiver} >> >> TID: [0] [STRATOS] [2014-10-30 19:34:58,072] INFO >> {org.apache.stratos.messaging.message.receiver.topology.TopologyEventMessageDelegator} >> - Topology event message delegator started >> {org.apache.stratos.messaging.message.receiver.topology.TopologyEventMessageDelegator} >> >> TID: [0] [STRATOS] [2014-10-30 19:35:01,024] DEBUG >> {org.apache.stratos.cloud.controller.topology.TopologySynchronizerTask} - >> Executing topology synchronization task >> {org.apache.stratos.cloud.controller.topology.TopologySynchronizerTask} >> >> TID: [0] [STRATOS] [2014-10-30 19:35:01,025] DEBUG >> {org.apache.stratos.cloud.controller.topology.TopologyManager} - Trying to >> retrieve topology from registry >> {org.apache.stratos.cloud.controller.topology.TopologyManager} >> >> TID: [0] [STRATOS] [2014-10-30 19:35:01,027] DEBUG >> {org.apache.stratos.manager.publisher.TenantSynzhronizerTask} - Publishing >> complete tenant event >> {org.apache.stratos.manager.publisher.TenantSynzhronizerTask} >> >> TID: [0] [STRATOS] [2014-10-30 19:35:01,032] DEBUG >> {org.apache.stratos.cloud.controller.topology.TopologyManager} - Topology >> not found in registry, creating new >> {org.apache.stratos.cloud.controller.topology.TopologyManager} >> >> TID: [0] [STRATOS] [2014-10-30 19:35:01,033] DEBUG >> {org.apache.stratos.cloud.controller.topology.TopologyManager} - Topology >> initialized {org.apache.stratos.cloud.controller.topology.TopologyManager} >> >> TID: [0] [STRATOS] [2014-10-30 19:35:01,035] DEBUG >> {org.apache.stratos.cloud.controller.topology.TopologyEventPublisher} - >> Publishing complete topology event >> {org.apache.stratos.cloud.controller.topology.TopologyEventPublisher} >> >> TID: [0] [STRATOS] [2014-10-30 19:35:01,155] INFO >> {org.apache.stratos.messaging.message.processor.topology.CompleteTopologyMessageProcessor} >> - Topology initialized >> {org.apache.stratos.messaging.message.processor.topology.CompleteTopologyMessageProcessor} >> >> TID: [0] [STRATOS] [2014-10-30 19:35:01,156] INFO >> {org.apache.stratos.messaging.message.processor.topology.CompleteTopologyMessageProcessor} >> - Topology initialized >> {org.apache.stratos.messaging.message.processor.topology.CompleteTopologyMessageProcessor} >> >> TID: [0] [STRATOS] [2014-10-30 19:35:01,156] INFO >> {org.apache.stratos.messaging.message.processor.topology.CompleteTopologyMessageProcessor} >> - Topology initialized >> {org.apache.stratos.messaging.message.processor.topology.CompleteTopologyMessageProcessor} >> >> TID: [0] [STRATOS] [2014-10-30 19:35:01,158] INFO >> {org.apache.stratos.autoscaler.message.receiver.topology.AutoscalerTopologyEventReceiver} >> - [CompleteTopologyEvent] Received: class >> org.apache.stratos.messaging.event.topology.CompleteTopologyEvent >> {org.apache.stratos.autoscaler.message.receiver.topology.AutoscalerTopologyEventReceiver} >> >> TID: [0] [STRATOS] [2014-10-30 19:35:01,158] INFO >> {org.apache.stratos.messaging.message.processor.topology.CompleteTopologyMessageProcessor} >> - Topology initialized >> {org.apache.stratos.messaging.message.processor.topology.CompleteTopologyMessageProcessor} >> >> TID: [0] [STRATOS] [2014-10-30 19:35:01,159] INFO >> {org.apache.stratos.manager.topology.receiver.StratosManagerTopologyEventReceiver} >> - [CompleteTopologyEventListener] Received: class >> org.apache.stratos.messaging.event.topology.CompleteTopologyEvent >> {org.apache.stratos.manager.topology.receiver.StratosManagerTopologyEventReceiver} >> >> TID: [0] [STRATOS] [2014-10-30 19:35:01,160] INFO >> {org.apache.stratos.cep.extension.CEPTopologyEventReceiver} - Complete >> topology event received to fault handling window processor. >> {org.apache.stratos.cep.extension.CEPTopologyEventReceiver} >> >> TID: [0] [STRATOS] [2014-10-30 19:35:01,160] INFO >> {org.apache.stratos.cep.extension.FaultHandlingWindowProcessor} - Member >> time stamp map was successfully loaded from the topology. >> {org.apache.stratos.cep.extension.FaultHandlingWindowProcessor} >> >> TID: [0] [STRATOS] [2014-10-30 19:35:04,168] DEBUG >> {org.apache.stratos.manager.publisher.TenantEventPublisher} - Publishing >> tenant created event: [tenant-id] 1 [tenant-domain] cloud1.qmog.cisco.com >> {org.apache.stratos.manager.publisher.TenantEventPublisher} >> >> TID: [0] [STRATOS] [2014-10-30 19:35:06,045] ERROR >> {org.apache.stratos.messaging.broker.heartbeat.TopicHealthChecker} - >> instance-status topic health checker is failed and will try to subscribe >> again in 30 seconds. >> {org.apache.stratos.messaging.broker.heartbeat.TopicHealthChecker} >> >> TID: [0] [STRATOS] [2014-10-30 19:35:06,104] ERROR >> {org.apache.stratos.messaging.broker.heartbeat.TopicHealthChecker} - >> instance-status topic health checker is failed and will try to subscribe >> again in 30 seconds. >> {org.apache.stratos.messaging.broker.heartbeat.TopicHealthChecker} >> >> TID: [0] [STRATOS] [2014-10-30 19:35:06,106] ERROR >> {org.apache.stratos.messaging.broker.heartbeat.TopicHealthChecker} - >> topology topic health checker is failed and will try to subscribe again in >> 30 seconds. >> {org.apache.stratos.messaging.broker.heartbeat.TopicHealthChecker} >> >> TID: [0] [STRATOS] [2014-10-30 19:35:06,225] ERROR >> {org.apache.stratos.messaging.broker.heartbeat.TopicHealthChecker} - >> topology topic health checker is failed and will try to subscribe again in >> 30 seconds. >> {org.apache.stratos.messaging.broker.heartbeat.TopicHealthChecker} >> >> TID: [0] [STRATOS] [2014-10-30 19:35:06,229] ERROR >> {org.apache.stratos.messaging.broker.heartbeat.TopicHealthChecker} - >> topology topic health checker is failed and will try to subscribe again in >> 30 seconds. >> {org.apache.stratos.messaging.broker.heartbeat.TopicHealthChecker} >> >> TID: [0] [STRATOS] [2014-10-30 19:35:06,319] ERROR >> {org.apache.stratos.messaging.broker.heartbeat.TopicHealthChecker} - >> summarized-health-stats topic health checker is failed and will try to >> subscribe again in 30 seconds. >> {org.apache.stratos.messaging.broker.heartbeat.TopicHealthChecker} >> >> TID: [0] [STRATOS] [2014-10-30 19:35:06,397] ERROR >> {org.apache.stratos.messaging.broker.heartbeat.TopicHealthChecker} - >> topology topic health checker is failed and will try to subscribe again in >> 30 seconds. >> {org.apache.stratos.messaging.broker.heartbeat.TopicHealthChecker} >> >> TID: [0] [STRATOS] [2014-10-30 19:35:06,656] INFO >> {org.apache.stratos.cloud.controller.application.status.receiver.ApplicationStatusTopicReceiver} >> - Cloud controller application status thread started >> {org.apache.stratos.cloud.controller.application.status.receiver.ApplicationStatusTopicReceiver} >> >> TID: [0] [STRATOS] [2014-10-30 19:35:06,658] INFO >> {org.apache.stratos.messaging.message.receiver.application.status.ApplicationStatusEventMessageDelegator} >> - Application status event message delegator started >> {org.apache.stratos.messaging.message.receiver.application.status.ApplicationStatusEventMessageDelegator} >> >> TID: [0] [STRATOS] [2014-10-30 19:35:06,663] ERROR >> {org.apache.stratos.messaging.broker.subscribe.TopicSubscriber} - Error >> while subscribing to the topic: application-status >> {org.apache.stratos.messaging.broker.subscribe.TopicSubscriber} >> >> javax.jms.JMSException: Could not connect to broker URL: tcp:// >> octl.qmog.cisco.com:61616. Reason: java.net.ConnectException: Connection >> refused >> >> at >> org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:36) >> >> at >> org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:358) >> >> at >> org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:303) >> >> at >> org.apache.activemq.ActiveMQConnectionFactory.createTopicConnection(ActiveMQConnectionFactory.java:277) >> >> at >> org.apache.stratos.messaging.broker.connect.TopicConnector.init(TopicConnector.java:68) >> >> at >> org.apache.stratos.messaging.broker.subscribe.TopicSubscriber.doSubscribe(TopicSubscriber.java:64) >> >> at >> org.apache.stratos.messaging.broker.subscribe.TopicSubscriber.run(TopicSubscriber.java:117) >> >> at java.lang.Thread.run(Thread.java:745) >> >> Caused by: java.net.ConnectException: Connection refused >> >> at java.net.PlainSocketImpl.socketConnect(Native Method) >> >> at >> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) >> >> at >> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) >> >> at >> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) >> >> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) >> >> at java.net.Socket.connect(Socket.java:579) >> >> at >> org.apache.activemq.transport.tcp.TcpTransport.connect(TcpTransport.java:501) >> >> at >> org.apache.activemq.transport.tcp.TcpTransport.doStart(TcpTransport.java:464) >> >> at >> org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55) >> >> at >> org.apache.activemq.transport.AbstractInactivityMonitor.start(AbstractInactivityMonitor.java:138) >> >> at >> org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:58) >> >> at >> org.apache.activemq.transport.WireFormatNegotiator.start(WireFormatNegotiator.java:72) >> >> at >> org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:58) >> >> at >> org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:58) >> >> at >> org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:338) >> >> ... 6 more >> >> TID: [0] [STRATOS] [2014-10-30 19:35:06,666] INFO >> {org.apache.stratos.messaging.broker.subscribe.TopicSubscriber} - Will try >> to subscribe again in 30 sec >> {org.apache.stratos.messaging.broker.subscribe.TopicSubscriber} >> >> TID: [0] [STRATOS] [2014-10-30 19:35:07,146] INFO >> {org.wso2.carbon.core.init.CarbonServerManager} - Shutdown hook >> triggered.... {org.wso2.carbon.core.init.CarbonServerManager} >> >> TID: [0] [STRATOS] [2014-10-30 19:35:07,152] INFO >> {org.wso2.carbon.core.init.CarbonServerManager} - Gracefully shutting down >> Apache Stratos... {org.wso2.carbon.core.init.CarbonServerManager} >> >> TID: [0] [STRATOS] [2014-10-30 19:35:07,156] INFO >> {org.wso2.carbon.core.ServerManagement} - Starting to switch to >> maintenance mode... {org.wso2.carbon.core.ServerManagement} >> >> TID: [0] [STRATOS] [2014-10-30 19:35:07,158] INFO >> {org.wso2.carbon.core.ServerManagement} - Stopped all transport listeners >> {org.wso2.carbon.core.ServerManagement} >> >> TID: [0] [STRATOS] [2014-10-30 19:35:07,158] INFO >> {org.wso2.carbon.core.ServerManagement} - Waiting for request service >> completion... {org.wso2.carbon.core.ServerManagement} >> >> TID: [0] [STRATOS] [2014-10-30 19:35:07,162] INFO >> {org.wso2.carbon.core.ServerManagement} - All requests have been served. >> {org.wso2.carbon.core.ServerManagement} >> >> TID: [0] [STRATOS] [2014-10-30 19:35:07,162] INFO >> {org.wso2.carbon.core.ServerManagement} - Waiting for deployment >> completion... {org.wso2.carbon.core.ServerManagement} >> >> TID: [0] [STRATOS] [2014-10-30 19:35:07,180] INFO >> {org.wso2.carbon.webapp.mgt.WebApplication} - Unloaded webapp: >> StandardEngine[Catalina].StandardHost[localhost].StandardContext[/STRATOS_ROOT] >> {org.wso2.carbon.webapp.mgt.WebApplication} >> >> TID: [0] [STRATOS] [2014-10-30 19:35:07,181] INFO >> {org.springframework.web.context.support.XmlWebApplicationContext} >> >> >> >> >> >> >> -- >> >> Imesh Gunaratne >> >> >> >> Technical Lead, WSO2 >> >> Committer & PMC Member, Apache Stratos >> > > > > -- > Best Regards, > Nirmal > > Nirmal Fernando. > PPMC Member & Committer of Apache Stratos, > Senior Software Engineer, WSO2 Inc. > > Blog: http://nirmalfdo.blogspot.com/ >
