Hi, Can you mark the mgmt server as Down (or set "removed" to "now()") in the "mshost" table , and retry?
-Wei On Wed, 17 Jan 2024 at 10:29, <[email protected]> wrote: > Looks like restoring on the exact same OS version is not fixing the > issue... at starting the cloudstack I get this error : > > 2024-01-17 09:26:26,067 DEBUG [c.c.u.s.Script] (main:null) (logid:) System > resource: > file:/usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh > 2024-01-17 09:26:26,067 DEBUG [c.c.u.s.Script] (main:null) (logid:) > Absolute path = > /usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh > 2024-01-17 09:26:26,068 DEBUG [c.c.s.ConfigurationServerImpl] (main:null) > (logid:) Executing: /bin/bash > /usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh > /var/lib/cloudstack/management/.ssh/id_rsa > 2024-01-17 09:26:26,070 DEBUG [c.c.s.ConfigurationServerImpl] (main:null) > (logid:) Executing while with timeout : 3600000 > 2024-01-17 09:26:26,078 DEBUG [c.c.s.ConfigurationServerImpl] (main:null) > (logid:) Execution is successful. > 2024-01-17 09:26:26,083 INFO [c.c.s.ConfigurationServerImpl] (main:null) > (logid:) The script injectkeys.sh was run with result : null > 2024-01-17 09:26:26,106 DEBUG [c.c.u.d.DbProperties] (main:null) (logid:) > DB properties were already loaded > 2024-01-17 09:26:26,106 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] > (main:null) (logid:) Start AsyncJobManager API executor thread pool in size > 125 > 2024-01-17 09:26:26,107 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] > (main:null) (logid:) Start AsyncJobManager Work executor thread pool in > size 166 > 2024-01-17 09:26:26,108 INFO [c.c.c.ClusterManagerImpl] (main:null) > (logid:) Start configuring cluster manager : ClusterManagerImpl > 2024-01-17 09:26:26,108 DEBUG [c.c.u.d.DbProperties] (main:null) (logid:) > DB properties were already loaded > 2024-01-17 09:26:26,109 INFO [c.c.c.ClusterManagerImpl] (main:null) > (logid:) Cluster node IP : 10.60.0.2 > 2024-01-17 09:26:26,122 INFO [c.c.c.ClusterManagerImpl] (main:null) > (logid:) Trying to connect to 10.60.0.2 > 2024-01-17 09:26:26,133 ERROR [c.c.c.ClusterManagerImpl] (main:null) > (logid:) Unable to ping management server at 10.60.0.2:9090 due to > ConnectException > 2024-01-17 09:26:26,134 DEBUG [c.c.c.ClusterManagerImpl] (main:null) > (logid:) Unable to ping management server at 10.60.0.2:9090 due to > ConnectException > java.net.ConnectException: Connection refused > at java.base/sun.nio.ch.Net.connect0(Native Method) > at java.base/sun.nio.ch.Net.connect(Net.java:483) > at java.base/sun.nio.ch.Net.connect(Net.java:472) > at java.base/sun.nio.ch > .SocketChannelImpl.connect(SocketChannelImpl.java:692) > at > com.cloud.cluster.ClusterManagerImpl.pingManagementNode(ClusterManagerImpl.java:1190) > at > com.cloud.cluster.ClusterManagerImpl.pingManagementNode(ClusterManagerImpl.java:1159) > at > com.cloud.cluster.ClusterManagerImpl.checkConflicts(ClusterManagerImpl.java:1240) > at > com.cloud.cluster.ClusterManagerImpl.configure(ClusterManagerImpl.java:1115) > at > org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle$3.with(CloudStackExtendedLifeCycle.java:114) > at > org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.with(CloudStackExtendedLifeCycle.java:153) > at > org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.configure(CloudStackExtendedLifeCycle.java:110) > at > org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.start(CloudStackExtendedLifeCycle.java:55) > at > org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:178) > at > org.springframework.context.support.DefaultLifecycleProcessor.access$200(DefaultLifecycleProcessor.java:54) > at > org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:356) > at java.base/java.lang.Iterable.forEach(Iterable.java:75) > at > org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:155) > at > org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:123) > at > org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:937) > at > org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:586) > at > org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.loadContext(DefaultModuleDefinitionSet.java:144) > at > org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet$2.with(DefaultModuleDefinitionSet.java:121) > at > org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:244) > at > org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:249) > at > org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:249) > at > org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:232) > at > org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.loadContexts(DefaultModuleDefinitionSet.java:116) > at > org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.load(DefaultModuleDefinitionSet.java:78) > at > org.apache.cloudstack.spring.module.factory.ModuleBasedContextFactory.loadModules(ModuleBasedContextFactory.java:37) > at > org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.init(CloudStackSpringContext.java:70) > at > org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.<init>(CloudStackSpringContext.java:57) > at > org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.<init>(CloudStackSpringContext.java:61) > at > org.apache.cloudstack.spring.module.web.CloudStackContextLoaderListener.contextInitialized(CloudStackContextLoaderListener.java:52) > at > org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1073) > at > org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:572) > at > org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:1002) > at > org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:765) > at > org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:379) > at > org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1449) > at > org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1414) > at > org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:916) > at > org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:288) > at > org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524) > at > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) > at > org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) > at > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110) > at > org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97) > at > org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:426) > at > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) > at > org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) > at > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) > at > org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97) > at > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) > at > org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) > at org.eclipse.jetty.server.Server.start(Server.java:423) > at > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110) > at > org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97) > at org.eclipse.jetty.server.Server.doStart(Server.java:387) > at > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) > at org.apache.cloudstack.ServerDaemon.start(ServerDaemon.java:192) > at org.apache.cloudstack.ServerDaemon.main(ServerDaemon.java:107) > 2024-01-17 09:26:26,137 INFO [c.c.c.ClusterManagerImpl] (main:null) > (logid:) Detected that another management node with the same IP 10.60.0.2 > is considered as running in DB, however it is not pingable, we will > continue cluster initialization with this management server node > 2024-01-17 09:26:26,137 INFO [c.c.c.ClusterManagerImpl] (main:null) > (logid:) Cluster manager is configured. > > > Regards, > Cristian > > > > > -----Original Message----- > From: [email protected] <[email protected]> > Sent: Wednesday, January 17, 2024 9:24 AM > To: [email protected] > Subject: RE: MgmtId 345050133919: Req: Resource [Host:11] is unreachable: > Host 11: Link is closed > > Hi Jithin, > > I changed the status to down and set as removed. I still have the same > issue... So, I think I must create a new server with previous OS version, > in any case not sure where is the logic. > > Thank you! > Cristian > > -----Original Message----- > From: Jithin Raju <[email protected]> > Sent: Wednesday, January 17, 2024 6:16 AM > To: [email protected] > Subject: Re: MgmtId 345050133919: Req: Resource [Host:11] is unreachable: > Host 11: Link is closed > > Hi Cristian, > > Could you locate any stale management server record in the mshost table? > If there is any ,specifically the one in the error ? update it as removed. > > -Jithin > > From: Cristian Ciobanu <[email protected]> > Date: Wednesday, 17 January 2024 at 4:16 AM > To: [email protected] <[email protected]> > Subject: Re: MgmtId 345050133919: Req: Resource [Host:11] is unreachable: > Host 11: Link is closed Looks like I have the same issue as I had some time > ago... > > "host time out after cloudstack management restore". Old topic, 2022. > When I restored the backup on a different os, from Rocky to Ubuntu.. but > now I used the same os... Different version.. > > This looks like a bug. I'm trying to identify the route cause. > > > Regards, > Cristian > > On Wed, Jan 17, 2024, 00:20 <[email protected]> wrote: > > > Hello, > > > > > > > > I have a CloudStack 4.18.1 restored from a DB backup due to server > > failure. The CloudStack is running; I see all the hosts with state > > 'Up', but it looks like it's stuck, nothing is working. I get a > > similar error for all hosts: “MgmtId 345050133919: Req: Resource > [Host:11] is unreachable: > > Host 11: Link is closed,” even if the hosts are reachable from > > CloudStack and vSphere, where they are connected and manageable. I do > > not see any events in vSphere. > > > > > > > > Any suggestions as to why CloudStack does not see that the servers are > > reachable or why I get a timeout? > > > > > > > > Thank you, > > > > Cristian > > > > > > > >
