Looks like restoring on the exact same OS version is not fixing the issue... at
starting the cloudstack I get this error :
2024-01-17 09:26:26,067 DEBUG [c.c.u.s.Script] (main:null) (logid:) System
resource: file:/usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh
2024-01-17 09:26:26,067 DEBUG [c.c.u.s.Script] (main:null) (logid:) Absolute
path = /usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh
2024-01-17 09:26:26,068 DEBUG [c.c.s.ConfigurationServerImpl] (main:null)
(logid:) Executing: /bin/bash
/usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh
/var/lib/cloudstack/management/.ssh/id_rsa
2024-01-17 09:26:26,070 DEBUG [c.c.s.ConfigurationServerImpl] (main:null)
(logid:) Executing while with timeout : 3600000
2024-01-17 09:26:26,078 DEBUG [c.c.s.ConfigurationServerImpl] (main:null)
(logid:) Execution is successful.
2024-01-17 09:26:26,083 INFO [c.c.s.ConfigurationServerImpl] (main:null)
(logid:) The script injectkeys.sh was run with result : null
2024-01-17 09:26:26,106 DEBUG [c.c.u.d.DbProperties] (main:null) (logid:) DB
properties were already loaded
2024-01-17 09:26:26,106 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (main:null)
(logid:) Start AsyncJobManager API executor thread pool in size 125
2024-01-17 09:26:26,107 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (main:null)
(logid:) Start AsyncJobManager Work executor thread pool in size 166
2024-01-17 09:26:26,108 INFO [c.c.c.ClusterManagerImpl] (main:null) (logid:)
Start configuring cluster manager : ClusterManagerImpl
2024-01-17 09:26:26,108 DEBUG [c.c.u.d.DbProperties] (main:null) (logid:) DB
properties were already loaded
2024-01-17 09:26:26,109 INFO [c.c.c.ClusterManagerImpl] (main:null) (logid:)
Cluster node IP : 10.60.0.2
2024-01-17 09:26:26,122 INFO [c.c.c.ClusterManagerImpl] (main:null) (logid:)
Trying to connect to 10.60.0.2
2024-01-17 09:26:26,133 ERROR [c.c.c.ClusterManagerImpl] (main:null) (logid:)
Unable to ping management server at 10.60.0.2:9090 due to ConnectException
2024-01-17 09:26:26,134 DEBUG [c.c.c.ClusterManagerImpl] (main:null) (logid:)
Unable to ping management server at 10.60.0.2:9090 due to ConnectException
java.net.ConnectException: Connection refused
at java.base/sun.nio.ch.Net.connect0(Native Method)
at java.base/sun.nio.ch.Net.connect(Net.java:483)
at java.base/sun.nio.ch.Net.connect(Net.java:472)
at
java.base/sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:692)
at
com.cloud.cluster.ClusterManagerImpl.pingManagementNode(ClusterManagerImpl.java:1190)
at
com.cloud.cluster.ClusterManagerImpl.pingManagementNode(ClusterManagerImpl.java:1159)
at
com.cloud.cluster.ClusterManagerImpl.checkConflicts(ClusterManagerImpl.java:1240)
at
com.cloud.cluster.ClusterManagerImpl.configure(ClusterManagerImpl.java:1115)
at
org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle$3.with(CloudStackExtendedLifeCycle.java:114)
at
org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.with(CloudStackExtendedLifeCycle.java:153)
at
org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.configure(CloudStackExtendedLifeCycle.java:110)
at
org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.start(CloudStackExtendedLifeCycle.java:55)
at
org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:178)
at
org.springframework.context.support.DefaultLifecycleProcessor.access$200(DefaultLifecycleProcessor.java:54)
at
org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:356)
at java.base/java.lang.Iterable.forEach(Iterable.java:75)
at
org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:155)
at
org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:123)
at
org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:937)
at
org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:586)
at
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.loadContext(DefaultModuleDefinitionSet.java:144)
at
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet$2.with(DefaultModuleDefinitionSet.java:121)
at
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:244)
at
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:249)
at
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:249)
at
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:232)
at
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.loadContexts(DefaultModuleDefinitionSet.java:116)
at
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.load(DefaultModuleDefinitionSet.java:78)
at
org.apache.cloudstack.spring.module.factory.ModuleBasedContextFactory.loadModules(ModuleBasedContextFactory.java:37)
at
org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.init(CloudStackSpringContext.java:70)
at
org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.<init>(CloudStackSpringContext.java:57)
at
org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.<init>(CloudStackSpringContext.java:61)
at
org.apache.cloudstack.spring.module.web.CloudStackContextLoaderListener.contextInitialized(CloudStackContextLoaderListener.java:52)
at
org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1073)
at
org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:572)
at
org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:1002)
at
org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:765)
at
org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:379)
at
org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1449)
at
org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1414)
at
org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:916)
at
org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:288)
at
org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at
org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:426)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at org.eclipse.jetty.server.Server.start(Server.java:423)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at org.eclipse.jetty.server.Server.doStart(Server.java:387)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.apache.cloudstack.ServerDaemon.start(ServerDaemon.java:192)
at org.apache.cloudstack.ServerDaemon.main(ServerDaemon.java:107)
2024-01-17 09:26:26,137 INFO [c.c.c.ClusterManagerImpl] (main:null) (logid:)
Detected that another management node with the same IP 10.60.0.2 is considered
as running in DB, however it is not pingable, we will continue cluster
initialization with this management server node
2024-01-17 09:26:26,137 INFO [c.c.c.ClusterManagerImpl] (main:null) (logid:)
Cluster manager is configured.
Regards,
Cristian
-----Original Message-----
From: [email protected] <[email protected]>
Sent: Wednesday, January 17, 2024 9:24 AM
To: [email protected]
Subject: RE: MgmtId 345050133919: Req: Resource [Host:11] is unreachable: Host
11: Link is closed
Hi Jithin,
I changed the status to down and set as removed. I still have the same
issue... So, I think I must create a new server with previous OS version, in
any case not sure where is the logic.
Thank you!
Cristian
-----Original Message-----
From: Jithin Raju <[email protected]>
Sent: Wednesday, January 17, 2024 6:16 AM
To: [email protected]
Subject: Re: MgmtId 345050133919: Req: Resource [Host:11] is unreachable: Host
11: Link is closed
Hi Cristian,
Could you locate any stale management server record in the mshost table? If
there is any ,specifically the one in the error ? update it as removed.
-Jithin
From: Cristian Ciobanu <[email protected]>
Date: Wednesday, 17 January 2024 at 4:16 AM
To: [email protected] <[email protected]>
Subject: Re: MgmtId 345050133919: Req: Resource [Host:11] is unreachable: Host
11: Link is closed Looks like I have the same issue as I had some time ago...
"host time out after cloudstack management restore". Old topic, 2022.
When I restored the backup on a different os, from Rocky to Ubuntu.. but now
I used the same os... Different version..
This looks like a bug. I'm trying to identify the route cause.
Regards,
Cristian
On Wed, Jan 17, 2024, 00:20 <[email protected]> wrote:
> Hello,
>
>
>
> I have a CloudStack 4.18.1 restored from a DB backup due to server
> failure. The CloudStack is running; I see all the hosts with state
> 'Up', but it looks like it's stuck, nothing is working. I get a
> similar error for all hosts: “MgmtId 345050133919: Req: Resource [Host:11] is
> unreachable:
> Host 11: Link is closed,” even if the hosts are reachable from
> CloudStack and vSphere, where they are connected and manageable. I do
> not see any events in vSphere.
>
>
>
> Any suggestions as to why CloudStack does not see that the servers are
> reachable or why I get a timeout?
>
>
>
> Thank you,
>
> Cristian
>
