GitHub user TheKunalSen added a comment to the discussion: Cloudstack Migrate
VM Query
i have turned off the host and the host went to down and maintainance but the
vm is still on that host, the vm ha did not started
and i am getting this message in the logs
Unable to find next HA state for current HA state=[Checking]
for event=[HealthCheckFailed]
the full log
2026-02-16 05:51:46,745 WARN [o.a.c.h.HAManagerImpl] (pool-2-thread-49:[])
(logid:) Unable to find next HA state for current HA state=[Checking] for
event=[HealthCheckFailed] for host Host
{"id":8,"name":"compute3","type":"Routing","uuid":"c64a6f6b-3b43-43c7-b6b5-9b2510a056b0"}
with id 8. com.cloud.utils.fsm.NoTransitionException: Unable to transition to
a new state from Checking via HealthCheckFailed at
com.cloud.utils.fsm.StateMachine2.getTransition(StateMachine2.java:108) at
com.cloud.utils.fsm.StateMachine2.getNextState(StateMachine2.java:94) at
org.apache.cloudstack.ha.HAManagerImpl.transitionHAState(HAManagerImpl.java:153)
at jdk.internal.reflect.GeneratedMethodAccessor136.invoke(Unknown Source) at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569) at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.
aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at jdk.proxy3/jdk.proxy3.$Proxy417.transitionHAState(Unknown Source) at
org.apache.cloudstack.ha.task.HealthCheckTask.processResult(HealthCheckTask.java:57)
at org.apache.cloudstack.ha.task.BaseHATask.call(BaseHATask.java:105) at
org.apache.cloudstack.ha.task.BaseHATask.call(BaseHATask.java:37) at
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.
concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at
java.base/java.lang.Thread.run(Thread.java:840)
outing","uuid":"c64a6f6b-3b43-43c7-b6b5-9b2510a056b0"} not being up 2026-02-16
05:53:39,242 DEBUG [o.a.c.k.h.KVMHostActivityChecker] (pool-2-thread-27:[])
(logid:) Setting Host
{"id":8,"name":"compute3","type":"Routing","uuid":"c64a6f6b-3b43-43c7-b6b5-9b2510a056b0"}
to "Disconnected" status. 2026-02-16 05:53:39,244 DEBUG
[o.a.c.k.h.KVMHostActivityChecker] (pool-2-thread-27:[]) (logid:) Investigating
Host
{"id":8,"name":"compute3","type":"Routing","uuid":"c64a6f6b-3b43-43c7-b6b5-9b2510a056b0"}
via neighbouring Host
{"id":5,"name":"compute1","type":"Routing","uuid":"54056d1f-2ba5-4361-950d-63490eae20ba"}.
2026-02-16 05:53:39,245 DEBUG [o.a.c.k.h.KVMHostActivityChecker]
(BackgroundTaskPollManager-6:[ctx-046cf5e7]) (logid:807a4423) Retrieving all
"Up" hosts from cluster [Cluster {id: "3", name: "IEM_Compute_Cluster_1", uuid:
"ba7c3ced-1cad-4d86-b868-4163d5fba7dd"}]...
2026-02-16 05:53:39,246 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
(pool-2-thread-27:[]) (logid:) Wait time setting on
com.cloud.agent.api.CheckOnHostCommand is 20 seconds 2026-02-16 05:53:39,246
DEBUG [o.a.c.k.h.KVMHostActivityChecker] (pool-2-thread-10:[]) (logid:)
Checking Host
{"id":5,"name":"compute1","type":"Routing","uuid":"54056d1f-2ba5-4361-950d-63490eae20ba"}
status... 2026-02-16 05:53:39,247 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
(pool-2-thread-10:[]) (logid:) Wait time setting on
com.cloud.agent.api.CheckOnHostCommand is 20 seconds 2026-02-16 05:53:39,248
DEBUG [c.c.a.m.ClusteredAgentAttache] (pool-2-thread-27:[]) (logid:) Seq
6-6009209277795865144: Routed from 163763490546348 2026-02-16 05:53:39,248
DEBUG [c.c.a.t.Request] (pool-2-thread-27:[]) (logid:) Seq
5-6009209277795865144: Sending { Cmd , MgmtId: 163763490546348, via:
5(compute1), Ver: v1, Flags: 100011,
[{"com.cloud.agent.api.CheckOnHostCommand":{"host":{"guid":"5cbb4794-681d-349e-8632-69348bdd67e2-LibvirtCom
putingResource","privateNetwork":{"ip":"172.16.17.205","netmask":"255.255.255.0","mac":"56:f2:7a:a7:be:54","securityGroupEnabled":"false"},"publicNetwork":{"ip":"172.16.17.205","netmask":"255.255.255.0","mac":"56:f2:7a:a7:be:54","securityGroupEnabled":"false"},"storageNetwork1":{"ip":"172.16.17.205","netmask":"255.255.255.0","mac":"56:f2:7a:a7:be:54","securityGroupEnabled":"false"}},"reportCheckFailureIfOneStorageIsDown":"true","wait":"20","bypassHostMaintenance":"false"}}]
} 2026-02-16 05:53:39,249 DEBUG [c.c.a.m.ClusteredAgentAttache]
(pool-2-thread-10:[]) (logid:) Seq 6-6009209277795865145: Routed from
163763490546348 2026-02-16 05:53:39,249 DEBUG [c.c.a.t.Request]
(pool-2-thread-10:[]) (logid:) Seq 5-6009209277795865145: Sending { Cmd ,
MgmtId: 163763490546348, via: 5(compute1), Ver: v1, Flags: 100011,
[{"com.cloud.agent.api.CheckOnHostCommand":{"host":{"guid":"acd4f02e-1b9f-39dc-bf36-9532c2463d62-LibvirtComputingResource","privateNetwork":{"ip":"172.16.17.203","netmask":"255.25
5.255.0","mac":"5a:41:d9:58:bd:d1","securityGroupEnabled":"false"},"publicNetwork":{"ip":"172.16.17.203","netmask":"255.255.255.0","mac":"5a:41:d9:58:bd:d1","securityGroupEnabled":"false"},"storageNetwork1":{"ip":"172.16.17.203","netmask":"255.255.255.0","mac":"5a:41:d9:58:bd:d1","securityGroupEnabled":"false"}},"reportCheckFailureIfOneStorageIsDown":"true","wait":"20","bypassHostMaintenance":"false"}}]
} 2026-02-16 05:53:39,301 DEBUG [c.c.a.t.Request] (AgentManager-Handler-14:[])
(logid:) Seq 9-6636617000883919100: Processing: { Ans: , MgmtId:
163763490546348, via: 9, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.Answer":{"result":"false","details":"Heart is
beating...","wait":"0","bypassHostMaintenance":"false"}}] } 2026-02-16
05:53:39,301 DEBUG [c.c.a.t.Request] (pool-2-thread-20:[]) (logid:) Seq
9-6636617000883919100: Received: { Ans: , MgmtId: 163763490546348, via:
9(compute2), Ver: v1, Flags: 10, { Answer } } 2026-02-16 05:53:39,301 DEBUG
[c.c.a.m.ClusteredAgentManagerImpl] (pool
-2-thread-20:[]) (logid:) Details from executing class
com.cloud.agent.api.CheckOnHostCommand: Heart is beating... 2026-02-16
05:53:39,302 DEBUG [o.a.c.k.h.KVMHostActivityChecker] (pool-2-thread-20:[])
(logid:) Host
{"id":9,"name":"compute2","type":"Routing","uuid":"6a96b74f-b114-4b26-b810-018ab626e575"}
has the status [Up]. 2026-02-16 05:53:39,302 DEBUG [o.a.c.h.HAManagerImpl]
(pool-2-thread-40:[]) (logid:) HA state pre-transition:: new state=[Available],
old state=[Available], for resource id=[9], status=[true], ha config
state=[Available]. 2026-02-16 05:53:39,307 DEBUG [c.c.a.t.Request]
(AgentManager-Handler-4:[]) (logid:) Seq 5-6009209277795865144: Processing: {
Ans: , MgmtId: 163763490546348, via: 5, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.Answer":{"result":"true","wait":"0","bypassHostMaintenance":"false"}}]
} 2026-02-16 05:53:39,307 DEBUG [c.c.a.t.Request] (pool-2-thread-27:[])
(logid:) Seq 5-6009209277795865144: Received: { Ans: , MgmtId: 163763490546348,
via: 5(compute1)
, Ver: v1, Flags: 10, { Answer } } 2026-02-16 05:53:39,308 DEBUG
[o.a.c.k.h.KVMHostActivityChecker] (pool-2-thread-27:[]) (logid:) Neighbouring
Host
{"id":5,"name":"compute1","type":"Routing","uuid":"54056d1f-2ba5-4361-950d-63490eae20ba"}
returned status [Down] for the investigated Host
{"id":8,"name":"compute3","type":"Routing","uuid":"c64a6f6b-3b43-43c7-b6b5-9b2510a056b0"}.
2026-02-16 05:53:39,308 DEBUG [c.c.a.t.Request] (AgentManager-Handler-10:[])
(logid:) Seq 5-6009209277795865145: Processing: { Ans: , MgmtId:
163763490546348, via: 5, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.Answer":{"result":"false","details":"Heart is
beating...","wait":"0","bypassHostMaintenance":"false"}}] } 2026-02-16
05:53:39,308 DEBUG [c.c.a.t.Request] (pool-2-thread-10:[]) (logid:) Seq
5-6009209277795865145: Received: { Ans: , MgmtId: 163763490546348, via:
5(compute1), Ver: v1, Flags: 10, { Answer } } 2026-02-16 05:53:39,309 DEBUG
[c.c.a.m.ClusteredAgentManagerImpl] (pool-2-thread-10:[]) (logid:) Deta
ils from executing class com.cloud.agent.api.CheckOnHostCommand: Heart is
beating... 2026-02-16 05:53:39,309 DEBUG [o.a.c.k.h.KVMHostActivityChecker]
(pool-2-thread-40:[]) (logid:) Retrieving all "Up" hosts from cluster [Cluster
{id: "3", name: "IEM_Compute_Cluster_1", uuid:
"ba7c3ced-1cad-4d86-b868-4163d5fba7dd"}]... 2026-02-16 05:53:39,309 DEBUG
[o.a.c.k.h.KVMHostActivityChecker] (pool-2-thread-27:[]) (logid:) Investigating
Host
{"id":8,"name":"compute3","type":"Routing","uuid":"c64a6f6b-3b43-43c7-b6b5-9b2510a056b0"}
via neighbouring Host
{"id":9,"name":"compute2","type":"Routing","uuid":"6a96b74f-b114-4b26-b810-018ab626e575"}.
2026-02-16 05:53:39,309 DEBUG [o.a.c.k.h.KVMHostActivityChecker]
(pool-2-thread-10:[]) (logid:) Host
{"id":5,"name":"compute1","type":"Routing","uuid":"54056d1f-2ba5-4361-950d-63490eae20ba"}
has the status [Up]. 2026-02-16 05:53:39,309 DEBUG [o.a.c.h.HAManagerImpl]
(pool-2-thread-11:[]) (logid:) HA state pre-transition:: new state=[Available],
old state=[Av
ailable], for resource id=[5], status=[true], ha config state=[Available].
2026-02-16 05:53:39,310 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
(pool-2-thread-27:[]) (logid:) Wait time setting on
com.cloud.agent.api.CheckOnHostCommand is 20 seconds 2026-02-16 05:53:39,311
DEBUG [c.c.a.m.ClusteredAgentAttache] (pool-2-thread-27:[]) (logid:) Seq
6-6636617000883919101: Routed from 163763490546348 2026-02-16 05:53:39,312
DEBUG [c.c.a.t.Request] (pool-2-thread-27:[]) (logid:) Seq
9-6636617000883919101: Sending { Cmd , MgmtId: 163763490546348, via:
9(compute2), Ver: v1, Flags: 100011,
[{"com.cloud.agent.api.CheckOnHostCommand":{"host":{"guid":"5cbb4794-681d-349e-8632-69348bdd67e2-LibvirtComputingResource","privateNetwork":{"ip":"172.16.17.205","netmask":"255.255.255.0","mac":"56:f2:7a:a7:be:54","securityGroupEnabled":"false"},"publicNetwork":{"ip":"172.16.17.205","netmask":"255.255.255.0","mac":"56:f2:7a:a7:be:54","securityGroupEnabled":"false"},"storageNetwork1":{"ip":"172.16.17.205","netma
sk":"255.255.255.0","mac":"56:f2:7a:a7:be:54","securityGroupEnabled":"false"}},"reportCheckFailureIfOneStorageIsDown":"true","wait":"20","bypassHostMaintenance":"false"}}]
} 2026-02-16 05:53:39,316 DEBUG [o.a.c.k.h.KVMHostActivityChecker]
(pool-2-thread-11:[]) (logid:) Retrieving all "Up" hosts from cluster [Cluster
{id: "3", name: "IEM_Compute_Cluster_1", uuid:
"ba7c3ced-1cad-4d86-b868-4163d5fba7dd"}]... 2026-02-16 05:53:39,345 DEBUG
[c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) Ping
from Routing host AgentAttache
{"_id":9,"_name":"compute2","_uuid":"6a96b74f-b114-4b26-b810-018ab626e575"}
2026-02-16 05:53:39,347 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(AgentManager-Handler-13:[]) (logid:) Process host VM state report from ping
process. host: Host
{"id":9,"name":"compute2","type":"Routing","uuid":"6a96b74f-b114-4b26-b810-018ab626e575"}
2026-02-16 05:53:39,348 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(AgentManager-Handler-13:[]) (logid:) Proce
ss VM state report. Host
{"id":9,"name":"compute2","type":"Routing","uuid":"6a96b74f-b114-4b26-b810-018ab626e575"},
number of records in report: 9. VMs: [321:PowerOn, 197:PowerOn, 248:PowerOn,
25:PowerOn, 314:PowerOn, 187:PowerOn, 316:PowerOn, 318:PowerOn, 319:PowerOn]]
2026-02-16 05:53:39,351 DEBUG
GitHub link:
https://github.com/apache/cloudstack/discussions/12629#discussioncomment-15819802
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
