kiranchavala commented on issue #12411:
URL: https://github.com/apache/cloudstack/issues/12411#issuecomment-3743070850
@yadvr Not able to reproduce the issue
1. Launched a 2 vm's one with HA enabled and another not ha enable on a KVM
host 1
2. Made sure the global setting 'commands.timeout' value to a suitable value
like "CheckHealthCommand=5,CheckOnHostCommand=5"
3. Shutdown the kvm host 1
4. The router and HA enabled vm restarted on another kvm host
5. The non-ha enabled vm went into stopped state
logs of ha enabled vm
```
root@acs420:~# cat /var/log/cloudstack/management/management-server.log
|grep -i "logid:48d451aa"
2026-01-13 08:38:08,653 INFO [c.c.h.HighAvailabilityManagerExtImpl]
(HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Processing work
HAWork[577-HA-155-Running-Investigating]
2026-01-13 08:38:08,654 DEBUG [c.c.h.HighAvailabilityManagerExtImpl]
(HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) RESTART with HAWORK
2026-01-13 08:38:08,656 DEBUG [c.c.h.HighAvailabilityManagerExtImpl]
(HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) SimpleInvestigator
unable to determine the state of the host. Moving on.
2026-01-13 08:38:08,656 DEBUG [c.c.h.HighAvailabilityManagerExtImpl]
(HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) XenServerInvestigator
unable to determine the state of the host. Moving on.
2026-01-13 08:38:08,657 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
(HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Can not send command
com.cloud.agent.api.CheckOnHostCommand due to Host Host
{"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"}
not being up
2026-01-13 08:38:08,658 DEBUG [c.c.h.KVMInvestigator]
(HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Investigating host:Host
{"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"}
via neighbouring host:Host
{"id":7,"name":"kvmhost2","type":"Routing","uuid":"c30b8e0d-0a48-40fa-bb57-c2de13513ca5"}
2026-01-13 08:38:08,658 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
(HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Wait time setting on
com.cloud.agent.api.CheckOnHostCommand is 5 seconds
2026-01-13 08:38:08,659 DEBUG [c.c.a.m.ClusteredAgentAttache]
(HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Seq
21-5292855462067175506: Routed from 206863097656310
2026-01-13 08:38:08,659 DEBUG [c.c.a.t.Request] (HA-Worker-0:[ctx-0d486e07,
work-577]) (logid:48d451aa) Seq 7-5292855462067175506: Sending { Cmd , MgmtId:
206863097656310, via: 7(kvmhost2), Ver: v1, Flags: 100011,
[{"com.cloud.agent.api.CheckOnHostCommand":{"host":{"guid":"0218d19d-3cbe-3827-bb03-7fd636ad202b-LibvirtComputingResource","privateNetwork":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"},"publicNetwork":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"},"storageNetwork1":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"}},"reportCheckFailureIfOneStorageIsDown":"false","wait":"5","bypassHostMaintenance":"false"}}]
}
2026-01-13 08:38:08,674 DEBUG [c.c.a.t.Request] (HA-Worker-0:[ctx-0d486e07,
work-577]) (logid:48d451aa) Seq 7-5292855462067175506: Received: { Ans: ,
MgmtId: 206863097656310, via: 7(kvmhost2), Ver: v1, Flags: 10, { Answer } }
2026-01-13 08:38:08,674 DEBUG [c.c.h.KVMInvestigator]
(HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Neighbouring host:Host
{"id":7,"name":"kvmhost2","type":"Routing","uuid":"c30b8e0d-0a48-40fa-bb57-c2de13513ca5"}
returned status:Down for the investigated host:Host
{"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"}
2026-01-13 08:38:08,675 DEBUG [c.c.h.KVMInvestigator]
(HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Investigating host:Host
{"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"}
via neighbouring host:Host
{"id":13,"name":"kvmhost3","type":"Routing","uuid":"3cfc4b1f-1aa9-40e5-a549-484ab4fcc921"}
2026-01-13 08:38:08,675 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
(HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Wait time setting on
com.cloud.agent.api.CheckOnHostCommand is 5 seconds
2026-01-13 08:38:08,676 DEBUG [c.c.a.m.ClusteredAgentAttache]
(HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Seq
21-5194902170171867174: Routed from 206863097656310
2026-01-13 08:38:08,676 DEBUG [c.c.a.t.Request] (HA-Worker-0:[ctx-0d486e07,
work-577]) (logid:48d451aa) Seq 13-5194902170171867174: Sending { Cmd ,
MgmtId: 206863097656310, via: 13(kvmhost3), Ver: v1, Flags: 100011,
[{"com.cloud.agent.api.CheckOnHostCommand":{"host":{"guid":"0218d19d-3cbe-3827-bb03-7fd636ad202b-LibvirtComputingResource","privateNetwork":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"},"publicNetwork":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"},"storageNetwork1":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"}},"reportCheckFailureIfOneStorageIsDown":"false","wait":"5","bypassHostMaintenance":"false"}}]
}
2026-01-13 08:38:08,723 DEBUG [c.c.a.t.Request] (HA-Worker-0:[ctx-0d486e07,
work-577]) (logid:48d451aa) Seq 13-5194902170171867174: Received: { Ans: ,
MgmtId: 206863097656310, via: 13(kvmhost3), Ver: v1, Flags: 10, { Answer } }
2026-01-13 08:38:08,723 DEBUG [c.c.h.KVMInvestigator]
(HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Neighbouring host:Host
{"id":13,"name":"kvmhost3","type":"Routing","uuid":"3cfc4b1f-1aa9-40e5-a549-484ab4fcc921"}
returned status:Down for the investigated host:Host
{"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"}
2026-01-13 08:38:08,723 DEBUG [c.c.h.KVMInvestigator]
(HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) HA: HOST is ineligible
legacy state Down for host Host
{"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"}
2026-01-13 08:38:08,723 DEBUG [c.c.h.HighAvailabilityManagerExtImpl]
(HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) KVMInvestigator was
able to determine host Host
{"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"}
is in Down
2026-01-13 08:38:08,723 INFO [c.c.h.HighAvailabilityManagerExtImpl]
(HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) HA on VM instance
{"id":155,"instanceName":"i-2-155-VM","state":"Running","type":"User","uuid":"0597fa15-2dc4-4a92-9c0d-4025bd5568c6"}
2026-01-13 08:38:08,725 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
(HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Wait time setting on
com.cloud.agent.api.CheckVirtualMachineCommand is 20 seconds
2026-01-13 08:38:08,725 DEBUG [c.c.h.CheckOnAgentInvestigator]
(HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Unable to reach the
agent for VM instance
{"id":155,"instanceName":"i-2-155-VM","state":"Running","type":"User","uuid":"0597fa15-2dc4-4a92-9c0d-4025bd5568c6"}:
Resource [Host:6] is unreachable: Host 6: Host with specified id is not in the
right state: Down
2026-01-13 08:38:08,725 INFO [c.c.h.HighAvailabilityManagerExtImpl]
(HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) SimpleInvestigator
could not find VM instance
{"id":155,"instanceName":"i-2-155-VM","state":"Running","type":"User","uuid":"0597fa15-2dc4-4a92-9c0d-4025bd5568c6"}
2026-01-13 08:38:08,725 INFO [c.c.h.HighAvailabilityManagerExtImpl]
(HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) XenServerInvestigator
could not find VM instance
{"id":155,"instanceName":"i-2-155-VM","state":"Running","type":"User","uuid":"0597fa15-2dc4-4a92-9c0d-4025bd5568c6"}
2026-01-13 08:38:08,727 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
(HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Can not send command
com.cloud.agent.api.CheckOnHostCommand due to Host Host
{"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"}
not being up
2026-01-13 08:38:08,727 DEBUG [c.c.h.KVMInvestigator]
(HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Investigating host:Host
{"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"}
via neighbouring host:Host
{"id":7,"name":"kvmhost2","type":"Routing","uuid":"c30b8e0d-0a48-40fa-bb57-c2de13513ca5"}
2026-01-13 08:38:08,728 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
(HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Wait time setting on
com.cloud.agent.api.CheckOnHostCommand is 5 seconds
2026-01-13 08:38:08,728 DEBUG [c.c.a.m.ClusteredAgentAttache]
(HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Seq
21-5292855462067175509: Routed from 206863097656310
2026-01-13 08:38:08,728 DEBUG [c.c.a.t.Request] (HA-Worker-0:[ctx-0d486e07,
work-577]) (logid:48d451aa) Seq 7-5292855462067175509: Sending { Cmd , MgmtId:
206863097656310, via: 7(kvmhost2), Ver: v1, Flags: 100011,
[{"com.cloud.agent.api.CheckOnHostCommand":{"host":{"guid":"0218d19d-3cbe-3827-bb03-7fd636ad202b-LibvirtComputingResource","privateNetwork":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"},"publicNetwork":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"},"storageNetwork1":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"}},"reportCheckFailureIfOneStorageIsDown":"false","wait":"5","bypassHostMaintenance":"false"}}]
}
2026-01-13 08:38:08,771 DEBUG [c.c.a.t.Request] (HA-Worker-0:[ctx-0d486e07,
work-577]) (logid:48d451aa) Seq 7-5292855462067175509: Received: { Ans: ,
MgmtId: 206863097656310, via: 7(kvmhost2), Ver: v1, Flags: 10, { Answer } }
2026-01-13 08:38:08,771 DEBUG [c.c.h.KVMInvestigator]
(HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Neighbouring host:Host
{"id":7,"name":"kvmhost2","type":"Routing","uuid":"c30b8e0d-0a48-40fa-bb57-c2de13513ca5"}
returned status:Down for the investigated host:Host
{"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"}
2026-01-13 08:38:08,771 DEBUG [c.c.h.KVMInvestigator]
(HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Investigating host:Host
{"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"}
via neighbouring host:Host
{"id":13,"name":"kvmhost3","type":"Routing","uuid":"3cfc4b1f-1aa9-40e5-a549-484ab4fcc921"}
2026-01-13 08:38:08,771 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
(HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Wait time setting on
com.cloud.agent.api.CheckOnHostCommand is 5 seconds
2026-01-13 08:38:08,772 DEBUG [c.c.a.m.ClusteredAgentAttache]
(HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Seq
21-5194902170171867176: Routed from 206863097656310
2026-01-13 08:38:08,772 DEBUG [c.c.a.t.Request] (HA-Worker-0:[ctx-0d486e07,
work-577]) (logid:48d451aa) Seq 13-5194902170171867176: Sending { Cmd ,
MgmtId: 206863097656310, via: 13(kvmhost3), Ver: v1, Flags: 100011,
[{"com.cloud.agent.api.CheckOnHostCommand":{"host":{"guid":"0218d19d-3cbe-3827-bb03-7fd636ad202b-LibvirtComputingResource","privateNetwork":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"},"publicNetwork":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"},"storageNetwork1":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"}},"reportCheckFailureIfOneStorageIsDown":"false","wait":"5","bypassHostMaintenance":"false"}}]
}
2026-01-13 08:38:08,780 DEBUG [c.c.a.t.Request] (HA-Worker-0:[ctx-0d486e07,
work-577]) (logid:48d451aa) Seq 13-5194902170171867176: Received: { Ans: ,
MgmtId: 206863097656310, via: 13(kvmhost3), Ver: v1, Flags: 10, { Answer } }
2026-01-13 08:38:08,780 DEBUG [c.c.h.KVMInvestigator]
(HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Neighbouring host:Host
{"id":13,"name":"kvmhost3","type":"Routing","uuid":"3cfc4b1f-1aa9-40e5-a549-484ab4fcc921"}
returned status:Down for the investigated host:Host
{"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"}
2026-01-13 08:38:08,780 DEBUG [c.c.h.KVMInvestigator]
(HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) HA: HOST is ineligible
legacy state Down for host Host
{"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"}
2026-01-13 08:38:08,780 DEBUG [c.c.h.KVMInvestigator]
(HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) HA: HOST is ineligible
legacy state Down for host Host
{"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"}
2026-01-13 08:38:08,780 INFO [c.c.h.HighAvailabilityManagerExtImpl]
(HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) KVMInvestigator could
not find VM instance
{"id":155,"instanceName":"i-2-155-VM","state":"Running","type":"User","uuid":"0597fa15-2dc4-4a92-9c0d-4025bd5568c6"}
2026-01-13 08:38:08,780 INFO [c.c.h.HighAvailabilityManagerExtImpl]
(HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) HypervInvestigator
could not find VM instance
{"id":155,"instanceName":"i-2-155-VM","state":"Running","type":"User","uuid":"0597fa15-2dc4-4a92-9c0d-4025bd5568c6"}
2026-01-13 08:38:08,781 INFO [c.c.h.HighAvailabilityManagerExtImpl]
(HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) VMwareInvestigator
could not find VM instance
{"id":155,"instanceName":"i-2-155-VM","state":"Running","type":"User","uuid":"0597fa15-2dc4-4a92-9c0d-4025bd5568c6"}
2026-01-13 08:38:08,781 DEBUG [c.c.h.UserVmDomRInvestigator]
(HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) testing if VM instance
{"id":155,"instanceName":"i-2-155-VM","state":"Running","type":"User","uuid":"0597fa15-2dc4-4a92-9c0d-4025bd5568c6"}
is alive
2026-01-13 08:38:08,783 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
(HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Can not send command
com.cloud.agent.api.PingTestCommand due to Host Host
{"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"}
not being up
2026-01-13 08:38:08,783 DEBUG [c.c.h.UserVmDomRInvestigator]
(HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) VM instance
{"id":155,"instanceName":"i-2-155-VM","state":"Running","type":"User","uuid":"0597fa15-2dc4-4a92-9c0d-4025bd5568c6"}
could not be pinged, returning that it is unknown
2026-01-13 08:38:08,783 DEBUG [c.c.h.UserVmDomRInvestigator]
(HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Returning null since
we're unable to determine state of VM instance
{"id":155,"instanceName":"i-2-155-VM","state":"Running","type":"User","uuid":"0597fa15-2dc4-4a92-9c0d-4025bd5568c6"}
2026-01-13 08:38:08,783 INFO [c.c.h.HighAvailabilityManagerExtImpl]
(HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) PingInvestigator could
not find VM instance
{"id":155,"instanceName":"i-2-155-VM","state":"Running","type":"User","uuid":"0597fa15-2dc4-4a92-9c0d-4025bd5568c6"}
2026-01-13 08:38:08,783 DEBUG [c.c.h.ManagementIPSystemVMInvestigator]
(HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Not a System Vm, unable
to determine state of VM instance
{"id":155,"instanceName":"i-2-155-VM","state":"Running","type":"User","uuid":"0597fa15-2dc4-4a92-9c0d-4025bd5568c6"}
returning null
2026-01-13 08:38:08,784 DEBUG [c.c.h.ManagementIPSystemVMInvestigator]
(HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Testing if VM instance
{"id":155,"instanceName":"i-2-155-VM","state":"Running","type":"User","uuid":"0597fa15-2dc4-4a92-9c0d-4025bd5568c6"}
is alive
2026-01-13 08:38:08,784 DEBUG [c.c.h.ManagementIPSystemVMInvestigator]
(HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Unable to find a
management nic, cannot ping this system VM, unable to determine state of VM
instance
{"id":155,"instanceName":"i-2-155-VM","state":"Running","type":"User","uuid":"0597fa15-2dc4-4a92-9c0d-4025bd5568c6"}
returning null
2026-01-13 08:38:08,784 INFO [c.c.h.HighAvailabilityManagerExtImpl]
(HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa)
ManagementIPSysVMInvestigator could not find VM instance
{"id":155,"instanceName":"i-2-155-VM","state":"Running","type":"User","uuid":"0597fa15-2dc4-4a92-9c0d-4025bd5568c6"}
2026-01-13 08:38:08,784 DEBUG [c.c.h.HighAvailabilityManagerExtImpl]
(HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Fencing off VM that we
don't know the state of
2026-01-13 08:38:08,785 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
(HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Wait time setting on
com.cloud.agent.api.FenceCommand is 1800 seconds
2026-01-13 08:38:08,785 DEBUG [c.c.a.m.ClusteredAgentAttache]
(HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Seq
21-5292855462067175512: Routed from 206863097656310
2026-01-13 08:38:08,786 DEBUG [c.c.a.t.Request] (HA-Worker-0:[ctx-0d486e07,
work-577]) (logid:48d451aa) Seq 7-5292855462067175512: Sending { Cmd , MgmtId:
206863097656310, via: 7(kvmhost2), Ver: v1, Flags: 100011,
[{"com.cloud.agent.api.FenceCommand":{"vmName":"i-2-155-VM","hostGuid":"0218d19d-3cbe-3827-bb03-7fd636ad202b-LibvirtComputingResource","hostIp":"192.168.55.164","inSeq":"false","host":{"guid":"0218d19d-3cbe-3827-bb03-7fd636ad202b-LibvirtComputingResource","privateNetwork":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"},"publicNetwork":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"},"storageNetwork1":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"}},"reportCheckFailureIfOneStorageIsDown":"false","wait":"0","bypassHostMaintenance":"false"}}]
}
2026-01-13 08:38:08,833 DEBUG [c.c.a.t.Request] (HA-Worker-0:[ctx-0d486e07,
work-577]) (logid:48d451aa) Seq 7-5292855462067175512: Received: { Ans: ,
MgmtId: 206863097656310, via: 7(kvmhost2), Ver: v1, Flags: 10, { FenceAnswer } }
2026-01-13 08:38:08,833 INFO [c.c.h.HighAvailabilityManagerExtImpl]
(HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Fencer KVMFenceBuilder
returned true
2026-01-13 08:38:08,833 WARN [o.a.c.f.j.AsyncJobExecutionContext]
(HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Job is executed without
a context, setup psudo job for the executing thread
2026-01-13 08:38:08,840 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Sync job-2541 execution
on object VmWorkJobQueue.155
2026-01-13 08:38:10,544 DEBUG [c.c.v.ClusteredVirtualMachineManagerImpl]
(HA-Worker-0:[ctx-0d486e07, work-577, ctx-99fd3c33]) (logid:48d451aa) start
parameter value of enterHardwareSetup == null during processing of queued job
2026-01-13 08:38:10,548 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(HA-Worker-0:[ctx-0d486e07, work-577, ctx-99fd3c33]) (logid:48d451aa) Sync
job-2543 execution on object VmWorkJobQueue.155
2026-01-13 08:40:14,151 INFO [c.c.h.HighAvailabilityManagerExtImpl]
(HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) HA is now restarting VM
instance
{"id":155,"instanceName":"i-2-155-VM","state":"Running","type":"User","uuid":"0597fa15-2dc4-4a92-9c0d-4025bd5568c6"}
on Host
{"id":7,"name":"kvmhost2","type":"Routing","uuid":"c30b8e0d-0a48-40fa-bb57-c2de13513ca5"}
2026-01-13 08:40:14,152 WARN [c.c.a.AlertManagerImpl]
(HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) alertType=[8]
dataCenterId=[1] podId=[1] clusterId=[null] message=[HA starting VM:
vm2-haenabled (i-2-155-VM)].
2026-01-13 08:40:14,155 WARN [c.c.a.AlertManagerImpl]
(HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) No recipients set in
global setting 'alert.email.addresses', skipping sending alert with subject [HA
starting VM: vm2-haenabled (i-2-155-VM)] and content [HA starting VM:
vm2-haenabled (i-2-155-VM)].
2026-01-13 08:40:14,155 INFO [c.c.h.HighAvailabilityManagerExtImpl]
(HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Completed work
HAWork[577-HA-155-Running-Scheduled]. Took 1/5 attempts.
```
Logs for Non ha enabled vm
```
root@acs420:~# cat /var/log/cloudstack/management/management-server.log
|grep -i "logid:62b9b072"
2026-01-13 08:38:08,648 INFO [c.c.h.HighAvailabilityManagerExtImpl]
(HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Processing work
HAWork[576-HA-153-Running-Investigating]
2026-01-13 08:38:08,649 DEBUG [c.c.h.HighAvailabilityManagerExtImpl]
(HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) RESTART with HAWORK
2026-01-13 08:38:08,651 DEBUG [c.c.h.HighAvailabilityManagerExtImpl]
(HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) SimpleInvestigator
unable to determine the state of the host. Moving on.
2026-01-13 08:38:08,651 DEBUG [c.c.h.HighAvailabilityManagerExtImpl]
(HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) XenServerInvestigator
unable to determine the state of the host. Moving on.
2026-01-13 08:38:08,652 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
(HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Can not send command
com.cloud.agent.api.CheckOnHostCommand due to Host Host
{"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"}
not being up
2026-01-13 08:38:08,653 DEBUG [c.c.h.KVMInvestigator]
(HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Investigating host:Host
{"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"}
via neighbouring host:Host
{"id":7,"name":"kvmhost2","type":"Routing","uuid":"c30b8e0d-0a48-40fa-bb57-c2de13513ca5"}
2026-01-13 08:38:08,654 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
(HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Wait time setting on
com.cloud.agent.api.CheckOnHostCommand is 5 seconds
2026-01-13 08:38:08,654 DEBUG [c.c.a.m.ClusteredAgentAttache]
(HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Seq
21-5292855462067175505: Routed from 206863097656310
2026-01-13 08:38:08,654 DEBUG [c.c.a.t.Request] (HA-Worker-4:[ctx-9060fa2b,
work-576]) (logid:62b9b072) Seq 7-5292855462067175505: Sending { Cmd , MgmtId:
206863097656310, via: 7(kvmhost2), Ver: v1, Flags: 100011,
[{"com.cloud.agent.api.CheckOnHostCommand":{"host":{"guid":"0218d19d-3cbe-3827-bb03-7fd636ad202b-LibvirtComputingResource","privateNetwork":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"},"publicNetwork":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"},"storageNetwork1":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"}},"reportCheckFailureIfOneStorageIsDown":"false","wait":"5","bypassHostMaintenance":"false"}}]
}
2026-01-13 08:38:08,664 DEBUG [c.c.a.t.Request] (HA-Worker-4:[ctx-9060fa2b,
work-576]) (logid:62b9b072) Seq 7-5292855462067175505: Received: { Ans: ,
MgmtId: 206863097656310, via: 7(kvmhost2), Ver: v1, Flags: 10, { Answer } }
2026-01-13 08:38:08,664 DEBUG [c.c.h.KVMInvestigator]
(HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Neighbouring host:Host
{"id":7,"name":"kvmhost2","type":"Routing","uuid":"c30b8e0d-0a48-40fa-bb57-c2de13513ca5"}
returned status:Down for the investigated host:Host
{"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"}
2026-01-13 08:38:08,664 DEBUG [c.c.h.KVMInvestigator]
(HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Investigating host:Host
{"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"}
via neighbouring host:Host
{"id":13,"name":"kvmhost3","type":"Routing","uuid":"3cfc4b1f-1aa9-40e5-a549-484ab4fcc921"}
2026-01-13 08:38:08,665 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
(HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Wait time setting on
com.cloud.agent.api.CheckOnHostCommand is 5 seconds
2026-01-13 08:38:08,665 DEBUG [c.c.a.m.ClusteredAgentAttache]
(HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Seq
21-5194902170171867173: Routed from 206863097656310
2026-01-13 08:38:08,665 DEBUG [c.c.a.t.Request] (HA-Worker-4:[ctx-9060fa2b,
work-576]) (logid:62b9b072) Seq 13-5194902170171867173: Sending { Cmd ,
MgmtId: 206863097656310, via: 13(kvmhost3), Ver: v1, Flags: 100011,
[{"com.cloud.agent.api.CheckOnHostCommand":{"host":{"guid":"0218d19d-3cbe-3827-bb03-7fd636ad202b-LibvirtComputingResource","privateNetwork":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"},"publicNetwork":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"},"storageNetwork1":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"}},"reportCheckFailureIfOneStorageIsDown":"false","wait":"5","bypassHostMaintenance":"false"}}]
}
2026-01-13 08:38:08,676 DEBUG [c.c.a.t.Request] (HA-Worker-4:[ctx-9060fa2b,
work-576]) (logid:62b9b072) Seq 13-5194902170171867173: Received: { Ans: ,
MgmtId: 206863097656310, via: 13(kvmhost3), Ver: v1, Flags: 10, { Answer } }
2026-01-13 08:38:08,676 DEBUG [c.c.h.KVMInvestigator]
(HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Neighbouring host:Host
{"id":13,"name":"kvmhost3","type":"Routing","uuid":"3cfc4b1f-1aa9-40e5-a549-484ab4fcc921"}
returned status:Down for the investigated host:Host
{"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"}
2026-01-13 08:38:08,677 DEBUG [c.c.h.KVMInvestigator]
(HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) HA: HOST is ineligible
legacy state Down for host Host
{"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"}
2026-01-13 08:38:08,677 DEBUG [c.c.h.HighAvailabilityManagerExtImpl]
(HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) KVMInvestigator was
able to determine host Host
{"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"}
is in Down
2026-01-13 08:38:08,677 INFO [c.c.h.HighAvailabilityManagerExtImpl]
(HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) HA on VM instance
{"id":153,"instanceName":"i-2-153-VM","state":"Running","type":"User","uuid":"669a5d32-0b9a-4e84-8e9f-16597722cf98"}
2026-01-13 08:38:08,678 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
(HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Wait time setting on
com.cloud.agent.api.CheckVirtualMachineCommand is 20 seconds
2026-01-13 08:38:08,679 DEBUG [c.c.h.CheckOnAgentInvestigator]
(HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Unable to reach the
agent for VM instance
{"id":153,"instanceName":"i-2-153-VM","state":"Running","type":"User","uuid":"669a5d32-0b9a-4e84-8e9f-16597722cf98"}:
Resource [Host:6] is unreachable: Host 6: Host with specified id is not in the
right state: Down
2026-01-13 08:38:08,679 INFO [c.c.h.HighAvailabilityManagerExtImpl]
(HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) SimpleInvestigator
could not find VM instance
{"id":153,"instanceName":"i-2-153-VM","state":"Running","type":"User","uuid":"669a5d32-0b9a-4e84-8e9f-16597722cf98"}
2026-01-13 08:38:08,679 INFO [c.c.h.HighAvailabilityManagerExtImpl]
(HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) XenServerInvestigator
could not find VM instance
{"id":153,"instanceName":"i-2-153-VM","state":"Running","type":"User","uuid":"669a5d32-0b9a-4e84-8e9f-16597722cf98"}
2026-01-13 08:38:08,681 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
(HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Can not send command
com.cloud.agent.api.CheckOnHostCommand due to Host Host
{"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"}
not being up
2026-01-13 08:38:08,682 DEBUG [c.c.h.KVMInvestigator]
(HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Investigating host:Host
{"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"}
via neighbouring host:Host
{"id":7,"name":"kvmhost2","type":"Routing","uuid":"c30b8e0d-0a48-40fa-bb57-c2de13513ca5"}
2026-01-13 08:38:08,682 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
(HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Wait time setting on
com.cloud.agent.api.CheckOnHostCommand is 5 seconds
2026-01-13 08:38:08,683 DEBUG [c.c.a.m.ClusteredAgentAttache]
(HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Seq
21-5292855462067175508: Routed from 206863097656310
2026-01-13 08:38:08,683 DEBUG [c.c.a.t.Request] (HA-Worker-4:[ctx-9060fa2b,
work-576]) (logid:62b9b072) Seq 7-5292855462067175508: Sending { Cmd , MgmtId:
206863097656310, via: 7(kvmhost2), Ver: v1, Flags: 100011,
[{"com.cloud.agent.api.CheckOnHostCommand":{"host":{"guid":"0218d19d-3cbe-3827-bb03-7fd636ad202b-LibvirtComputingResource","privateNetwork":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"},"publicNetwork":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"},"storageNetwork1":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"}},"reportCheckFailureIfOneStorageIsDown":"false","wait":"5","bypassHostMaintenance":"false"}}]
}
2026-01-13 08:38:08,771 DEBUG [c.c.a.t.Request] (HA-Worker-4:[ctx-9060fa2b,
work-576]) (logid:62b9b072) Seq 7-5292855462067175508: Received: { Ans: ,
MgmtId: 206863097656310, via: 7(kvmhost2), Ver: v1, Flags: 10, { Answer } }
2026-01-13 08:38:08,771 DEBUG [c.c.h.KVMInvestigator]
(HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Neighbouring host:Host
{"id":7,"name":"kvmhost2","type":"Routing","uuid":"c30b8e0d-0a48-40fa-bb57-c2de13513ca5"}
returned status:Down for the investigated host:Host
{"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"}
2026-01-13 08:38:08,771 DEBUG [c.c.h.KVMInvestigator]
(HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Investigating host:Host
{"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"}
via neighbouring host:Host
{"id":13,"name":"kvmhost3","type":"Routing","uuid":"3cfc4b1f-1aa9-40e5-a549-484ab4fcc921"}
2026-01-13 08:38:08,771 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
(HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Wait time setting on
com.cloud.agent.api.CheckOnHostCommand is 5 seconds
2026-01-13 08:38:08,772 DEBUG [c.c.a.m.ClusteredAgentAttache]
(HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Seq
21-5194902170171867177: Routed from 206863097656310
2026-01-13 08:38:08,772 DEBUG [c.c.a.t.Request] (HA-Worker-4:[ctx-9060fa2b,
work-576]) (logid:62b9b072) Seq 13-5194902170171867177: Sending { Cmd ,
MgmtId: 206863097656310, via: 13(kvmhost3), Ver: v1, Flags: 100011,
[{"com.cloud.agent.api.CheckOnHostCommand":{"host":{"guid":"0218d19d-3cbe-3827-bb03-7fd636ad202b-LibvirtComputingResource","privateNetwork":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"},"publicNetwork":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"},"storageNetwork1":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"}},"reportCheckFailureIfOneStorageIsDown":"false","wait":"5","bypassHostMaintenance":"false"}}]
}
2026-01-13 08:38:08,780 DEBUG [c.c.a.t.Request] (HA-Worker-4:[ctx-9060fa2b,
work-576]) (logid:62b9b072) Seq 13-5194902170171867177: Received: { Ans: ,
MgmtId: 206863097656310, via: 13(kvmhost3), Ver: v1, Flags: 10, { Answer } }
2026-01-13 08:38:08,780 DEBUG [c.c.h.KVMInvestigator]
(HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Neighbouring host:Host
{"id":13,"name":"kvmhost3","type":"Routing","uuid":"3cfc4b1f-1aa9-40e5-a549-484ab4fcc921"}
returned status:Down for the investigated host:Host
{"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"}
2026-01-13 08:38:08,780 DEBUG [c.c.h.KVMInvestigator]
(HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) HA: HOST is ineligible
legacy state Down for host Host
{"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"}
2026-01-13 08:38:08,780 DEBUG [c.c.h.KVMInvestigator]
(HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) HA: HOST is ineligible
legacy state Down for host Host
{"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"}
2026-01-13 08:38:08,780 INFO [c.c.h.HighAvailabilityManagerExtImpl]
(HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) KVMInvestigator could
not find VM instance
{"id":153,"instanceName":"i-2-153-VM","state":"Running","type":"User","uuid":"669a5d32-0b9a-4e84-8e9f-16597722cf98"}
2026-01-13 08:38:08,781 INFO [c.c.h.HighAvailabilityManagerExtImpl]
(HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) HypervInvestigator
could not find VM instance
{"id":153,"instanceName":"i-2-153-VM","state":"Running","type":"User","uuid":"669a5d32-0b9a-4e84-8e9f-16597722cf98"}
2026-01-13 08:38:08,781 INFO [c.c.h.HighAvailabilityManagerExtImpl]
(HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) VMwareInvestigator
could not find VM instance
{"id":153,"instanceName":"i-2-153-VM","state":"Running","type":"User","uuid":"669a5d32-0b9a-4e84-8e9f-16597722cf98"}
2026-01-13 08:38:08,781 DEBUG [c.c.h.UserVmDomRInvestigator]
(HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) testing if VM instance
{"id":153,"instanceName":"i-2-153-VM","state":"Running","type":"User","uuid":"669a5d32-0b9a-4e84-8e9f-16597722cf98"}
is alive
2026-01-13 08:38:08,783 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
(HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Can not send command
com.cloud.agent.api.PingTestCommand due to Host Host
{"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"}
not being up
2026-01-13 08:38:08,783 DEBUG [c.c.h.UserVmDomRInvestigator]
(HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) VM instance
{"id":153,"instanceName":"i-2-153-VM","state":"Running","type":"User","uuid":"669a5d32-0b9a-4e84-8e9f-16597722cf98"}
could not be pinged, returning that it is unknown
2026-01-13 08:38:08,783 DEBUG [c.c.h.UserVmDomRInvestigator]
(HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Returning null since
we're unable to determine state of VM instance
{"id":153,"instanceName":"i-2-153-VM","state":"Running","type":"User","uuid":"669a5d32-0b9a-4e84-8e9f-16597722cf98"}
2026-01-13 08:38:08,783 INFO [c.c.h.HighAvailabilityManagerExtImpl]
(HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) PingInvestigator could
not find VM instance
{"id":153,"instanceName":"i-2-153-VM","state":"Running","type":"User","uuid":"669a5d32-0b9a-4e84-8e9f-16597722cf98"}
2026-01-13 08:38:08,783 DEBUG [c.c.h.ManagementIPSystemVMInvestigator]
(HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Not a System Vm, unable
to determine state of VM instance
{"id":153,"instanceName":"i-2-153-VM","state":"Running","type":"User","uuid":"669a5d32-0b9a-4e84-8e9f-16597722cf98"}
returning null
2026-01-13 08:38:08,784 DEBUG [c.c.h.ManagementIPSystemVMInvestigator]
(HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Testing if VM instance
{"id":153,"instanceName":"i-2-153-VM","state":"Running","type":"User","uuid":"669a5d32-0b9a-4e84-8e9f-16597722cf98"}
is alive
2026-01-13 08:38:08,784 DEBUG [c.c.h.ManagementIPSystemVMInvestigator]
(HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Unable to find a
management nic, cannot ping this system VM, unable to determine state of VM
instance
{"id":153,"instanceName":"i-2-153-VM","state":"Running","type":"User","uuid":"669a5d32-0b9a-4e84-8e9f-16597722cf98"}
returning null
2026-01-13 08:38:08,784 INFO [c.c.h.HighAvailabilityManagerExtImpl]
(HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072)
ManagementIPSysVMInvestigator could not find VM instance
{"id":153,"instanceName":"i-2-153-VM","state":"Running","type":"User","uuid":"669a5d32-0b9a-4e84-8e9f-16597722cf98"}
2026-01-13 08:38:08,784 DEBUG [c.c.h.HighAvailabilityManagerExtImpl]
(HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Fencing off VM that we
don't know the state of
2026-01-13 08:38:08,785 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
(HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Wait time setting on
com.cloud.agent.api.FenceCommand is 1800 seconds
2026-01-13 08:38:08,785 DEBUG [c.c.a.m.ClusteredAgentAttache]
(HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Seq
21-5292855462067175511: Routed from 206863097656310
2026-01-13 08:38:08,785 DEBUG [c.c.a.t.Request] (HA-Worker-4:[ctx-9060fa2b,
work-576]) (logid:62b9b072) Seq 7-5292855462067175511: Sending { Cmd , MgmtId:
206863097656310, via: 7(kvmhost2), Ver: v1, Flags: 100011,
[{"com.cloud.agent.api.FenceCommand":{"vmName":"i-2-153-VM","hostGuid":"0218d19d-3cbe-3827-bb03-7fd636ad202b-LibvirtComputingResource","hostIp":"192.168.55.164","inSeq":"false","host":{"guid":"0218d19d-3cbe-3827-bb03-7fd636ad202b-LibvirtComputingResource","privateNetwork":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"},"publicNetwork":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"},"storageNetwork1":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"}},"reportCheckFailureIfOneStorageIsDown":"false","wait":"0","bypassHostMaintenance":"false"}}]
}
2026-01-13 08:38:08,793 DEBUG [c.c.a.t.Request] (HA-Worker-4:[ctx-9060fa2b,
work-576]) (logid:62b9b072) Seq 7-5292855462067175511: Received: { Ans: ,
MgmtId: 206863097656310, via: 7(kvmhost2), Ver: v1, Flags: 10, { FenceAnswer } }
2026-01-13 08:38:08,793 INFO [c.c.h.HighAvailabilityManagerExtImpl]
(HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Fencer KVMFenceBuilder
returned true
2026-01-13 08:38:08,793 WARN [o.a.c.f.j.AsyncJobExecutionContext]
(HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Job is executed without
a context, setup psudo job for the executing thread
2026-01-13 08:38:08,807 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Sync job-2539 execution
on object VmWorkJobQueue.153
2026-01-13 08:38:10,517 DEBUG [c.c.h.HighAvailabilityManagerExtImpl]
(HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) VM is not HA enabled so
we're done.
2026-01-13 08:38:10,517 INFO [c.c.h.HighAvailabilityManagerExtImpl]
(HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Completed work
HAWork[576-HA-153-Running-Scheduled]. Took 1/5 attempts.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
