RosiKyu commented on PR #14450:
URL: https://github.com/apache/cloudstack/pull/14450#issuecomment-6836031447
@Pearl1594 HA VMs are not restarted after a KVM host is powered off in a
cluster with CLVM pools
### Summary
After a host is powered off, the management server correctly decides the
host is Down and schedules the HA restarts, but fencing fails on every attempt.
The fence check on the other hosts has no sanlock host id for the dead host
("host id [null]"), so the new CLVM check "assumes the host is alive", and that
wins over the NFS heartbeat, which says the host is dead. The VMs stay
"Running" on the dead host and are never restarted.
### Environment
**Build:** CloudStack 24.0.0-shapeblue22663 (PR #14450 head f53764c),
upgraded from 4.23.0.0; management server and agents on the PR build; 4 KVM
hosts on Oracle Linux 8 in one cluster; 1 CLVM and 2 CLVM_NG pools on lvmlockd
+ sanlock shared VGs (iSCSI), plus 1 NFS primary pool; VM HA only (Host HA not
enabled).
### Steps to Reproduce
1. In a cluster with CLVM/CLVM_NG pools and an NFS pool, deploy HA-enabled
VMs (offering with `offerha=true`) on host B: one with its disks on CLVM_NG,
one on CLVM, one on NFS.
2. Check that every host has the `host.sanlock.id` detail.
3. Power off host B (`echo o > /proc/sysrq-trigger`).
4. Wait 15 minutes and watch the VMs and `op_ha_work`.
### Expected Result
Host B is marked Down, the HA VMs are fenced and started on other hosts
within a few minutes (as happens with the 4.23 agents on the same setup).
### Actual Result
Host B is marked Down after about 5 minutes and HA work is created for all 4
VMs, but every attempt fails with "Unable to fence off host" and "Fencer
KVMFenceBuilder returned false" and is rescheduled 10 minutes later. The VMs
stay "Running" on the dead host. The agents on the other hosts log "host id
[null] is unknown, assuming the host is alive" for each CLVM pool, while the
NFS heartbeat on the same host says the dead host is DEAD.
From the code: `KVMFencer` builds `new FenceCommand(vm, host)` and the
`HostTO` in it never gets `sanlockHostId`; only `KVMHostActivityChecker` sets
it (`addSanlockHostId`). `LibvirtFenceCommandWrapper` passes that `HostTO` to
`KVMHAChecker`, so `ClvmSanlockHaChecker.isHostLeaseAlive` always gets `null`
on the fence path.
### Evidence
Lab mapping: `management server M = 10.0.35.155`
(**ref-trl-12929-k-Mol8-rositsa-kyuchukova**), `host A = 10.0.35.51` (kvm1),
`host B = 10.0.32.132` (kvm2, powered off at 20:40:07 UTC), NFS heartbeat pool
`kvm-pri1`, CLVM VGs `clvm`, `clvmng1`, `clvmng2`, VM P = `vm-ha-ng1`
(i-2-3-VM).
**Management server M, step 3: host B is Down, the restart is scheduled,
then fencing fails and the work is rescheduled**
```
2026-11-12 20:44:57,659 DEBUG [c.c.h.HighAvailabilityManagerExtImpl]
(AgentTaskPool-8:[ctx-b52020f6]) (logid:4f2d1afa) KVMInvestigator was able to
determine host Host
{"id":2,"name":"ref-trl-12929-k-mol8-rositsa-kyuchukova-kvm2","type":"Routing","uuid":"c1887c6f-8e6e-487a-9351-8a5929e6f465"}
is in Down
2026-11-12 20:44:57,759 INFO [c.c.h.HighAvailabilityManagerExtImpl]
(AgentTaskPool-8:[ctx-b52020f6]) (logid:4f2d1afa) Schedule vm for HA: VM
instance
{"id":3,"instanceName":"i-2-3-VM","state":"Running","type":"User","uuid":"9dd23d74-5c2b-4966-b17d-b974c8fd804e"}
2026-11-12 20:45:08,371 WARN [c.c.a.AlertManagerImpl]
(HA-Worker-0:[ctx-d0fcc3c3, work-5]) (logid:0ae2c17d) alertType=[7]
dataCenterId=[1] podId=[1] clusterId=[null] message=[Unable to fence off host:
2].
2026-11-12 20:45:08,381 ERROR [c.c.h.KVMFencer] (HA-Worker-0:[ctx-d0fcc3c3,
work-5]) (logid:0ae2c17d) Unable to fence off VM instance
{"id":3,"instanceName":"i-2-3-VM","state":"Running","type":"User","uuid":"9dd23d74-5c2b-4966-b17d-b974c8fd804e"}
on Host
{"id":2,"name":"ref-trl-12929-k-mol8-rositsa-kyuchukova-kvm2","type":"Routing","uuid":"c1887c6f-8e6e-487a-9351-8a5929e6f465"}
2026-11-12 20:45:08,381 INFO [c.c.h.HighAvailabilityManagerExtImpl]
(HA-Worker-0:[ctx-d0fcc3c3, work-5]) (logid:0ae2c17d) Fencer KVMFenceBuilder
returned false
2026-11-12 20:45:08,395 WARN [c.c.h.HighAvailabilityManagerExtImpl]
(HA-Worker-0:[ctx-d0fcc3c3, work-5]) (logid:0ae2c17d) Rescheduling work
HAWork[5-HA-3-Running-Investigating] to try again at
2026-11-12T20:55:22.368+0000. Finished attempt 1/5 times.
2026-11-12 20:56:03,313 INFO [c.c.h.HighAvailabilityManagerExtImpl]
(HA-Worker-1:[ctx-8232bcc5, work-5]) (logid:6640dfc7) Fencer KVMFenceBuilder
returned false
2026-11-12 20:56:03,329 WARN [c.c.h.HighAvailabilityManagerExtImpl]
(HA-Worker-1:[ctx-8232bcc5, work-5]) (logid:6640dfc7) Rescheduling work
HAWork[5-HA-3-Running-Investigating] to try again at
2026-11-12T21:06:17.728+0000. Finished attempt 2/5 times.
```
**Host A, step 4: the fence check has no sanlock id for host B and assumes
it is alive, while the NFS heartbeat says DEAD**
```
[root@ref-trl-12929-k-mol8-rositsa-kyuchukova-kvm1 ~]# grep -E "^2026-11-12
20:56:0[0-3]" /var/log/cloudstack/agent/agent.log | grep -E
"ClvmSanlockHaChecker|KVMHAChecker command|Fence"
2026-11-12 20:56:03,168 WARN [storage.clvm.ClvmSanlockHaChecker]
(pool-21-thread-1:[]) (logid:) Cannot check the sanlock lease: VG name
[clvmng1] or host id [null] is unknown, assuming the host is alive.
2026-11-12 20:56:03,186 WARN [kvm.storage.LibvirtStoragePool]
(pool-21-thread-1:[]) (logid:) Checking heart beat for host IP 10.0.32.132 with
KVMHAChecker command
[/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh -i
10.0.32.4 -p
/acs/primary/ref-trl-12929-k-Mol8-rositsa-kyuchukova/ref-trl-12929-k-Mol8-rositsa-kyuchukova-kvm-pri1
-m /mnt/a67112f8-e383-317e-8a32-30141ecd251d -h 10.0.32.132 -r -t 60 ]
returned [=====> Considering host as DEAD because last write on
[/mnt/a67112f8-e383-317e-8a32-30141ecd251d/KVMHA//hb-10.0.32.132] was [223]
seconds ago, but the max interval is [60] <======]. It may cause a shutdown of
the host.
2026-11-12 20:56:03,186 WARN [storage.clvm.ClvmSanlockHaChecker]
(pool-21-thread-1:[]) (logid:) Cannot check the sanlock lease: VG name
[clvmng2] or host id [null] is unknown, assuming the host is alive.
2026-11-12 20:56:03,186 WARN [storage.clvm.ClvmSanlockHaChecker]
(pool-21-thread-1:[]) (logid:) Cannot check the sanlock lease: VG name [clvm]
or host id [null] is unknown, assuming the host is alive.
```
**Management server M, step 4: all 4 restarts stuck in Investigating (ids 1
to 4 are the same test with the 4.23 agents, restarted fine)**
```
+----+-------------+------+---------+---------------+---------+---------------------+---------------------+
| id | instance_id | type | state | step | host_id | taken
| created |
+----+-------------+------+---------+---------------+---------+---------------------+---------------------+
| 1 | 3 | HA | Running | Done | 2 | 2026-11-12
19:28:58 | 2026-11-12 19:28:58 |
| 2 | 5 | HA | Running | Done | 2 | 2026-11-12
19:28:58 | 2026-11-12 19:28:58 |
| 3 | 6 | HA | Running | Done | 2 | 2026-11-12
19:28:58 | 2026-11-12 19:28:58 |
| 4 | 7 | HA | Running | Done | 2 | 2026-11-12
19:28:58 | 2026-11-12 19:28:58 |
| 5 | 3 | HA | Running | Investigating | 2 | NULL
| 2026-11-12 20:44:58 |
| 6 | 5 | HA | Running | Investigating | 2 | NULL
| 2026-11-12 20:44:58 |
| 7 | 6 | HA | Running | Investigating | 2 | NULL
| 2026-11-12 20:44:58 |
| 8 | 7 | HA | Running | Investigating | 2 | NULL
| 2026-11-12 20:44:58 |
+----+-------------+------+---------+---------------+---------+---------------------+---------------------+
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
