rchicote72 opened a new issue, #14178:
URL: https://github.com/apache/cloudstack/issues/14178
### problem
During Host HA testing on a KVM cluster using CLVM_NG Primary Storage, we
observed that the VM activity check performed against CLVM_NG may incorrectly
report activity for a host that has been completely powered off.
When the failed host only runs HA-enabled VMs whose volumes reside on
CLVM_NG, CloudStack continues reporting that the host has VM activity. As a
result, Host HA may enter Degraded and repeat the activity-check cycle instead
of progressing to fencing.
The other KVM hosts correctly detect that the failed host is unreachable.
As a comparison, when the same failed host also runs an HA-enabled VM whose
volume resides on NFS Primary Storage, CloudStack checks both storage types and
we observe:
CLVM_NG activity check -> host has activity
NFS activity check -> host does not have activity
In this case, Host HA is able to progress to fencing.
Once the host reaches Fenced, the HA-enabled VMs stored on CLVM_NG are
successfully restarted on other KVM hosts.
This indicates that CLVM_NG itself works correctly as shared storage for VM
HA once fencing has occurred. The problem appears to be specifically related to
the Host HA VM activity detection performed against CLVM_NG storage.
As a workaround, we currently use a small shared NFS Primary Storage with
one small HA-enabled NFS-backed VM per KVM host. These VMs belong to a host
anti-affinity group so that one VM is normally present on each host.
With this workaround, Host HA correctly detects the absence of activity
through NFS, reaches fencing, and the CLVM_NG-backed HA VMs are successfully
restarted on the remaining hosts.
### versions
Apache CloudStack: 4.23.0.0
Hypervisor: KVM
Operating System: Ubuntu 24.04
Cluster: 4 KVM hosts
Primary Storage:
- CLVM_NG backed by shared Fibre Channel SAN
- sanlock + lvmlockd
- Additional shared NFS Primary Storage
Host HA provider: KVMHAProvider
OOBM: IPMI
VM HA: enabled on the test VMs
Host HA activity-check configuration:
kvm.ha.activity.check.failure.ratio = 0.7
kvm.ha.activity.check.interval = 60
kvm.ha.activity.check.max.attempts = 10
kvm.ha.activity.check.timeout = 60
kvm.ha.degraded.max.period = 300
The activity-check parameters above use their default values.
The following parameter is changed from its default value:
kvm.ha.recover.failure.threshold = 0
The default value is 1.
We use 0 to skip host recovery attempts and proceed to fencing once Host HA
determines that the host must be recovered.
### The steps to reproduce the bug
1. Create a KVM cluster with multiple hosts.
2. Configure Host HA using KVMHAProvider and configure OOBM/IPMI for the KVM
hosts.
3. Configure shared CLVM_NG Primary Storage backed by Fibre Channel SAN
using sanlock and lvmlockd.
4. Deploy one or more HA-enabled VMs with their volumes on CLVM_NG.
5. Ensure that the test host only runs HA-enabled VMs whose volumes reside
on CLVM_NG.
6. Completely power off the KVM host to simulate an unexpected host failure.
7. Monitor the Host HA activity checks in the CloudStack Management Server
log.
8. CloudStack detects that the host is unreachable, but the CLVM_NG VM
activity check reports that the host still has activity.
9. Host HA may enter Degraded and repeat the activity-check process instead
of progressing to fencing.
10. Power the host back on and deploy an additional HA-enabled VM whose
volume resides on shared NFS Primary Storage.
11. Repeat the hard power-off test.
12. CloudStack now performs activity checks against both CLVM_NG and NFS.
The observed result is:
CLVM_NG -> host has activity
NFS -> host does not have activity
13. Host HA is now able to progress from Checking to Recovering, Fencing and
finally Fenced.
14. After fencing, the HA-enabled VMs stored on CLVM_NG are successfully
restarted on the remaining KVM hosts.
15. When the failed host becomes available again, CloudStack also
automatically starts the pending NFS-backed auxiliary VM on that host because
the host anti-affinity constraint can be satisfied again.
CLVM_NG-backed VMs can also be live migrated normally between healthy hosts.
Therefore, the problem does not appear to be related to CLVM_NG volume
activation, locking, or general access to the shared storage.
### What to do about it?
The Host HA VM activity-check mechanism should correctly determine that a
completely powered-off KVM host no longer has VM activity when the affected VM
volumes reside on CLVM_NG.
Expected behavior:
Host powered off
|
v
CLVM_NG activity check
|
v
No VM activity
|
v
Fencing
|
v
Fenced
|
v
VM HA
The CLVM_NG activity-check implementation should be reviewed to determine
why it reports activity for a host that has been completely powered off.
The issue appears to be limited to activity detection. Once fencing is
reached, CloudStack successfully restarts the CLVM_NG-backed HA VMs on other
hosts.
For comparison, an NFS-backed VM on the same failed host correctly reports
no activity and allows Host HA to proceed to fencing.
It may therefore be useful to compare how VM activity is determined for
CLVM_NG storage with the NFS activity-check path.
Management Server and KVM agent logs from both scenarios can be provided if
required.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
