Pearl1594 opened a new pull request, #14450: URL: https://github.com/apache/cloudstack/pull/14450
### Description This PR fixes: https://github.com/apache/cloudstack/issues/14178 This PR does two things for CLVM and CLVM_NG primary storage (lvmlockd + sanlock). 1. Host HA support CLVM and CLVM_NG pools were not in `LIBVIRT_STORAGE_POOL_TYPES_WITH_HA_SUPPORT`, so the KVM HA activity check never gave a usable answer for them and a failed host was never resolved. - Both pool types are now HA-capable. - No heartbeat file is written for CLVM pools. The sanlock delta lease is the heartbeat, and wdmd resets a host that stops renewing it. - A neighbour host reads the lease with `sanlock client host_status -s lvm_<vg> -D`. The host is dead only when `last_check - last_live >= 8 * io_timeout + 60` (140s with the default io_timeout). Missing or unparsable state is treated as alive, so a host is never declared dead without evidence. - The agent reports its sanlock host id (`lvmconfig local/host_id`) as the host detail `host.sanlock.id`. `KVMHostActivityChecker` passes it to the checking host in `HostTO.sanlockHostId`. Sanlock host ids are host-wide and can repeat across clusters, so the lease is looked up by VG name plus host id. - `KVMHAMonitor` no longer unmounts or checks libvirt for CLVM pools, and deleting a CLVM pool removes it from the monitor. - When recovery needs an exclusive activation target, hosts that did not answer the lock-state query are skipped. 2. Retry of transient lock refusals After a holder failed or rebooted, the first exclusive activation (`lvchange -aey`) can be refused for a short time: either the previous holder's sanlock lease has not expired yet (`LV locked by other host`), or the lockspace is still being joined on a rebooted host (`lock start in progress`). Previously the activation was tried once, the lookup returned no holder, and the VM start went ahead with the LV inactive. The agent then waits for the device node (`KVMStoragePoolManager.getPhysicalDisk`, 100 tries every 3 seconds). That wait only checks whether `/dev/<vg>/<lv>` exists. It never requests the lock, and `ClvmStorageAdaptor.handleMissingDeviceNode` activates only `template-` volumes. So even after the lease expires nothing activates the LV and the start fails after about 6-7 minutes. HA then retries only after `restart.retry.interval` (600s by default). `ClvmPoolManager` now retries the exclusive activation on the chosen host until sanlock grants it or the budget runs out: - Only `locked by other host` and `lock start in progress` are retried. Any other failure or exception is a single attempt, as before. - New settings: `clvm.lock.recovery.retry.timeout` (default 180s, counted from the start of the lookup; 0 disables the retry) and `clvm.lock.recovery.retry.interval` (default 10s). The lease expires 8 x io_timeout + 60 seconds after the holder stops renewing, so raise the timeout if lvmlockd runs with a larger `--sanlock-timeout`. - The budget is shared by all volumes of one VM start, so a VM with several volumes cannot wait several times the timeout. - sanlock still decides who may hold a lock. An LV was never exclusive on two hosts at once in any test. **Behaviour changes to be aware of** - Existing CLVM/CLVM_NG clusters become HA-eligible on upgrade. When a host disconnects, KVMInvestigator can now conclude it is down, send the "Host is down" alert, and HA can restart HA-enabled VMs elsewhere. Before this, investigation was skipped for CLVM-only clusters. - Requirements on every host: `sanlock`, `wdmd`, `lvmlockd` and `lvmlocks` enabled at boot, and a working watchdog device. <!--- Describe your changes in DETAIL - And how has behaviour functionally changed. --> <!-- For new features, provide link to FS, dev ML discussion etc. --> <!-- In case of bug fix, the expected and actual behaviours, steps to reproduce. --> <!-- When "Fixes: #<id>" is specified, the issue/PR will automatically be closed when this PR gets merged --> <!-- For addressing multiple issues/PRs, use multiple "Fixes: #<id>" --> <!-- Fixes: # --> <!--- ******************************************************************************* --> <!--- NOTE: AUTOMATION USES THE DESCRIPTIONS TO SET LABELS AND PRODUCE DOCUMENTATION. --> <!--- PLEASE PUT AN 'X' in only **ONE** box --> <!--- ******************************************************************************* --> ### Types of changes - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] New feature (non-breaking change which adds functionality) - [ ] Bug fix (non-breaking change which fixes an issue) - [X] Enhancement (improves an existing feature and functionality) - [ ] Cleanup (Code refactoring and cleanup, that may add test cases) - [ ] Build/CI - [ ] Test (unit or integration test code) ### Feature/Enhancement Scale or Bug Severity #### Feature/Enhancement Scale - [X] Major - [ ] Minor ### Screenshots (if appropriate): ### How Has This Been Tested? HA support - Failover with Host HA enabled. - Failover with VM HA only, Host HA off. - Ablocked the host's connection to management, and HA correctly left the host alone because its storage lease was still alive - no fence - Mixed NFS and CLVM pools, stale NFS heartbeat with a live lease: host stays alive, no fence. - sanlock timing: another host's lvchange -aey is refused until about 140s after the last renewal. - Skipping unreachable hosts as the activation target, in four scenarios: 1. Host unreachable but still Up in the DB (kvm1's agent port blocked): recovery no longer picked kvm1 and chose kvm1, where sanlock refused at once. 2. Host already marked Alert: no commands were sent to kvm1 at all, and the StartCommand went out within 2 seconds. 3. Healthy cluster: a start on the holder took ~10s and the DB lock host was corrected. A start on the other host did a normal lock transfer in ~15s. 4. Holder fully isolated, VM started on the other host 8 seconds later, DB host still Up: the first query waited 130s, and then the activation succeeded at once because the wait ended about 20 seconds after the lease expired. The VM was Running about 5 minutes after the request. - Alive lock holder, VM start on another host: safe, the start fails. Retry - Healthy path (stop, start, start on the other host, live migrate): no retry lines, normal times, lock host followed the VM. - Holder rebooted, HA restarts the VM, 2 volumes: root volume retried (11 refusals, `lock start in progress`) and the data volume activated at once. VM Running in 232s. Without the retry the same scenario failed after 6m47s and the VM stayed stopped. - Holder hard-reset and kept down, VM started on the survivor inside the lease window: 5 refusals (`locked by other host`) over 48s, activated about 136s after the reset, VM Running 142s after the reset. - Holder alive but unreachable from management, VM started on the other host: the start fails as expected, the LVs never became active on the other host, and the second volume got no extra retry wait. - Retry disabled (`clvm.lock.recovery.retry.timeout=0`): one refusal per volume, no retry, the start fails as before. Setting back to 180 restored the retry without a restart. - Two HA VMs (3 volumes) failing over together: both Running within 10 seconds of each other, both root volumes retried in parallel, the data volume activated at once, two HA worker threads used. - In all runs LV state was polled on both hosts every 3 seconds: no LV was ever exclusively active on two hosts, and there were no CLVM-related errors in the management server log. <!-- Please describe in detail how you tested your changes. --> <!-- Include details of your testing environment, and the tests you ran to --> #### How did you try to break this feature and the system with this change? <!-- see how your change affects other areas of the code, etc. --> <!-- Please read the [CONTRIBUTING](https://github.com/apache/cloudstack/blob/main/CONTRIBUTING.md) document --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
