danivogel90 opened a new pull request, #14331:
URL: https://github.com/apache/cloudstack/pull/14331
### Description
This PR fixes the KVM HA heartbeat for SharedMountPoint pools whose path is
a subdirectory of a mounted filesystem, which is common with clustered/parallel
filesystems (IBM Storage Scale/GPFS, WEKA, Quobyte, GFS2, OCFS2, CephFS). It
also fixes two related problems in `kvmsmpheartbeat.sh` (introduced in #12773):
1. **Mount point check** – `mountpoint -q "$MountPoint"` rejects valid pools
such as `/data01/vol01` on a GPFS filesystem mounted at `/data01`. Every
heartbeat write fails with `Mount point is not a mounted filesystem`. A new
helper `is_on_mounted_fs()` accepts any path on a mounted filesystem other than
`/` (`findmnt -n -o TARGET -T`, fallback `df -P`). The safety intent is kept:
if the shared filesystem is not mounted, the heartbeat is not written to the
local root filesystem.
The same helper now also replaces the `/proc/mounts` check in front of
`deleteVMs`. **Without this, relaxing only the first check would make every
heartbeat write call `deleteVMs`, which `kill -9`s all qemu processes using the
pool.**
2. **False DEAD on read** – `check_hbLog` used ``diff=`expr $now - $hb` ``
followed by `if [ $? -ne 0 ]`. `expr` exits with 1 when the result is 0, so a
host whose heartbeat was written in the same second as the check was reported
as DEAD (`999997`). The timestamp is now validated and the difference computed
with shell arithmetic.
3. **Temp file left behind** – if the script is killed (e.g. on timeout
while the clustered FS is recovering), the `hb-<ip>.<pid>` temp file remained.
A `trap` now removes it.
Fixes: #14326
Broader context: #14327
### Types of changes
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] New feature (non-breaking change which adds functionality)
- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] Enhancement (improves an existing feature and functionality)
- [ ] Cleanup (Code refactoring and cleanup, that may add test cases)
- [ ] Build/CI
- [ ] Test (unit or integration test code)
### Feature/Enhancement Scale or Bug Severity
#### Bug Severity
- [ ] BLOCKER
- [ ] Critical
- [x] Major
- [ ] Minor
- [ ] Trivial
### How Has This Been Tested?
Environment: 3 KVM hosts, Ubuntu 24.04, CloudStack 4.23.0.0 (script
identical on 4.22 and main), hyperconverged IBM Storage Scale 5.2.3-5 (3-way
replication). SharedMountPoint pool `/data01/vol01` (subdirectory of GPFS mount
`/data01`), plus an NFS primary storage pool.
**Script tests** (copy of the script with `kill -9` replaced by an echo):
| Case | Expected | Result |
|---|---|---|
| unpatched, subdirectory of GPFS mount | fails | exit 1 |
| patched, subdirectory of GPFS mount, write + read | ALIVE, no deleteVMs |
OK |
| patched, real mount point (tmpfs) | ALIVE | OK |
| patched, NFS pool mount point (read) | ALIVE | OK |
| patched, path on root FS | refuse, nothing written | exit 1, dir empty |
| patched, path does not exist | refuse | exit 1 |
| unpatched, read right after write (diff=0) | ALIVE | **DEAD** |
| patched, read right after write / after 3 s, `-t 60` | ALIVE | OK / OK |
| patched, after 3 s with `-t 1` / corrupt hb file | DEAD | OK / OK |
| patched, SIGTERM during write | temp file removed | OK |
**Live:** the patched script has been running in the agent on one host since
2026-10-07. Heartbeat writes succeed, there are no errors in agent.log, and
nothing was killed.
**HA failover test:** HA-enabled VM (offering with `offerha=true`) on the
SharedMountPoint pool on host A. Agent autostart disabled on A, then A
hard-reset via sysrq. Results:
- after ~85 s the neighbour with the patched script returned "Heart is not
beating" for both pools (NFS + SharedMountPoint), and the host was marked Down
- after ~110 s the VM was running on another host
- host A rejoined normally after the agent was re-enabled
#### How did you try to break this feature and the system with this change?
- Path on the local root filesystem / non-existent path: refused, nothing
written, no deleteVMs.
- Corrupt and stale heartbeat files: reported as DEAD as before.
- Script interrupted during write: no leftover temp file.
- Known limitation (unchanged from before): a path on a *different local*
filesystem (e.g. a separate `/var` LV) is not detected as "not shared". The
previous `mountpoint -q` check did not cover this either.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
