nikolauseppinger commented on issue #13376:
URL: https://github.com/apache/cloudstack/issues/13376#issuecomment-4970516468
Independent reproduction on a **second site with the `ipmitool` driver**
(this issue and #12921 are both Redfish — this confirms the failure mode is
driver-agnostic, as the issue text anticipated).
**Environment**
- CloudStack **4.22.1.0**, KVM, management clustered on 2 nodes.
- OOBM driver: **ipmitool** (Dell iDRAC, LAN/623). Host-HA + OOBM enabled,
VM-HA enabled.
- Primary storage: **SharedMountPoint**, zone-wide, GFS2
(`isStorageSupportHA() == true`, so the legacy investigator is not the
bottleneck — same as your Linstor note).
**What we observed — two distinct failure modes**
*Mode 1 — real power loss (both PSUs pulled), BMC therefore unreachable →
OOBM fence can never complete → stuck in `Fencing`:*
```
11:03:24 WARN FenceTask Exception occurred while running FenceTask ...
HAFenceException: OBM service is not configured or enabled for this
host <host>
```
This repeated every ~4 s for the whole outage. As you note in "Secondary Bug
#1", the message is misleading — OOBM *is* configured; the real cause is that
the `ipmitool` power-off against a **powerless BMC** throws (the BMC/iDRAC
itself had no power), and `KVMHAProvider.fence()`'s catch-all masks it. The
host only reached `Fenced` **once the iDRAC/BMC regained power and finished
booting** — at that point the still-retrying `ipmitool` power-off finally
succeeded — after which:
```
11:17:38 WARN HAManagerImpl Unable to find next HA state for current HA
state=[Fenced]
for event=[Ineligible] ... NoTransitionException: ... from Fenced
via Ineligible
```
(repeating every ~4 s — this is the spam addressed by #13588). VMs were only
restarted after the host came back — exactly the "perverse result" described
here.
*Mode 2 — BMC reachable (graceful iDRAC shutdown, and separately a kernel
panic `echo c > /proc/sysrq-trigger`):* host down, storage heartbeat stale,
**no failover for a full 20 minutes** with Host-HA enabled. (We did not have
DEBUG on `org.apache.cloudstack.ha` enabled, so we cannot yet pin the exact FSM
state for this mode — will re-run with DEBUG.)
**A/B control** — the clearest single piece of evidence: same kernel-panic
failure, with **Host-HA disabled**, only legacy VM-HA active:
```
ClusteredAgentManagerImpl Host <id> is down. Starting HA on the VMs
KVMInvestigator could not find VM ... (NOT "alive?
true")
Fencer KVMFenceBuilder returned true
HighAvailabilityManagerExtImpl HA is now restarting VM ... on Host <other>
```
VMs back within **~5–6 minutes**. So on this hardware the legacy path works
fine; **enabling Host-HA is what suppresses it** — while the host-HA state is
not `Fenced`, `KVMInvestigator.isVmAlive` →
`HAManagerImpl.isVMAliveOnHost`/`getHostStatusFromHAConfig` reports
`Up`/`Disconnected`, so VM-HA defers indefinitely.
**Takeaways**
1. Confirms the bug is **not Redfish-specific** — the `ipmitool` path fails
the same way when the BMC is unreachable. IPMI's "already-off returns exit 0"
only helps when the BMC *still has power*; in a real power loss (BMC down with
the host) the power-off simply throws.
2. The `Fenced → Ineligible` NoTransition spam and the misleading `fence()`
catch-all ("Secondary Bug #1") both reproduce here verbatim.
3. **Power-state confirmation (#13377) and a storage-heartbeat fallback
(#13589) are complementary — for total power loss you need the latter:** #13377
confirms death by reading the OOBM power state, but when the BMC itself is
powerless that read is unavailable too, so the host still can't be confirmed
off. The only positive signal left is the cluster's view of the host's
**storage heartbeat** via its neighbours — which is exactly what #13589
(`kvm.ha.fence.on.storage.heartbeat`) uses as a fence fallback when OOBM fails.
Between them they cover both "off but BMC alive" (#13377) and "BMC dead with
the host" (#13589).
We're glad to help validate the reworked #13377 on a setup it hasn't been
exercised on yet: **`ipmitool` driver + SharedMountPoint (GFS2) zone-wide
storage**, incl. the real power-loss case. Happy to share full logs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
