akoskuczi-bw opened a new issue, #14436:
URL: https://github.com/apache/cloudstack/issues/14436
### The required feature described as a wish
Currently, Apache CloudStack KVM Host HA relies on Out-of-Band Management
(IPMI/Redfish) to fence failed hosts.
If the OOB management interface is unavailable, the fencing operation fails,
potentially leaving the host in the Fencing state and preventing VM HA from
restarting affected instances on healthy hosts.
For example, if a physical KVM host experiences a power outage and its BMC
becomes unreachable, CloudStack may be unable to confirm that the host is
powered off, even though the host has physically stopped operating.
I propose introducing support for user-defined fencing scripts, allowing
administrators to implement infrastructure-specific fencing verification
without requiring CloudStack to support every possible hardware, storage, or
network configuration.
### Current Behavior
During a simulated failure, CloudStack attempts to fence an unavailable KVM
host using OOB management.
The fencing operation fails with the following exception:
```
WARN [o.a.c.h.t.FenceTask]
Exception occurred while running FenceTask on a resource:
org.apache.cloudstack.ha.provider.HAFenceException:
OBM service is not configured or enabled for this host
acsh2.hu.abc.com
at org.apache.cloudstack.kvm.ha.KVMHAProvider.fence(KVMHAProvider.java:99)
```
The current KVM HA provider executes:
```
outOfBandManagementService.executePowerOperation(
host, PowerOperation.OFF, null
);
```
If the operation fails, the host may remain in the Fencing state, with
subsequent fencing attempts failing for the same reason.
As a result, VM HA recovery may be blocked even when the physical host is
already powered off.
### Proposed Enhancement
Introduce a configurable custom fencing script mechanism for KVM Host HA.
Administrators should be able to specify an external executable that
CloudStack invokes when fencing a host.
The script would receive information identifying the affected host and
return a result indicating whether fencing has been successfully verified.
This would allow users to implement fencing verification based on their own
infrastructure, without requiring CloudStack to integrate every possible
fencing technology.
1. Configurable fencing script:
For example, a CloudStack configuration option could specify:
`kvm.ha.custom.fencing.script=/opt/cloudstack/ha/fence-host.sh`
The script could be invoked with arguments identifying the affected host:
```
/opt/cloudstack/ha/fence-host.sh \
--host-id 2 \
--host-uuid e1f21bff-9033-4823-aec2-329a14a06453 \
--host-name acsh2.hu.abc.com
```
The exact interface and configuration mechanism would be subject to
implementation.
**2. Script return values**
A simple exit-code-based interface could be used:
> <html><head></head><body>
Exit code | Meaning | CloudStack behavior
-- | -- | --
0 | Host successfully fenced or confirmed unable to access shared storage |
Transition to Fenced
1 | Host is still active or fencing failed | Do not transition to Fenced
2 | Host state cannot be determined | Keep recovery blocked and retry
according to policy
Other | Script execution error | Log error and treat fencing as unsuccessful
</body></html>
The important requirement is that **only an explicit successful result
should allow CloudStack to transition the host to Fenced**.
A script timeout, execution failure, or inconclusive result must never be
interpreted as successful fencing.
**3. Configurable execution policy**
Administrators should be able to select how custom fencing integrates with
the existing OOB mechanism.
For example:
```
Fencing strategy:
OOB_ONLY
CUSTOM_ONLY
OOB_THEN_CUSTOM
```
This would allow the existing OOB-based behavior to remain the default,
preserving backward compatibility.
**4. Infrastructure-independent implementation**
The custom script could use any verification or fencing mechanism
appropriate for the environment, including:
- Querying an external power management system or intelligent PDU.
- Checking hardware power status through an independent management interface.
- Verifying successful storage-level fencing.
- Querying an external cluster manager or fencing service.
- Checking an infrastructure-specific quorum or watchdog mechanism.
- Integrating with third-party automation or monitoring systems that provide
authoritative fencing status.
The responsibility for implementing reliable fencing verification would
remain with the administrator.
CloudStack would only need to provide a standardized script execution
interface and interpret the result.
**5. Security and execution requirements**
The implementation should provide:
- Configurable execution timeout.
- Controlled script execution permissions.
- Proper argument escaping and input validation.
- Detailed logging of execution results.
- Safe handling of script failures.
- A clearly documented exit-code contract.
- Protection against concurrent fencing operations on the same host.
Since a successful fencing result may trigger VM recovery on another host,
the script must provide reliable evidence that the original host cannot
continue running the affected workloads.
### Expected Behavior
When a KVM host becomes unavailable:
1. CloudStack detects the host failure.
2. Host HA initiates fencing.
3. Depending on the configured strategy, CloudStack attempts OOB fencing
and/or executes the custom fencing script.
4. The script performs infrastructure-specific fencing or verifies that the
host is already safely fenced.
5. If the script returns success, CloudStack transitions the host to Fenced.
6. VM HA can proceed with restarting affected instances on healthy hosts.
7. If fencing cannot be verified, CloudStack does not automatically restart
the affected VMs.
### Example Use Case: Single Host Power Failure
Consider a conventional CloudStack cluster with three KVM hosts using shared
primary storage.
One host experiences a complete power failure, and its BMC is also
unreachable.
Although the host is physically powered off, CloudStack cannot complete OOB
fencing.
With custom fencing scripts, an administrator could implement an alternative
method to verify that the host is powered off or otherwise safely isolated.
Once the script confirms successful fencing, CloudStack can proceed with VM
HA recovery.
### Example Use Case: Multi-Site Disaster Recovery
In a geographically distributed deployment, an entire site may become
unreachable, including its OOB management infrastructure.
An infrastructure-specific fencing script could verify that the affected
site's hosts have been isolated through an independent mechanism.
This would allow CloudStack to continue automatic VM recovery without
depending exclusively on BMC connectivity.
### Benefits
- Eliminates the exclusive dependency on OOB management for KVM Host HA
fencing.
- Supports both single-host failures and complete site outages.
- Allows infrastructure-specific fencing implementations.
- Avoids adding hardware-specific or storage-specific logic to the
CloudStack core.
- Enables integration with existing fencing solutions.
- Maintains split-brain protection by requiring explicit fencing
confirmation.
- Preserves backward compatibility with the existing OOB fencing mechanism.
Related Issues
- [#11627 – Host HA and VM HA
interaction](https://github.com/apache/cloudstack/issues/11627)
- [#13376 – KVM Host HA fencing
failure](https://github.com/apache/cloudstack/issues/13376)
### Questions
1. Would a configurable custom fencing script be acceptable as an extension
to the existing KVM HA provider?
2. Could the existing KVMHAProvider.fence() implementation be extended to
execute an administrator-defined script?
3. Would it be preferable to implement this as a separate HA provider rather
than extending the existing KVM HA provider?
4. Could this functionality be implemented without modifying the existing
Host HA state machine?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
