bhouse-nexthop opened a new issue, #14744:
URL: https://github.com/apache/cloudstack/issues/14744
### problem
When an HA-enabled VM stops unexpectedly while its host stays Up, the HA
restart first tries the VM's last host, which is usually the host that just
stopped it. If that host is Up and passes the capacity check, the VM goes
straight back there. The deployment planner and `vm.allocation.algorithm` are
never consulted.
That's the right choice when the VM itself failed, for example a guest
crash. It's the wrong one when the host stopped the VM, for example when the
host's kernel OOM-killed the QEMU process because the host's VMs together used
more memory than it has. With memory overcommit, the capacity check counts
allocated memory times the overcommit ratio, so the host that killed the VM
still "has capacity". The VM is restarted on the same full host, where it, or
one of its neighbours, can be killed again.
We hit this on KVM with `memoryOvercommitRatio` 4: Kubernetes worker VMs
killed by the host's OOM killer were restarted by HA on the hosts that had
killed them.
Cause:
- `HighAvailabilityManagerImpl.restart()` starts the VM with `HaTag` (when
`ha.tag` is set) and `HaOperation`, but never sets
`VirtualMachineProfile.Param.ConsiderLastHost`.
- `DeploymentPlanningManagerImpl.deployInVmLastHost()` therefore tries the
last host before the planner.
- A successful start sets the VM's last host to the host it starts on, and a
stop the VM didn't ask for doesn't change it, so the last host is the host the
VM was stopped on (or, after a migration, the migration's source host).
When the host itself fails, this doesn't arise: a host that isn't Up is
never used as the last host.
### versions
4.22 and main. The code involved is the same on both.
### The steps to reproduce the bug
1. Use a KVM cluster with two or more hosts, and an HA-enabled VM running on
host A.
2. Kill the VM's QEMU process on A (`kill -9 <pid>`), the same thing the OOM
killer does.
3. CloudStack reports the VM stopped out of band and schedules an HA restart.
4. The VM is restarted on A, even with `vm.allocation.algorithm` set to an
algorithm that would choose another host. The management server log shows `Last
host [Host {... A ...}] of VM [VM instance {...}] is UP and has enough
capacity`.
Reproduced on a three-host lab cluster with
`vm.allocation.algorithm=firstfitleastconsumed`, the VM's host A being the most
consumed: HA restarted it on A, with an empty host available.
### What to do about it?
Changing the default would surprise people who rely on HA restarting a VM in
place, so make it a setting. Add a cluster-scoped
`vm.ha.restart.consider.last.host`, defaulting to `true` (today's behaviour).
When it is `false`, the HA restart sets `ConsiderLastHost=false`, so the
planner and allocators choose the host. The planner may still choose the same
host when it is the best fit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]