bhouse-nexthop opened a new issue, #14742:
URL: https://github.com/apache/cloudstack/issues/14742
### problem
When a KVM host is put into maintenance, each VM that is live-migrated off
it goes back to its **last host** whenever that host is Up and passes the
capacity check. The planner and the configured `vm.allocation.algorithm` are
skipped for those VMs.
Every migration records the source host as the VM's last host, so during a
rolling maintenance of a cluster the VMs pile up on the hosts that were
evacuated earlier:
1. Host A is put into maintenance. Its VMs migrate to B, C, D... and now
have last host = A.
2. A is rebooted and comes back Up and Enabled.
3. Host B is put into maintenance. Every VM on B whose last host is A goes
straight back to A, whatever the allocation algorithm would have chosen.
4. Each later host repeats this, so the hosts evacuated first collect most
of the VMs.
With memory overcommit, the capacity check lets the last host take far more
VMs than it has memory. On a cluster with `memoryOvercommitRatio` 4, a rolling
maintenance of 12 hosts left one host with about 2.3 TiB of VM memory on 1.1
TiB of RAM, while three others had 700+ GiB free each. VMs that used their
memory (Kubernetes nodes) were then OOM-killed on that host. In the management
server's log for that day, 376 of 447 migrations off hosts in maintenance were
placed by the last-host check (`Last host [...] of VM [...] is UP and has
enough capacity`).
Cause:
- `DeploymentPlanningManagerImpl.planDeployment()` calls
`deployInVmLastHost()` before the planner. That returns the VM's last host
whenever the host is usable and has capacity, unless the profile sets
`ConsiderLastHost=false` or an HA tag.
- The two paths that evacuate a host both mark their plan as a migration
plan (`plan.setMigrationPlan(true)`):
- `VirtualMachineManagerImpl.orchestrateMigrateAway()`, used by host
maintenance and HA;
- `ResourceManagerImpl.migrateAwayVmWithVolumes()`, used by host
maintenance for VMs on local storage.
But nothing reads `isMigrationPlan()`, and neither path sets
`ConsiderLastHost`.
- `migrate()` sets the VM's last host to its source host on every migration
(`vm.setLastHostId(srcHostId)`).
A user-initiated migration without a destination host already avoids this.
`UserVmManagerImpl.chooseVmMigrationDestination()` clears the last host first
("Last host does not have higher priority in vm migration").
### versions
4.22 and main. The code involved is the same on both, and older releases
have the same flow.
### The steps to reproduce the bug
1. Use a cluster of three or more KVM hosts, A, B and C, with
`vm.allocation.algorithm` set to one that spreads VMs, such as
`firstfitleastconsumed`.
2. Put A into maintenance and wait until its VMs have migrated to B and C.
Cancel maintenance on A.
3. Put B into maintenance.
4. The VMs on B that came from A in step 2 all migrate back to A, even when
C has more free capacity. The management server log shows `Last host [... A
...] of VM [...] is UP and has enough capacity` for each of them.
### What to do about it?
For a migration plan, don't try the last host in `deployInVmLastHost()`, so
the planner and allocators choose the destination. This covers both evacuation
paths, and the HA planner fallback of `migrateAway()`. The source host is still
excluded as before.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]