bhouse-nexthop commented on PR #14108:
URL: https://github.com/apache/cloudstack/pull/14108#issuecomment-6740297499
### Test report: DRS honours anti-affinity when planning and executing
migrations
**Environment**
- A CloudStack 24.0 (main) build with this PR applied, alongside some of my
other open PRs, including #14111 (`weighted` DRS) and #14109.
- This PR targets 4.22, and I have not run it on a 4.22 build. The affinity
processors and the execution-time check are the same on 4.22 and main. One
thing differs in this build: #14111 also changes `ClusterDrsServiceImpl`. It
groups equivalent VMs for the candidate-host lookup, and after the first
iteration it re-evaluates affinity only for the moved VM's group peers. So
these runs cover this PR together with that change, not this PR alone. Nothing
failed, so I did not need to separate the two.
- Three KVM hosts on Ubuntu 24.04 (libvirt 10.0.0, QEMU 8.2.2), nested
virtualisation; Ceph RBD primary storage; one management server. DEBUG logging
for `com.cloud` and `org.apache.cloudstack`.
- Cluster settings: `drs.algorithm=balanced`, `drs.metric=memory`,
`drs.max.migrations=10`, `drs.imbalance=1.0`. `drs.imbalance` is the share of
imbalance *not* tolerated (threshold = 1 - value), so 1.0 is what makes DRS
always rebalance.
- Test VMs are CirrOS: 2 GB for the DRS steps and 256 MB for the deploy-path
step. Strict group GS is `host anti-affinity`; non-strict group GN is
`non-strict host anti-affinity`.
- The cluster also runs long-lived VMs that are not part of this test. I
gave them the `skipFromDRS` VM detail for the duration, so that plans contain
only test VMs. Their memory still counts towards host load. I removed the
detail afterwards.
**S1: in-memory placements during plan generation**
- A (GS) on host 2 and B (GS) on host 3. Fillers on both hosts, also marked
`skipFromDRS`, so that A and B were the only movable VMs. Host 1 was the
lightest: allocated memory was 11 / 20.5 / 26.5 GiB.
- `generateClusterDrsPlan` returned one migration, B host 3 → host 1. A was
not moved onto host 1 after it:
```
needs DRS. Imbalance: 0.3389 Threshold: 0.0 Algorithm: balanced DRS
metric: memory
Plan for VM ... i-2-220-VM (B) to migrate from host host3 to host host1
needs DRS. Imbalance: 0.2541 Threshold: 0.0 ...
Added host 1 to avoid set, since VM ... i-2-220-VM ... is placed on the
host by the plan being built
```
- Final placement: A on host 2, B on host 1. No two GS members share a host.
- Before this PR the second iteration read B's host from the database (host
3), so host 1 was still open to A.
**S2: re-check at execution time**
- I migrated B to host 1 by hand. `migrateVirtualMachine` with a host id
doesn't enforce affinity groups.
- `executeClusterDrsPlan migrateto[0].vm=A migrateto[0].host=host1` gave:
- plan `COMPLETED`, with the migration `CANCELLED`;
- A stayed on host 2;
- a WARN `CLUSTER.DRS` event:
```
WARN Completed Skipped DRS migration of VM instance {..."i-2-219-VM"...}
to Host {..."host1"...}: the destination no longer satisfies the placement
rules for that VM. The plan was generated against older state.
```
**S3: a swap between unrelated VMs**
- One plan with two migrations of VMs in no group: X host 2 → host 1, and Y
host 3 → host 2, the host X was leaving.
- Both ended `SUCCEEDED`, and X and Y were on their new hosts. This was the
review regression, where the second move was refused because the database still
showed X on host 2.
**S4: a non-strict group with more VMs than hosts**
- GN had 4 VMs: A and D on host 3 (the heaviest, 26.5 GiB), B on host 1, and
C on host 2.
- The plan moved A from host 3 to host 1. GN members are not frozen.
- After that move, D (host 3, now with no GN peer) was not moved onto host 1
(2 GN members) or host 2 (1). The log shows the exclusions, for example:
```
Host 1 is dispreferred by a non-strict affinity group more than the
current host 3, so DRS will not migrate onto it
```
So DRS spreads a non-strict group where it can, and never adds to the
members a VM shares a host with.
**S5: deploy path, stopped member with reserved capacity**
- GS member B deployed on host 2 and stopped.
`capacity.skipcounting.hours=3600`, so its capacity on host 2 was still
reserved.
- Three GS deploys without a host id:
- C went to host 3;
- D went to host 1;
- E was refused, `No destination found for a deployment`. Hosts 1 and 3
hold members, and host 2 is reserved by the stopped B:
```
3 x Added host 2 to avoid set, since VM i-2-235-VM is in Stopped state
on the host but still has reserved capacity
```
- After B was expunged, the next GS deploy without a host id went to host 2.
- Before the PR, this branch could not be reached, because it was an `else`
of a non-null check. A stopped member's last host was never avoided.
- A deploy *with* `hostid=` bypasses affinity processing altogether.
`DeploymentPlanningManagerImpl` returns through
`deployInSpecifiedHostWithoutHA` before the affinity chain runs. So a GS member
deployed with `hostid` of a host that already held a GS member started there.
That is stock behaviour, an admin override, and this PR does not touch it.
**Result:** pass.
- Plans don't break a strict group, including through placements made
earlier in the same plan.
- A migration that a strict group no longer allows is cancelled at execution
time, with a WARN event.
- Unrelated swaps still run.
- Non-strict members can move but are not stacked.
- A stopped member's last host is avoided while its capacity is reserved.
**Not covered**
- A 4.22 build, and this PR without #14111's `ClusterDrsServiceImpl` changes.
- `host affinity` and `non-strict host affinity` groups, automatic DRS
(`drs.automatic.enable`), and DPDK or dedicated-resource exclusions in the
execution-time check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
