bhouse-nexthop commented on PR #14111:
URL: https://github.com/apache/cloudstack/pull/14111#issuecomment-6746837890
### Test report: the `weighted` DRS algorithm, on a live 3-host cluster
**Summary:** the first lab run found a defect, which "Sample host load for
weighted DRS, not only for the weighted allocator" now fixes. On a build with
that commit, `weighted` sees CPU load with `vm.allocation.algorithm` left at
its default. It moves VMs off the loaded host, and it says in the log when it
can't.
**Environment**
- A CloudStack 24.0 (main) build with this PR, including the commit above.
Some of my other open PRs are on the same build, among them #14108 and #14109,
which this PR is stacked on.
- Three KVM hosts on Ubuntu 24.04 (libvirt 10.0.0, QEMU 8.2.2, 32 vCPU
each), nested virtualisation, one cluster. Ceph RBD primary storage, one
management server, DEBUG logging for `com.cloud` and `org.apache.cloudstack`.
- `vm.allocation.algorithm` was left at its default (`random`) for every
result below except the first-run rows that say otherwise. The global
`drs.algorithm` stayed `balanced`; only the test cluster was set to `weighted`.
- `host.load.half.life=60`, which speeds the test up.
`host.load.sample.interval` stayed at 60 and `host.stats.interval` at 60.
- Cluster settings: `drs.imbalance=1.0` (threshold 0),
`drs.max.migrations=10`.
- The cluster also runs long-lived VMs that aren't part of this test. They
had the `skipFromDRS` detail for the duration, so plans only contain test VMs,
but their load and allocation still count.
- Memory allocation was balanced to 22.5-24 GiB per host with 2 GiB CirrOS
filler VMs.
- CPU load came from 30 busy loops in host 1's hypervisor OS, not from
stress VMs. One-vCPU guests can't take a 32-core host past 85%, and loops on
the host give the same "load without allocation" picture. Host 1 sat at about
97% CPU, hosts 2 and 3 at 3-7%.
- Every setting was put back afterwards, and the test VMs were removed.
**What the first run found, and what changed**
The first run had the same setup, on a build without the commit above:
- With the allocator at its default, `weighted` saw no load at all:
imbalance 0.009, 0 migrations, with host 1 at 98% CPU.
- `HostLoadTracker` only sampled while `vm.allocation.algorithm` was
`balancedweighted`, so every host read as unmeasured, and `weighted` quietly
fell back to allocation only.
- Switching only the allocator to `balancedweighted` gave imbalance 0.25 and
4 moves off host 1.
The commit makes two changes:
- The tracker now also samples while `drs.algorithm` is `weighted` globally
or on any live cluster.
- `Weighted.prepare` logs once per plan when it drops the utilisation terms:
WARN when no host has samples, INFO when only some do.
**Results**
| Scenario | Build | Result |
|---|---|---|
| S1: moves off the CPU-loaded host, allocator at its default | with the fix
| pass |
| W1: right after `weighted` is set, before any sample: WARN fallback, once
per plan | with the fix | pass |
| N1: `weighted` unset, allocator at its default: no sampling | with the fix
| pass |
| S2: a host's agent stopped (host Disconnected): INFO fallback, nothing
moved | with the fix (also passed on the first run, without the log line) |
pass |
| S2b: a hung agent (`kill -STOP`) on an Up host | with the fix (also passed
on the first run) | pass, see the note |
| S3: equivalent VMs share one candidate-host lookup | first run (the fix
doesn't touch this path) | pass |
| S4: storage-motion cost | | not tested: shared RBD storage |
**N1 and W1: no sampling while nothing reads the figures, and the WARN
fallback**
- With the cluster on `balanced` and the allocator on `random`, I left host
1 at 97% CPU for 150 s, which is more than two sample intervals.
- I then set the cluster's `drs.algorithm=weighted` and generated a plan 4 s
later, before the tracker's next pass. It had no samples to offer:
```
WARN No host in cluster Cluster {id: "1", name: "cluster1", ...} has
current load samples, so DRS algorithm weighted is balancing on allocation only
and ignoring host.weighted.cpu.used.weight and
host.weighted.memory.used.weight. Hosts are sampled every
host.load.sample.interval seconds, so there are none yet just after a
management server starts or this algorithm is first selected; otherwise no host
is Up and reporting stats.
needs DRS. Imbalance: 0.0275 Threshold: 0.0 Algorithm: weighted
```
The plan had 0 migrations. A second plan 3 s later logged the WARN once
more, so it is logged once per plan.
- Later in the run, after the tracker had been sampling, I reset the
cluster's `drs.algorithm` for 150 s and then set `weighted` again. The plan
generated at once logged the same WARN and had 0 migrations. So the tracker
stops sampling and drops its averages once nothing reads them. 150 s after
that, the plan was back to 4 moves off host 1.
**S1: moves off the loaded host, with the allocator at its default**
- 240 s after `weighted` was set, with no other change:
```
host1 cpuused=97.38% host2 3.18% host3 6.94%
needs DRS. Imbalance: 0.2535 Threshold: 0.0 Algorithm: weighted
i-2-379-VM: host1 -> host2
i-2-380-VM: host1 -> host2
i-2-381-VM: host1 -> host3
i-2-382-VM: host1 -> host3
```
That is 4 migrations, all off host 1, and no fallback line was logged.
- Control, the same load with `balanced` and `drs.metric=memory`: imbalance
0.027, 0 migrations.
- These are the same numbers as the first run with `balancedweighted`, which
gave imbalance 0.2516 and the same 4 moves. Now the allocator doesn't have to
change.
**S2: a host that isn't Up**
- After the measured plan above (4 moves off host 1), I stopped host 3's
agent. The host was `Disconnected` within a second.
- The plan 71 s later:
```
INFO 1 of 3 hosts in cluster Cluster {id: "1", ...} have no current load
samples (not yet sampled, not Up, or older than host.load.stale.after), so DRS
algorithm weighted is balancing every host on allocation only for this plan.
needs DRS. Imbalance: 0.0275 Threshold: 0.0 Algorithm: weighted
```
It had 0 migrations: host 3 was not evacuated for lack of data, and host
1's load was not acted on from a partial picture.
**S2b: a hung agent on a host that stays Up**
- `host.load.stale.after=60`. After a measured plan (4 moves off host 1), I
sent `kill -STOP` to host 3's agent. Host 3 stayed `Up` throughout.
- The plans at 90 s and 154 s both had 0 migrations, imbalance 0.0275, and
logged the **WARN** (no host measured), not the INFO.
- **Why all three hosts.** This is stock `StatsCollector` behaviour, not
this PR. Its host-stats pass sends `GetHostStatsCommand` to each host in turn,
with a 1800 s wait. It stopped at host 3 (`HostStatsCollector is running` at
14:00:50, no answer until the agent was continued at 14:03:06). No host got a
fresh reading during that time, so all three went stale.
- For DRS this is the safe outcome: nothing is evacuated, and nothing
moves on stale load.
- The WARN's last clause ("otherwise no host is Up and reporting stats")
doesn't quite describe this case. The hosts are Up; their stats just aren't
arriving.
- In one of two runs, the 90 s plan was still measured (4 moves off host 1,
no fallback line). A host goes stale `host.load.stale.after` seconds after the
tracker last *folded* a reading, not after the reading was taken. That can lag
the agent's stop by up to a stats interval plus a sample interval, here up to
120 s. With the defaults (600 s stale, 60 s intervals), the lag is small next
to the threshold.
- In the first run the allocator was `balancedweighted`. That plan was
allocation only as well (imbalance 0.0052, 0 moves).
**S3: equivalence grouping (first run)**
- 30 identical VMs (15 each on hosts 1 and 2), 11 fillers, and 2
custom-offering VMs:
```
Worked out candidate hosts for 43 VMs in 6 passes
```
- The fillers took one pass per host (2).
- The identical VMs took one pass per host (2).
- Each custom-offering VM took a pass of its own (2).
- `generateClusterDrsPlan` took 550 ms with `weighted` and 1220 ms with
`balanced`. Without this PR, every VM gets its own lookup, which is 43 here.
**Result:** pass. `weighted` balances on measured CPU load without any
change to the allocator. It falls back to allocation only when a host has no
current samples, and it now logs why. The tracker samples only while an
allocator or a cluster reads the figures.
**Not covered**
- Storage-motion cost (`drs.weighted.storage.motion.cost`): with shared RBD,
no migration needs storage motion.
- Memory-utilisation imbalance (only CPU was loaded).
- The global `drs.algorithm=weighted` path of the sampling gate. Only the
per-cluster setting was used.
- Planning speed on a large cluster. S3 shows the number of lookups, not the
effect at scale.
- More than one management server, where each has its own averages.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
