bhouse-nexthop commented on PR #14109:
URL: https://github.com/apache/cloudstack/pull/14109#issuecomment-6740297953
### Test report: the `balancedweighted` VM allocation algorithm
**Environment**
- A CloudStack 24.0 (main) build with this PR applied, alongside some of my
other open PRs, including #14111 (which reuses this PR's host load tracker) and
#14108.
- Three KVM hosts on Ubuntu 24.04 (libvirt 10.0.0, QEMU 8.2.2), 32 vCPU
each, nested virtualisation; Ceph RBD primary storage; one management server.
DEBUG logging for `com.cloud`.
- `vm.allocation.algorithm=balancedweighted`, `host.load.half.life=60`,
`host.load.sample.interval=30` (management server restarted).
`host.stats.interval` stayed at its default of 60 s. Every other
`host.weighted.*` setting was at its default unless a step says otherwise.
- Test VMs are CirrOS on a 1 vCPU / 200 MHz / 256 MB offering, deployed with
no host id. Host 1 had the lowest allocation throughout: 10.5 GiB of memory
allocated, against 12.5 and 22.5 GiB on the others.
- CPU load came from 30 busy loops in host 1's hypervisor OS, not from
stress VMs. 1-vCPU guests can't take a 32-core host past 85%, and loops on the
host give the same "high load, no allocation" picture. Lower scores rank better.
**S1: utilisation hold-back**
- Host 1 was at 96% CPU (`listHostsMetrics` cpuused 95.96%). The other two
were at 2% and 7%.
- After 8 minutes of load, 6 sequential deploys: none landed on host 1. Each
deploy logged:
```
Holding back 1 host(s) above their utilisation threshold: [host1]
Weighted host ranking: host2=0.3259, host3=0.5682, host1=0.6588
```
- Controls on the same load, 2 deploys each: `firstfitleastconsumed` put
both on host 1, and so did `firstfit`.
- **Timing note.** My first pass deployed after 4 minutes of load, which is
3 half-lives as the test plan says. Its first deploy went to host 1 and was not
held back. Host 1 scored 0.5867 then, and was held back from the second deploy
on, 11 s later, when its score rose to 0.6280. The other 5 deploys avoided it.
- This is the average still converging, not a defect. The tracker only
folds a reading when `StatsCollector` has a new one, which is every 60 s here.
- So 4 minutes of load is about 3 effective half-lives: 0.875 × 96% ≈ 84%,
just under the 0.85 threshold. Load is held back once it has persisted for a
few half-lives, by design.
**S2: burst spread**
- 12 deploys at once with the default `host.weighted.selection.spread=3`: 3
/ 5 / 4 per host.
- The same with `spread=1`: all 12 on host 1. Concurrent deploys all saw the
same ranking, `host1=0.2857, host2=0.3278, host3=0.5700`.
- So the spread is what keeps a burst from piling onto one host. With 3
hosts and spread 3, healthy hosts are picked at random.
**S3: backlog of starting VMs**
- On host 1, a libvirt `qemu` hook wrapper slept 90 s on `prepare begin` for
guest domains, then ran the original hook.
- Settings: `host.weighted.starting.vms.threshold=2` and `spread=1`.
`host.weighted.recent.start.weight` and `vm.count.weight` were set to 0, so
that the hard threshold could be seen on its own and not hidden by the soft
recent-start penalty. All were restored afterwards.
- 6 deploys, 6 s apart:
- Host 1 took 2, which then sat in Starting.
- The rest went to host 2:
```
Holding back 1 host(s) with too many VMs still starting: [host1]
Weighted host ranking: host2=0.3647, host3=0.6327, host1=0.3096
```
Host 1 still had the best score, but was ranked last while it had 2 VMs
Starting.
- The first deploy in this step also ranked host 1 last without a hold-back
line. Host 1's agent had just reconnected after the libvirtd restart, and the
tracker drops samples for hosts that are not Up. So host 1 was briefly in the
"unmeasured" tier, behind the measured hosts. One sample later it was back on
top.
**S4: a host carrying two tags**
- Host 3 tagged `a,b`. 5 new isolated networks, each with a VM, so 5 routers
and 5 user VMs went through the allocator.
- All 10 `Weighted host ranking` lines listed host 3 once. Example:
`host3=0.5745, host1=0.3100, host2=0.3471`.
- The allocator's next line, `FirstFitAllocator has 3 hosts to check`, had 3
entries in every case.
- The tags were removed afterwards.
**S5: a freshly restarted management server**
- Deploys started 1 s after the API came back:
```
11:53:14 Weighted host ranking: host3=0.5070, host1=0.2413, host2=0.2802
```
- No `No candidate host is within its utilisation thresholds` warning.
Unmeasured hosts are not treated as a capacity problem.
**Result:** pass.
- Hosts above the utilisation threshold are held back, unlike the
allocation-only algorithms on the same load.
- A burst spreads across hosts.
- A host queueing starts stops attracting more until they finish.
- A multi-tagged host is ranked once.
- A fresh management server places VMs without warning.
**Not covered**
- Memory-utilisation hold-back (only CPU was loaded).
- Non-KVM hypervisors, whose CPU figure means something else (see the class
javadoc).
- More than one management server, where each has its own average.
- I did not capture the raw candidate list before de-duplication in S4, so
this shows the ranking output is de-duplicated, not how often the input carried
repeats.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
