simanchou commented on PR #13561: URL: https://github.com/apache/cloudstack/pull/13561#issuecomment-5391467989
We can confirm this bug on **ACS 4.22.1.0** (2 management servers + 5 KVM hosts, all hosts connected to MS1), and we have evidence it is **more severe than "Minor"** — it does affect the authenticity of the data. ## Symptoms - `vm_stats` gets **two rows per timestamp** (one per MS). One side is garbage, and *which* side is garbage flips over time as the two collectors' schedules drift. - On a **1-vCPU VM** we observe `cpuUtilization` of **100% and 200%** — physically impossible — while `top` inside the guest shows ~0-6% sustained. - All read paths share the polluted table: `listVirtualMachinesUsageHistory` (reads the table), `listVirtualMachines` / `listVirtualMachinesMetrics` (`ApiDBUtils.getVmStats` → `vmStatsDao.findByVmIdOrderByTimestampDesc` → latest row). With two rows at the same timestamp and no secondary sort key, APIs can return the garbage row nondeterministically. ## Root cause (matches the PR description, with one addition) `LibvirtComputingResource.getVmStat` keeps a **single per-VM `oldStats` entry** shared by all requesters. When both MS collectors are phase-aligned (both fire within the same second), the later request computes utilization over a **sub-second window**, and nanosecond-scale `cpuTime` jitter gets amplified into tens/hundreds of percent. So beyond ignoring the configured interval, the data itself becomes wrong — hence we'd suggest severity **Major**. ## Verification that single-collector fixes it (controlled experiment) 1. Baseline, 2 MS: 2 rows/min per VM; the 1-vCPU VM shows `200, 0, 29.41, 0, 100` in the garbage column (which side is garbage flips sides over time). 2. Stopped MS2 (all hosts owned by MS1) for ~5 minutes: **1 row/min**, values rock-steady at the sane baseline (4-vCPU VM: 4.56–4.60%; 1-vCPU VM: 5.91–6.34%) — zero anomalies. 3. Restarted MS2: dual rows and garbage values return immediately (e.g. 23.44% on an idle VM). This PR's approach (each MS collects only from hosts connected to it) reproduces exactly the single-collector condition without downtime. :+1: We'd appreciate a merge + backport to 4.22 — happy to provide more data if useful. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
