jansyren commented on issue #13586:
URL: https://github.com/apache/cloudstack/issues/13586#issuecomment-4958065705
# PrometheusExporterImpl calls deployment-global `recalculateCapacity()`
once per zone, causing O(zones) redundant full-infrastructure capacity
recomputes on every scrape
### Summary
`PrometheusExporterImpl.updateMetrics()` invokes
`alertManager.recalculateCapacity()` **inside** the per-datacenter loop.
`AlertManagerImpl.recalculateCapacity()` is deployment-global — it takes no
zone/scope argument and recomputes capacity for *all* hosts, storage pools,
pods and datacenters every time it runs. Calling it once per zone therefore
repeats the entire global recompute N times per scrape, where N = number of
zones.
On a multi-zone deployment this makes the `/metrics` scrape duration scale
with `zones × (hosts + pools + pods)` instead of `hosts + pools + pods`, and it
runs synchronously on the scrape-serving thread.
### Affected code
**`plugins/integrations/prometheus/.../PrometheusExporterImpl.java`**,
`updateMetrics()`:
```java
for (final DataCenterVO dc : dcDao.listAll()) {
final String zoneName = dc.getName();
final String zoneUuid = dc.getUuid();
alertManager.recalculateCapacity(); // <-- deployment-global,
called once PER ZONE
addHostMetrics(latestMetricsItems, dc.getId(), zoneName, zoneUuid);
addVMMetrics(...);
...
}
```
**`server/.../AlertManagerImpl.java`**, `recalculateCapacity()` — takes no
zone parameter and iterates the entire deployment:
```java
public void recalculateCapacity() {
...
recalculateHostCapacities(); // hostDao.listIdsByType(Routing) ->
ALL routing hosts
recalculateStorageCapacities(); // _storagePoolDao.listAllIds() ->
ALL storage pools
List<DataCenterVO> datacenters = _dcDao.listAll();
for (DataCenterVO datacenter : datacenters) { ... } // ALL datacenters
List<HostPodVO> pods = _podDao.listAll();
for (HostPodVO pod : pods) { ... } // ALL pods
}
```
Because the method is global, the per-zone invocation in the exporter
recomputes the same all-zones capacity set once for every zone.
### Impact / measurements
Environment: CloudStack 4.22.0.0, 7 zones, 30 routing hosts.
- Observed `Com_select` delta for a single `/metrics` scrape: **~20,839
queries**.
- Scrape duration climbs from **~0.9 s to ~4.0 s over ~3 days** of uptime,
resetting on management-server restart.
- Thread dumps during a scrape show the serving thread predominantly in
`PrometheusExporterImpl.updateMetrics()` at the `recalculateCapacity()` call
site and in the downstream host-capacity recompute.
- JVM heap is flat (no leak); MariaDB executes the individual capacity query
in <1 ms. The cost is the sheer number of redundant recomputes per scrape, not
slow individual queries or GC.
With a default Prometheus `scrape_timeout` of 10s, a deployment on this
trajectory eventually crosses the timeout and drops scrapes.
### Suggested fix
`recalculateCapacity()` is deployment-global and is **already run on a
timer** by the background capacity checker (`capacity.check.period`, default
300000 ms). The exporter does not need to force a synchronous recompute at all,
and certainly not once per zone.
Options, in order of preference:
1. **Remove the call from the exporter.** Let the metrics read whatever the
scheduled `recalculateCapacity()` background pass last wrote. Capacity gauges
tolerate being up to `capacity.check.period` stale, and the scrape becomes a
pure read.
2. **If a fresh recompute at scrape time is considered necessary, hoist it
out of the loop** so it runs at most once per scrape:
```java
alertManager.recalculateCapacity(); // once, before the loop
for (final DataCenterVO dc : dcDao.listAll()) {
addHostMetrics(...);
...
}
```
Either change reduces the recompute work per scrape by a factor of N(zones)
with no loss of correctness (single-zone deployments are unaffected).
### Secondary observations (not the primary defect)
- `addHostMetrics()` iterates `hostDao.listAll()` for every zone and
`continue`s on hosts not in the current `dcId`, so all hosts are scanned once
per zone. Filtering by datacenter at the DAO level
(`hostDao.listByDataCenterId(dcId)` or equivalent) would avoid the repeated
full scan.
- Within the surviving per-host body, several single-row DAO lookups are
issued per host (`findByHostId`, `getHostTags`, `getHostStatistics`,
`findByHostIdType` ×3, `listByHostId`). These could be batched, though they are
a minor cost next to the per-zone global recompute.
### Environment
- CloudStack: 4.22.0.0
- Hypervisor mgmt: KVM
- DB: MariaDB
- Zones: 7, Routing hosts: 30
- JDK: OpenJDK 17
Added some troubleshooting I have done regarding this with Claude Opus, at
least it didn't destroy anything and it seems to have found some good pointers.
I hope this will help since I am not a JAVA programmer I have to pass the torch
to the experts.
Best of luck and thanks for looking into this
Jan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
