jansyren commented on issue #13586:
URL: https://github.com/apache/cloudstack/issues/13586#issuecomment-4957909584
I have been working with this for a few days and have some findings (yes a
lot is ai generated but they can be helpful sometimes, if they are to you in
this case I don't know)
1. The one-line lever that helps most — move recalculateCapacity() out of
the loop. It's deployment-global, so calling it per-zone is pure waste. In a
patched build it belongs once, before the loop, or removed entirely since
capacity.check.period already runs it every 300s in the background. That alone
likely cuts you from 8 zones × recompute to 1.
You have the source checked out. The change is mechanical:
PrometheusExporterImpl.java
```
alertManager.recalculateCapacity(); // hoist to here, before
the for
for (final DataCenterVO dc : dcDao.listAll()) {
// remove line 511 from inside
```
Why it grows over time — this is the part that matters
recalculateCapacity() in AlertManager walks every host and every VM to
rebuild op_host_capacity. Its cost scales with the number of VMs it has to
examine — including ones in transitional states.
Look at your earlier data:
vm_instance: Expunging = 5, Running = 118
volumes: Expunged = 113, Destroy = 10, Ready = 163
Those 5 Expunging VMs and especially the accumulating Destroy/Expunged
volumes are the growth vector. recalculateCapacity and the capacity aggregation
queries (findCapacityBy with its GROUP BY over host/VM joins) scan rows that
pile up between cleanup cycles. As destroyed-but-not-purged resources
accumulate over days, each recompute examines more rows. That's your 0.9 → 4.0s
slope — not host count (fixed at 44), but transitional-state resource
accumulation feeding the recompute.
The restart resets it because recalculateCapacity rebuilds from live
in-memory capacity state that's cheaper right after boot.
The per-host waste is real too, just secondary
The addHostMetrics loop (line 153) iterates hostDao.listAll() — all 44 hosts
across all zones — then continues on the ones not in dcId. So for each zone you
scan all 44 hosts to process the ~5-6 in that zone. Per host that survives the
filter, it fires:
_dedicatedDao.findByHostId (165)
_hostTagsDao.getHostTags (276, inside markTagMaps)
ApiDBUtils.getHostStatistics (181)
capacityDao.findByHostIdType ×3 (188, 206, 226)
vmDao.listByHostId (222)
~7 queries × 44 hosts × number of zones. Plus line 321-324 rebuilds all host
tags via flatMap over listAllIds() — another full walk. It adds up, but it's
constant per scrape. The growth is the recompute.
Hopefully this will be of some help. I have more query data and system data
collected if you want. The database seems fine and so does the JAVA heap.
Thanks
//Jan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
