SviridoffA opened a new issue, #14729:
URL: https://github.com/apache/cloudstack/issues/14729
### problem
Domain resource-count recalculation fails with a NullPointerException when a
tagged domain counter is absent and recalculated usage is nonzero. This affects
both `updateResourceCount` and the periodic `ResourceCountChecker`.
The issue was investigated using production logs, read-only SQL queries and
source-code inspection. No database repair or destructive reproduction was
performed during diagnosis.
The API returns **HTTP 530**. The relevant stack trace is:
```text
java.lang.NullPointerException: Cannot invoke
"com.cloud.configuration.ResourceCountVO.setCount(long)" because "domainRC" is
null
at
com.cloud.resourcelimit.ResourceLimitManagerImpl.lambda$recalculateDomainResourceCount$2(ResourceLimitManagerImpl.java:1221)
at com.cloud.utils.db.Transaction$2.doInTransaction(Transaction.java:48)
at com.cloud.utils.db.Transaction.execute(Transaction.java:38)
at com.cloud.utils.db.Transaction.execute(Transaction.java:45)
at
com.cloud.resourcelimit.ResourceLimitManagerImpl.recalculateDomainResourceCount(ResourceLimitManagerImpl.java:1184)
at
com.cloud.resourcelimit.ResourceLimitManagerImpl.recalculateDomainTaggedResourceCount(ResourceLimitManagerImpl.java:1063)
at
com.cloud.resourcelimit.ResourceLimitManagerImpl.recalculateResourceCount(ResourceLimitManagerImpl.java:1128)
at
org.apache.cloudstack.api.command.user.resource.UpdateResourceCountCmd.execute(UpdateResourceCountCmd.java:130)
```
Read-only inspection confirmed:
- Positive account-level `volume/SSD` counters with corresponding domain
counters absent.
- A missing domain `primary_storage/SSD` counter with positive account-level
usage.
- A disabled but non-removed account contributing positive tagged usage in a
failing domain.
- Successful recalculation for zero-usage domains even though their tagged
domain counters remain absent.
- Parent-domain failure during recursive child-domain recalculation.
- Repeated failures of the periodic checker instead of automatic recovery.
- No duplicate counters, negative values or invalid owners in the diagnostic
SELECT results. Unique indexes on `(type, tag, domain_id)` and `(type, tag,
account_id)` are present.
The failure does not require child domains or an explicitly configured
tagged limit on the affected domain.
Potential impact is incomplete domain resource accounting. An actual
end-to-end quota bypass has **not** been demonstrated; account and untagged
limits may still reject allocations.
**Disclosure:** This report was generated with ChatGPT using
operator-provided SQL results, inspected production logs and public Apache
CloudStack source code. Confirmed observations and unverified hypotheses are
identified separately. Internal addresses, customer names, UUIDs and
authentication parameters have been omitted.
### versions
- Apache CloudStack management package: **4.20.3.1** on all three management
servers.
- Management operating system: Ubuntu; exact release not included in this
report.
- Hypervisor: KVM.
- Database: separate MySQL cluster; exact version not yet collected.
Relevant global settings:
```text
resource.limit.storage.tags = SSD,HDD
resource.limit.host.tags = <empty>
resourcecount.check.interval = 300
```
Runtime observations are from 4.20.3.1. No runtime reproduction on another
CloudStack version is claimed.
### The steps to reproduce the bug
Observed preconditions and action:
1. A storage tag is configured in `resource.limit.storage.tags`.
2. A non-removed account has positive resource usage for that tag and
resource type.
3. The corresponding domain row in `cloud.resource_count` is absent.
4. Call `updateResourceCount` for the domain without an `account` parameter,
or let the periodic resource-count checker run.
5. Observe the NullPointerException and failed domain recalculation.
**Expected:** CloudStack safely initializes the missing domain counter and
records the recalculated usage.
**Actual:** The API returns HTTP 530, or the periodic checker logs an
exception. Repeated periodic attempts can fail again.
For zero usage, recalculation can return successfully while leaving the
domain counter absent.
A possible clean-lab scenario is to create domains and storage-tagged
resources before enabling those tags for resource-limit accounting, then enable
the tags and recalculate domain usage. **This scenario has not been
independently reproduced.** The exact historical action that originally left
the production rows absent has not been established.
### What to do about it?
Source inspection identifies the failing path in
[ResourceLimitManagerImpl.java, version
4.20.3.1](https://github.com/apache/cloudstack/blob/4.20.3.1/server/src/main/java/com/cloud/resourcelimit/ResourceLimitManagerImpl.java#L1169-L1227):
1. Accounts and child domains are recalculated.
2. Existing counter rows are selected and locked.
3. If the current domain/type/tag row is absent, `domainRC` remains `null`
and `oldResourceCount` remains zero.
4. A nonzero `newResourceCount` reaches
`domainRC.setCount(newResourceCount)` at line 1221.
5. With zero usage, that branch is skipped and the missing row is not
created.
By contrast, [account
recalculation](https://github.com/apache/cloudstack/blob/4.20.3.1/server/src/main/java/com/cloud/resourcelimit/ResourceLimitManagerImpl.java#L1275-L1285)
can persist a missing account counter.
[ResourceCountDaoImpl](https://github.com/apache/cloudstack/blob/4.20.3.1/engine/schema/src/main/java/com/cloud/configuration/dao/ResourceCountDaoImpl.java#L198-L243)
also contains create-on-miss logic for tagged domain counters, but the domain
recalculation path does not use it.
**Potential regression lead:** PR #8903 removed `lockDomainRows()`, which
called `listAllRowsToUpdate()`, and introduced selection of existing rows. This
conclusion is based on diff inspection; it has not been verified by reproducing
the scenario against the parent revision.
PR #11138 fixes a different failure, `Unable to find account null`; the
corresponding guard is already present in 4.20.3.1.
A fix should initialize missing domain counters safely and calculate correct
usage, including:
- Positive and zero usage.
- Disabled but non-removed accounts.
- Nested domains.
- Repeated recalculation.
- Concurrent creation of the same counter by multiple management servers.
Simply skipping a `null` counter would hide the exception while leaving
accounting incomplete.
Is there an existing fix or recommended backport for this
missing-domain-counter path?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]