SviridoffA opened a new issue, #14729:
URL: https://github.com/apache/cloudstack/issues/14729

   ### problem
   
   Domain resource-count recalculation fails with a NullPointerException when a 
tagged domain counter is absent and recalculated usage is nonzero. This affects 
both `updateResourceCount` and the periodic `ResourceCountChecker`.
   
   The issue was investigated using production logs, read-only SQL queries and 
source-code inspection. No database repair or destructive reproduction was 
performed during diagnosis.
   
   The API returns **HTTP 530**. The relevant stack trace is:
   
   ```text
   java.lang.NullPointerException: Cannot invoke 
"com.cloud.configuration.ResourceCountVO.setCount(long)" because "domainRC" is 
null
       at 
com.cloud.resourcelimit.ResourceLimitManagerImpl.lambda$recalculateDomainResourceCount$2(ResourceLimitManagerImpl.java:1221)
       at com.cloud.utils.db.Transaction$2.doInTransaction(Transaction.java:48)
       at com.cloud.utils.db.Transaction.execute(Transaction.java:38)
       at com.cloud.utils.db.Transaction.execute(Transaction.java:45)
       at 
com.cloud.resourcelimit.ResourceLimitManagerImpl.recalculateDomainResourceCount(ResourceLimitManagerImpl.java:1184)
       at 
com.cloud.resourcelimit.ResourceLimitManagerImpl.recalculateDomainTaggedResourceCount(ResourceLimitManagerImpl.java:1063)
       at 
com.cloud.resourcelimit.ResourceLimitManagerImpl.recalculateResourceCount(ResourceLimitManagerImpl.java:1128)
       at 
org.apache.cloudstack.api.command.user.resource.UpdateResourceCountCmd.execute(UpdateResourceCountCmd.java:130)
   ```
   
   Read-only inspection confirmed:
   
   - Positive account-level `volume/SSD` counters with corresponding domain 
counters absent.
   - A missing domain `primary_storage/SSD` counter with positive account-level 
usage.
   - A disabled but non-removed account contributing positive tagged usage in a 
failing domain.
   - Successful recalculation for zero-usage domains even though their tagged 
domain counters remain absent.
   - Parent-domain failure during recursive child-domain recalculation.
   - Repeated failures of the periodic checker instead of automatic recovery.
   - No duplicate counters, negative values or invalid owners in the diagnostic 
SELECT results. Unique indexes on `(type, tag, domain_id)` and `(type, tag, 
account_id)` are present.
   
   The failure does not require child domains or an explicitly configured 
tagged limit on the affected domain.
   
   Potential impact is incomplete domain resource accounting. An actual 
end-to-end quota bypass has **not** been demonstrated; account and untagged 
limits may still reject allocations.
   
   **Disclosure:** This report was generated with ChatGPT using 
operator-provided SQL results, inspected production logs and public Apache 
CloudStack source code. Confirmed observations and unverified hypotheses are 
identified separately. Internal addresses, customer names, UUIDs and 
authentication parameters have been omitted.
   
   ### versions
   
   - Apache CloudStack management package: **4.20.3.1** on all three management 
servers.
   - Management operating system: Ubuntu; exact release not included in this 
report.
   - Hypervisor: KVM.
   - Database: separate MySQL cluster; exact version not yet collected.
   
   Relevant global settings:
   
   ```text
   resource.limit.storage.tags = SSD,HDD
   resource.limit.host.tags = <empty>
   resourcecount.check.interval = 300
   ```
   
   Runtime observations are from 4.20.3.1. No runtime reproduction on another 
CloudStack version is claimed.
   
   ### The steps to reproduce the bug
   
   Observed preconditions and action:
   
   1. A storage tag is configured in `resource.limit.storage.tags`.
   2. A non-removed account has positive resource usage for that tag and 
resource type.
   3. The corresponding domain row in `cloud.resource_count` is absent.
   4. Call `updateResourceCount` for the domain without an `account` parameter, 
or let the periodic resource-count checker run.
   5. Observe the NullPointerException and failed domain recalculation.
   
   **Expected:** CloudStack safely initializes the missing domain counter and 
records the recalculated usage.
   
   **Actual:** The API returns HTTP 530, or the periodic checker logs an 
exception. Repeated periodic attempts can fail again.
   
   For zero usage, recalculation can return successfully while leaving the 
domain counter absent.
   
   A possible clean-lab scenario is to create domains and storage-tagged 
resources before enabling those tags for resource-limit accounting, then enable 
the tags and recalculate domain usage. **This scenario has not been 
independently reproduced.** The exact historical action that originally left 
the production rows absent has not been established.
   
   ### What to do about it?
   
   Source inspection identifies the failing path in 
[ResourceLimitManagerImpl.java, version 
4.20.3.1](https://github.com/apache/cloudstack/blob/4.20.3.1/server/src/main/java/com/cloud/resourcelimit/ResourceLimitManagerImpl.java#L1169-L1227):
   
   1. Accounts and child domains are recalculated.
   2. Existing counter rows are selected and locked.
   3. If the current domain/type/tag row is absent, `domainRC` remains `null` 
and `oldResourceCount` remains zero.
   4. A nonzero `newResourceCount` reaches 
`domainRC.setCount(newResourceCount)` at line 1221.
   5. With zero usage, that branch is skipped and the missing row is not 
created.
   
   By contrast, [account 
recalculation](https://github.com/apache/cloudstack/blob/4.20.3.1/server/src/main/java/com/cloud/resourcelimit/ResourceLimitManagerImpl.java#L1275-L1285)
 can persist a missing account counter. 
[ResourceCountDaoImpl](https://github.com/apache/cloudstack/blob/4.20.3.1/engine/schema/src/main/java/com/cloud/configuration/dao/ResourceCountDaoImpl.java#L198-L243)
 also contains create-on-miss logic for tagged domain counters, but the domain 
recalculation path does not use it.
   
   **Potential regression lead:** PR #8903 removed `lockDomainRows()`, which 
called `listAllRowsToUpdate()`, and introduced selection of existing rows. This 
conclusion is based on diff inspection; it has not been verified by reproducing 
the scenario against the parent revision.
   
   PR #11138 fixes a different failure, `Unable to find account null`; the 
corresponding guard is already present in 4.20.3.1.
   
   A fix should initialize missing domain counters safely and calculate correct 
usage, including:
   
   - Positive and zero usage.
   - Disabled but non-removed accounts.
   - Nested domains.
   - Repeated recalculation.
   - Concurrent creation of the same counter by multiple management servers.
   
   Simply skipping a `null` counter would hide the exception while leaving 
accounting incomplete.
   
   Is there an existing fix or recommended backport for this 
missing-domain-counter path?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to