0z5a opened a new pull request, #14730:
URL: https://github.com/apache/cloudstack/pull/14730

   ### Description
   
   This PR makes `recalculateDomainResourceCount()` create the domain's 
`resource_count` row when it is missing, instead of failing with a 
`NullPointerException`.
   
   Since #8903 the method only selects existing rows for the domain, its 
immediate child domains and its accounts. If the domain has no row for the type 
and tag, `domainRC` stays `null`. That is the case for a tag that was added to 
`resource.limit.storage.tags` or `resource.limit.host.tags` after the domain 
was created. As soon as the domain's accounts have any usage for that tag, 
`oldResourceCount != newResourceCount` and `domainRC.setCount()` throws. 
`recalculateAccountResourceCount()` already persists a new row in the same 
situation for accounts. This change does the same for the domain: a new row 
with the recalculated count, including 0. When the row exists, nothing changes. 
The discrepancy warning is still logged when the count changes.
   
   Fixes: #14729
   
   ### How I found it
   
   I started from #14729 and checked it against the code. The method, and the 
`// TODO domainRC may be null ...` comment right above the failing line, are 
the same in 4.20.3.1, on the 4.20 branch and on main. Before #8903, 
`lockDomainRows()` went through `listAllRowsToUpdate()`, which created missing 
rows. The new code only selects rows that already exist.
   
   ### Where this can show up in practice
   
   This is my estimate from the code. The only field report I know of is #14729.
   
   - On 4.20 and later, an admin adds a storage or host tag to the limit tags 
while domains already exist. Accounts get their tagged rows on the next 
recalculation, but the domain does not. `updateResourceCount` for that domain 
then fails with error 530.
   - `ResourceCountCheckTask` runs every 300 seconds by default 
(`resourcecount.check.interval`) and has no try/catch around its loop over 
types and domains. It stops at the first domain that hits this, so the types 
and domains after it, and the root accounts, are not rechecked. This repeats on 
every run until a resource with that tag is allocated or released in the 
domain, which creates the row through `listRowsToUpdateForDomain()`.
   
   ### Types of changes
   
   - [ ] Breaking change (fix or feature that would cause existing 
functionality to change)
   - [ ] New feature (non-breaking change which adds functionality)
   - [x] Bug fix (non-breaking change which fixes an issue)
   - [ ] Enhancement (improves an existing feature and functionality)
   - [ ] Cleanup (Code refactoring and cleanup, that may add test cases)
   - [ ] build/CI
   - [ ] test (unit or integration test code)
   
   ### Feature/Enhancement Scale or Bug Severity
   
   #### Feature/Enhancement Scale
   
   - [ ] Major
   - [ ] Minor
   
   #### Bug Severity
   
   - [ ] BLOCKER
   - [ ] Critical
   - [x] Major
   - [ ] Minor
   - [ ] Trivial
   
   ### Screenshots (if appropriate):
   
   ### How Has This Been Tested?
   
   On macOS with JDK 27, without a Maven build. I compiled 
`ResourceLimitManagerImpl` and `ResourceLimitManagerImplTest` from the 4.20.3.1 
sources against the server jar from the 4.20.3.1 `cloudstack-management` 
package, and ran them with JUnit 4.13.2 and Mockito 4.11.0. To run on JDK 27 I 
used Byte Buddy 1.18.17 with `-Dnet.bytebuddy.experimental=true` instead of the 
version Mockito 4.11 depends on.
   
   Three new tests in `ResourceLimitManagerImplTest`, each with one account 
that has usage for a storage tag:
   - `testRecalculateDomainResourceCountCreatesMissingTaggedRow`: no domain 
row, account usage 10. Without the fix it fails with the error from #14729 
(`Cannot invoke "com.cloud.configuration.ResourceCountVO.setCount(long)" 
because "domainRC" is null`). With the fix, the method returns 10, a domain row 
with that type, tag and count 10 is persisted, and `update()` is not called.
   - `testRecalculateDomainResourceCountCreatesMissingTaggedRowWithZeroCount`: 
no domain row, usage 0. Without the fix it does not throw, but nothing is 
persisted. With the fix, a 0 row is persisted.
   - `testRecalculateDomainResourceCountUpdatesExistingRow` (control): existing 
domain row with count 5. It passes with and without the fix: the row is updated 
to 10 and nothing is persisted.
   
   Results: before the fix, 56 of 58 tests pass (the first two above fail). 
After the fix, all 58 pass (55 existing and 3 new).
   
   I then ran the same check on the 4.20 branch head (e6f193db), where the 
method is unchanged. The head uses 
`ResourceLimitDao.listByDomainIdsAndTypeAndTag()`, which is not in the 4.20.3.1 
jar, so I compiled `ResourceLimitDao` and `ResourceLimitDaoImpl` from the head 
in front of the jar. `ResourceLimitManagerImplTest` from the head with the 
three new tests: 64 of 66 pass without the fix (the same two fail), and 66 pass 
with it.
   
   Not run: the Maven build, checkstyle, the simulator and Marvin tests, and a 
real management server with MySQL.
   
   #### How did you try to break this feature and the system with this change?
   
   Going through the cases listed in #14729:
   - Positive and zero usage: both create the row (the first two tests).
   - Disabled accounts that are not removed: `findActiveAccountsForDomain()` 
only leaves out removed accounts, so their usage is counted. This PR does not 
change that.
   - Nested domains: child domains are recalculated first, each in its own 
transaction, so their rows exist before the parent adds up the rows of its 
immediate children. There is no new test for this.
   - Repeated recalculation: the next run finds the row and takes the existing 
update path (the control test).
   - Concurrent creation: the periodic task only runs on one management server, 
under the `ResourceCheckTask` global lock, but the row can also be created at 
the same time by a manual `updateResourceCount`, or by 
`listRowsToUpdateForDomain()` when a resource with that tag is allocated or 
released on any management server. That path also uses a plain `persist()` 
without a lock. `schema-41910to42000.sql` adds the unique index 
`i_resource_count__type_tag_domainId` on `(type, tag, domain_id)`, so the 
second insert fails with a duplicate key error and no second row is created. If 
the recalculation is the one that fails, its transaction is rolled back and 
that checker run stops at this domain, as it does today with the NPE, but the 
next run finds the row and updates it. If the allocation fails, it fails the 
same way it already can when two allocations create the row at once. I did not 
add locking or a retry, to keep this the same as the existing create-on-miss 
path. If you want a retry on 
 duplicate key here, I can add one.
   - Untagged rows have `tag` NULL, which the unique index does not cover, but 
those rows are created together with the domain, so this path should not 
normally need to create them.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to