0z5a opened a new pull request, #14730: URL: https://github.com/apache/cloudstack/pull/14730
### Description This PR makes `recalculateDomainResourceCount()` create the domain's `resource_count` row when it is missing, instead of failing with a `NullPointerException`. Since #8903 the method only selects existing rows for the domain, its immediate child domains and its accounts. If the domain has no row for the type and tag, `domainRC` stays `null`. That is the case for a tag that was added to `resource.limit.storage.tags` or `resource.limit.host.tags` after the domain was created. As soon as the domain's accounts have any usage for that tag, `oldResourceCount != newResourceCount` and `domainRC.setCount()` throws. `recalculateAccountResourceCount()` already persists a new row in the same situation for accounts. This change does the same for the domain: a new row with the recalculated count, including 0. When the row exists, nothing changes. The discrepancy warning is still logged when the count changes. Fixes: #14729 ### How I found it I started from #14729 and checked it against the code. The method, and the `// TODO domainRC may be null ...` comment right above the failing line, are the same in 4.20.3.1, on the 4.20 branch and on main. Before #8903, `lockDomainRows()` went through `listAllRowsToUpdate()`, which created missing rows. The new code only selects rows that already exist. ### Where this can show up in practice This is my estimate from the code. The only field report I know of is #14729. - On 4.20 and later, an admin adds a storage or host tag to the limit tags while domains already exist. Accounts get their tagged rows on the next recalculation, but the domain does not. `updateResourceCount` for that domain then fails with error 530. - `ResourceCountCheckTask` runs every 300 seconds by default (`resourcecount.check.interval`) and has no try/catch around its loop over types and domains. It stops at the first domain that hits this, so the types and domains after it, and the root accounts, are not rechecked. This repeats on every run until a resource with that tag is allocated or released in the domain, which creates the row through `listRowsToUpdateForDomain()`. ### Types of changes - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] New feature (non-breaking change which adds functionality) - [x] Bug fix (non-breaking change which fixes an issue) - [ ] Enhancement (improves an existing feature and functionality) - [ ] Cleanup (Code refactoring and cleanup, that may add test cases) - [ ] build/CI - [ ] test (unit or integration test code) ### Feature/Enhancement Scale or Bug Severity #### Feature/Enhancement Scale - [ ] Major - [ ] Minor #### Bug Severity - [ ] BLOCKER - [ ] Critical - [x] Major - [ ] Minor - [ ] Trivial ### Screenshots (if appropriate): ### How Has This Been Tested? On macOS with JDK 27, without a Maven build. I compiled `ResourceLimitManagerImpl` and `ResourceLimitManagerImplTest` from the 4.20.3.1 sources against the server jar from the 4.20.3.1 `cloudstack-management` package, and ran them with JUnit 4.13.2 and Mockito 4.11.0. To run on JDK 27 I used Byte Buddy 1.18.17 with `-Dnet.bytebuddy.experimental=true` instead of the version Mockito 4.11 depends on. Three new tests in `ResourceLimitManagerImplTest`, each with one account that has usage for a storage tag: - `testRecalculateDomainResourceCountCreatesMissingTaggedRow`: no domain row, account usage 10. Without the fix it fails with the error from #14729 (`Cannot invoke "com.cloud.configuration.ResourceCountVO.setCount(long)" because "domainRC" is null`). With the fix, the method returns 10, a domain row with that type, tag and count 10 is persisted, and `update()` is not called. - `testRecalculateDomainResourceCountCreatesMissingTaggedRowWithZeroCount`: no domain row, usage 0. Without the fix it does not throw, but nothing is persisted. With the fix, a 0 row is persisted. - `testRecalculateDomainResourceCountUpdatesExistingRow` (control): existing domain row with count 5. It passes with and without the fix: the row is updated to 10 and nothing is persisted. Results: before the fix, 56 of 58 tests pass (the first two above fail). After the fix, all 58 pass (55 existing and 3 new). I then ran the same check on the 4.20 branch head (e6f193db), where the method is unchanged. The head uses `ResourceLimitDao.listByDomainIdsAndTypeAndTag()`, which is not in the 4.20.3.1 jar, so I compiled `ResourceLimitDao` and `ResourceLimitDaoImpl` from the head in front of the jar. `ResourceLimitManagerImplTest` from the head with the three new tests: 64 of 66 pass without the fix (the same two fail), and 66 pass with it. Not run: the Maven build, checkstyle, the simulator and Marvin tests, and a real management server with MySQL. #### How did you try to break this feature and the system with this change? Going through the cases listed in #14729: - Positive and zero usage: both create the row (the first two tests). - Disabled accounts that are not removed: `findActiveAccountsForDomain()` only leaves out removed accounts, so their usage is counted. This PR does not change that. - Nested domains: child domains are recalculated first, each in its own transaction, so their rows exist before the parent adds up the rows of its immediate children. There is no new test for this. - Repeated recalculation: the next run finds the row and takes the existing update path (the control test). - Concurrent creation: the periodic task only runs on one management server, under the `ResourceCheckTask` global lock, but the row can also be created at the same time by a manual `updateResourceCount`, or by `listRowsToUpdateForDomain()` when a resource with that tag is allocated or released on any management server. That path also uses a plain `persist()` without a lock. `schema-41910to42000.sql` adds the unique index `i_resource_count__type_tag_domainId` on `(type, tag, domain_id)`, so the second insert fails with a duplicate key error and no second row is created. If the recalculation is the one that fails, its transaction is rolled back and that checker run stops at this domain, as it does today with the NPE, but the next run finds the row and updates it. If the allocation fails, it fails the same way it already can when two allocations create the row at once. I did not add locking or a retry, to keep this the same as the existing create-on-miss path. If you want a retry on duplicate key here, I can add one. - Untagged rows have `tag` NULL, which the unique index does not cover, but those rows are created together with the domain, so this path should not normally need to create them. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
