Daniel-Pereira-Linux opened a new pull request, #14612: URL: https://github.com/apache/cloudstack/pull/14612
### Description This PR backports #13936 to 4.22, targeting 4.22.2. Since #11531 (4.22.1), Usage writes two `usage_volume` helper rows for an attached volume: the cumulative row (`vm_id` NULL) and a per-VM row. Each one generates a `usage_type = 6` record, and Quota bills both, so attached volumes are charged twice. #13936 makes Quota ignore the per-VM records (VOLUME with `vm_instance_id` set): they are marked as calculated, and no `quota_usage` row is created for them. It is a clean cherry-pick (`-x`) of ba67a627f7. The only conflict was in `QuotaManagerImplTest`: 4.22 does not have `QuotaTariffUsageDao`, so only the `ConfigDepotImpl` mock was kept. This matters more for 4.22.2: #13933 (already in 4.22) widened the unique key of `usage_volume`, so ROOT volumes now get the per-VM row too. Without this backport, 4.22.2 would also bill ROOT volumes twice. Fixes: #14550 ### Types of changes - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] New feature (non-breaking change which adds functionality) - [x] Bug fix (non-breaking change which fixes an issue) - [ ] Enhancement (improves an existing feature and functionality) - [ ] Cleanup (Code refactoring and cleanup, that may add test cases) - [ ] Build/CI - [ ] Test (unit or integration test code) ### Feature/Enhancement Scale or Bug Severity #### Bug Severity - [ ] BLOCKER - [ ] Critical - [x] Major - [ ] Minor - [ ] Trivial ### How Has This Been Tested? **Unit tests** (JDK 21): `framework/quota` with checkstyle (123 tests), `plugins/database/quota` (76) and `usage` (21), all passing. Without the fix, `shouldCalculateUsageRecordTestQuotaIsEnabledForAccountAndUsageRecordIsVolumeWithVmInstanceIdReturnFalse` fails. **End to end**, on a lab with a management server and a nested KVM host (Rocky 9), using the official 4.22.1.1 packages, a 5-minute aggregation range and a VOLUME tariff. I compared `cloud_usage` with `quota_usage` record by record: 1. Official 4.22.1.1: a data disk attached to a running VM was billed twice in the same window (236.11 for the cumulative record and 195.83 for the per-VM one). 2. Same setup with the patched `cloud-framework-quota` jar in the Usage server: per-VM records are no longer billed, and the cumulative record is billed once. RUNNING_VM is still billed. 3. Usage server replaced by the build of this branch, after applying `schema-42210to42220.sql` (management server still on 4.22.1.1). The ROOT volume now gets the per-VM row and is billed once. 4. Advanced zone (isolated network with VR, VLAN isolation), same Usage build. I went through: data disk created with the VM, detach and attach to another VM in the same window, stop with the disk attached, resize from 5 to 20 GB, destroy with expunge (implicit detach) and delete. In 30 volume windows, no per-VM record was billed and the cumulative record was billed once in each. In the resize window there are two cumulative records, one per size, together covering the 5 minutes. Also checked: - All per-VM records end up with `quota_calculated = 1`, so they are not picked up again. - `quotaStatement` and `quota_balance` match the sum of the cumulative records only. Not tested: VMware, XenServer and multiple management servers. #### How did you try to break this feature and the system with this change? - Moving a volume between VMs inside one aggregation window, resizing it, and removing it through VM expunge, to check that every transition that creates or ends a per-VM row is covered. - Swapping the official and patched jars on the same environment and comparing record by record, instead of comparing totals. #### Notes @winterhazel, about existing records: on an unmodified 4.22.1.1 install I did get duplicates for data disks. The old unique key (`volume_id`, `created`) only collides for ROOT volumes, where both rows share the same `created`. A data disk attached after it was created gets a per-VM row with the attach time, so it is inserted and billed. So 4.22.1.x environments that use Quota may already have duplicated `quota_usage` rows for data disks. This PR does not change them. Do you think it is worth a note in the release notes? Also on 4.22.1.1, the ROOT per-VM insert fails with `Duplicate entry ... for key 'id'`. The rollback leaves earlier events with `processed = 0`, so every job moves its start date back to the oldest event and parses old windows again. This is what #13933 fixes: after applying the new schema, the errors stopped and the jobs went back to consecutive windows. But the windows that were parsed again stay in `cloud_usage`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
