olivierlemasle commented on pull request #5017:
URL: https://github.com/apache/cloudstack/pull/5017#issuecomment-846629382
As I expected, this causes a regression when the usage aggregation range is
greater than the backup sync interval.
@Pearl1594, you wrote:
> I don't think we would lose usage records if
usage.stats.job.aggregation.range < backup.framework.sync.interval, as I had
tested (with the dummy backup provider) with usage.stats.job.aggregation.range
= 2mins and backup.framework.sync.interval = 5mins.
Perhaps you did missed converting the backup sync interval to seconds?
On my side, I did a test on this PR with:
- `usage.stats.job.aggregation.range` = 5 (5 minutes)
- `backup.framework.sync.interval` = 600 (10 minutes)
I have a (non-deleted, running) VM, with a backup (using the dummy B&R
provider).
Here is the usage records for type=28:
<details>
<summary>Backup usages (click to open)</summary>
```
{
"listusagerecordsresponse": {
"count": 6,
"usagerecord": [
{
"account": "admin",
"accountid": "b4d9535b-bc01-11eb-a756-02000b4c0001",
"domainid": "9654cd27-bc01-11eb-a756-02000b4c0001",
"domain": "ROOT",
"zoneid": "06fb3b55-38c6-4f44-a5a2-ec1a167413a6",
"description": "Backup usage of size 0.0000 GiB for VM test-ole
(195bb990-63a1-4805-87d5-f9d6a2f4122d) and backup offering Silver
(0be952ea-b9ab-42b0-ab40-99cece5acc25, user ad-hoc/scheduled backup allowed:
true)",
"usage": "0.0000 GiB",
"usagetype": 28,
"rawusage": "1000",
"virtualmachineid": "195bb990-63a1-4805-87d5-f9d6a2f4122d",
"name": "i-2-3-VM",
"usageid": "195bb990-63a1-4805-87d5-f9d6a2f4122d",
"startdate": "2021-05-23'T'20:30:00+00:00",
"enddate": "2021-05-23'T'20:35:00+00:00",
"tags": []
},
{
"account": "admin",
"accountid": "b4d9535b-bc01-11eb-a756-02000b4c0001",
"domainid": "9654cd27-bc01-11eb-a756-02000b4c0001",
"domain": "ROOT",
"zoneid": "06fb3b55-38c6-4f44-a5a2-ec1a167413a6",
"description": "Backup usage of size 0.0000 GiB for VM test-ole
(195bb990-63a1-4805-87d5-f9d6a2f4122d) and backup offering Silver
(0be952ea-b9ab-42b0-ab40-99cece5acc25, user ad-hoc/scheduled backup allowed:
true)",
"usage": "0.0000 GiB",
"usagetype": 28,
"rawusage": "1000",
"virtualmachineid": "195bb990-63a1-4805-87d5-f9d6a2f4122d",
"name": "i-2-3-VM",
"usageid": "195bb990-63a1-4805-87d5-f9d6a2f4122d",
"startdate": "2021-05-23'T'20:40:00+00:00",
"enddate": "2021-05-23'T'20:45:00+00:00",
"tags": []
},
{
"account": "admin",
"accountid": "b4d9535b-bc01-11eb-a756-02000b4c0001",
"domainid": "9654cd27-bc01-11eb-a756-02000b4c0001",
"domain": "ROOT",
"zoneid": "06fb3b55-38c6-4f44-a5a2-ec1a167413a6",
"description": "Backup usage of size 0.0000 GiB for VM test-ole
(195bb990-63a1-4805-87d5-f9d6a2f4122d) and backup offering Silver
(0be952ea-b9ab-42b0-ab40-99cece5acc25, user ad-hoc/scheduled backup allowed:
true)",
"usage": "0.0000 GiB",
"usagetype": 28,
"rawusage": "1000",
"virtualmachineid": "195bb990-63a1-4805-87d5-f9d6a2f4122d",
"name": "i-2-3-VM",
"usageid": "195bb990-63a1-4805-87d5-f9d6a2f4122d",
"startdate": "2021-05-23'T'20:50:00+00:00",
"enddate": "2021-05-23'T'20:55:00+00:00",
"tags": []
},
{
"account": "admin",
"accountid": "b4d9535b-bc01-11eb-a756-02000b4c0001",
"domainid": "9654cd27-bc01-11eb-a756-02000b4c0001",
"domain": "ROOT",
"zoneid": "06fb3b55-38c6-4f44-a5a2-ec1a167413a6",
"description": "Backup usage of size 0.0000 GiB for VM test-ole
(195bb990-63a1-4805-87d5-f9d6a2f4122d) and backup offering Silver
(0be952ea-b9ab-42b0-ab40-99cece5acc25, user ad-hoc/scheduled backup allowed:
true)",
"usage": "0.0000 GiB",
"usagetype": 28,
"rawusage": "1000",
"virtualmachineid": "195bb990-63a1-4805-87d5-f9d6a2f4122d",
"name": "i-2-3-VM",
"usageid": "195bb990-63a1-4805-87d5-f9d6a2f4122d",
"startdate": "2021-05-23'T'21:00:00+00:00",
"enddate": "2021-05-23'T'21:05:00+00:00",
"tags": []
},
{
"account": "admin",
"accountid": "b4d9535b-bc01-11eb-a756-02000b4c0001",
"domainid": "9654cd27-bc01-11eb-a756-02000b4c0001",
"domain": "ROOT",
"zoneid": "06fb3b55-38c6-4f44-a5a2-ec1a167413a6",
"description": "Backup usage of size 0.0000 GiB for VM test-ole
(195bb990-63a1-4805-87d5-f9d6a2f4122d) and backup offering Silver
(0be952ea-b9ab-42b0-ab40-99cece5acc25, user ad-hoc/scheduled backup allowed:
true)",
"usage": "0.0000 GiB",
"usagetype": 28,
"rawusage": "1000",
"virtualmachineid": "195bb990-63a1-4805-87d5-f9d6a2f4122d",
"name": "i-2-3-VM",
"usageid": "195bb990-63a1-4805-87d5-f9d6a2f4122d",
"startdate": "2021-05-23'T'21:10:00+00:00",
"enddate": "2021-05-23'T'21:15:00+00:00",
"tags": []
},
{
"account": "admin",
"accountid": "b4d9535b-bc01-11eb-a756-02000b4c0001",
"domainid": "9654cd27-bc01-11eb-a756-02000b4c0001",
"domain": "ROOT",
"zoneid": "06fb3b55-38c6-4f44-a5a2-ec1a167413a6",
"description": "Backup usage of size 0.0000 GiB for VM test-ole
(195bb990-63a1-4805-87d5-f9d6a2f4122d) and backup offering Silver
(0be952ea-b9ab-42b0-ab40-99cece5acc25, user ad-hoc/scheduled backup allowed:
true)",
"usage": "0.0000 GiB",
"usagetype": 28,
"rawusage": "1000",
"virtualmachineid": "195bb990-63a1-4805-87d5-f9d6a2f4122d",
"name": "i-2-3-VM",
"usageid": "195bb990-63a1-4805-87d5-f9d6a2f4122d",
"startdate": "2021-05-23'T'21:20:00+00:00",
"enddate": "2021-05-23'T'21:25:00+00:00",
"tags": []
}
]
}
}
```
</details>
As you can see, we have usage for 20:30 to 20:35, then 20:40 to 20:45, etc.,
but one usage record over 2 is missing.
The cause is that the usage server generates an usage record for a 5-minutes
period only if there is [at least one
event](https://github.com/apache/cloudstack/blob/c53b7652b9d2a4373ef8fe7dad272af24f390bef/usage/src/main/java/com/cloud/usage/parser/BackupUsageParser.java#L96)
on that period, and the events are generated every 10 minutes:
```
+---------------------+---------------------+------------------------------------------------------+------+
| created | type | resource_name
| size |
+---------------------+---------------------+------------------------------------------------------+------+
| 2021-05-23 20:34:56 | BACKUP.USAGE.METRIC |
Backup-test-ole-195bb990-63a1-4805-87d5-f9d6a2f4122d | 1000 |
| 2021-05-23 20:44:56 | BACKUP.USAGE.METRIC |
Backup-test-ole-195bb990-63a1-4805-87d5-f9d6a2f4122d | 1000 |
| 2021-05-23 20:54:56 | BACKUP.USAGE.METRIC |
Backup-test-ole-195bb990-63a1-4805-87d5-f9d6a2f4122d | 1000 |
| 2021-05-23 21:04:56 | BACKUP.USAGE.METRIC |
Backup-test-ole-195bb990-63a1-4805-87d5-f9d6a2f4122d | 1000 |
| 2021-05-23 21:14:56 | BACKUP.USAGE.METRIC |
Backup-test-ole-195bb990-63a1-4805-87d5-f9d6a2f4122d | 1000 |
| 2021-05-23 21:24:56 | BACKUP.USAGE.METRIC |
Backup-test-ole-195bb990-63a1-4805-87d5-f9d6a2f4122d | 1000 |
+---------------------+---------------------+------------------------------------------------------+------+
```
So, when the usage aggregation range is greater than the backup sync
interval, we're **missing some usage records** when the VM is still existing.
However, **when the VM destroyed, the usages stop as expected**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
