GutoVeronezi opened a new issue #5891:
URL: https://github.com/apache/cloudstack/issues/5891
##### ISSUE TYPE
* Enhancement Request
##### COMPONENT NAME
~~~
Quota, billing
~~~
##### CLOUDSTACK VERSION
4.16/main
##### SUMMARY
This spec changes the Cloudstack's Quota Plugin to allow operators to
customize tariffs based on characteristics of the billed resources.
---
# Table of Contents
- [Problem description](#problem-description)
- [Current workflows](#problem-description--current-workflows)
- [Calculate
usage](#problem-description--current-workflows--calculate-usage)
- [List tariff](#problem-description--current-workflows--list-tariff)
- [Update
tariff](#problem-description--current-workflows--update-tariff)
- [Proposed changes](#proposed-changes)
- [Proposed workflows](#proposed-changes--proposed-workflows)
- [Calculate
usage](#proposed-changes--proposed-workflows--calculate-usage)
- [List tariff](#proposed-changes--proposed-workflows--list-tariff)
- [Update tariff](#proposed-changes--proposed-workflows--update-tariff)
- [Create tariff](#proposed-changes--proposed-workflows--create-tariff)
- [Delete tariff](#proposed-changes--proposed-workflows--delete-tariff)
- [Rules processing and
variables](#proposed-changes--rules-processing-and-variables)
- [Default](#proposed-changes--rules-processing-and-variables--default)
-
[RUNNING_VM](#proposed-changes--rules-processing-and-variables--running-vm)
-
[ALLOCATED_VM](#proposed-changes--rules-processing-and-variables--allocated-vm)
- [VOLUME](#proposed-changes--rules-processing-and-variables--volume)
-
[TEMPLATE/ISO](#proposed-changes--rules-processing-and-variables--template-iso)
-
[SNAPSHOT](#proposed-changes--rules-processing-and-variables--snapshot)
-
[NETWORK_OFFERING](#proposed-changes--rules-processing-and-variables--network-offering)
-
[VM_SNAPSHOT](#proposed-changes--rules-processing-and-variables--vm-snapshot)
- [Others
resources](#proposed-changes--rules-processing-and-variables--other-resources)
- [Script samples](#proposed-changes--script-samples)
- [Billing example](#proposed-changes--billing-example)
- [Work items](#work-items)
- [Future works](#future-works)
---
<a name="problem-description"/>
Problem description
===============
Currently, ACS's **Quota Plugin** accounts for different resources:
```sh
RUNNING_VM
ALLOCATED_VM
IP_ADDRESS
NETWORK_BYTES_SENT
NETWORK_BYTES_RECEIVED
VOLUME
TEMPLATE
ISO
SNAPSHOT
SECURITY_GROUP
LOAD_BALANCER_POLICY
PORT_FORWARDING_RULE
NETWORK_OFFERING
VPN_USERS
CPU_SPEED
vCPU
MEMORY
VM_DISK_IO_READ
VM_DISK_IO_WRITE
VM_DISK_BYTES_READ
VM_DISK_BYTES_WRITE
VM_SNAPSHOT
```
Each element assumes only one tariff/price, turning it into an one to one
relationship:

Therefore, for every **RUNNING_VM**, we must apply the same tariff/price, as
well as every **ALLOCATED_VM**. However, there are situations where one
resource needs to have different tariffs/prices. These situations can be
related to characteristics of the business, such as Windows (or other O.S.)
licensing, performance of the primary storage where volumes are allocated, and
who is the owner of the resource. Examples of mapped cases till now:
| **Characteristic** | **Example** |
| ------ | ------ |
| owner (account/domain/project) | Owner **X** has a special contract and
will pay a different price per resource. |
| volume of allocated resource for the owner | If owner **X** has less than
10 VMs, the owner will pay **Y**, otherwise, **Z**. |
| O.S. | VMs with Windows (or other built-in licensing) costs more. |
| storage tags | Volumes with tag **SSD NVME** costs more. |
| host tags | VMs with tag **CPU platinum** costs more. |
<a name="problem-description--current-workflows"/>
## Current workflows
Current **calculate usage**, **list tariff** and **update tariff** workflows
are:
<a name="problem-description--current-workflows--calculate-usage"/>
- Calculate usage:
<img
src="https://user-images.githubusercontent.com/38945620/150794770-8ff87aca-9953-4808-b7e6-211e07db33c0.png"
alt="calculate-usage" width="400"/>
---
<a name="problem-description--current-workflows--list-tariff"/>
- List tariff:
<img
src="https://user-images.githubusercontent.com/38945620/150794788-1ce8a577-e28f-4fc1-ba5f-c7c9606ea3a9.png"
alt="list-tariff" width="400"/>
---
<a name="problem-description--current-workflows--update-tariff"/>
- Update tariff:
<img
src="https://user-images.githubusercontent.com/38945620/150794809-0cbdcdb7-8495-4ec8-8ce7-b8dedd595498.png"
alt="update-tariff" width="400"/>
<a name="proposed-changes"/>
Proposed changes
==============
This proposal intends to change the paradigm of the feature by allowing
tariff customization and making the relationship between resource and tariff
**one to zero or many**:

For each one of the resources (listed in the section **Problem
description**), operators will be able to create many tariffs as needed and
define (or not) the activation rules for each tariff and its duration (the
start date will always be required). Activation rules will be used to define if
a tariff should be applied to a resource being rated; if no activation rule is
provided, we assume that it is applied to all resources of the given type. When
updating the tariff, the previous one will be removed and a new one will be
created. However, to ensure traceability of the record, a common identifier
will be kept.
<a name="proposed-changes--proposed-workflows"/>
## Proposed workflows
Proposed **calculate usage**, **list tariff** and **update tariff**
workflows are:
<a name="proposed-changes--proposed-workflows--calculate-usage"/>
- Calculate usage:
<img
src="https://user-images.githubusercontent.com/38945620/150795343-7004aa1d-3845-41e6-9366-9dcf6302441c.png"
alt="calculate-usage" width="800"/>
---
<a name="proposed-changes--proposed-workflows--list-tariff"/>
- List tariff:
<img
src="https://user-images.githubusercontent.com/38945620/150795395-d2633557-cb76-4e58-87e6-c3c47c08f23f.png"
alt="list-tariff" width="400"/>
This API will receive three (3) new parameters:
| **Parameters** | **Description** | **Required** | **Value** |
| ------ | ------ | ------ | ------ |
| name | To retrieve tariff by its name. | No | The name of the tariff. |
| enddate | To retrieve tariffs with end date less or equal to the
parameter. | No | Any date (format yyyy-MM-dd). |
| listall | To retrieve even removed tariffs. | No | **true** or **false**.
|
---
<a name="proposed-changes--proposed-workflows--update-tariff"/>
- Update tariff:
<img
src="https://user-images.githubusercontent.com/38945620/150795819-e28eccef-6714-48ce-baf1-18118e09ce55.png"
alt="update-tariff" width="400"/>
| **Parameters** | **Description** | **Required** | **Value** |
| ------ | ------ | ------ | ------ |
| id | UUID of the tariff to update. | Yes | The UUID of the tariff. |
| description | Description of the tariff. | No | Any string (max 65535
characters). |
| value | The price of the tariff. | No | Any float value. |
| activationrule | The rule to apply the tariff. Null means that it will
always by applied. | No | Any JavaScript code (max 65535 characters). |
| enddate | Date when the tariff will stop to be applied. | No | Any date
(format yyyy-MM-dd) from the current date and after or equal **startdate**. |
The parameter **usagetype** will be kept, however it will not be used and a
warning message is going to show that it is ignored for the request. Moreover,
it will be removed in future releases.
---
Also, will be necessary to create two (2) new APIs:
<a name="proposed-changes--proposed-workflows--create-tariff"/>
- Create tariff: this API will allow operators to create new tariffs for
the listed resources;
<img
src="https://user-images.githubusercontent.com/38945620/150801454-02d64eb9-88d1-44a1-986b-667a150324b0.png"
alt="create-tariff" width="400"/>
| **Parameters** | **Description** | **Required** | **Value** |
| ------ | ------ | ------ | ------ |
| name | An unique name for the tariff. | Yes | Any string (max 65535
characters). |
| description | Description of the tariff. | No | Any string (max 65535
characters). |
| usagetype | Resource type of the tariff. | Yes | Any of the resource types
(listed in the section **Problem description**). |
| value | The price of the tariff. | Yes | Any float value. |
| activationrule | The rule to apply the tariff. Null means that it will
always by applied. | No | Any JavaScript code (max 65535 characters). |
| startdate | Date when the tariff will start to be applied. | No | Any date
from the current date. If this parameter is not informed, the default value
will be D+1. |
| enddate | Date when the tariff will stop to be applied. | No | Any date
from the current date and after or equal **startdate**. |
---
<a name="proposed-changes--proposed-workflows--delete-tariff"/>
- Delete tariff: this API will mark the tariff as removed;
<img
src="https://user-images.githubusercontent.com/38945620/150802258-e1845088-3373-45c9-90ae-95e84b800e9c.png"
alt="delete-tariff" width="400"/>
| **Parameters** | **Description** | **Required** | **Value** |
| ------ | ------ | ------ | ------ |
| id | The UUID of the tariff. | Yes | The UUID of the tariff. |
<a name="proposed-changes--rules-processing-and-variables"/>
## Rules processing and variables
To process the activation rules, it will be used the library
[J2V8](https://github.com/eclipsesource/J2V8) which "...is a set of Java
bindings for V8. J2V8 focuses on performance and tight integration with V8...".
It has a considerable relevance, good performance and is easy to implement.
Therefore, the activation rule expressions must be written in **JavaScript
code**[^1] and return a boolean or number value[^2]. If there is no expression
to be evaluated or the expression is empty, the tariff will always be applied.
Some variables will be pre-created into the code's context to give more
flexibility to operators. Each resource type will have a series of
variables corresponding to their characteristics:
---
<a name="proposed-changes--rules-processing-and-variables--default"/>
### Default
| **Variable** | **Description** |
| ------ | ------ |
| account.id | UUID of the account owner of the resource.|
| account.name | Name of the account owner of the resource.|
| account.role.id | UUID of the role of the account owner of the resource
(if exists).|
| account.role.name | Name of the role of the account owner of the resource
(if exists).|
| account.role.type | Type of the role of the account owner of the resource
(if exists).|
| domain.id | UUID of the domain owner of the resource.|
| domain.name | Name of the domain owner of the resource.|
| domain.path | Path of the domain owner of the resource.|
| project.id | UUID of the project owner of the resource (if exists).|
| project.name | Name of the project owner of the resource (if exists).|
| resourceType | Type of the record.|
| value.accountResources | List of resources of the account between the
start and end date of the usage record being calculated (i.e.: **[{zoneId: ...,
domainId:...}]**).|
| zone.id | UUID of the zone owner of the resource.|
| zone.name | Name of the zone owner of the resource.|
---
<a name="proposed-changes--rules-processing-and-variables--running-vm"/>
### RUNNING_VM
| **Variable** | **Description** |
| ------ | ------ |
| value.host.id | UUID of the host where the VM is
running. |
| value.host.name | Name of the host where the VM is
running. |
| value.host.tags | List of tags of the host where the VM
is running (i.e.: **["a", "b"]**). |
| value.id | UUID of the VM. |
| value.name | Name of the VM. |
| value.osName | Name of the OS of the VM. |
| value.computeOffering.customized | A boolean informing if the compute
offering is customized or not. |
| value.computeOffering.id | UUID of the compute offering with
which VM was created. |
| value.computeOffering.name | Name of the compute offering with
which VM was created. |
| value.tags | List of tags of the VM in the format
**key:value** (i.e.: **{"a":"b", "c":"d"}**). |
| value.template.id | UUID of the template with which VM was
created. |
| value.template.name | Name of the template with which VM was
created. |
---
<a name="proposed-changes--rules-processing-and-variables--allocated-vm"/>
### ALLOCATED_VM
| **Variable** | **Description** |
| ------ | ------ |
| value.id | UUID of the VM. |
| value.name | Name of the VM. |
| value.osName | Name of the OS of the VM. |
| value.computeOffering.customized | A boolean informing if the compute
offering is customized or not. |
| value.computeOffering.id | UUID of the compute offering with which
VM was created. |
| value.computeOffering.name | Name of the compute offering with which
VM was created. |
| value.tags | List of tags of the VM in the format
**key:value** (i.e.: **{"a":"b", "c":"d"}**). |
| value.template.id | UUID of the template with which VM was
created. |
| value.template.name | Name of the template with which VM was
created. |
---
<a name="proposed-changes--rules-processing-and-variables--volume"/>
### VOLUME
| **Variable** | **Description** |
| ------ | ------ |
| value.diskOffering.id | UUID of the disk offering with which volume was
created. |
| value.diskOffering.name | Name of the disk offering with which volume was
created. |
| value.id | UUID of the volume. |
| value.name | Name of the volume. |
| value.provisioningType | Provisioning type of the resource. Values can
be: **thin**, **sparse** or **fat**. |
| value.storage.id | UUID of the storage where the volume is. |
| value.storage.name | Name of the storage where the volume is. |
| value.storage.scope | Scope of the storage where the volume is.
Values can be: **ZONE** or **CLUSTER**. |
| value.storage.tags | List of tags of the storage where the volume is
(i.e.: **["a", "b"]**). |
| value.tags | List of tags of the volume in the format
**key:value** (i.e.: **{"a":"b", "c":"d"}**). |
| value.size | Size of the volume (in MiB). |
---
<a name="proposed-changes--rules-processing-and-variables--template-iso"/>
### TEMPLATE / ISO
| **Variable** | **Description** |
| ------ | ------ |
| value.id | UUID of the template/ISO. |
| value.name | Name of the template/ISO. |
| value.osName | Name of the OS of the template/ISO. |
| value.tags | List of tags of the template/ISO in the format
**key:value** (i.e.: **{"a":"b", "c":"d"}**). |
| value.size | Size of the template/ISO (in MiB). |
---
<a name="proposed-changes--rules-processing-and-variables--snapshot"/>
### SNAPSHOT
| **Variable** | **Description** |
| ------ | ------ |
| value.id | UUID of the snapshot. |
| value.name | Name of the snapshot. |
| value.size | Size of the snapshot (in MiB). |
| value.snapshotType | Type of the snapshot. Values can be: **MANUAL**,
**HOURLY**, **DAILY**, **WEEKLY** and **MONTHLY**. |
| value.storage.id | UUID of the storage where the snapshot is. The data
will be from the primary storage if the global setting
**snapshot.backup.to.secondary** is **false**, otherwise it will be from
secondary storage. |
| value.storage.name | Name of the storage where the snapshot is. The data
will be from the primary storage if the global setting
**snapshot.backup.to.secondary** is **false**, otherwise it will be from
secondary storage. |
| value.storage.scope | If the global setting
**snapshot.backup.to.secondary** is **false**, the scope of the primary storage
where the snapshot is (values can be: **ZONE** or **CLUSTER**), otherwise it
will not exist. |
| value.storage.tags | List of tags of the storage where the snapshot is
(i.e.: **["a", "b"]**). The data will be from the primary storage if the global
setting **snapshot.backup.to.secondary** is **false**, otherwise it will not
exist. |
| value.tags | List of tags of the snapshot in the format
**key:value** (i.e.: **{"a":"b", "c":"d"}**). |
---
<a
name="proposed-changes--rules-processing-and-variables--network-offering"/>
### NETWORK_OFFERING
| **Variable** | **Description** |
| ------ | ------ |
| value.id | UUID of the network offering. |
| value.name | Name of the network offering. |
| value.tag | Tag of the network offering. |
---
<a name="proposed-changes--rules-processing-and-variables--vm-snapshot"/>
### VM_SNAPSHOT
| **Variable** | **Description** |
| ------ | ------ |
| value.id | UUID of the VM snapshot. |
| value.name | Name of the VM snapshot. |
| value.tags | List of tags of the VM snapshot in the format
**key:value** (i.e.: **{"a":"b", "c":"d"}**). |
| value.vmSnapshotType | Type of the VM snapshot. Values can be: **Disk** or
**DiskAndMemory**. |
---
<a name="proposed-changes--rules-processing-and-variables--other-resources"/>
### Others resources
Others resources will have only the **Default** preset variables.
Others resources:
```sh
IP_ADDRESS
NETWORK_BYTES_SENT
NETWORK_BYTES_RECEIVED
SECURITY_GROUP
LOAD_BALANCER_POLICY
PORT_FORWARDING_RULE
VPN_USERS
CPU_SPEED
vCPU
MEMORY
```
---
<a name="proposed-changes--script-samples"/>
## Script samples
1. Owner (account/domain/project) of the resource (available to **ALL**
resources):
```js
if (account.id == 'b29e84da-ed2e-47dc-9785-49231de8ff07') {
true
} else {
false
}
```
Or just:
```js
account.id == 'b29e84da-ed2e-47dc-9785-49231de8ff07'
```
2. Volume of allocated resource for the owner (available to **ALL**
resources):
```js
value.accountResources.filter(resource =>
resource.domainId == 'b5ea6ffb-fa80-455e-8b38-c9b7e3900cfd'
).length > 20
```
3. Volume of allocated resource for the owner, resulting in the value of
the tariff (available to **ALL** resources)[^3]:
```js
const resourcesLength = value.accountResources.filter(resource =>
resource.domainId == 'b5ea6ffb-fa80-455e-8b38-c9b7e3900cfd'
).length
if (resourcesLength > 40) {
20
} else if (resourcesLength > 10) {
25
} else {
30
}
```
4. Name of the O.S (available to resources **RUNNING_VM** and
**ALLOCATED_VM**):
```js
['Windows 10 (32-bit)',
'Windows 10 (64-bit)',
'Windows 2000 Advanced Server'].includes(value.osName)
```
5. Storage tags (available to resources **VOLUME** and **SNAPSHOT**):
```js
value.storage.tags.includes('SSD')
&& value.storage.tags.includes('NVME')
```
6. Host tags (available to resource **RUNNING_VM**):
```js
value.host.tags.includes('CPU platinum')
```
7. Billing the public IP[^4]. Therefore, if we want to provide one public
IP free of charge to users, we can avoid billing **source
NAT** IPs (available to resource **IP_ADDRESS**):
```js
resourceType !== 'SourceNat'
```
A setting will be created to define the timeout of the scripts. The default
value will be two (2) seconds.
---
<a name="proposed-changes--billing-example"/>
## Billing exampe
The **RUNNING_VM** tariff costs 10 and there are 2 VMs.
The VM **A** belongs to **af7bfdef-2c8f-44a7-9a0e-eb817d6cf821** and has the
name **promo-123-PersonalCloud**.
The VM **B** belongs to **1e4100b8-e28b-4e76-814b-d0d77b27d7a7**, has the
name **CompanyCloud** and the host tag **Best Performance**.
With the current workflow, both VMs would be accounted with the same
tariff/price. With the proposal, operators will be able to create
different tariffs, like:
1. Price: **-1.5**
Rule:
```js
value.name.includes('promo-123-')
```
2. Price: **-1.0**
Rule:
```js
account.id == '1e4100b8-e28b-4e76-814b-d0d77b27d7a7'
```
3. Price: **5.0**
Rule:
```js
value.host.tags.includes('Best Performance')
```
At the end, both VMs will have different costs:
- VM **A**: 10 (**base**) - 1.5 (**rule 1**) = 8.5
- VM **B**: 10 (**base**) - 1.0 (**rule 2**) + 5.0 (**rule 3**) = 14.0
<a name="work-items"/>
Work items
---------------
- Create six (6) new columns in table **cloud_usage.quota_tariff**:
| **Column** | **Nullable** | **Updatable** | **Description** |
| ------ | ------ | ------ | ------ |
| uuid | No | No | To identify the tariff. |
| name | No | No | A name, defined by the user, to the
tariff. This column will be used as common identifier along the tariff updates.
|
| description | Yes | Yes | To describe the tariff. |
| activation_rule | Yes | Yes | To define when the tariff should be
activated. Use **null** if the tariff is always activated. |
| removed | Yes | Yes | To mark tariff as removed. |
| end_date | Yes | Yes | To define the end of the tariff. |
- Migrate tariffs to new paradigm:
- Tariffs before of the current will be marked as **removed** and will
have the **end_date** equal to the **effective_on** of its next;
- Tariffs after of the current will have its value as the difference
between current tariff and its original value;
- Change **QuotaTariffVO**;
- Change API **quotaTariffList**:
- Create parameter **name**, **enddate** and **listall**.
- Change API behavior to, if parameter **listAll** is not informed,
only retrieve not removed tariffs;
- Change API **quotaTariffUpdate**:
- Mark previous tariff as **removed**, create a new one and copy the
identifier (column **name**) to it.
- Create API **quotaTariffCreate**:
- This API will create a new tariff.
- Create API **quotaTariffDelete**:
- This API will mark the tariff as **removed**.
- Create global setting to define the scripts' timeout.
- Change API **quotaUpdate**:
- Validate resources against tariffs' rules, sum the total price and
calculate the usage.
<a name="future-works"/>
Future works
-----------------
This proposal regards to backend and database, therefore, front-end will not
be addressed. In the future, we must change the UI to be compatible with the
APIs.
Other proposals arising from this spec are:
- Bill the Network resource in Quota.
- Bill the VPC resource in Quota.
- Change APIs
[quotaCredits](https://cloudstack.apache.org/api/apidocs-4.16/apis/quotaCredits.html)
and
[quotaBalance](https://cloudstack.apache.org/api/apidocs-4.16/apis/quotaBalance.html)
to allow operators to inform the credits' date and the payment's date or the
processing's date. It is necessary due to how some payment methods work.
- Change resource Tags behavior to indicate when it is an administrative
Tag (created by the operator) or an user Tag.
- Create account/domain setting to granular enabling of Quota.
- Add VM details on **RUNNING_VM**. This item will need a special
attention as we retrieve the values while processing the rating, meaning that
users are able to bypass the tariff by changing the attributes that
activate/deactivate a tariff right before the rating is executed.
- Change the account balance calculation in the APIs **quotaSummary** and
[quotaBalance](https://cloudstack.apache.org/api/apidocs-4.16/apis/quotaBalance.html),
as the current calculations is incorrect and does not represent the balance.
- Improve API
[quotaStatement](https://cloudstack.apache.org/api/apidocs-4.16/apis/quotaStatement.html)
to, when using the parameter **type**, retrieve detailed data of the type
instead of listing the other types with value 0.
[^1]: As the V8 object will be instantiated only once per cycle, operators
must avoid declaring variables with the keywords
**const**, **var** or **let**, otherwise it will throw the error
`Identifier has already been declared` between the iterations. Instead of using
[const a = 1;]{style="background-color: lightgray"} , one must use [a =
1;]{style="background-color: lightgray"} .
[^2]: It will automatically infer the type of the result: If the result is a
number, like **1**, **2.5**, **-3.0** and so on, it will consider the result as
a **number** and will use it as the value of the tariff. Otherwise, it will try
to convert the result to a boolean (**true** or **false**). If the result is
**true**, it will use the tariffs value in the calculation. If the result is
**false** or not a valid boolean, it will not add the tariff value to the
calculation.
[^3]: If the **else** value (in this example, 30) is not provided, the
script's result will be `undefined` and the the tariff won't be applied.
[^4]: Public IPs are bound to VPCs or isolated networks (not user VMs
directly). Every first IP of a VPC or isolated network is a **source NAT**;
additional, if added/allocated by the user, IPs have a null **resourceType**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]