weizhouapache commented on issue #14302:
URL: https://github.com/apache/cloudstack/issues/14302#issuecomment-6315340968
I tried to reproduce this on KVM and VMware (4.23.0, a nested lab, not
4.22.0 itself), and the numbers are explained by what the column actually
contains.
## What "Disk IOPS" is on KVM
`diskiopstotal` is `diskioread + diskiowrite`. Those are the libvirt
`rd_req` / `wr_req` counters of the disk, i.e. the **number of I/O requests
since the Instance was started**, not a number per second. It never goes down
while the VM runs. A disk doing 5,000 requests/s for ~18.5 hours reads 333
million, which matches what is reported here. The value says nothing about
whether an IOPS limit is applied.
KVM, 600 s of a fixed-rate fio load (about 140 read + 60 write requests/s)
on a data disk of each storage type:
| Volume | Storage | `diskioread` | `diskiowrite` | "Disk IOPS" | libvirt
`rd_req + wr_req` |
|---|---|---|---|---|---|
| nfsdisk | NFS | 84,129 | 36,000 | 120,129 | 120,129 |
| cephdisk | RBD | 84,128 | 36,000 | 120,128 | 120,128 |
After 10 minutes idle, then exactly 10,240 more 4 KiB writes, the NFS volume
went from 240,215 to 250,541 (+10,326: the 10,240 writes plus 86 small reads of
the guest). `diskkbsread` / `diskkbswrite` behave the same way (totals in KiB).
NFS and Ceph volumes give the same kind of values; the storage type makes no
difference.
## Is the IOPS limit applied?
Only the **hypervisor QoS** of the offering is: `iopsreadrate` /
`iopswriterate` (the "QoS type: hypervisor" fields) become `<read_iops_sec>` /
`<write_iops_sec>` in the libvirt XML. With 300/300 the disk stayed at ~300
reads/s and ~301 writes/s in fio (fio-measured, on RBD).
The offering's **Min/Max IOPS** (`miniops` / `maxiops`) are not applied on
KVM: a disk with only Min IOPS 100 / Max IOPS 300 had no `iotune` in the
libvirt XML and ran at 4,500 (mixed) to 7,000 (read) IOPS. If that is what the
offering uses here, nothing throttles the disk, which could be the cause of the
Ceph latency, independently of the displayed number.
@tatay188 could you check:
1. whether the numbers come from the volume list / volumes metrics
(`listVolumesMetrics`) or from the Instance list,
2. which fields are set on the disk offering
(`iopsreadrate`/`iopswriterate`, or only Min/Max IOPS),
3. `virsh dumpxml <instance>` for the disk: is there an `<iotune>` section?
## VMware (ESXi 8.0.3), for comparison
The VMware values are **not** cumulative. The agent reads vCenter's
`numberReadAveraged` / `numberWriteAveraged`, which are averages per second
over the sample window (`vmware.stats.time.window`, 300 s). Guest `dd` load for
10 minutes (~90 writes/s, ~45 reads/s; the guest did 53,900 writes and 26,950
reads in total):
| Collection | `diskioread` | `diskiowrite` | "Disk IOPS" |
|---|---|---|---|
| during the load (first sample) | 39 | 79 | 118 |
| during the load | 48 | 96 | 144 |
| load ended | 3 | 7 | 10 |
So on VMware the column is already a rate. The server still handles it as if
it were a counter: an idle disk is skipped ("IO/bytes read and write are all 0.
Not updating vm_disk_statistics"), so the last non-zero value stays; and a
lower value is treated as a counter reset, adding the previous value to
`net_io_read/write` (48 / 96 in my test). That is a separate problem.
## Suggestions
- On KVM, the column should not be called IOPS. I will propose relabelling /
renaming it (`diskiopstotal` → `diskiototal`, "Disk IO (requests)"), or
alternatively computing a real rate from two collections.
- Separate bug found on the way: on KVM, volumes on **local** storage
(`/var/lib/libvirt/images`) always showed 0, because the disk is matched with
the wrong part of its path. A fix is ready.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
