This series implements the RISC-V Quality-of-Service Identifiers
(Ssqosid) extension [1] which adds the srmcfg register. It also
implements the RISC-V Capacity and Bandwidth Controller QoS Register
Interface (CBQRI) specification [2]. Quality of Service (QoS) in this
context is concerned with shared resources on an SoC such as cache
capacity and memory bandwidth.
Sssqosid srmcfg CSR
-------------------
The srmcfg CSR configures a hart with two identifiers:
- Resource Control ID (RCID)
- Monitoring Counter ID (MCID)
These identifiers accompany each request issued by the hart to shared
resource controllers. This allows the capacity and bandwidth resources
used by a software workload (e.g. a process or a set of processes) to be
controlled and monitored.
CBQRI controllers
-----------------
CBQRI defines operations to configure resource usage limits, in the form
of capacity or bandwidth, for an RCID. CBQRI also defines operations to
configure counters to track the resource utilization per MCID.
This series implements an CBQRI capacity controller and an CBQRI
bandwidth controller which can be configured from the command line:
$ qemu-system-riscv64 -M virt ... \
-device riscv.cbqri.capacity,mmio_base=0x04828000[,...] \
-device riscv.cbqri.bandwidth,mmio_base=0x04829000[,...]
The mmio_base option is mandatory, the others are optional.
As many -device arguments as wanted can be provided as long as their
mmio regions don't conflict.
To see all possible options:
$ qemu-system-riscv64 -device riscv.cbqri.capacity,help
riscv.cbqri.capacity options:
alloc_op_config_limit=<bool> - (default: true)
alloc_op_flush_rcid=<bool> - (default: true)
alloc_op_read_limit=<bool> - (default: true)
at_code=<bool> - (default: true)
at_data=<bool> - (default: true)
cunits=<bool> - on/off (default: on)
max_mcids=<uint16> - (default: 256)
max_rcids=<uint16> - (default: 64)
mmio_base=<uint64> - (default: 0)
mon_evt_id_none=<bool> - (default: true)
mon_evt_id_occupancy=<bool> - (default: true)
mon_op_config_event=<bool> - (default: true)
mon_op_read_counter=<bool> - (default: true)
ncblks=<uint16> - (default: 16)
p=<uint8> - (default: 4)
rpfx=<bool> - on/off (default: on)
target=<str>
$ qemu-system-riscv64 -device riscv.cbqri.bandwidth,help
riscv.cbqri.bandwidth options:
alloc_op_config_limit=<bool> - (default: true)
alloc_op_read_limit=<bool> - (default: true)
at_code=<bool> - (default: true)
at_data=<bool> - (default: true)
max_mcids=<uint16> - (default: 256)
max_rcids=<uint16> - (default: 64)
mmio_base=<uint64> - (default: 0)
mon_evt_id_none=<bool> - (default: true)
mon_evt_id_rdonly_count=<bool> - (default: true)
mon_evt_id_rdwr_count=<bool> - (default: true)
mon_evt_id_wronly_count=<bool> - (default: true)
mon_op_config_event=<bool> - (default: true)
mon_op_read_counter=<bool> - (default: true)
nbwblks=<uint16> - (default: 1024)
p=<uint8> - (default: 4)
rpfx=<bool> - on/off (default: on)
target=<str>
Boolean options correspond to hardware capabilities that can be disabled
Example SoC for CBQRI
---------------------
An example SoC with the following CBQRI controller configuration can be
used to test the implementation:
- L2 cache controllers
- Resource type: Capacity
- Number of capacity blocks (NCBLKS): 12
- In the context of a set-associative cache, the number of
capacity blocks can be thought of as the number of ways
- Number of access types: 2 (code and data)
- Usage monitoring not supported
- Capacity allocation operations: CONFIG_LIMIT, READ_LIMIT
- Last-level cache (LLC) controller
- Resource type: Capacity
- Number of capacity blocks (NCBLKS): 16
- Number of access types: 2 (code and data)
- Usage monitoring operations: CONFIG_EVENT, READ_COUNTER
- Event IDs supported: None, Occupancy
- Capacity allocation ops: CONFIG_LIMIT, READ_LIMIT, FLUSH_RCID
- Memory controllers
- Resource type: Bandwidth
- Number of bandwidth blocks (NBWBLKS): 1024
- Bandwidth blocks do not have a unit but instead represent a
portion of the total bandwidth resource. For NWBLKS of 1024,
each block represents about 0.1% of the bandwidth resource.
- Maximum reserved bandwidth blocks (MRBWB): 819 [80% of NBWBLKS]
- Number of access types: 1 (no code/data differentiation)
- Usage monitoring operations: CONFIG_EVENT, READ_COUNTER
- Event IDs supported: None, Total read/write byte count, Total
read byte count, Total write byte count
- Bandwidth allocation operations: CONFIG_LIMIT, READ_LIMIT
The memory map used for the example SoC:
Base addr Size
0x4820000 4KB Cluster 0 L2 cache controller
0x4821000 4KB Cluster 1 L2 cache controller
0x4828000 4KB Memory controller 0
0x4829000 4KB Memory controller 1
0x482a000 4KB Memory controller 2
0x482b000 4KB Shared LLC cache controller
This configuration is meant to provide a "concrete" example for software
(like Linux) to test against. It represents just one of many possible
ways for hardware to implement the CBQRI spec.
The example SoC configuration is created with the following:
qemu-system-riscv64 \
-M virt \
-nographic \
-smp 8 \
-device
riscv.cbqri.capacity,max_mcids=256,max_rcids=64,ncblks=12,alloc_op_flush_rcid=false,mon_op_config_event=false,mon_op_read_counter=false,mon_evt_id_none=false,mon_evt_id_occupancy=false,mmio_base=0x04820000
\
-device
riscv.cbqri.capacity,max_mcids=256,max_rcids=64,ncblks=12,alloc_op_flush_rcid=false,mon_op_config_event=false,mon_op_read_counter=false,mon_evt_id_none=false,mon_evt_id_occupancy=false,mmio_base=0x04821000
\
-device
riscv.cbqri.capacity,max_mcids=256,max_rcids=64,ncblks=16,mmio_base=0x0482B000 \
-device
riscv.cbqri.bandwidth,max_mcids=256,max_rcids=64,nbwblks=1024,mrbwb=819,mmio_base=0x04828000
\
-device
riscv.cbqri.bandwidth,max_mcids=256,max_rcids=64,nbwblks=1024,mrbwb=819,mmio_base=0x04829000
\
-device
riscv.cbqri.bandwidth,max_mcids=256,max_rcids=64,nbwblks=1024,mrbwb=819,mmio_base=0x0482a000
In addition, please note that this series only implements the register
interface that CBQRI specifies. It does not attempt to emulate the
performance impact of configuring limits on shared resources like cache
and memory bandwidth. Similarly, the code does not attempt to emulate
cache and memory bandwidth utilization, like what would be observed on a
real hardware system implementing CBQRI.
There is a branch of Linux [3] which adds support for Ssqosid and CBQRI
along with resctrl integration. I still need to do some more cleanup of
the code and intend to post the Linux patch series soon.
Changes since v2
----------------
- Rebase on master which is currently at version v10.1.50
- Add fields that were not in the draft used for the proof of concept
and introduced in final CBQRI 1.0 spec: capacity units (cunits) and
RCID-prefixed mode (RPFX) along with parameter P (prefixed bits)
- Fix check_srmcfg() to check if virtualization is enabled, and if so,
return virt instruction fault, otherwise return smode()
- Fix indentation in read_srmcfg()
- Add SPDX headers
- Add MAINTAINERS enteries
- Link to v2:
https://lore.kernel.org/qemu-devel/[email protected]/
Changes since v1
----------------
- Rebase on current master (8.0.50) instead of 8.0.0-rc4
- Configure CBQRI controllers based on device properties in command line
arguments instead of a fixed example SoC configuration.
- Move TYPE_RISCV_CBQRI_BC and TYPE_RISCV_CBQRI_CC into header so that
machines may use it (suggested by Alistair).
- Change 'select RISC_CBQRI' to 'imply RISCV_CBQRI' for RISCV_VIRT.
- Patches 8/9 could be dropped as they are not needed for the CBQRI
proof-of-concept to work. They are only meant to serve as an example
for those implementing new machines.
- Link to v1:
https://lore.kernel.org/qemu-devel/[email protected]/
[1] https://github.com/riscv/riscv-ssqosid/releases/tag/v1.0
[2] https://github.com/riscv-non-isa/riscv-cbqri/releases/tag/v1.0
[3]
https://git.kernel.org/pub/scm/linux/kernel/git/fustini/linux.git/log/?h=b4/ssqosid-cbqri
Signed-off-by: Drew Fustini <[email protected]>
---
Kornel Dulęba (1):
riscv: implement Ssqosid extension and srmcfg CSR
Nicolas Pitre (6):
hw/riscv: define capabilities of CBQRI controllers
hw/riscv: implement CBQRI capacity controller
hw/riscv: implement CBQRI bandwidth controller
hw/riscv: Kconfig: add CBQRI options
hw/riscv: meson: add CBQRI controllers to the build
hw/riscv: add CBQRI controllers to virt machine
MAINTAINERS | 9 +
disas/riscv.c | 1 +
hw/riscv/Kconfig | 4 +
hw/riscv/cbqri_bandwidth.c | 613 ++++++++++++++++++++++++++++++++++++
hw/riscv/cbqri_capacity.c | 634 ++++++++++++++++++++++++++++++++++++++
hw/riscv/meson.build | 1 +
hw/riscv/virt.c | 3 +
include/hw/riscv/cbqri.h | 89 ++++++
target/riscv/cpu.c | 2 +
target/riscv/cpu.h | 3 +
target/riscv/cpu_bits.h | 8 +
target/riscv/cpu_cfg_fields.h.inc | 1 +
target/riscv/csr.c | 34 ++
13 files changed, 1402 insertions(+)
---
base-commit: e88510fcdc13380bd4895a17d6f8a0b3a3325b85
change-id: 20251114-riscv-ssqosid-cbqri-8fd61be020d5
Best regards,
--
Drew Fustini <[email protected]>
