On 2026/08/17 12:19, Huang, Honglei wrote:
Hi Michael, Alex, Dmitry, Akihiko,
Hi Honglei,
I'm bringing AMD GPU compute ROCm based on virtio. I posted a ROCm over
virtio
implementation to virglrenderer nine months ago (MR !1568 [1]). The ROCm
side has
been supportted by ROCm offical.
Current implementation is a virtio gpu context type capset handled inside
virglrenderer, sharing the display path. That's an awkward fit, many
compute GPUs have no display engine at all.
I think "sharing the display path" conflates several layers and makes
the problem difficult to assess. It would help to identify the concrete
constraint behind "awkward fit."
End-to-end, there are four relevant layers:
1. Host GPU stack: hardware, host kernel, and host userspace
2. Paravirtualization stack: virglrenderer and QEMU
3. Host/guest interface: virtio and the capset-specific command stream
4. Guest GPU stack: guest kernel and guest userspace
Orthogonally, acceleration is separate from display and scanout. A
physical device may provide both, but acceleration does not require a
display engine. Linux likewise exposes render and compute interfaces
separately from modesetting. The userspace interface virglrenderer uses
is messy; there is Vulkan, EGL, OpenGL, and now you are adding ROCm. But
there is one thing I must note is that acceleration and display is
decoupled, and acceleration does not require display.
At the protocol layer, context command buffers are carried by
VIRTIO_GPU_CMD_SUBMIT_3D. Scanout uses separate core virtio-gpu
commands, and VIRTIO_GPU_CMD_GET_DISPLAY_INFO may report no enabled
displays. At the implementation layer, QEMU handles scanout
presentation. virgl_cmd_set_scanout() obtains resource information
through virgl_renderer_resource_get_info() or
virgl_renderer_resource_get_info_ext(). That does not make scanout a
virglrenderer-owned display path.
Therefore, if "sharing the display path" means sharing the same device,
control queue, and QEMU execution context, that identifies a possible
source of contention. If it means that capsets or virglrenderer are
inherently tied to display, I do not think that is accurate. Vulkan
compute is already used through Venus with libkrun [2], and VCL proposes
OpenCL support through virglrenderer [3].
Beyond that, sharing the display path is increasingly painful:
- Compute hammers the queues more than graphics, so sharing
virtio gpu's single control queue with display/virgl causes contention
and display stutter.
All non-cursor commands do share one control queue, but a fence avoids
serialization.
There may still be implementation-level contention, and it is not
necessarily specific to compute. A sufficiently busy graphics workload
could expose the same bottlenecks. Possible contributors in current QEMU
include:
a) qemu_console_hw_gl_block() blocks the entire queue when QEMU only
needs to fence scanout commands.
b) virtio_gpu_virgl_unmap_resource_blob() may also block the entire
queue just to delay one command.
c) QEMU dispatches the control queue and calls into virglrenderer from
its main-loop thread along with display work and many other things.
Venus's render server can offload renderer work, but control-queue
dispatch remains in QEMU's main loop.
In any case, I think you need to do some experiments to track down the
real cause. a) is easy to check: just comment out all
qemu_console_hw_gl_block() calls; it may corrupt display but removes the
blocking. b) can also be tested by leaking the mappings instead of
blocking the whole queue. Using a different display device like qxl
tells whether c) is causing contention.
- Compute contexts need far more blob / shared memory than a display
one.
It is not a problem by itself. Frequent mapping and unmapping might
amplify the second issue above, but that needs to be measured.
- Maybe needs a wider ROCm / compute stack, cause the render model
fits poorly:
rocprofiler (PC sampling, SQTT/SPM, counters, high bandwidth streams)
and ROCgdb (wave control, address watch, async exceptions an
out of band channel that must not block display).
virglrenderer does not impose a particular render model. That's why
Vulkan Compute just works with Venus.
- Events, faults and GPU reset/SMI are async and don't map onto fences.>
- All of this is hard to extend cleanly inside a display capset.
Capset is not about display but determines the protocol of the
VIRTIO_GPU_CMD_SUBMIT_3D command stream. You have described events,
faults and GPU reset/SMI are async don't map onto fences that may be
associated with VIRTIO_GPU_CMD_SUBMIT_3D which is dictated by capset. An
additional feature may be necessary, and it may or may not be dictated
by capset. The other things are irrelevant with the protocol capset
represents; they are either behavioral or about different commands.
On the QEMU/host side, would something like this be OK? One step, two
parts:
- a dedicated headless virtio gpu instance for compute.
A second device would isolate its virtqueues and device-wide
renderer_blocked state. That may be useful if measurements show that
these are the bottlenecks, but it is not yet clear that they are or that
a second device is the appropriate solution.
- that instance served by a separate ROCm backend library loaded
in-process by QEMU.
First, I think we need to establish why ROCm cannot or should not remain
in virglrenderer. The virglrenderer, Venus, and VCL maintainers are
likely better placed to advise on that boundary. Once the protocol
requirements and performance measurements are clear, we can assess the
appropriate QEMU integration.
[2]
https://developers.redhat.com/articles/2025/06/05/how-we-improved-ai-inference-macos-podman-containers
[3]
https://www.qualcomm.com/developer/blog/2024/10/vcl-virtio-gpu-opencl-driver
Regards,
Akihiko Odaki
That reuses the existing pluggable backend model, a second virtio gpu + a
backend library. It doesn't add dedicated queues for debug/profiling
currently.
Waiting for reply and happy to share more detail. Thanks!
[1] https://gitlab.freedesktop.org/virgl/virglrenderer/-/
merge_requests/1568
Regards,
Honglei
