On 8/20/26 13:49, Huang, Honglei wrote:
>
>
> On 8/18/2026 7:27 PM, Akihiko Odaki wrote:
>> On 2026/08/18 17:53, Huang, Honglei wrote:
>>>
>>>
>>> On 8/18/2026 3:50 PM, Akihiko Odaki wrote:
>>>> On 2026/08/18 13:26, Huang, Honglei wrote:
>>>>>
>>>>>
>>>>> On 8/18/2026 12:05 PM, Akihiko Odaki wrote:
>>>>>> On 2026/08/18 11:50, Huang, Honglei wrote:
>>>>>>>
>>>>>>>
>>>>>>> On 8/18/2026 12:29 AM, Akihiko Odaki wrote:
>>>>>>>> On 2026/08/17 22:44, Huang, Honglei wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 8/17/2026 7:44 PM, Akihiko Odaki wrote:
>>>>>>>>>> On 2026/08/17 12:19, Huang, Honglei wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi Michael, Alex, Dmitry, Akihiko,
>>>>>>>>>>
>>>>>>>>>> Hi Honglei,
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I'm bringing AMD GPU compute ROCm based on virtio. I posted a
>>>>>>>>>>> ROCm over virtio
>>>>>>>>>>> implementation to virglrenderer nine months ago (MR !1568
>>>>>>>>>>> [1]). The ROCm side has
>>>>>>>>>>> been supportted by ROCm offical.
>>>>>>>>>>>
>>>>>>>>>>> Current implementation is a virtio gpu context type capset
>>>>>>>>>>> handled inside
>>>>>>>>>>> virglrenderer, sharing the display path. That's an awkward
>>>>>>>>>>> fit, many
>>>>>>>>>>> compute GPUs have no display engine at all.
>>>>>>>>>>
>>>>>>>>>> I think "sharing the display path" conflates several layers
>>>>>>>>>> and makes the problem difficult to assess. It would help to
>>>>>>>>>> identify the concrete constraint behind "awkward fit."
>>>>>>>>>>
>>>>>>>>>> End-to-end, there are four relevant layers:
>>>>>>>>>>
>>>>>>>>>> 1. Host GPU stack: hardware, host kernel, and host userspace
>>>>>>>>>> 2. Paravirtualization stack: virglrenderer and QEMU
>>>>>>>>>
>>>>>>>>> Yes we are asking can we add a new file like virtio-gpu
>>>>>>>>> specific for compute, but maybe we can only add a new backend
>>>>>>>>> like virglrenderer specific for compute.
>>>>>>>>>
>>>>>>>>>> 3. Host/guest interface: virtio and the capset-specific
>>>>>>>>>> command stream
>>>>>>>>>
>>>>>>>>> In this plan we may need just add a capset id.
>>>>>>>>>
>>>>>>>>>> 4. Guest GPU stack: guest kernel and guest userspace
>>>>>>>>>
>>>>>>>>> Won't modify the guest kernel in this plan, this email list.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Orthogonally, acceleration is separate from display and
>>>>>>>>>> scanout. A
>>>>>>>>>> physical device may provide both, but acceleration does not
>>>>>>>>>> require a
>>>>>>>>>> display engine. Linux likewise exposes render and compute
>>>>>>>>>> interfaces
>>>>>>>>>> separately from modesetting. The userspace interface
>>>>>>>>>> virglrenderer uses is messy; there is Vulkan, EGL, OpenGL, and
>>>>>>>>>> now you are adding ROCm. But there is one thing I must note is
>>>>>>>>>> that acceleration and display is decoupled, and acceleration
>>>>>>>>>> does not require display.
>>>>>>>>>
>>>>>>>>> Yes totally agreed.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> At the protocol layer, context command buffers are carried by
>>>>>>>>>> VIRTIO_GPU_CMD_SUBMIT_3D. Scanout uses separate core virtio-
>>>>>>>>>> gpu commands, and VIRTIO_GPU_CMD_GET_DISPLAY_INFO may report
>>>>>>>>>> no enabled displays. At the implementation layer, QEMU handles
>>>>>>>>>> scanout presentation. virgl_cmd_set_scanout() obtains resource
>>>>>>>>>> information through virgl_renderer_resource_get_info() or
>>>>>>>>>> virgl_renderer_resource_get_info_ext(). That does not make
>>>>>>>>>> scanout a virglrenderer-owned display path.
>>>>>>>>>
>>>>>>>>> Yes, agreed.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Therefore, if "sharing the display path" means sharing the
>>>>>>>>>> same device, control queue, and QEMU execution context, that
>>>>>>>>>> identifies a possible source of contention. If it means that
>>>>>>>>>> capsets or virglrenderer are inherently tied to display, I do
>>>>>>>>>> not think that is accurate. Vulkan compute is already used
>>>>>>>>>> through Venus with libkrun [2], and VCL proposes OpenCL
>>>>>>>>>> support through virglrenderer [3].
>>>>>>>>>
>>>>>>>>> Yes，but the vulkan is for GFX originally, and for some formal
>>>>>>>>> AI frame work like pytorch, it's support is limited, and it
>>>>>>>>> performance is lower than ROCm, and vulkan also lacks many AI
>>>>>>>>> infrastructure, like composable kernel.
>>>>>>>>> And for virCL, actually it is came from same project with ROCm
>>>>>>>>> native context, but the original author didn't continue to
>>>>>>>>> support it, they handed it over to someone else to take over.
>>>>>>>>> And in the first version of
>>>>>>>>> virCL, it didn't pass the test of actual projects.
>>>>>>>>>
>>>>>>>>> And it seems like virCL didn't upstream into virglrenderer
>>>>>>>>> also, correct me if I am wrong.
>>>>>>>>
>>>>>>>> I cited Venus and VCL only as examples showing that virtio-gpu and
>>>>>>>> virglrenderer are not intrinsically tied to display. I did not
>>>>>>>> suggest
>>>>>>>> either as a substitute for ROCm.
>>>>>>>>
>>>>>>>>>
>>>>>>>>>>> Beyond that, sharing the display path is increasingly painful:
>>>>>>>>>>>
>>>>>>>>>>> - Compute hammers the queues more than graphics, so sharing
>>>>>>>>>>> virtio gpu's single control queue with display/virgl
>>>>>>>>>>> causes contention
>>>>>>>>>>> and display stutter.
>>>>>>>>>>
>>>>>>>>>> All non-cursor commands do share one control queue, but a
>>>>>>>>>> fence avoids serialization.
>>>>>>>>>
>>>>>>>>> yes, agreed.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> There may still be implementation-level contention, and it is not
>>>>>>>>>> necessarily specific to compute. A sufficiently busy graphics
>>>>>>>>>> workload
>>>>>>>>>> could expose the same bottlenecks. Possible contributors in
>>>>>>>>>> current QEMU
>>>>>>>>>> include:
>>>>>>>>>>
>>>>>>>>>> a) qemu_console_hw_gl_block() blocks the entire queue when
>>>>>>>>>> QEMU only
>>>>>>>>>> needs to fence scanout commands.
>>>>>>>>>
>>>>>>>>> Yes, agreed.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> b) virtio_gpu_virgl_unmap_resource_blob() may also block the
>>>>>>>>>> entire
>>>>>>>>>> queue just to delay one command.
>>>>>>>>>
>>>>>>>>> Yes, but it is seems like it is must, someone else in AMD tried
>>>>>>>>> to use async method to relase blob, but it failed to
>>>>>>>>> consistency issue, then
>>>>>>>>> reverted to sync version.
>>>>>>>>
>>>>>>>> Queue-wide suspension is not inherently required. Commit
>>>>>>>> 4eb0aace85f5 ("virtio-gpu: Support mapping hostmem blobs with
>>>>>>>> map_fixed") added a path that avoids per-blob MemoryRegion
>>>>>>>> teardown when virgl_renderer_resource_map_fixed() succeeds. The
>>>>>>>> remaining path is also being improved with:
>>>>>>>>
>>>>>>>> https://lore.kernel.org/qemu-devel/20260424-force_rcu-v4-0-
>>>>>>>> [email protected]/
>>>>>>>> ("[PATCH v4 0/6] virtio-gpu: Force RCU when unmapping blob")
>>>>>>>
>>>>>>> Thanks. force_rcu is a clean fix for the RCU-reclamation part,
>>>>>>> but it still keeps the unmap synchronous and serial.
>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> c) QEMU dispatches the control queue and calls into
>>>>>>>>>> virglrenderer from
>>>>>>>>>> its main-loop thread along with display work and many
>>>>>>>>>> other things.
>>>>>>>>>> Venus's render server can offload renderer work, but
>>>>>>>>>> control- queue
>>>>>>>>>> dispatch remains in QEMU's main loop.
>>>>>>>>>
>>>>>>>>> Yes, we did some async optimization in ROCm context, but its
>>>>>>>>> effectiveness is limited, see bellow.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> In any case, I think you need to do some experiments to track
>>>>>>>>>> down the real cause. a) is easy to check: just comment out all
>>>>>>>>>> qemu_console_hw_gl_block() calls; it may corrupt display but
>>>>>>>>>> removes the blocking. b) can also be tested by leaking the
>>>>>>>>>> mappings instead of blocking the whole queue. Using a
>>>>>>>>>> different display device like qxl tells whether c) is causing
>>>>>>>>>> contention.
>>>>>>>>>
>>>>>>>>> Yes, totally agreed. following is my findings. In short words:
>>>>>>>>>
>>>>>>>>> Optimization can reduce queue pressure, but it can't withstand
>>>>>>>>> absolute overload because each command has some overhead.
>>>>>>>>> Making all commands asynchronous would lead to a debugging hell
>>>>>>>>> about asynchronous issues.
>>>>>>>>> And we have high load applications rocmprofiler that
>>>>>>>>> continuously catch information need virtio queue to handle. But
>>>>>>>>> create a new backend can not solve it simply, we are trying to
>>>>>>>>> find a way. like shmem between guest and host, then use cpu
>>>>>>>>> polling, bypass the virtqueue.
>>>>>>>>
>>>>>>>> Most commands are fast on the CPU side, while heavy processing
>>>>>>>> happens asynchronously on the GPU. Cases (a) and (b) are
>>>>>>>> exceptions.
>>>>>>>>
>>>>>>>>>
>>>>>>>>> The load is mostly memory management. Running an AI model
>>>>>>>>> allocates and frees a large number of blobs. We already did
>>>>>>>>> some optimization release them asynchronously, but the host
>>>>>>>>> processing is a single queue one
>>>>>>>>> process_cmdq, this is where the main bottleneck in my debugging
>>>>>>>>> work / my understanding so far. I'm not certain it's the whole
>>>>>>>>> picture, so please correct if I am wrong.
>>>>>>>>>
>>>>>>>>> A model load or unload frees a large batch of BOs and allocates
>>>>>>>>> another. Some of those commands are async in the virtio-gpu
>>>>>>>>> guest driver, but QEMU still has to work through them on the
>>>>>>>>> one queue, which takes time; so even though any single command
>>>>>>>>> is quick, there are simply too many of them, the single queue
>>>>>>>>> backs up, and everything behind it, gets delayed.
>>>>>>>>>
>>>>>>>>> real work load (a few downstream customisations): loading one
>>>>>>>>> 16 GB model (gemm4 e4b), drives ~1200 blob creates, a burst of
>>>>>>>>> ~1400 resource frees at teardown, ~3700 submits and ~6000
>>>>>>>>> virtqueue notifies, caused a 22 s guest soft lockup. And the
>>>>>>>>> behavior of memory operations are controlled by upper layer
>>>>>>>>> like pytorch / HIP / runtime,
>>>>>>>>> we can not control it.
>>>>>>>>>
>>>>>>>>> To be honest, a separate backend won't fix this. But the real
>>>>>>>>> solution maybe is compute specific. That logic is only useful
>>>>>>>>> to the compute path, and folding it into the shared display
>>>>>>>>> device / renderer would mean churning code that is mature and
>>>>>>>>> stable for graphics, with regression risk. Keeping compute on
>>>>>>>>> its own instance and backend lets us iterate on these compute
>>>>>>>>> only optimisations.
>>>>>>>>
>>>>>>>> A 22-second lockup is too long for those command counts.
>>>>>>>>
>>>>>>>> The most probable explanation I have is that the ROCm
>>>>>>>> integration blocks QEMU's main loop thread while synchronously
>>>>>>>> waiting for GPU execution. Creating separate devices won't
>>>>>>>> resolve this because the main loop thread is shared, and
>>>>>>>> synchronously waiting on the GPU should be avoided in the first
>>>>>>>> place.
>>>>>>>
>>>>>>> No synchronously waiting in ROCm backend, we are using user
>>>>>>> queue, and event waiting, no sync operation in CMD wait. all the
>>>>>>> resource release in ROCm are all async now.
>>>>>>> Only the sync thing is memory thing mapping/unmapping in qemu, as
>>>>>>> long as it remains synchronous, it will be overwhelmed by the
>>>>>>> massive number of requests.
>>>>>>
>>>>>> Mapping and unmapping should not block QEMU's main-loop thread for
>>>>>> that
>>>>>> long. The command counts you reported are relatively small. That
>>>>>> is why
>>>>>> I suspect something else went wrong, such as the main-loop thread
>>>>>> being
>>>>>> inadvertently blocked while waiting for the GPU.
>>>>>
>>>>> Will investigate it.
>>>>>
>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> In any case, profiling is necessary before touching the
>>>>>>>> implementation.
>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> - Compute contexts need far more blob / shared memory than
>>>>>>>>>>> a display one.
>>>>>>>>>>
>>>>>>>>>> It is not a problem by itself. Frequent mapping and unmapping
>>>>>>>>>> might amplify the second issue above, but that needs to be
>>>>>>>>>> measured.
>>>>>>>>>
>>>>>>>>> Yes, agreed. I can give more detailed information.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> - Maybe needs a wider ROCm / compute stack, cause the
>>>>>>>>>>> render model fits poorly:
>>>>>>>>>>> rocprofiler (PC sampling, SQTT/SPM, counters, high
>>>>>>>>>>> bandwidth streams)
>>>>>>>>>>> and ROCgdb (wave control, address watch, async
>>>>>>>>>>> exceptions an
>>>>>>>>>>> out of band channel that must not block display).
>>>>>>>>>>
>>>>>>>>>> virglrenderer does not impose a particular render model.
>>>>>>>>>> That's why Vulkan Compute just works with Venus.
>>>>>>>>>
>>>>>>>>> Yes but vulkan is used for GFX initally. And can not support
>>>>>>>>> many AI application.>
>>>>>>>>>>> - Events, faults and GPU reset/SMI are async and don't map
>>>>>>>>>>> onto fences.> - All of this is hard to extend cleanly
>>>>>>>>>>> inside a display capset.
>>>>>>>>>> Capset is not about display but determines the protocol of the
>>>>>>>>>> VIRTIO_GPU_CMD_SUBMIT_3D command stream. You have described
>>>>>>>>>> events, faults and GPU reset/SMI are async don't map onto
>>>>>>>>>> fences that may be associated with VIRTIO_GPU_CMD_SUBMIT_3D
>>>>>>>>>> which is dictated by capset. An additional feature may be
>>>>>>>>>> necessary, and it may or may not be dictated by capset. The
>>>>>>>>>> other things are irrelevant with the protocol capset
>>>>>>>>>> represents; they are either behavioral or about different
>>>>>>>>>> commands.
>>>>>>>>>
>>>>>>>>> A fence is the one shot, but event is stateful and repeatable.
>>>>>>>>> That may or may not be tied to capset. Agreed.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On the QEMU/host side, would something like this be OK? One
>>>>>>>>>>> step, two parts:
>>>>>>>>>>>
>>>>>>>>>>> - a dedicated headless virtio gpu instance for compute.
>>>>>>>>>>
>>>>>>>>>> A second device would isolate its virtqueues and device-wide
>>>>>>>>>> renderer_blocked state. That may be useful if measurements
>>>>>>>>>> show that
>>>>>>>>>> these are the bottlenecks, but it is not yet clear that they
>>>>>>>>>> are or that
>>>>>>>>>> a second device is the appropriate solution.
>>>>>>>>>>
>>>>>>>>>>> - that instance served by a separate ROCm backend library
>>>>>>>>>>> loaded
>>>>>>>>>>> in-process by QEMU.
>>>>>>>>>>
>>>>>>>>>> First, I think we need to establish why ROCm cannot or should
>>>>>>>>>> not remain
>>>>>>>>>> in virglrenderer. The virglrenderer, Venus, and VCL
>>>>>>>>>> maintainers are
>>>>>>>>>> likely better placed to advise on that boundary. Once the
>>>>>>>>>> protocol
>>>>>>>>>> requirements and performance measurements are clear, we can
>>>>>>>>>> assess the
>>>>>>>>>> appropriate QEMU integration.
>>>>>>>>>
>>>>>>>>> venus is borned for GFX.
>>>>>>>>> virCL not merged.
>>>>>>>>>
>>>>>>>>> To be clear, I'm not saying virglrenderer can't host a ROCm
>>>>>>>>> native context it clearly can. My hesitation is more about fit
>>>>>>>>> and direction: virglrenderer has grown up around GL/graphics,
>>>>>>>>> and I haven't yet found compute oriented plumbing there to
>>>>>>>>> build on, while ROCm moves very fast and I need something I can
>>>>>>>>> keep current with low friction.
>>>>>>>>
>>>>>>>> Whether keeping ROCm in virglrenderer would create extra
>>>>>>>> friction is primarily a question for the virglrenderer
>>>>>>>> maintainers. Its graphics origins do not by themselves motivate
>>>>>>>> adding a separate backend interface to QEMU.
>>>>>>>
>>>>>>> Fair. The first draft version in virglrenderer was in May 2024,
>>>>>>> and ROCm has gone 5.7 → 7.14 in that window.
>>>>>>
>>>>>> One point to note is that virtio-gpu development in QEMU is somewhat
>>>>>> less active. crosvm is the most active user of virglrenderer, and
>>>>>> QEMU
>>>>>> sometimes lags behind it. If you are considering moving the ROCm
>>>>>> integration from virglrenderer to QEMU solely because ROCm evolves
>>>>>> rapidly, I do not think that would be a good idea. A rapidly
>>>>>> evolving component is better kept in virglrenderer unless there is
>>>>>> another reason to place it in QEMU.
>>>>>
>>>>> Actually didn't see something new about compute merged in to
>>>>> virglrenderer this recently 2 years.
>>>>
>>>> Neither QEMU nor virglrenderer has seen new compute-related
>>>> additions in the past two years.
>>>
>>> Maybe that is the reason we need a compute specific path? But I think
>>> the VFIO or vDPA are all can be used for compute, they are really
>>> active. We only need a small file for compute, providing the basic
>>> mechanisms, this code will also benefit other computing devices, such
>>> as NPU, I believe there will be more and more computing devices in
>>> the future.
>>
>> I think this highlights that you are pioneering device-specific
>> paravirtualization here. Venus demonstrates a vendor-neutral approach,
>> but it didn't get compute-specific plumbing simply because Venus
>> didn't need one. Pushing improvements through virglrenderer will help
>> both graphics and compute accelerators.
>>
>>>
>>>
>>>>
>>>> While you have regularly updated the merge request, initiating
>>>> discussions around it is necessary to move review forward. Open-
>>>> source projects like QEMU and virglrenderer need proactive driving
>>>> to complete reviews. Simply shifting the ROCm integration to QEMU
>>>> will not resolve this bottleneck.
>>>
>>> Yes I have actively promoted it, but I haven't received substantial
>>> reviews regarding virtio gpu userptr and virglrenderer. Hard to make
>>> MR move forward without a substantial review. So I am finding a
>>> another way.
>>
>> I don't see recent engagement on the MR. Reaching out publicly on the
>> MR would be a good way to show that you are actively driving it forward.
>>
>>>
>>>>
>>>> Besides, looking at the "Architecture Components" in the
>>>> description, most of them haven't been merged yet. The virglrenderer
>>>> code cannot be merged in its current state, so focusing on those
>>>> dependencies first is essential.
>>>
>>> Yes, I must admit that most of them not merged.
>>> But actually for para virtualization those components are need merged
>>> together because they are closely connected.
>>>
>>>>
>>>> However, taking a naive approach can lead to a chicken-and-egg
>>>> problem: component maintainers want the virglrenderer side
>>>> stabilized first, while virglrenderer maintainers want the component
>>>> side stabilized. To break this deadlock, I suggest seeking consensus
>>>> on the interfaces before completing the implementation. Once an
>>>> interface agreement is reached, changes to each component can land
>>>> independently:
>>>
>>> I've seen amdgpu native context (for GFX) and msm native context
>>> merged quickly. And actually ROCm native context is using the same
>>> method. That's why I think this is a problem of direction.
>>
>> It is understandable that those merged more quickly given the state of
>> the "Architecture Components" mentioned earlier. For both the amdgpu
>> and msm native contexts, the UAPI was already in place, allowing those
>> changes to be reviewed and merged in isolation.
>>
>>>
>>>>
>>>> - virtio interface: I raised a concern regarding the interface [1][2]
>>>> that needs to be addressed.
>>>
>>> I'm open to any feedback from the virtio maintainer, but since it's
>>> really just you and me discussing this, I can immediately modify your
>>> proposal if the maintainer agrees.
>>
>> A pattern sometimes I observe is that maintainers often skip patches
>> with unaddressed discussion items. Updating the code to resolve known
>> feedback often serves as a good trigger to draw in new reviewers.
>>
>>>
>>>> - amdkfd patches: There are interface-level concerns [3] that still
>>>> need
>>>> resolution.
>>>
>>> We have a another solution to solve it, it is already done in
>>> virglrender.
>>>
>>>> - ROCm runtime: The description lists this as "90% complete," but the
>>>> linked pull requests were closed due to inactivity. They need to be
>>>> reopened and seek for a consensus on its interface.
>>>
>>> It was completed using another PR, so it was closed.
>>
>> These are great news. I suggest updating the MR description then so
>> that potential reviewers can see the progress.
>
> Will update the virtio userptr and virglrenderer PR in future. But
> unclear how compute will eventually evolve on virtio. The virgl version
> may only support old ROCm version.
>
> And a single control queue is the design of the virtio GPU, and it is
> indeed a bottleneck, modifying it is difficult. However, some other
> virtio devices have multi queues for different usages.
>
> And actually such as NPUs, TPUs, or custom ASICs, no display compute
> card all go to the virglrenderer seem like not a natural long term
> abstraction.
It's not clear to me if you're still wanting to have current virtio-gpu
ROCm context version landed upstream, or you now looking for a better
alternative?
Virtio-gpu multi-queue support was in discussions over past years,
primary driven by ChromeOS+crosvm needs. Yes, it's not a trivial task to
support multi-queue, no progress was made on it AFAIK.
--
Best regards,
Dmitry
