Hi all, This patch series introduces initial support for emulating the Arm SMMUv3 Secure State.
As Pierrick pointed out in a previous discussion [1], full Secure SMMUv3 emulation is a notable missing piece in QEMU. While the FVP model has some support, its limited PCIe capabilities make it challenging for complex use cases. The ability to properly manage device DMA from a secure context is a critical prerequisite for enabling device assignment (passthrough) for confidential computing solutions like Arm CCA and related research such as virtCCA [2]. This series aims to build that foundational support in QEMU. This work is being proposed as an RFC. It introduces a significant amount of new logic, including the core concept of modeling parallel secure and non-secure contexts within a single SMMUv3 device. I am seeking feedback on the overall approach, the core refactoring, and the implementation details before proceeding further. The series begins by implementing the components of the secure programming interface, then progressively refactors the core SMMU logic to handle secure and non-secure contexts in parallel. Secure Interface Implementation: The initial patches add the secure-side registers, implement their read/write logic, and enable the secure command and event queues. This includes the S_INIT mechanism and the new secure TLB invalidation commands. Core Logic Refactoring: The next set of patches makes the core SMMU functions security-state aware. This involves plumbing an is_secure context flag through the main code paths and adding logic to route SMMU-originated memory accesses to the correct (Secure or Non-secure) address space. Cache Isolation: With the core logic now aware of security states, the following patches refactor the configuration and translation lookup caches. The cache keys are modified to include the security context, ensuring that secure and non-secure entries for the same device or address are properly isolated and preventing aliasing. Framework Integration: The final patch connects the SMMU's internal security context to the generic QEMU IOMMU framework by using the iommu_index to represent the architectural SEC_SID. To validate this work, I performed the following tests: Non-Secure Regression: To ensure that existing functionality remains intact, I ran a nested virtualization test. A TCG guest was created on the host, with iommu=smmuv3 and with an emulated PCIe NVMe device assigned. Command line of TCG VM is below: qemu-system-aarch64 \ -machine virt,virtualization=on,gic-version=3,iommu=smmuv3 \ -cpu max -smp 1 -m 4080M \ -accel tcg,thread=single,tb-size=512 \ -kernel Image \ -append 'nokaslr root=/dev/vda rw rootfstype=ext4 iommu.passthrough=on' \ -device pcie-root-port,bus=pcie.0,id=rp0,addr=0x4.0,chassis=1,port=0x10 \ -device pcie-root-port,bus=pcie.0,id=rp1,addr=0x5.0,chassis=2,port=0x11 \ -drive if=none,file=u2204fs.img.qcow2,format=qcow2,id=hd0 \ -device virtio-blk-device,drive=hd0 \ -qmp unix:/tmp/qmp-sock12,server=on,wait=off \ -netdev user,id=eth0,hostfwd=tcp::10022-:22,hostfwd=tcp::59922-:5922 \ -device virtio-net-device,netdev=eth0 \ -drive if=none,file=nvme.img,format=raw,id=nvme0 \ -device nvme,drive=nvme0,serial=deadbeef \ -d unimp,guest_errors -trace events=smmu-events.txt -D qemu.log -nographic Inside this TCG VM, a KVM guest was launched, and the same NVMe device was re-assigned to it via VFIO. Command line of KVM VM inside TCG VM is below: sudo qemu-system-aarch64 \ -enable-kvm -m 1024 -cpu host -M virt \ -machine virt,gic-version=3 \ -cpu max -append "nokaslr" -smp 1 \ -monitor stdio \ -kernel 5.15.Image \ -initrd rootfs.cpio.gz \ -display vnc=:22,id=primary \ -device vfio-pci,host=00:01.0 The KVM guest was able to perform I/O on the device correctly, confirming that the non-secure path is not broken. Secure Register/Command Interface: I set up an OP-TEE + Hafnium environment. Hafnium's smmuv3_driver_init function was used to test the secure register I/O and command queue functionality (excluding translation). As Hafnium assumes larger queue and StreamID sizes than are practical without TTST support, I temporarily patched Hafnium to use smaller values, allowing its driver to initialize the emulated secure SMMU successfully. Secure Translation Path: Since the TF-A SMMUv3 Test Engine does not support QEMU, and no secure device assignment feature exists yet, I created a custom platform device to test the secure translation flow. To trigger the translation logic, I initiated MMIO writes to this device from within Hafnium. The device's MMIO callback handler then performed DMA accesses via its IOMMU region, exercising the secure translation path. While SMMUv3 is typically used for PCIe on physical SoCs, the architecture allows its use with platform devices via a stream-id binding in the device tree. The test harness required some non-standard modifications to decouple the SMMU from its tight integration with PCIe. The code for this test device is available for review at [3]. README.md with detailed instructions is also provided. I've attempted to follow all of the guidance in the "Submitting a Patch" guide, but as this is my first series of this scale, I apologize if I missed anything and welcome all feedback. Thanks, Tang [1] https://lists.nongnu.org/archive/html/qemu-devel/2025-06/msg02940.html [2] https://arxiv.org/abs/2306.11011 [3] https://github.com/hnusdr/qemu Tao Tang (11): hw/arm/smmuv3: Introduce secure registers and commands hw/arm/smmuv3: Implement read/write logic for secure registers hw/arm/smmuv3: Implement S_INIT for secure initialization hw/arm/smmuv3: Enable command processing for the Secure state hw/arm/smmuv3: Support secure event queue and error handling hw/arm/smmuv3: Plumb security state through core functions hw/arm/smmuv3: Add separate address space for secure SMMU accesses hw/arm/smmuv3: Enable secure-side stage 2 TLB invalidations hw/arm/smmuv3: Make the configuration cache security-state aware hw/arm/smmuv3: Differentiate secure TLB entries via keying hw/arm/smmuv3: Use iommu_index to represent SEC_SID hw/arm/smmu-common.c | 74 ++- hw/arm/smmuv3-internal.h | 128 +++++- hw/arm/smmuv3.c | 844 ++++++++++++++++++++++++++++++----- hw/arm/trace-events | 7 +- hw/arm/virt.c | 5 + include/hw/arm/smmu-common.h | 23 +- include/hw/arm/smmuv3.h | 27 ++ 7 files changed, 968 insertions(+), 140 deletions(-) -- 2.34.1
