Re: [vfio-users] Direct Assignment of Non-PCI devices to VM
Hi Roja, On 11/4/20 4:27 PM, Roja Eswaran wrote: > I am using Jetson Xavier NX which integrates a Realtek RTL8211FDI > Gigabit Ethernet controller. The on-module Ethernet controller > supports 10/100/1000 Gigabit Ethernet IEEE 802.3u Media Access > Controller (MAC). > I am trying to assign Non-PCI NIC mentioned above directly to the > Vanilla VM. As it's not a PCI device, I have no idea how I can achieve > this. > > 1) Is it possible to do a direct assignment on Non-PCI devise to VM > using VFIO drivers? > 2) If not, are there any other drivers that would help me achieve the > direct assignment? > The direct assignment is important because we are trying to reduce the > CPU utilization even though virtio drivers give us a very small > network latency overhead. Any comments or insights are really > appreciated. Thank you so much for your time! Is the NIC protected by an IOMMU? In the positive you may try to use VFIO platform supported by Linux and QEMU. But this is not as straighforward as PCI. You need to add a kernel reset module for that NIC (rather simple). On QEMU side you need to generate the DT node for that NIC. The feasability depends on the complexity of the host DT node. Thanks Eric > > > -- > Thanks, > Roja Eswaran > Ph.D. Student, Operating Systems and Networks Laboratory > State University of New York, Binghamton ___ vfio-users mailing list vfio-users@redhat.com https://listman.redhat.com/mailman/listinfo/vfio-users
Re: [vfio-users] Direct Assignment of Non-PCI devices to VM
Hi Mica, Roja, On 11/4/20 9:25 PM, Micah Morton wrote: > On Wed, Nov 4, 2020 at 5:30 AM Roja Eswaran wrote: >> >> I am using Jetson Xavier NX which integrates a Realtek RTL8211FDI Gigabit >> Ethernet controller. The on-module Ethernet controller supports 10/100/1000 >> Gigabit Ethernet IEEE 802.3u Media Access Controller (MAC). >> I am trying to assign Non-PCI NIC mentioned above directly to the Vanilla >> VM. As it's not a PCI device, I have no idea how I can achieve this. >> >> 1) Is it possible to do a direct assignment on Non-PCI devise to VM using >> VFIO drivers? > > I don't see any mention of an IOMMU in any of these 3 DT files: > > https://elixir.bootlin.com/linux/latest/source/arch/arm64/boot/dts/nvidia/tegra194-p2972-.dts > https://elixir.bootlin.com/linux/latest/source/arch/arm64/boot/dts/nvidia/tegra194-p2888.dtsi > https://elixir.bootlin.com/linux/latest/source/arch/arm64/boot/dts/nvidia/tegra194.dtsi Jason AGX Xavier seems to have one but I don't know if it is a different SOC. https://forums.developer.nvidia.com/t/pcie-smmu-issue/67027 https://en.miivii.com/uploads/file/20200511/NV_Jetson_Xavier_NX_DataSheet_v0.1.pdf > > Does the Jetson Xavier NX board have an iommu? If not, this is the > most fundamental reason you wouldn't be able to use VFIO for > assignment on that board -- especially if the ethernet controller is > DMA-capable, since you won't be able to ensure the ethernet device's > DMAs operate on the memory assigned to the guest. > > You could check for this by doing "dmesg | grep -i -e iommu -e smmu" > on the device and seeing if it prints anything, or looking at > /sys/kernel/iommu_groups to see if that dir exists. > > Beyond the IOMMU, vfio-platform/QEMU in their current state wouldn't > be able to handle the embedded PHY device here: > https://elixir.bootlin.com/linux/latest/source/arch/arm64/boot/dts/nvidia/tegra194-p2888.dtsi#L43 > , which would be another hurdle. wondering if the phy-mode couldn' be changed or if some interaction with the phy couldn't be managed in the vfio-platform reset driver. As micah said SMMU is a must have for VFIO device assignment. A basic reset module must be written on kernel side and in QEMU, if you use this latter, you must generate a device tree node for the guest. To evaluate the complexity of this work you should look at the dt on the host side and look at the ethernet node. Please look at /proc/device-tree or /sys/firmware/devicetree/ to find out your device node. Thanks Eric > >> 2) If not, are there any other drivers that would help me achieve the direct >> assignment? >> The direct assignment is important because we are trying to reduce the CPU >> utilization even though virtio drivers give us a very small network latency >> overhead. Any comments or insights are really appreciated. Thank you so much >> for your time! >> >> >> -- >> Thanks, >> Roja Eswaran >> Ph.D. Student, Operating Systems and Networks Laboratory >> State University of New York, Binghamton >> ___ >> vfio-users mailing list >> vfio-users@redhat.com >> https://www.redhat.com/mailman/listinfo/vfio-users > ___ vfio-users mailing list vfio-users@redhat.com https://www.redhat.com/mailman/listinfo/vfio-users
Re: [vfio-users] Direct Assignment of Non-PCI devices to VM
On Wed, Nov 4, 2020 at 5:30 AM Roja Eswaran wrote: > > I am using Jetson Xavier NX which integrates a Realtek RTL8211FDI Gigabit > Ethernet controller. The on-module Ethernet controller supports 10/100/1000 > Gigabit Ethernet IEEE 802.3u Media Access Controller (MAC). > I am trying to assign Non-PCI NIC mentioned above directly to the Vanilla > VM. As it's not a PCI device, I have no idea how I can achieve this. > > 1) Is it possible to do a direct assignment on Non-PCI devise to VM using > VFIO drivers? I don't see any mention of an IOMMU in any of these 3 DT files: https://elixir.bootlin.com/linux/latest/source/arch/arm64/boot/dts/nvidia/tegra194-p2972-.dts https://elixir.bootlin.com/linux/latest/source/arch/arm64/boot/dts/nvidia/tegra194-p2888.dtsi https://elixir.bootlin.com/linux/latest/source/arch/arm64/boot/dts/nvidia/tegra194.dtsi Does the Jetson Xavier NX board have an iommu? If not, this is the most fundamental reason you wouldn't be able to use VFIO for assignment on that board -- especially if the ethernet controller is DMA-capable, since you won't be able to ensure the ethernet device's DMAs operate on the memory assigned to the guest. You could check for this by doing "dmesg | grep -i -e iommu -e smmu" on the device and seeing if it prints anything, or looking at /sys/kernel/iommu_groups to see if that dir exists. Beyond the IOMMU, vfio-platform/QEMU in their current state wouldn't be able to handle the embedded PHY device here: https://elixir.bootlin.com/linux/latest/source/arch/arm64/boot/dts/nvidia/tegra194-p2888.dtsi#L43 , which would be another hurdle. > 2) If not, are there any other drivers that would help me achieve the direct > assignment? > The direct assignment is important because we are trying to reduce the CPU > utilization even though virtio drivers give us a very small network latency > overhead. Any comments or insights are really appreciated. Thank you so much > for your time! > > > -- > Thanks, > Roja Eswaran > Ph.D. Student, Operating Systems and Networks Laboratory > State University of New York, Binghamton > ___ > vfio-users mailing list > vfio-users@redhat.com > https://www.redhat.com/mailman/listinfo/vfio-users ___ vfio-users mailing list vfio-users@redhat.com https://www.redhat.com/mailman/listinfo/vfio-users
[vfio-users] Direct Assignment of Non-PCI devices to VM
I am using Jetson Xavier NX which integrates a Realtek RTL8211FDI Gigabit Ethernet controller. The on-module Ethernet controller supports 10/100/1000 Gigabit Ethernet IEEE 802.3u Media Access Controller (MAC). I am trying to assign Non-PCI NIC mentioned above directly to the Vanilla VM. As it's not a PCI device, I have no idea how I can achieve this. 1) Is it possible to do a direct assignment on Non-PCI devise to VM using VFIO drivers? 2) If not, are there any other drivers that would help me achieve the direct assignment? The direct assignment is important because we are trying to reduce the CPU utilization even though virtio drivers give us a very small network latency overhead. Any comments or insights are really appreciated. Thank you so much for your time! -- Thanks, Roja Eswaran Ph.D. Student, Operating Systems and Networks Laboratory State University of New York, Binghamton ___ vfio-users mailing list vfio-users@redhat.com https://www.redhat.com/mailman/listinfo/vfio-users
