Re: [vfio-users] [External Email] Re: Direct Assignment of Non-PCI devices to VM
On Wed, Nov 4, 2020 at 1:32 PM Roja Eswaran wrote: > > Sorry for the previous email. Here is the email with Plaintext. > > > -- > Thanks, > Roja Eswaran > PhD Student, Operating Systems and Networks Laboratory > State University of New York, Binghamton > > > -- Forwarded message - > From: Roja Eswaran > Date: Wed, Nov 4, 2020 at 6:31 PM > Subject: Re: [External Email] Re: [vfio-users] Direct Assignment of > Non-PCI devices to VM > To: Micah Morton > Cc: Auger Eric , > > > Mr. Morton, > Thank you so much for your insights. I have one quick question. Are > there any drivers other than VFIO that could be used with QEMU for the > direct assignment? Please let me know if you're aware of any such > drivers. Thanks again! AFAIK no -- the UIO driver is the next closest thing but QEMU doesn't have any support for using it to assign HW devices to guest VMs. > -- > Thanks, > Roja Eswaran > PhD Student, Operating Systems and Networks Laboratory > State University of New York, Binghamton > > > On Wed, Nov 4, 2020 at 3:25 PM Micah Morton wrote: > > > > On Wed, Nov 4, 2020 at 5:30 AM Roja Eswaran wrote: > > > > > > I am using Jetson Xavier NX which integrates a Realtek RTL8211FDI Gigabit > > > Ethernet controller. The on-module Ethernet controller supports > > > 10/100/1000 Gigabit Ethernet IEEE 802.3u Media Access Controller (MAC). > > > I am trying to assign Non-PCI NIC mentioned above directly to the > > > Vanilla VM. As it's not a PCI device, I have no idea how I can achieve > > > this. > > > > > > 1) Is it possible to do a direct assignment on Non-PCI devise to VM using > > > VFIO drivers? > > > > I don't see any mention of an IOMMU in any of these 3 DT files: > > > > https://elixir.bootlin.com/linux/latest/source/arch/arm64/boot/dts/nvidia/tegra194-p2972-.dts > > https://elixir.bootlin.com/linux/latest/source/arch/arm64/boot/dts/nvidia/tegra194-p2888.dtsi > > https://elixir.bootlin.com/linux/latest/source/arch/arm64/boot/dts/nvidia/tegra194.dtsi > > > > Does the Jetson Xavier NX board have an iommu? If not, this is the > > most fundamental reason you wouldn't be able to use VFIO for > > assignment on that board -- especially if the ethernet controller is > > DMA-capable, since you won't be able to ensure the ethernet device's > > DMAs operate on the memory assigned to the guest. > > > > You could check for this by doing "dmesg | grep -i -e iommu -e smmu" > > on the device and seeing if it prints anything, or looking at > > /sys/kernel/iommu_groups to see if that dir exists. > > > > Beyond the IOMMU, vfio-platform/QEMU in their current state wouldn't > > be able to handle the embedded PHY device here: > > https://elixir.bootlin.com/linux/latest/source/arch/arm64/boot/dts/nvidia/tegra194-p2888.dtsi#L43 > > , which would be another hurdle. > > > > > 2) If not, are there any other drivers that would help me achieve the > > > direct assignment? > > > The direct assignment is important because we are trying to reduce the > > > CPU utilization even though virtio drivers give us a very small network > > > latency overhead. Any comments or insights are really appreciated. Thank > > > you so much for your time! > > > > > > > > > -- > > > Thanks, > > > Roja Eswaran > > > Ph.D. Student, Operating Systems and Networks Laboratory > > > State University of New York, Binghamton > > > ___ > > > vfio-users mailing list > > > vfio-users@redhat.com > > > https://www.redhat.com/mailman/listinfo/vfio-users ___ vfio-users mailing list vfio-users@redhat.com https://www.redhat.com/mailman/listinfo/vfio-users
[vfio-users] Fwd: [External Email] Re: Direct Assignment of Non-PCI devices to VM
Sorry for the previous email. Here is the email with Plaintext. -- Thanks, Roja Eswaran PhD Student, Operating Systems and Networks Laboratory State University of New York, Binghamton -- Forwarded message - From: Roja Eswaran Date: Wed, Nov 4, 2020 at 6:31 PM Subject: Re: [External Email] Re: [vfio-users] Direct Assignment of Non-PCI devices to VM To: Micah Morton Cc: Auger Eric , Mr. Morton, Thank you so much for your insights. I have one quick question. Are there any drivers other than VFIO that could be used with QEMU for the direct assignment? Please let me know if you're aware of any such drivers. Thanks again! -- Thanks, Roja Eswaran PhD Student, Operating Systems and Networks Laboratory State University of New York, Binghamton On Wed, Nov 4, 2020 at 3:25 PM Micah Morton wrote: > > On Wed, Nov 4, 2020 at 5:30 AM Roja Eswaran wrote: > > > > I am using Jetson Xavier NX which integrates a Realtek RTL8211FDI Gigabit > > Ethernet controller. The on-module Ethernet controller supports > > 10/100/1000 Gigabit Ethernet IEEE 802.3u Media Access Controller (MAC). > > I am trying to assign Non-PCI NIC mentioned above directly to the Vanilla > > VM. As it's not a PCI device, I have no idea how I can achieve this. > > > > 1) Is it possible to do a direct assignment on Non-PCI devise to VM using > > VFIO drivers? > > I don't see any mention of an IOMMU in any of these 3 DT files: > > https://elixir.bootlin.com/linux/latest/source/arch/arm64/boot/dts/nvidia/tegra194-p2972-.dts > https://elixir.bootlin.com/linux/latest/source/arch/arm64/boot/dts/nvidia/tegra194-p2888.dtsi > https://elixir.bootlin.com/linux/latest/source/arch/arm64/boot/dts/nvidia/tegra194.dtsi > > Does the Jetson Xavier NX board have an iommu? If not, this is the > most fundamental reason you wouldn't be able to use VFIO for > assignment on that board -- especially if the ethernet controller is > DMA-capable, since you won't be able to ensure the ethernet device's > DMAs operate on the memory assigned to the guest. > > You could check for this by doing "dmesg | grep -i -e iommu -e smmu" > on the device and seeing if it prints anything, or looking at > /sys/kernel/iommu_groups to see if that dir exists. > > Beyond the IOMMU, vfio-platform/QEMU in their current state wouldn't > be able to handle the embedded PHY device here: > https://elixir.bootlin.com/linux/latest/source/arch/arm64/boot/dts/nvidia/tegra194-p2888.dtsi#L43 > , which would be another hurdle. > > > 2) If not, are there any other drivers that would help me achieve the > > direct assignment? > > The direct assignment is important because we are trying to reduce the CPU > > utilization even though virtio drivers give us a very small network latency > > overhead. Any comments or insights are really appreciated. Thank you so > > much for your time! > > > > > > -- > > Thanks, > > Roja Eswaran > > Ph.D. Student, Operating Systems and Networks Laboratory > > State University of New York, Binghamton > > ___ > > vfio-users mailing list > > vfio-users@redhat.com > > https://www.redhat.com/mailman/listinfo/vfio-users ___ vfio-users mailing list vfio-users@redhat.com https://www.redhat.com/mailman/listinfo/vfio-users
Re: [vfio-users] [External Email] Re: Direct Assignment of Non-PCI devices to VM
Mr. Morton, Thank you so much for your insights. I have one quick question. Are there any drivers other than VFIO that could be used with QEMU for the direct assignment? Please let me know if you're aware of any such drivers. Thanks again! -- Thanks, Roja Eswaran PhD Student, Operating Systems and Networks Laboratory State University of New York, Binghamton On Wed, Nov 4, 2020 at 3:25 PM Micah Morton wrote: > On Wed, Nov 4, 2020 at 5:30 AM Roja Eswaran > wrote: > > > > I am using Jetson Xavier NX which integrates a Realtek RTL8211FDI > Gigabit Ethernet controller. The on-module Ethernet controller supports > 10/100/1000 Gigabit Ethernet IEEE 802.3u Media Access Controller (MAC). > > I am trying to assign Non-PCI NIC mentioned above directly to the > Vanilla VM. As it's not a PCI device, I have no idea how I can achieve this. > > > > 1) Is it possible to do a direct assignment on Non-PCI devise to VM > using VFIO drivers? > > I don't see any mention of an IOMMU in any of these 3 DT files: > > > https://elixir.bootlin.com/linux/latest/source/arch/arm64/boot/dts/nvidia/tegra194-p2972-.dts > > https://elixir.bootlin.com/linux/latest/source/arch/arm64/boot/dts/nvidia/tegra194-p2888.dtsi > > https://elixir.bootlin.com/linux/latest/source/arch/arm64/boot/dts/nvidia/tegra194.dtsi > > Does the Jetson Xavier NX board have an iommu? If not, this is the > most fundamental reason you wouldn't be able to use VFIO for > assignment on that board -- especially if the ethernet controller is > DMA-capable, since you won't be able to ensure the ethernet device's > DMAs operate on the memory assigned to the guest. > > You could check for this by doing "dmesg | grep -i -e iommu -e smmu" > on the device and seeing if it prints anything, or looking at > /sys/kernel/iommu_groups to see if that dir exists. > > Beyond the IOMMU, vfio-platform/QEMU in their current state wouldn't > be able to handle the embedded PHY device here: > > https://elixir.bootlin.com/linux/latest/source/arch/arm64/boot/dts/nvidia/tegra194-p2888.dtsi#L43 > , which would be another hurdle. > > > 2) If not, are there any other drivers that would help me achieve the > direct assignment? > > The direct assignment is important because we are trying to reduce the > CPU utilization even though virtio drivers give us a very small network > latency overhead. Any comments or insights are really appreciated. Thank > you so much for your time! > > > > > > -- > > Thanks, > > Roja Eswaran > > Ph.D. Student, Operating Systems and Networks Laboratory > > State University of New York, Binghamton > > ___ > > vfio-users mailing list > > vfio-users@redhat.com > > https://www.redhat.com/mailman/listinfo/vfio-users > ___ vfio-users mailing list vfio-users@redhat.com https://www.redhat.com/mailman/listinfo/vfio-users
Re: [vfio-users] Direct Assignment of Non-PCI devices to VM
On Wed, Nov 4, 2020 at 5:30 AM Roja Eswaran wrote: > > I am using Jetson Xavier NX which integrates a Realtek RTL8211FDI Gigabit > Ethernet controller. The on-module Ethernet controller supports 10/100/1000 > Gigabit Ethernet IEEE 802.3u Media Access Controller (MAC). > I am trying to assign Non-PCI NIC mentioned above directly to the Vanilla > VM. As it's not a PCI device, I have no idea how I can achieve this. > > 1) Is it possible to do a direct assignment on Non-PCI devise to VM using > VFIO drivers? I don't see any mention of an IOMMU in any of these 3 DT files: https://elixir.bootlin.com/linux/latest/source/arch/arm64/boot/dts/nvidia/tegra194-p2972-.dts https://elixir.bootlin.com/linux/latest/source/arch/arm64/boot/dts/nvidia/tegra194-p2888.dtsi https://elixir.bootlin.com/linux/latest/source/arch/arm64/boot/dts/nvidia/tegra194.dtsi Does the Jetson Xavier NX board have an iommu? If not, this is the most fundamental reason you wouldn't be able to use VFIO for assignment on that board -- especially if the ethernet controller is DMA-capable, since you won't be able to ensure the ethernet device's DMAs operate on the memory assigned to the guest. You could check for this by doing "dmesg | grep -i -e iommu -e smmu" on the device and seeing if it prints anything, or looking at /sys/kernel/iommu_groups to see if that dir exists. Beyond the IOMMU, vfio-platform/QEMU in their current state wouldn't be able to handle the embedded PHY device here: https://elixir.bootlin.com/linux/latest/source/arch/arm64/boot/dts/nvidia/tegra194-p2888.dtsi#L43 , which would be another hurdle. > 2) If not, are there any other drivers that would help me achieve the direct > assignment? > The direct assignment is important because we are trying to reduce the CPU > utilization even though virtio drivers give us a very small network latency > overhead. Any comments or insights are really appreciated. Thank you so much > for your time! > > > -- > Thanks, > Roja Eswaran > Ph.D. Student, Operating Systems and Networks Laboratory > State University of New York, Binghamton > ___ > vfio-users mailing list > vfio-users@redhat.com > https://www.redhat.com/mailman/listinfo/vfio-users ___ vfio-users mailing list vfio-users@redhat.com https://www.redhat.com/mailman/listinfo/vfio-users
[vfio-users] Direct Assignment of Non-PCI devices to VM
I am using Jetson Xavier NX which integrates a Realtek RTL8211FDI Gigabit Ethernet controller. The on-module Ethernet controller supports 10/100/1000 Gigabit Ethernet IEEE 802.3u Media Access Controller (MAC). I am trying to assign Non-PCI NIC mentioned above directly to the Vanilla VM. As it's not a PCI device, I have no idea how I can achieve this. 1) Is it possible to do a direct assignment on Non-PCI devise to VM using VFIO drivers? 2) If not, are there any other drivers that would help me achieve the direct assignment? The direct assignment is important because we are trying to reduce the CPU utilization even though virtio drivers give us a very small network latency overhead. Any comments or insights are really appreciated. Thank you so much for your time! -- Thanks, Roja Eswaran Ph.D. Student, Operating Systems and Networks Laboratory State University of New York, Binghamton ___ vfio-users mailing list vfio-users@redhat.com https://www.redhat.com/mailman/listinfo/vfio-users
