Re: Query on VFIO in Virtual machine
On Thu, Jun 22, 2017 at 11:27:09AM -0600, Alex Williamson wrote: > On Thu, 22 Jun 2017 22:42:19 +0530 > Nitin Saxenawrote: > > > Thanks Alex. > > > > >> Without an iommu in the VM, you'd be limited to no-iommu support for VM > > >> userspace, > > So are you trying to say VFIO NO-IOMMU should work inside VM. Does > > that mean VFIO NO-IOMMU in VM and VFIO IOMMU in host for same device > > is a legitimate configuration? I did tried this configuration and the > > application (in VM) seems to get container_fd, group_fd, device_fd > > successfully but after VFIO_DEVICE_RESET ioctl the PCI link breaks > > from VM as well as from host. This could be specific to PCI endpoint > > device which I can dig. > > > > I will be happy if VFIO NO-IOMMU in VM and IOMMU in host for same > > device is legitimate configuration. > > Using no-iommu in the guest should work in that configuration, however > there's no isolation from the user to the rest of VM memory, so the VM > kernel will be tainted. Host memory does have iommu isolation. Device > reset from VM userspace sounds like another bug to investigate. Thanks, > > Alex Besides what Alex has mentioned, there is a wiki page for the usage. The command line will be slightly different on QEMU side comparing to without vIOMMU: http://wiki.qemu.org/Features/VT-d#With_Assigned_Devices One more thing to mention is that, when vfio-pci devices in the guest are used with emulated VT-d, huge performance degradation will be expected for dynamic allocations at least for now. While for mostly static allocations (like DPDK) the performance should be merely the same as no-IOMMU mode. It's just a hint on performance, and I believe for your own case it should mostly depend on how the application is managing DMA map/unmaps. Thanks, -- Peter Xu
Re: Query on VFIO in Virtual machine
On Thu, Jun 22, 2017 at 11:27:09AM -0600, Alex Williamson wrote: > On Thu, 22 Jun 2017 22:42:19 +0530 > Nitin Saxena wrote: > > > Thanks Alex. > > > > >> Without an iommu in the VM, you'd be limited to no-iommu support for VM > > >> userspace, > > So are you trying to say VFIO NO-IOMMU should work inside VM. Does > > that mean VFIO NO-IOMMU in VM and VFIO IOMMU in host for same device > > is a legitimate configuration? I did tried this configuration and the > > application (in VM) seems to get container_fd, group_fd, device_fd > > successfully but after VFIO_DEVICE_RESET ioctl the PCI link breaks > > from VM as well as from host. This could be specific to PCI endpoint > > device which I can dig. > > > > I will be happy if VFIO NO-IOMMU in VM and IOMMU in host for same > > device is legitimate configuration. > > Using no-iommu in the guest should work in that configuration, however > there's no isolation from the user to the rest of VM memory, so the VM > kernel will be tainted. Host memory does have iommu isolation. Device > reset from VM userspace sounds like another bug to investigate. Thanks, > > Alex Besides what Alex has mentioned, there is a wiki page for the usage. The command line will be slightly different on QEMU side comparing to without vIOMMU: http://wiki.qemu.org/Features/VT-d#With_Assigned_Devices One more thing to mention is that, when vfio-pci devices in the guest are used with emulated VT-d, huge performance degradation will be expected for dynamic allocations at least for now. While for mostly static allocations (like DPDK) the performance should be merely the same as no-IOMMU mode. It's just a hint on performance, and I believe for your own case it should mostly depend on how the application is managing DMA map/unmaps. Thanks, -- Peter Xu
Re: Query on VFIO in Virtual machine
On Thu, 22 Jun 2017 22:42:19 +0530 Nitin Saxenawrote: > Thanks Alex. > > >> Without an iommu in the VM, you'd be limited to no-iommu support for VM > >> userspace, > So are you trying to say VFIO NO-IOMMU should work inside VM. Does > that mean VFIO NO-IOMMU in VM and VFIO IOMMU in host for same device > is a legitimate configuration? I did tried this configuration and the > application (in VM) seems to get container_fd, group_fd, device_fd > successfully but after VFIO_DEVICE_RESET ioctl the PCI link breaks > from VM as well as from host. This could be specific to PCI endpoint > device which I can dig. > > I will be happy if VFIO NO-IOMMU in VM and IOMMU in host for same > device is legitimate configuration. Using no-iommu in the guest should work in that configuration, however there's no isolation from the user to the rest of VM memory, so the VM kernel will be tainted. Host memory does have iommu isolation. Device reset from VM userspace sounds like another bug to investigate. Thanks, Alex
Re: Query on VFIO in Virtual machine
On Thu, 22 Jun 2017 22:42:19 +0530 Nitin Saxena wrote: > Thanks Alex. > > >> Without an iommu in the VM, you'd be limited to no-iommu support for VM > >> userspace, > So are you trying to say VFIO NO-IOMMU should work inside VM. Does > that mean VFIO NO-IOMMU in VM and VFIO IOMMU in host for same device > is a legitimate configuration? I did tried this configuration and the > application (in VM) seems to get container_fd, group_fd, device_fd > successfully but after VFIO_DEVICE_RESET ioctl the PCI link breaks > from VM as well as from host. This could be specific to PCI endpoint > device which I can dig. > > I will be happy if VFIO NO-IOMMU in VM and IOMMU in host for same > device is legitimate configuration. Using no-iommu in the guest should work in that configuration, however there's no isolation from the user to the rest of VM memory, so the VM kernel will be tainted. Host memory does have iommu isolation. Device reset from VM userspace sounds like another bug to investigate. Thanks, Alex
Re: Query on VFIO in Virtual machine
Thanks Alex. >> Without an iommu in the VM, you'd be limited to no-iommu support for VM >> userspace, So are you trying to say VFIO NO-IOMMU should work inside VM. Does that mean VFIO NO-IOMMU in VM and VFIO IOMMU in host for same device is a legitimate configuration? I did tried this configuration and the application (in VM) seems to get container_fd, group_fd, device_fd successfully but after VFIO_DEVICE_RESET ioctl the PCI link breaks from VM as well as from host. This could be specific to PCI endpoint device which I can dig. I will be happy if VFIO NO-IOMMU in VM and IOMMU in host for same device is legitimate configuration. Thanks Nitin On Thu, Jun 22, 2017 at 10:29 PM, Alex Williamsonwrote: > [cc +qemu-devel, +peterx] > > On Thu, 22 Jun 2017 22:18:06 +0530 > Nitin Saxena wrote: > >> Hi, >> >> I have a PCI device connected as an endpoint to Intel host machine. >> The requirement is to run dpdk like user space data path application >> in VM using PCI PF passthrough (SRIOV disabled). This application >> works fine on host kernel and uses VFIO to get MSIX interrupts from >> PCI device. We are trying to run this existing application in VM using >> PCI passthrough. This application has capability to use >> VFIO_IOMMU_TYPE1 as wells as VFIO_NOIOMMU. >> >> On Intel host machine VT-d has been enabled and using virt-manager PCI >> device PF is assigned to the VM. This makes virt-manager to implicitly >> binds PCI device PF to vfio with vfio_iommu_type1. The VM LINUX kernel >> was booted with intel_iommu=on as boot parameter. >> >> My question: Is it possible that vfio can coexist in host (by >> virt-manager) as well as VM (by application)? If yes, does application >> running inside VM needs to configure VFIO with iommu_type=IOMMU or >> iommu_type=no-iommu. >> >> In VM I tried inserting vfio_iommu_type1.ko kernel module which failed >> with "No such device error". Thats why I am confused whether my >> requirement is legitimate or not. What could be the best solution? > > This is really more of a QEMU question. In order to use > vfio_iommu_type1 in the guest, you need an iommu in the guest. The > most recent release of QEMU supports this with an emulated VT-d > device. Therefore if you create a VM with emulated VT-d and a device > assigned through vfio-pci, you can expose it to userspace in the VM with > physical iommu protection. Without an iommu in the VM, you'd be > limited to no-iommu support for VM userspace, the physical iommu would > only protect the device to the extent of VM memory, no to specific > userspace mappings within the VM. Thanks, > > Alex
Re: Query on VFIO in Virtual machine
Thanks Alex. >> Without an iommu in the VM, you'd be limited to no-iommu support for VM >> userspace, So are you trying to say VFIO NO-IOMMU should work inside VM. Does that mean VFIO NO-IOMMU in VM and VFIO IOMMU in host for same device is a legitimate configuration? I did tried this configuration and the application (in VM) seems to get container_fd, group_fd, device_fd successfully but after VFIO_DEVICE_RESET ioctl the PCI link breaks from VM as well as from host. This could be specific to PCI endpoint device which I can dig. I will be happy if VFIO NO-IOMMU in VM and IOMMU in host for same device is legitimate configuration. Thanks Nitin On Thu, Jun 22, 2017 at 10:29 PM, Alex Williamson wrote: > [cc +qemu-devel, +peterx] > > On Thu, 22 Jun 2017 22:18:06 +0530 > Nitin Saxena wrote: > >> Hi, >> >> I have a PCI device connected as an endpoint to Intel host machine. >> The requirement is to run dpdk like user space data path application >> in VM using PCI PF passthrough (SRIOV disabled). This application >> works fine on host kernel and uses VFIO to get MSIX interrupts from >> PCI device. We are trying to run this existing application in VM using >> PCI passthrough. This application has capability to use >> VFIO_IOMMU_TYPE1 as wells as VFIO_NOIOMMU. >> >> On Intel host machine VT-d has been enabled and using virt-manager PCI >> device PF is assigned to the VM. This makes virt-manager to implicitly >> binds PCI device PF to vfio with vfio_iommu_type1. The VM LINUX kernel >> was booted with intel_iommu=on as boot parameter. >> >> My question: Is it possible that vfio can coexist in host (by >> virt-manager) as well as VM (by application)? If yes, does application >> running inside VM needs to configure VFIO with iommu_type=IOMMU or >> iommu_type=no-iommu. >> >> In VM I tried inserting vfio_iommu_type1.ko kernel module which failed >> with "No such device error". Thats why I am confused whether my >> requirement is legitimate or not. What could be the best solution? > > This is really more of a QEMU question. In order to use > vfio_iommu_type1 in the guest, you need an iommu in the guest. The > most recent release of QEMU supports this with an emulated VT-d > device. Therefore if you create a VM with emulated VT-d and a device > assigned through vfio-pci, you can expose it to userspace in the VM with > physical iommu protection. Without an iommu in the VM, you'd be > limited to no-iommu support for VM userspace, the physical iommu would > only protect the device to the extent of VM memory, no to specific > userspace mappings within the VM. Thanks, > > Alex
Re: Query on VFIO in Virtual machine
[cc +qemu-devel, +peterx] On Thu, 22 Jun 2017 22:18:06 +0530 Nitin Saxenawrote: > Hi, > > I have a PCI device connected as an endpoint to Intel host machine. > The requirement is to run dpdk like user space data path application > in VM using PCI PF passthrough (SRIOV disabled). This application > works fine on host kernel and uses VFIO to get MSIX interrupts from > PCI device. We are trying to run this existing application in VM using > PCI passthrough. This application has capability to use > VFIO_IOMMU_TYPE1 as wells as VFIO_NOIOMMU. > > On Intel host machine VT-d has been enabled and using virt-manager PCI > device PF is assigned to the VM. This makes virt-manager to implicitly > binds PCI device PF to vfio with vfio_iommu_type1. The VM LINUX kernel > was booted with intel_iommu=on as boot parameter. > > My question: Is it possible that vfio can coexist in host (by > virt-manager) as well as VM (by application)? If yes, does application > running inside VM needs to configure VFIO with iommu_type=IOMMU or > iommu_type=no-iommu. > > In VM I tried inserting vfio_iommu_type1.ko kernel module which failed > with "No such device error". Thats why I am confused whether my > requirement is legitimate or not. What could be the best solution? This is really more of a QEMU question. In order to use vfio_iommu_type1 in the guest, you need an iommu in the guest. The most recent release of QEMU supports this with an emulated VT-d device. Therefore if you create a VM with emulated VT-d and a device assigned through vfio-pci, you can expose it to userspace in the VM with physical iommu protection. Without an iommu in the VM, you'd be limited to no-iommu support for VM userspace, the physical iommu would only protect the device to the extent of VM memory, no to specific userspace mappings within the VM. Thanks, Alex
Re: Query on VFIO in Virtual machine
[cc +qemu-devel, +peterx] On Thu, 22 Jun 2017 22:18:06 +0530 Nitin Saxena wrote: > Hi, > > I have a PCI device connected as an endpoint to Intel host machine. > The requirement is to run dpdk like user space data path application > in VM using PCI PF passthrough (SRIOV disabled). This application > works fine on host kernel and uses VFIO to get MSIX interrupts from > PCI device. We are trying to run this existing application in VM using > PCI passthrough. This application has capability to use > VFIO_IOMMU_TYPE1 as wells as VFIO_NOIOMMU. > > On Intel host machine VT-d has been enabled and using virt-manager PCI > device PF is assigned to the VM. This makes virt-manager to implicitly > binds PCI device PF to vfio with vfio_iommu_type1. The VM LINUX kernel > was booted with intel_iommu=on as boot parameter. > > My question: Is it possible that vfio can coexist in host (by > virt-manager) as well as VM (by application)? If yes, does application > running inside VM needs to configure VFIO with iommu_type=IOMMU or > iommu_type=no-iommu. > > In VM I tried inserting vfio_iommu_type1.ko kernel module which failed > with "No such device error". Thats why I am confused whether my > requirement is legitimate or not. What could be the best solution? This is really more of a QEMU question. In order to use vfio_iommu_type1 in the guest, you need an iommu in the guest. The most recent release of QEMU supports this with an emulated VT-d device. Therefore if you create a VM with emulated VT-d and a device assigned through vfio-pci, you can expose it to userspace in the VM with physical iommu protection. Without an iommu in the VM, you'd be limited to no-iommu support for VM userspace, the physical iommu would only protect the device to the extent of VM memory, no to specific userspace mappings within the VM. Thanks, Alex
Query on VFIO in Virtual machine
Hi, I have a PCI device connected as an endpoint to Intel host machine. The requirement is to run dpdk like user space data path application in VM using PCI PF passthrough (SRIOV disabled). This application works fine on host kernel and uses VFIO to get MSIX interrupts from PCI device. We are trying to run this existing application in VM using PCI passthrough. This application has capability to use VFIO_IOMMU_TYPE1 as wells as VFIO_NOIOMMU. On Intel host machine VT-d has been enabled and using virt-manager PCI device PF is assigned to the VM. This makes virt-manager to implicitly binds PCI device PF to vfio with vfio_iommu_type1. The VM LINUX kernel was booted with intel_iommu=on as boot parameter. My question: Is it possible that vfio can coexist in host (by virt-manager) as well as VM (by application)? If yes, does application running inside VM needs to configure VFIO with iommu_type=IOMMU or iommu_type=no-iommu. In VM I tried inserting vfio_iommu_type1.ko kernel module which failed with "No such device error". Thats why I am confused whether my requirement is legitimate or not. What could be the best solution? Thanks in advance Nitin
Query on VFIO in Virtual machine
Hi, I have a PCI device connected as an endpoint to Intel host machine. The requirement is to run dpdk like user space data path application in VM using PCI PF passthrough (SRIOV disabled). This application works fine on host kernel and uses VFIO to get MSIX interrupts from PCI device. We are trying to run this existing application in VM using PCI passthrough. This application has capability to use VFIO_IOMMU_TYPE1 as wells as VFIO_NOIOMMU. On Intel host machine VT-d has been enabled and using virt-manager PCI device PF is assigned to the VM. This makes virt-manager to implicitly binds PCI device PF to vfio with vfio_iommu_type1. The VM LINUX kernel was booted with intel_iommu=on as boot parameter. My question: Is it possible that vfio can coexist in host (by virt-manager) as well as VM (by application)? If yes, does application running inside VM needs to configure VFIO with iommu_type=IOMMU or iommu_type=no-iommu. In VM I tried inserting vfio_iommu_type1.ko kernel module which failed with "No such device error". Thats why I am confused whether my requirement is legitimate or not. What could be the best solution? Thanks in advance Nitin