On Fri, Feb 2, 2018 at 12:21 PM, Ravi Kerur <rke...@gmail.com> wrote:
> > > On Fri, Feb 2, 2018 at 2:28 AM, Burakov, Anatoly < > anatoly.bura...@intel.com> wrote: > >> On 01-Feb-18 7:26 PM, Ravi Kerur wrote: >> >>> >>> >>> On Thu, Feb 1, 2018 at 2:10 AM, Burakov, Anatoly < >>> anatoly.bura...@intel.com <mailto:anatoly.bura...@intel.com>> wrote: >>> >>> On 31-Jan-18 9:51 PM, Ravi Kerur wrote: >>> >>> >>> Hi Anatoly, >>> >>> Thanks. I am following wiki link below which uses vIOMMU with >>> DPDK as a use-case and instantiate VM as specified with Q35 >>> chipset in Qemu. >>> >>> https://wiki.qemu.org/Features/VT-d >>> <https://wiki.qemu.org/Features/VT-d> >>> >>> Qemu-version is 2.11 >>> Host kernel 4.9 >>> Guest kernel 4.4 >>> >>> I can only guess that guest kernel needs an upgrade in my setup >>> to work correctly, if versions on my setup rings a bell on not >>> having support kindly let me know. >>> >>> When 'modprobe vfio enable_unsafe_noiommu_node=Y' is executed on >>> guest I get following error >>> ... >>> vfio: unknown parameter 'enable_unsafe_noiommu_node' ignored >>> ... >>> >>> in guest. >>> >>> Thanks. >>> >>> >>> AFAIK kernel 4.4 should have noiommu mode - it was introduced in >>> 3.1x days. However, in order for that to work, kernel also has to be >>> built with this mode enabled. My guess is, whoever is the supplier >>> of your kernel, did not do that. You should double-check the kernel >>> configuration of your distribution. >>> >>> However, if you have vIOMMU in QEMU, you shouldn't need noiommu mode >>> - "regular" vfio should work fine. noiommu mode should only be >>> needed if you know you don't have IOMMU enabled in your kernel, and >>> even if you can't enable it, you can still use igb_uio. >>> >>> Hi Anatoly, >>> >>> Do you suggest I take this discussion to kvm/qemu mailing list as I am >>> not sure which component has the issue? I check dmesg for BIOS physical >>> memory map and address reported as fault by DMAR is reported by BIOS as >>> usable on both host and vm. >>> >>> [ 4539.597737] DMAR: [DMA Read] Request device [04:10.0] fault addr >>> *33a128000 *[fault reason 06] PTE Read access is not set >>> >>> dmesg | grep BIOS >>> [ 0.000000] e820: BIOS-provided physical RAM map: >>> [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009afff] >>> usable >>> [ 0.000000] BIOS-e820: [mem 0x000000000009b000-0x000000000009ffff] >>> reserved >>> [ 0.000000] BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] >>> reserved >>> [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000007938afff] >>> usable >>> [ 0.000000] BIOS-e820: [mem 0x000000007938b000-0x000000007994bfff] >>> reserved >>> [ 0.000000] BIOS-e820: [mem 0x000000007994c000-0x000000007999cfff] >>> ACPI data >>> [ 0.000000] BIOS-e820: [mem 0x000000007999d000-0x0000000079f7dfff] >>> ACPI NVS >>> [ 0.000000] BIOS-e820: [mem 0x0000000079f7e000-0x000000007bd37fff] >>> reserved >>> [ 0.000000] BIOS-e820: [mem 0x000000007bd38000-0x000000007bd38fff] >>> usable >>> [ 0.000000] BIOS-e820: [mem 0x000000007bd39000-0x000000007bdbefff] >>> reserved >>> [ 0.000000] BIOS-e820: [mem 0x000000007bdbf000-0x000000007bffffff] >>> usable >>> [ 0.000000] BIOS-e820: [mem 0x000000007c000000-0x000000008fffffff] >>> reserved >>> [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed44fff] >>> reserved >>> [ 0.000000] BIOS-e820: [mem 0x00000000ff000000-0x00000000ffffffff] >>> reserved >>> [* 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000407fffffff] >>> usable* >>> * >>> * >>> Kindly let me know your inputs. >>> >>> Thanks. >>> >>> >>> -- Thanks, >>> Anatoly >>> >>> >>> >> The "PTE Read not set" error usually indicates that you are trying to use >> a non-IOMMU method when you have IOMMU enabled (i.e. trying to use igb_uio >> when IOMMU is on). That, to me, indicates that you do have IOMMU emulation >> enabled. >> >> I would go about it this way. >> >> First, i'd ensure that your VM has IOMMU emulation enabled and working. >> You have mentioned that your QEMU version should have IOMMU emulation, so >> let's assume that's the case. >> >> I am not sure of the exact command-line needed to activate the vIOMMU >> emulation, but assuming your VM emulates an Intel processor, your kernel >> command-line should have "iommu=on intel_iommu=on" in it. Check >> /etc/default/grub for GRUB_CMDLINE_LINUX_DEFAULT value, and if the above >> values are not in there, add the above changes, do "update-grub" and reboot >> your VM. >> >> If it already did have the necessary kernel configuration, do "dmesg | >> grep IOMMU" and look for "IOMMU Enabled". That should tell you that IOMMU >> is enabled and working in the kernel. >> >> After that, you can modprobe vfio and vfio-pci, bind NICs to it, and it >> should be working. Please bear in mind that all of that is how i would've >> gone about it if i had similar problems on baremetal, but i'm hoping all of >> it is applicable to VM's. So, either disable IOMMU and use igb_uio, or >> enable IOMMU and use VFIO. Both will work. >> >> > Thanks for your help so far on this. I have everything enabled as you have > mentioned in your email. Below is the error from EAL when DPDK is run on a > VM. I am referring EAL code to check why it is failing. > > ************VM DPDK errors*********** > EAL: Detected 14 lcore(s) > EAL: Probing VFIO support... > EAL: VFIO support initialized > EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using > unreliable clock cycles ! > EAL: PCI device 0000:00:07.0 on NUMA socket -1 > EAL: Invalid NUMA socket, default to 0 > EAL: probe driver: 8086:10ed net_ixgbe_vf > EAL: using IOMMU type 1 (Type 1) > > > *EAL: RTE_IOVA_VA 140051367329792EAL: cannot set up DMA remapping, > error 14 (Bad address)EAL: 0000:00:07.0 DMA remapping failed, error 14 > (Bad address)* > EAL: Requested device 0000:00:07.0 cannot be used > EAL: PCI device 0000:03:00.0 on NUMA socket -1 > EAL: Invalid NUMA socket, default to 0 > EAL: probe driver: 1af4:1041 net_virtio > EAL: PCI device 0000:04:00.0 on NUMA socket -1 > EAL: Invalid NUMA socket, default to 0 > EAL: probe driver: 1af4:1041 net_virtio > EAL: PCI device 0000:05:00.0 on NUMA socket -1 > EAL: Invalid NUMA socket, default to 0 > EAL: probe driver: 1af4:1041 net_virtio > EAL: No probed ethernet devices > Interactive-mode selected > USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=171456, size=2176, > socket=0 > > > ********************VM************ > root@sriov-ravi-01:~# lsmod | grep vfio > > > > > *vfio_pci 40960 1vfio_virqfd 16384 1 > vfio_pcivfio_iommu_type1 20480 1vfio 28672 5 > vfio_iommu_type1,vfio_pciirqbypass 16384 3 kvm,vfio_pci* > root@sriov-ravi-01:~# dmesg | grep -e DMAR -e IOMMU > [ 0.000000] ACPI: DMAR 0x000000007FFE209D 000050 (v01 BOCHS BXPCDMAR > 00000001 BXPC 00000001) > *[ 0.000000] DMAR: IOMMU enabled* > [ 9.423839] DMAR: Host address width 39 > [ 9.424640] DMAR: DRHD base: 0x000000fed90000 flags: 0x1 > [ 9.425754] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap > 12008c22260286 ecap f00f5e > [ 9.427314] DMAR: ATSR flags: 0x1 > [ 9.428047] DMAR-IR: IOAPIC id 0 under DRHD base 0xfed90000 IOMMU 0 > [ 9.429255] DMAR-IR: Queued invalidation will be enabled to support > x2apic and Intr-remapping. > [ 9.433096] DMAR-IR: Enabled IRQ remapping in x2apic mode > [ 11.167334] DMAR: No RMRR found > [ 11.168855] DMAR: dmar0: Using Queued invalidation > [ 12.119060] DMAR: Setting RMRR: > [ 12.123734] DMAR: Prepare 0-16MiB unity mapping for LPC > [ 12.124787] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 - > 0xffffff] > [ 12.392826] DMAR: Intel(R) Virtualization Technology for Directed I/O > root@sriov-ravi-01:~# dpdk-devbind -s > > Network devices using DPDK-compatible driver > ============================================ > *0000:00:07.0 '82599 Ethernet Controller Virtual Function 10ed' > drv=vfio-pci unused=* > > Network devices using kernel driver > =================================== > 0000:03:00.0 'Device 1041' if=eth0 drv=virtio-pci unused=vfio-pci *Active* > 0000:04:00.0 'Device 1041' if=eth1 drv=virtio-pci unused=vfio-pci > 0000:05:00.0 'Device 1041' if=eth2 drv=virtio-pci unused=vfio-pci > > Other Network devices > ===================== > <none> > > Crypto devices using DPDK-compatible driver > =========================================== > <none> > > Crypto devices using kernel driver > ================================== > <none> > > Other Crypto devices > ==================== > <none> > > Eventdev devices using DPDK-compatible driver > ============================================= > <none> > > Eventdev devices using kernel driver > ==================================== > <none> > > Other Eventdev devices > ====================== > <none> > > Mempool devices using DPDK-compatible driver > ============================================ > <none> > > Mempool devices using kernel driver > =================================== > <none> > > Other Mempool devices > ===================== > <none> > root@sriov-ravi-01:~# > > root@sriov-ravi-01:/home/deployuser/dpdk-17.11/usertools# dmesg | grep -i > vfio > [ 77.676711] VFIO - User Level meta-driver version: 0.3 > [ 149.013806] VFIO - User Level meta-driver version: 0.3 > [ 275.876440] VFIO - User Level meta-driver version: 0.3 > > Thanks, > Ravi > > > Additonal information. On VM following message is seen [ 298.714974] DMAR: intel_iommu_map: iommu width (39) is not sufficient for the mapped address (7f7300000000) [ 1494.673406] DMAR: intel_iommu_map: iommu width (39) is not sufficient for the mapped address (7fa580000000) ON VM, DMAR reports address width = 39 root@sriov-ravi-01:/home/deployuser/dpdk-17.11/usertools# dmesg | grep -e IOMMU -e DMAR [ 0.000000] ACPI: DMAR 0x000000007FFE209D 000050 (v01 BOCHS BXPCDMAR 00000001 BXPC 00000001) [ 0.000000] DMAR: IOMMU enabled [ 9.209145] DMAR: Host address width 39 On Host, DMAR reports address width = 46 # dmesg | grep -e DMAR -e IOMMU [ 0.000000] ACPI: DMAR 0x000000007999BAD0 0000E0 (v01 ALASKA A M I 00000001 INTL 20091013) [ 0.000000] DMAR: IOMMU enabled [ 0.519368] DMAR: Host address width 46 [ 0.527243] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0 Searching for IOMMU errors found following link from RedHat which points to DPDK 17.11 issue. https://bugzilla.redhat.com/show_bug.cgi?id=1530957 Thanks. > -- >> Thanks, >> Anatoly >> > >
