Correction: 1. get_user_pages_fast() for each hugepage start address for one page 2. sg_alloc_table_from_pages() using page array from #1 3. dma_map_sg() for num hugepages using sgt from #2
On Sun, Dec 1, 2019 at 5:46 PM Anand Misra <[email protected]> wrote: > Hello: > > I'm in process of adding iommu support in my driver for a PCIe device. The > device doesn't publish ACS/ATS via its config space. I've following config: > > Linux cmdline: "intel-iommu=on iommu=pt > vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream" > Centos kernel: 3.10.0-1062.1.2.el7.x86_64 > > I'm trying to use iommu for multiple hugepages (mmap'ed by process and > pushed to driver via ioctl). The expectation is to have multiple hugepages > mapped via iommu with each huge page having an entry in iommu (i.e. > minimize table walk for DMA). Is this possible? > > [1] The driver ioctl has the following sequence: > > 1. get_user_pages_fast() for each hugepage start address for one page > 2. sg_alloc_table_from_pages() using sgt from #3 > 3. dma_map_sg() for num hugepages using sgt from #4 > > I'm getting kernel crash at #3 for "domain_get_iommu+0x55/0x70": > > ---------------------- > [148794.896405] kernel BUG at drivers/iommu/intel-iommu.c:667! > [148794.896409] invalid opcode: 0000 [#1] SMP > [148794.896414] Modules linked in: mydrv(OE) nfsv3 nfs_acl nfs lockd grace > fscache xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 > nf_conntrack_netlink nfnetlink xt_addrtype iptable_filter iptable_nat > nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack libcrc32c > br_netfilter bridge stp llc overlay(T) ipmi_devintf ipmi_msghandler sunrpc > vfat fat iTCO_wdt mei_wdt iTCO_vendor_support sb_edac intel_powerclamp > coretemp intel_rapl iosf_mbi kvm_intel kvm snd_hda_codec_hdmi irqbypass > crc32_pclmul ghash_clmulni_intel aesni_intel snd_hda_codec_realtek lrw > gf128mul glue_helper ablk_helper cryptd dell_smbios snd_hda_codec_generic > intel_wmi_thunderbolt dcdbas dell_wmi_descriptor pcspkr snd_hda_intel > snd_hda_codec snd_hda_core snd_hwdep i2c_i801 snd_seq snd_seq_device sg > snd_pcm lpc_ich ftdi_sio snd_timer > [148794.896522] joydev snd mei_me mei soundcore pcc_cpufreq binfmt_misc > ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic sr_mod > cdrom nouveau video mxm_wmi i2c_algo_bit drm_kms_helper crct10dif_pclmul > crct10dif_common crc32c_intel serio_raw syscopyarea sysfillrect sysimgblt > fb_sys_fops ttm ahci drm libahci ata_generic e1000e pata_acpi libata ptp > pps_core drm_panel_orientation_quirks wmi [last unloaded: mydrv] > [148794.896587] CPU: 0 PID: 6020 Comm: TestIommu Kdump: loaded Tainted: G > OE ------------ T 3.10.0-1062.1.2.el7.x86_64 #1 > [148794.896592] Hardware name: Dell Inc. Precision Tower 5810/0HHV7N, BIOS > A25 02/02/2018 > [148794.896597] task: ffff8c82b6e0d230 ti: ffff8c8ac5b6c000 task.ti: > ffff8c8ac5b6c000 > [148794.896601] RIP: 0010:[<ffffffff8efff195>] [<ffffffff8efff195>] > domain_get_iommu+0x55/0x70 > [148794.896611] RSP: 0018:ffff8c8ac5b6fce8 EFLAGS: 00010202 > [148794.896614] RAX: ffff8c8adbeb0b00 RBX: ffff8c8ad4ac7600 RCX: > 0000000000000000 > [148794.896619] RDX: 00000000fffffff0 RSI: ffff8c8ace6e5940 RDI: > ffff8c8adbeb0b00 > [148794.896622] RBP: ffff8c8ac5b6fce8 R08: 000000000001f0a0 R09: > ffffffff8f00255e > [148794.896626] R10: ffff8c8bdfc1f0a0 R11: fffff941bc39b940 R12: > 0000000000000001 > [148794.896630] R13: ffff8c4ce6b9d098 R14: 0000000000000000 R15: > ffff8c8ac8f22a00 > [148794.896635] FS: 00007f1548320740(0000) GS:ffff8c8bdfc00000(0000) > knlGS:0000000000000000 > [148794.896639] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [148794.896642] CR2: 00007f1547373689 CR3: 00000036f17c8000 CR4: > 00000000003607f0 > [148794.896647] DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > [148794.896651] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: > 0000000000000400 > [148794.896654] Call Trace: > [148794.896660] [<ffffffff8f002ee5>] intel_map_sg+0x65/0x1e0 > [...] > > ---------------------- > > > [2] I've also tried using iommu APIs directly in my driver but I get "PTE > Read access is not set" for DMA read when attempting DMA from host to > device memory (size 1KB). > > DMAR: DRHD: handling fault status reg 2 > DMAR: [DMA Read] Request device [02:00.0] fault addr ffffc030b000 [fault > reason 06] PTE Read access is not set > > I see the following messages after DMA failure (and eventually system > crash): > > DMAR: DRHD: handling fault status reg 100 > DMAR: DRHD: handling fault status reg 100 > > > I've used the following sequence with iommu APIs: > > iommu_init: > > iommu_group = iommu_group_get(dev) > > iommu_domain = iommu_domain_alloc(&pci_bus_type) > > init_iova_domain(&iova_domain) > > iommu_attach_group(iommu_domain, iommu_group) > > iommu_map: > > iova = alloc_iova(&iova_domain, size >> shift, end >> shift, true); > > addr = iova_dma_addr(&iova_domain, iova); > > iommu_map_sg(iommu_domain, addr, sgl, sgt->nents, IOMMU_READ | > IOMMU_WRITE); > > > Thanks, > am > >
_______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
