Hi Alex, Thanks very much, I'll lean on IT a bit to try and test this with a newer kernel, but it looks like it's definitely been addressed in later kernels. What I really need is to get a new system with wider IOVA support.
Obviously your X79 board supports 48-bits, looking through Intel's datasheets it looks like the base level VT-d supports 39-bits, while some CPUs support 48-bits (the 4xxx 5xxx 6xxx and 7xxx series at least, but not 3xxx or older): "The processor supports the following added new Intel VT-d features: - 4-level Intel VT-d Pagewalk - bothe default Intel VT-d engine as well as the IGD VT-d engine are upgraded to support 4-level Intel VT-d tables (adjusted guest address width of 48 bits)" Do you know if the chipset would put any limit on this (so long as the chipset supports VT-d in the first place)? For full disclosure I'm the same Chris Thompson as emailed the list a while ago from [email protected], I initially used a personal email because I prefer to not clutter up my work inbox with mailing lists, but recent policy changes at work mean I can't actually access personal email from my work machine any more. Thanks and regards, Chris > -----Original Message----- > From: Alex Williamson [mailto:[email protected]] > Sent: 30 March 2017 17:18 > To: Christopher Thompson <[email protected]> > Cc: [email protected] > Subject: Re: [vfio-users] vfio dma map succeeds but dmesg indicates error > > On Thu, 30 Mar 2017 13:50:27 +0000 > Christopher Thompson <[email protected]> wrote: > > > Hi, > > > > I'm currently using VFIO for PCI passthrough in a custom simulation, and > have discovered that when I try to map IO virtual addresses larger than my > host supports, VFIO happily returns 0 from the dma map ioctl, but dmesg is > telling me something is wrong. > > > > I noticed this after a few days of wondering why this specific simulation > wasn't working - the simulated system is trying to setup dma addresses using > 40-bits of address space, starting at 0xfffffff000 and descending, it turns > out > my host only support 39 bits. > > > > For example a dma map of 0xffffffd000, size 0x1000, the ioctl returns 0 (but > if I inspect errno it has the value "Bad address"). Dmesg generates the error: > > [677414.285778] intel_iommu_map: iommu width (39) is not sufficient > > for the mapped address (ffffffe000) > > > > I can accept that I can't map addresses larger than my host IOMMU > supports, but shouldn't VFIO report an error from the mapping? > > > > If this has been fixed in a later version please let me know, my Google-fu > wasn't good enough to find much on this error. My kernel is a relatively > ancient 3.13 (Ubuntu 14.04.4). > > The kernel version may indeed be the issue, I just modified one of my test > programs to probe the address width using this function: > > int probe_iova_limit(int fd, unsigned long vaddr, unsigned long pagesize) { > struct vfio_iommu_type1_dma_map dma_map = { > .argsz = sizeof(dma_map), > .flags = VFIO_DMA_MAP_FLAG_READ | > VFIO_DMA_MAP_FLAG_WRITE, > .size = pagesize, > .vaddr = vaddr, > }; > struct vfio_iommu_type1_dma_unmap dma_unmap = { > .argsz = sizeof(dma_unmap), > .size = pagesize, > }; > int ret; > unsigned long iova = pagesize; > > while (iova) { > dma_map.iova = dma_unmap.iova = iova - pagesize; > > ret = ioctl(fd, VFIO_IOMMU_MAP_DMA, &dma_map); > if (ret) { > printf("IOMMU failed at 0x%lx (%m), address width > %d\n", > dma_map.iova, ffsll(iova) - 2); > return 0; > } > > ret = ioctl(fd, VFIO_IOMMU_UNMAP_DMA, &dma_unmap); > if (ret) { > printf("Unmap 0x%lx failed\n", dma_unmap.iova); > return ret; > } > > iova <<= 1; > } > > printf("IOMMU has full 64 bit address width, no failures\n"); > return 0; > } > > And my BDW Intel NUC reports: > > IOMMU failed at 0xfffffff000 (Bad address), address width 39 > > An X79 box reports: > > IOMMU failed at 0x1fffffffff000 (Bad address), address width 48 > > Tested both a v4.10 kernel and RHEL7 kernel on the NUC, RHEL7 on the X79. > Try a different kernel, let me know what you find. Thanks, > > Alex _______________________________________________ vfio-users mailing list [email protected] https://www.redhat.com/mailman/listinfo/vfio-users
