Re: [vfio-users] VFIO_IOMMU_MAP_DMA succeeds only on second try?
Many thanks!
mmap succeeds - I removed the checks only to post the code on this list.
I use this now:
if ( !ioctl(container, VFIO_IOMMU_MAP_DMA, &dma_map) )
{ if ( !ioctl(container, VFIO_IOMMU_MAP_DMA, &dma_map) )
{ perror("mmap ioctl"); return 1; }
}
It mostly succeeds only on second try but I never needed more than two
tries. So it's ok.
Re. IOMMU type 1 recognition: You are right: It is independent of kernel
version. It seems to be related to auto-loading vfio_pci via
/etc/modules.conf. Installing the module later gets it working.
*From:* Alex Williamson <mailto:alex.william...@redhat.com>
*Sent:* Monday, March 12, 2018 6:57AM
*To:* Heid, Oliver <mailto:oliver.h...@h-next.de>
*Cc:* Vfio-users <mailto:vfio-users@redhat.com>
*Subject:* Re: [vfio-users] VFIO_IOMMU_MAP_DMA succeeds only on second try?
On Mon, 5 Mar 2018 12:37:24 +0100
Oliver Heid wrote:
Do I miss something here? I want to allow RW access of a peripheral
device to a memory region via
struct vfio_iommu_type1_dma_map dma_map = { .argsz = sizeof(dma_map) };
__u32* mem = (__u32*)mmap(NULL,size,PROT_READ|PROT_WRITE,
MAP_SHARED|MAP_ANONYMOUS|MAP_LOCKED, -1, 0);
dma_map.argsz = sizeof(dma_map);
dma_map.flags = VFIO_DMA_MAP_FLAG_READ | VFIO_DMA_MAP_FLAG_WRITE;
dma_map.vaddr = (__u64)mem;
dma_map.iova = 0;
dma_map.size = size;
ioctl(container, VFIO_IOMMU_MAP_DMA, &dma_map);
Any idea why theVFIO_IOMMU_MAP_DMA ioctl fails with EINVAL, but then a
second, identical call succeeds? Does it actually succeed then? QEMU
re-tries if the first attempt fails with EBUSY, but not with EINVAL.
I don't think the QEMU -EBUSY behavior is related, that handles
previous mappings and may no longer even be necessary. One of my unit
tests[1] does a similar map with no issues. Your code excerpt doesn't
check whether mmap succeeds, so for all we know mmap is failing too.
The latest 4.15.7 kernel (4.15.6 too IIRC) does not recognize the AMD
Ryzen 3 1200 as VFIO_TYPE1_IOMMU so I am using version 4.15.4.
This doesn't make sense, anything implementing the kernel's IOMMU API in
the kernel is compatible with type1, that includes anything supporting
AMD-Vi. There is no whitelist or specific processor support
selection. There are no iommu or vfio changes between the kernels
you're quoting, perhaps you could do a bisect. Thanks,
Alex
[1] https://github.com/awilliam/tests/blob/master/vfio-iommu-map-unmap.c
___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users
[vfio-users] VFIO 128 byte TLP size?
Does VFIO somehow limit PCIe TLP size to 128 bytes on virtualized devices? In our case MaxPayload = MaxReadReq = 256 bytes in PCI config, and we use 4k aligned base addresses and do not cross 4k boundaries, but actual package size is only 128B. Any idea how to get 256 byte TLPs? Thanks! ___ vfio-users mailing list vfio-users@redhat.com https://www.redhat.com/mailman/listinfo/vfio-users
[vfio-users] VFIO_IOMMU_MAP_DMA succeeds only on second try?
Do I miss something here? I want to allow RW access of a peripheral
device to a memory region via
struct vfio_iommu_type1_dma_map dma_map = { .argsz = sizeof(dma_map) };
__u32* mem = (__u32*)mmap(NULL,size,PROT_READ|PROT_WRITE,
MAP_SHARED|MAP_ANONYMOUS|MAP_LOCKED, -1, 0);
dma_map.argsz = sizeof(dma_map);
dma_map.flags = VFIO_DMA_MAP_FLAG_READ | VFIO_DMA_MAP_FLAG_WRITE;
dma_map.vaddr = (__u64)mem;
dma_map.iova = 0;
dma_map.size = size;
ioctl(container, VFIO_IOMMU_MAP_DMA, &dma_map);
Any idea why theVFIO_IOMMU_MAP_DMA ioctl fails with EINVAL, but then a
second, identical call succeeds? Does it actually succeed then? QEMU
re-tries if the first attempt fails with EBUSY, but not with EINVAL.
The latest 4.15.7 kernel (4.15.6 too IIRC) does not recognize the AMD
Ryzen 3 1200 as VFIO_TYPE1_IOMMU so I am using version 4.15.4.
Thanks!
___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users
Re: [vfio-users] VFIO for PCIe streaming from/to user space
Problem solved: To enable MSI -> eventfd signalling we had to set the bus master enable bit in PCI config space (bit 2 at offset 4). Thanks! *From:* Alex Williamson <mailto:alex.william...@redhat.com> *Sent:* Thursday, February 22, 2018 10:03PM *To:* Heid, Oliver <mailto:oliver.h...@h-next.de> *Subject:* Re: [vfio-users] VFIO for PCIe streaming from/to user space On Thu, 22 Feb 2018 21:44:44 +0100 Oliver Heid wrote: After registering e.g. MSI 0 with __s32 msifd = eventfd(0, 0); struct vfio_irq_set *irq_set; irq_set=(struct vfio_irq_set *)malloc(sizeof(struct vfio_irq_set) + sizeof(__s32)); irq_set->argsz = sizeof(struct vfio_irq_set) + sizeof(__s32); irq_set->index = VFIO_PCI_MSI_IRQ_INDEX; irq_set->flags = VFIO_IRQ_SET_DATA_EVENTFD | VFIO_IRQ_SET_ACTION_TRIGGER; irq_set->start = 0; irq_set->count = 1; *(__s32*)&irq_set->data = msifd; ioctl(device, VFIO_DEVICE_SET_IRQS, irq_set); any read by __u64 value = 0; int ret = read(msifd,&value,sizeof(value)); should acknowledge the MSI, right? Or did I miss something? What do you mean by "acknowledge"? This effectively acknowledges the eventfd, but it's up to your driver to service the hardware upon receiving the eventfd notification. The eventfd is only the signaling mechanism. Thanks, Alex ___ vfio-users mailing list vfio-users@redhat.com https://www.redhat.com/mailman/listinfo/vfio-users
Re: [vfio-users] VFIO for PCIe streaming from/to user space
After registering e.g. MSI 0 with
__s32 msifd = eventfd(0, 0);
struct vfio_irq_set *irq_set;
irq_set=(struct vfio_irq_set *)malloc(sizeof(struct vfio_irq_set) +
sizeof(__s32));
irq_set->argsz = sizeof(struct vfio_irq_set) + sizeof(__s32);
irq_set->index = VFIO_PCI_MSI_IRQ_INDEX;
irq_set->flags = VFIO_IRQ_SET_DATA_EVENTFD |
VFIO_IRQ_SET_ACTION_TRIGGER;
irq_set->start = 0;
irq_set->count = 1;
*(__s32*)&irq_set->data = msifd;
ioctl(device, VFIO_DEVICE_SET_IRQS, irq_set);
any read by
__u64 value = 0;
int ret = read(msifd,&value,sizeof(value));
should acknowledge the MSI, right? Or did I miss something?
*From:* Alex Williamson <mailto:alex.william...@redhat.com>
*Sent:* Wednesday, February 14, 2018 10:18PM
*To:* Heid, Oliver <mailto:oliver.h...@h-next.de>
*Cc:* Vfio-users <mailto:vfio-users@redhat.com>
*Subject:* Re: [vfio-users] VFIO for PCIe streaming from/to user space
On Wed, 14 Feb 2018 22:06:35 +0100
Oliver Heid wrote:
Regarding the vfio_device_info struct: I get 9 device regions, of which
region 0, 2 and 4 seem to be the three BAR regions. Is that so? Always?
What do the other regions do?
This is all answered in the API:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/uapi/linux/vfio.h
/*
* The VFIO-PCI bus driver makes use of the following fixed region and
* IRQ index mapping. Unimplemented regions return a size of zero.
* Unimplemented IRQ types return a count of zero.
*/
enum {
VFIO_PCI_BAR0_REGION_INDEX,
VFIO_PCI_BAR1_REGION_INDEX,
VFIO_PCI_BAR2_REGION_INDEX,
VFIO_PCI_BAR3_REGION_INDEX,
VFIO_PCI_BAR4_REGION_INDEX,
VFIO_PCI_BAR5_REGION_INDEX,
VFIO_PCI_ROM_REGION_INDEX,
VFIO_PCI_CONFIG_REGION_INDEX,
/*
* Expose VGA regions defined for PCI base class 03, subclass 00.
* This includes I/O port ranges 0x3b0 to 0x3bb and 0x3c0 to 0x3df
* as well as the MMIO range 0xa to 0xb. Each implemented
* range is found at it's identity mapped offset from the region
* offset, for example 0x3b0 is region_info.offset + 0x3b0. Areas
* between described ranges are unimplemented.
*/
VFIO_PCI_VGA_REGION_INDEX,
VFIO_PCI_NUM_REGIONS = 9 /* Fixed user ABI, region indexes >=9 use */
/* device specific cap to define content. */
};
And by mmap of PCIe config space you mean mapping the files
/sys/bus/pci/devices/:xx:yy.z/resource0 ?!
No, use the VFIO_PCI_CONFIG_REGION_INDEX region.
___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users
Re: [vfio-users] VFIO for PCIe streaming from/to user space
Regarding the vfio_device_info struct: I get 9 device regions, of which region 0, 2 and 4 seem to be the three BAR regions. Is that so? Always? What do the other regions do? And by mmap of PCIe config space you mean mapping the files /sys/bus/pci/devices/:xx:yy.z/resource0 ?! Thanks a lot! Oliver *From:* Alex Williamson <mailto:alex.william...@redhat.com> *Sent:* Tuesday, January 30, 2018 10:00PM *To:* Heid, Oliver <mailto:oliver.h...@h-next.de> *Cc:* Vfio-users <mailto:vfio-users@redhat.com> *Subject:* Re: [vfio-users] VFIO for PCIe streaming from/to user space On Tue, 30 Jan 2018 21:41:30 +0100 Oliver Heid wrote: So the userspace -> IOVA mapping is not done automatically? No, see the API, user provides an IOVA. How do I get the MMU entries of user's pages from userspace and write them into the IOMMU? You don't need them, provide the virtual address to the mapping ioctl. For instance if your buffer is mmap'd, provide the mmap pointer. Isn't it quite a security hazard to let any user program the IOMMU? Any user cannot program the IOMMU, only users with permissions granted to access the vfio group file and only up to the locked page limit allowed for that user. Thanks, Alex *From:* Alex Williamson <mailto:alex.william...@redhat.com> *Sent:* Tuesday, January 30, 2018 4:06PM *To:* Heid, Oliver <mailto:oliver.h...@h-next.de> *Cc:* Vfio-users <mailto:vfio-users@redhat.com> *Subject:* Re: [vfio-users] VFIO for PCIe streaming from/to user space On Tue, 30 Jan 2018 12:02:32 +0100 Oliver Heid wrote: Hi, I want to user VFIO for direct streaming between user space and a PCIe device (FPGA). Device isolation works, but how do I re-assign the PCIe BARs to my user space memory and handle the IOMMU TLB remapping? Or is there some other/better way to do that? https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/vfio.txt PCI BARs are exposed as regions within the device file descriptor, you can read/write to them via the file descriptor or mmap them into the user address space where supported. DMA is configured via ioctls on the container, mapping a user virtual address to an IOVA. Note that some IOMMUs do not support a full 64bit IOVA space, so don't expect to be able to identity map the IOVA to the virtual address. We're working on an API to expose to the IOVA ranges to userspace. Thanks, Alex ___ vfio-users mailing list vfio-users@redhat.com https://www.redhat.com/mailman/listinfo/vfio-users
[vfio-users] VFIO for PCIe streaming from/to user space
Hi, I want to user VFIO for direct streaming between user space and a PCIe device (FPGA). Device isolation works, but how do I re-assign the PCIe BARs to my user space memory and handle the IOMMU TLB remapping? Or is there some other/better way to do that? Many thanks! ___ vfio-users mailing list vfio-users@redhat.com https://www.redhat.com/mailman/listinfo/vfio-users
