> Date: Mon, 1 Aug 2016 07:06:18 -0600
> From: [email protected]
> To: [email protected]
> CC: [email protected]
> Subject: Re: [vfio-users] Cannot register eventfd with MSI/MSI-X interrupts
>
> On Mon, 1 Aug 2016 10:24:52 +0000
> chris thompson <[email protected]> wrote:
>
> > Hi,
> >
> > I'm trying to talk to my PCI-e card (USB3 host controller) from userland,
> > and have been successful accessing the PCI Config registers and device
> > memory, and able to register eventfd file descriptors for legacy interrupts
> > (IRQx) and Error interrupts - including stimulating these via the ioctl
> > calls. Unfortunately the MSI and MSI-X interrupts return EINVAL (invalid
> > argument) when trying to register them.
> >
> > Sample from the code:
> > printf("\nInterrupts:\n");
> > int * irq_fds[VFIO_PCI_NUM_IRQS];
> > int num_irqs[VFIO_PCI_NUM_IRQS];
> > for (i = 0; i < device_info.num_irqs; i++) {
> > struct vfio_irq_info irq = { .argsz = sizeof(irq) };
> > irq.index = i;
> > ioctl(device, VFIO_DEVICE_GET_IRQ_INFO, &irq);
> > printf("argsz 0x%x, flags 0x%x, index 0x%x, count 0x%x,\n", irq.argsz,
> > irq.flags, irq.index, irq.count);
> > num_irqs[i] = irq.count;
> > irq_fds[i]=(int*)malloc(irq.count * sizeof(int));
> > int j;
> > for(j=0; j<irq.count; j++)
> > {
> > irq_fds[i][j] = eventfd(0, 0);
> > printf("config IRQ %u : %u\n", i, j);
> > struct vfio_irq_set *irq_setup;
> > unsigned int size = sizeof(struct vfio_irq_set) + sizeof(int);
> > irq_setup = malloc(size);
> > irq_setup->argsz = size;
> > irq_setup->flags = VFIO_IRQ_SET_DATA_EVENTFD |
> > VFIO_IRQ_SET_ACTION_TRIGGER ;
> > irq_setup->index = i;
> > irq_setup->start = j;
> > irq_setup->count = 1;
> > *(int*)(irq_setup->data)=irq_fds[i][j];
> > int ret = ioctl(device, VFIO_DEVICE_SET_IRQS, irq_setup);
> > if(ret){ printf("failed to register interrupt %u : %u, error %i %s\n",
> > i, j, ret, strerror(errno)); }
> > free(irq_setup);
> > }
> > }
> >
> > Which gives the output:
> >
> > Interrupts:
> > argsz 0x10, flags 0x7, index 0x0, count 0x1,
> > config IRQ 0 : 0
> > argsz 0x10, flags 0x9, index 0x1, count 0x40,
> > config IRQ 1 : 0
> > failed to register interrupt 1 : 0, error -1 Invalid argument
> > config IRQ 1 : 1
> > failed to register interrupt 1 : 1, error -1 Invalid argument
> > config IRQ 1 : 2
> > ...
> >
> > Does anyone know a likely reason that I can't register these? From what
> > I've read my IOMMUs all support IRQ remapping:
> > [ 0.016443] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap
> > c0000020e60262 ecap f0101a
> > [ 0.016450] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap
> > c9008020660262 ecap f0105a
> >
> > The hardware is an Intel i5 2500 in an HP 8200 Elite small form-factor PC,
> > running Ubuntu 14.04 on the 3.13 kernel.
>
>
> include/uapi/linux/vfio.h:
> * The NORESIZE flag indicates that the interrupt lines within the index
> * are setup as a set and new subindexes cannot be enabled without first
> * disabling the entire index. This is used for interrupts like PCI MSI
> * and MSI-X where the driver may only use a subset of the available
> * indexes, but VFIO needs to enable a specific number of vectors
> * upfront. In the case of MSI-X, where the user can enable MSI-X and
> * then add and unmask vectors, it's up to userspace to make the decision
> * whether to allocate the maximum supported number of vectors or tear
> * down setup and incrementally increase the vectors as each is enabled.
>
> struct vfio_irq_info {
> __u32 argsz;
> __u32 flags;
> #define VFIO_IRQ_INFO_EVENTFD (1 << 0)
> #define VFIO_IRQ_INFO_MASKABLE (1 << 1)
> #define VFIO_IRQ_INFO_AUTOMASKED (1 << 2)
> #define VFIO_IRQ_INFO_NORESIZE (1 << 3) <----
> __u32 index; /* IRQ index */
> __u32 count; /* Number of IRQs within this index */
> };
Hi Alex, Thanks for your suggestions, based on the same documentation I should
be able to disable the interrupts for the set using:"... or the index can be
disabled as a whole with: flags = (DATA_NONE|ACTION_TRIGGER), count = 0."and
then bind them to eventfd as a group (I was originally trying to bind them as a
group with the same results). However: printf("\nInterrupts:\n");
int * irq_fds[VFIO_PCI_NUM_IRQS];
int num_irqs[VFIO_PCI_NUM_IRQS];
for (i = 0; i < device_info.num_irqs; i++) {
struct vfio_irq_info irq = { .argsz = sizeof(irq) }; irq.index = i;
ioctl(device, VFIO_DEVICE_GET_IRQ_INFO, &irq);
printf("argsz 0x%x, flags 0x%x, index 0x%x, count 0x%x,\n", irq.argsz,
irq.flags, irq.index, irq.count); num_irqs[i] = irq.count;
irq_fds[i]=(int*)malloc(irq.count * sizeof(int));
int j;
for(j=0; j<irq.count; j++) irq_fds[i][j] = eventfd(0, 0); printf("config
IRQ set %u\n", i);
struct vfio_irq_set *irq_setup;
unsigned int size = sizeof(struct vfio_irq_set) + (irq.count*sizeof(int));
irq_setup = malloc(size);
irq_setup->argsz = size;
irq_setup->flags = VFIO_IRQ_SET_DATA_NONE | VFIO_IRQ_SET_ACTION_TRIGGER ;
irq_setup->index = i;
irq_setup->start = 0;
irq_setup->count = irq.count;
int ret = ioctl(device, VFIO_DEVICE_SET_IRQS, irq_setup);
if(ret){ printf("failed to unregister register interrupt set %u error %i
%s\n", i, ret, strerror(errno)); } irq_setup->flags =
VFIO_IRQ_SET_DATA_EVENTFD | VFIO_IRQ_SET_ACTION_TRIGGER ;
irq_setup->index = i;
irq_setup->start = 0;
irq_setup->count = irq.count;
int j;
for(j=0; j<irq.count; j++) ((int*)(irq_setup->data))[j]=irq_fds[i][j];
ret = ioctl(device, VFIO_DEVICE_SET_IRQS, irq_setup);
if(ret){ printf("failed to register interrupts %u : %u-%u, error %i %s\n",
i, 0, irq.count-1, ret, strerror(errno)); }
free(irq_setup);
}
Results in:
Interrupts:
argsz 0x10, flags 0x7, index 0x0, count 0x1,
config IRQ 0 : 0
failed to unregister register interrupt set 0 error -1 Invalid argument
argsz 0x10, flags 0x9, index 0x1, count 0x40,
config IRQ 1 : 0
failed to unregister register interrupt set 1 error -1 Invalid argument
failed to register interrupts 1 : 0-63, error -1 Invalid argument
argsz 0x10, flags 0x9, index 0x2, count 0x8,
config IRQ 2 : 0
failed to unregister register interrupt set 2 error -1 Invalid argument
failed to register interrupts 2 : 0-7, error -1 Invalid argument
argsz 0x10, flags 0x9, index 0x3, count 0x1,
config IRQ 3 : 0
Killed
This Killed at the IOCtl to the error IRQ line is concerning, and a second
attempt results in:
Interrupts:
argsz 0x10, flags 0x7, index 0x0, count 0x1,
config IRQ 0 : 0
then the process hangs in the unregister ioctl. If I force kill the process
then I can't open the device again without rebooting my machine.
Any ideas? I've probably done something obviously wrong.
Thanks and regards,
Chris
_______________________________________________
vfio-users mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/vfio-users
