From: [email protected] To: [email protected] Date: Mon, 1 Aug 2016 16:16:00 +0000 CC: [email protected] Subject: Re: [vfio-users] Cannot register eventfd with MSI/MSI-X interrupts
> Date: Mon, 1 Aug 2016 07:06:18 -0600 > From: [email protected] > To: [email protected] > CC: [email protected] > Subject: Re: [vfio-users] Cannot register eventfd with MSI/MSI-X interrupts > > On Mon, 1 Aug 2016 10:24:52 +0000 > chris thompson <[email protected]> wrote: > > > Hi, > > > > I'm trying to talk to my PCI-e card (USB3 host controller) from userland, > > and have been successful accessing the PCI Config registers and device > > memory, and able to register eventfd file descriptors for legacy interrupts > > (IRQx) and Error interrupts - including stimulating these via the ioctl > > calls. Unfortunately the MSI and MSI-X interrupts return EINVAL (invalid > > argument) when trying to register them. > > > > Sample from the code: > > printf("\nInterrupts:\n"); > > int * irq_fds[VFIO_PCI_NUM_IRQS]; > > int num_irqs[VFIO_PCI_NUM_IRQS]; > > for (i = 0; i < device_info.num_irqs; i++) { > > struct vfio_irq_info irq = { .argsz = sizeof(irq) }; > > irq.index = i; > > ioctl(device, VFIO_DEVICE_GET_IRQ_INFO, &irq); > > printf("argsz 0x%x, flags 0x%x, index 0x%x, count 0x%x,\n", irq.argsz, > > irq.flags, irq.index, irq.count); > > num_irqs[i] = irq.count; > > irq_fds[i]=(int*)malloc(irq.count * sizeof(int)); > > int j; > > for(j=0; j<irq.count; j++) > > { > > irq_fds[i][j] = eventfd(0, 0); > > printf("config IRQ %u : %u\n", i, j); > > struct vfio_irq_set *irq_setup; > > unsigned int size = sizeof(struct vfio_irq_set) + sizeof(int); > > irq_setup = malloc(size); > > irq_setup->argsz = size; > > irq_setup->flags = VFIO_IRQ_SET_DATA_EVENTFD | > > VFIO_IRQ_SET_ACTION_TRIGGER ; > > irq_setup->index = i; > > irq_setup->start = j; > > irq_setup->count = 1; > > *(int*)(irq_setup->data)=irq_fds[i][j]; > > int ret = ioctl(device, VFIO_DEVICE_SET_IRQS, irq_setup); > > if(ret){ printf("failed to register interrupt %u : %u, error %i %s\n", > > i, j, ret, strerror(errno)); } > > free(irq_setup); > > } > > } > > > > Which gives the output: > > > > Interrupts: > > argsz 0x10, flags 0x7, index 0x0, count 0x1, > > config IRQ 0 : 0 > > argsz 0x10, flags 0x9, index 0x1, count 0x40, > > config IRQ 1 : 0 > > failed to register interrupt 1 : 0, error -1 Invalid argument > > config IRQ 1 : 1 > > failed to register interrupt 1 : 1, error -1 Invalid argument > > config IRQ 1 : 2 > > ... > > > > Does anyone know a likely reason that I can't register these? From what > > I've read my IOMMUs all support IRQ remapping: > > [ 0.016443] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap > > c0000020e60262 ecap f0101a > > [ 0.016450] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap > > c9008020660262 ecap f0105a > > > > The hardware is an Intel i5 2500 in an HP 8200 Elite small form-factor PC, > > running Ubuntu 14.04 on the 3.13 kernel. > > > include/uapi/linux/vfio.h: > * The NORESIZE flag indicates that the interrupt lines within the index > * are setup as a set and new subindexes cannot be enabled without first > * disabling the entire index. This is used for interrupts like PCI MSI > * and MSI-X where the driver may only use a subset of the available > * indexes, but VFIO needs to enable a specific number of vectors > * upfront. In the case of MSI-X, where the user can enable MSI-X and > * then add and unmask vectors, it's up to userspace to make the decision > * whether to allocate the maximum supported number of vectors or tear > * down setup and incrementally increase the vectors as each is enabled. > > struct vfio_irq_info { > __u32 argsz; > __u32 flags; > #define VFIO_IRQ_INFO_EVENTFD (1 << 0) > #define VFIO_IRQ_INFO_MASKABLE (1 << 1) > #define VFIO_IRQ_INFO_AUTOMASKED (1 << 2) > #define VFIO_IRQ_INFO_NORESIZE (1 << 3) <---- > __u32 index; /* IRQ index */ > __u32 count; /* Number of IRQs within this index */ > }; Hi Alex, Thanks for your suggestions, based on the same documentation I should be able to disable the interrupts for the set using:"... or the index can be disabled as a whole with: flags = (DATA_NONE|ACTION_TRIGGER), count = 0."and then bind them to eventfd as a group (I was originally trying to bind them as a group with the same results). However: printf("\nInterrupts:\n"); int * irq_fds[VFIO_PCI_NUM_IRQS]; int num_irqs[VFIO_PCI_NUM_IRQS]; for (i = 0; i < device_info.num_irqs; i++) { struct vfio_irq_info irq = { .argsz = sizeof(irq) }; irq.index = i; ioctl(device, VFIO_DEVICE_GET_IRQ_INFO, &irq); printf("argsz 0x%x, flags 0x%x, index 0x%x, count 0x%x,\n", irq.argsz, irq.flags, irq.index, irq.count); num_irqs[i] = irq.count; irq_fds[i]=(int*)malloc(irq.count * sizeof(int)); int j; for(j=0; j<irq.count; j++) irq_fds[i][j] = eventfd(0, 0); printf("config IRQ set %u\n", i); struct vfio_irq_set *irq_setup; unsigned int size = sizeof(struct vfio_irq_set) + (irq.count*sizeof(int)); irq_setup = malloc(size); irq_setup->argsz = size; irq_setup->flags = VFIO_IRQ_SET_DATA_NONE | VFIO_IRQ_SET_ACTION_TRIGGER ; irq_setup->index = i; irq_setup->start = 0; irq_setup->count = irq.count; int ret = ioctl(device, VFIO_DEVICE_SET_IRQS, irq_setup); if(ret){ printf("failed to unregister register interrupt set %u error %i %s\n", i, ret, strerror(errno)); } irq_setup->flags = VFIO_IRQ_SET_DATA_EVENTFD | VFIO_IRQ_SET_ACTION_TRIGGER ; irq_setup->index = i; irq_setup->start = 0; irq_setup->count = irq.count; int j; for(j=0; j<irq.count; j++) ((int*)(irq_setup->data))[j]=irq_fds[i][j]; ret = ioctl(device, VFIO_DEVICE_SET_IRQS, irq_setup); if(ret){ printf("failed to register interrupts %u : %u-%u, error %i %s\n", i, 0, irq.count-1, ret, strerror(errno)); } free(irq_setup); } Results in: Interrupts: argsz 0x10, flags 0x7, index 0x0, count 0x1, config IRQ 0 : 0 failed to unregister register interrupt set 0 error -1 Invalid argument argsz 0x10, flags 0x9, index 0x1, count 0x40, config IRQ 1 : 0 failed to unregister register interrupt set 1 error -1 Invalid argument failed to register interrupts 1 : 0-63, error -1 Invalid argument argsz 0x10, flags 0x9, index 0x2, count 0x8, config IRQ 2 : 0 failed to unregister register interrupt set 2 error -1 Invalid argument failed to register interrupts 2 : 0-7, error -1 Invalid argument argsz 0x10, flags 0x9, index 0x3, count 0x1, config IRQ 3 : 0 Killed This Killed at the IOCtl to the error IRQ line is concerning, and a second attempt results in: Interrupts: argsz 0x10, flags 0x7, index 0x0, count 0x1, config IRQ 0 : 0 then the process hangs in the unregister ioctl. If I force kill the process then I can't open the device again without rebooting my machine. Any ideas? I've probably done something obviously wrong. Thanks and regards, Chris P.S. Missed something important from dmesg too: [ 1334.286951] VFIO - User Level meta-driver version: 0.3 [ 1383.701974] xhci_hcd 0000:02:00.0: remove, state 4 [ 1383.701984] usb usb4: USB disconnect, device number 1 [ 1383.705976] xhci_hcd 0000:02:00.0: USB bus 4 deregistered [ 1383.705984] xhci_hcd 0000:02:00.0: remove, state 4 [ 1383.705989] usb usb3: USB disconnect, device number 1 [ 1383.712050] xhci_hcd 0000:02:00.0: USB bus 3 deregistered [ 1498.085560] vfio_pin_pages: RLIMIT_MEMLOCK (65536) exceeded [ 1499.112900] BUG: unable to handle kernel NULL pointer dereference at (null) [ 1499.112954] IP: [<ffffffffa071b6f9>] vfio_pci_set_err_trigger+0x19/0x130 [vfio_pci] [ 1499.113000] PGD 362e7067 PUD 35172067 PMD 0 [ 1499.113028] Oops: 0000 [#1] SMP [ 1499.113049] Modules linked in: vfio_iommu_type1 vfio_pci vfio nfsv3 nfsv4 autofs4 cuse dm_crypt bnep rfcomm bluetooth gpio_ich hp_wmi sparse_keymap intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp snd_hda_codec_hdmi snd_hda_codec_realtek kvm_intel kvm crct10dif_pclmul crc32_pclmul snd_hda_intel snd_hda_codec snd_hwdep aesni_intel snd_pcm aes_x86_64 lrw snd_page_alloc gf128mul glue_helper ablk_helper snd_seq_midi cryptd snd_seq_midi_event snd_rawmidi snd_seq serio_raw snd_seq_device snd_timer lpc_ich shpchp snd nfsd tpm_infineon wmi mac_hid auth_rpcgss mei_me nfs_acl mei soundcore parport_pc nfs ppdev lockd sunrpc lp fscache parport binfmt_misc btrfs xor raid6_pq libcrc32c i915 video i2c_algo_bit e1000e drm_kms_helper psmouse ahci drm libahci ptp pps_core [ 1499.113526] CPU: 3 PID: 2229 Comm: vfio_test Not tainted 3.13.0-87-generic #133-Ubuntu [ 1499.113568] Hardware name: Hewlett-Packard HP Compaq 8200 Elite SFF PC/1495, BIOS J01 v02.15 11/10/2011 [ 1499.113618] task: ffff880222964800 ti: ffff88003630c000 task.ti: ffff88003630c000 [ 1499.113657] RIP: 0010:[<ffffffffa071b6f9>] [<ffffffffa071b6f9>] vfio_pci_set_err_trigger+0x19/0x130 [vfio_pci] [ 1499.113713] RSP: 0018:ffff88003630ddd0 EFLAGS: 00010246 [ 1499.113742] RAX: 00000000ffffffea RBX: ffff8800ca409800 RCX: 0000000000000001 [ 1499.113779] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8800ca409800 [ 1499.113816] RBP: ffff88003630dde8 R08: 0000000000000021 R09: 0000000000000000 [ 1499.113854] R10: 0000000000000021 R11: ffffffffa071b6e0 R12: 0000000000000000 [ 1499.113891] R13: ffff8800ca409858 R14: 00000000007a81d0 R15: 0000000000000000 [ 1499.113929] FS: 00007f3f57f07740(0000) GS:ffff88022e380000(0000) knlGS:0000000000000000 [ 1499.113971] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1499.114002] CR2: 0000000000000000 CR3: 00000000353c2000 CR4: 00000000000407e0 [ 1499.114039] Stack: [ 1499.114050] ffff8800ca409800 0000000000000000 ffff8800ca409858 ffff88003630ddf8 [ 1499.114093] ffffffffa071c9ec ffff88003630deb0 ffffffffa071ae7d ffff88003630de98 [ 1499.114135] 000000018145452f 0000000000000004 ffff88021f7fba10 0000000000000202 [ 1499.114178] Call Trace: [ 1499.114196] [<ffffffffa071c9ec>] vfio_pci_set_irqs_ioctl+0x8c/0xa0 [vfio_pci] [ 1499.114236] [<ffffffffa071ae7d>] vfio_pci_ioctl+0x31d/0xa30 [vfio_pci] [ 1499.114277] [<ffffffff810ad624>] ? __wake_up+0x44/0x50 [ 1499.114308] [<ffffffff812035e1>] ? fsnotify+0x241/0x320 [ 1499.114340] [<ffffffffa070e193>] vfio_device_fops_unl_ioctl+0x23/0x30 [vfio] [ 1499.114379] [<ffffffff811d46e0>] do_vfs_ioctl+0x2e0/0x4c0 [ 1499.114412] [<ffffffff8172d854>] ? __schedule+0x384/0x7f0 [ 1499.114442] [<ffffffff811d4941>] SyS_ioctl+0x81/0xa0 [ 1499.114472] [<ffffffff8173a3dd>] system_call_fastpath+0x1a/0x1f [ 1499.114503] Code: 48 89 e5 e8 8a 21 af e0 b8 01 00 00 00 5d c3 0f 1f 00 66 66 66 66 90 55 83 fe 03 b8 ea ff ff ff 48 89 e5 41 55 41 54 53 48 89 fb <41> 8b 39 4c 8b 23 75 64 41 f6 c0 07 74 5e 41 f6 c0 01 75 63 41 [ 1499.114673] RIP [<ffffffffa071b6f9>] vfio_pci_set_err_trigger+0x19/0x130 [vfio_pci] [ 1499.114717] RSP <ffff88003630ddd0> [ 1499.114735] CR2: 0000000000000000 [ 1499.258683] ---[ end trace 978702c604a203ce ]--- _______________________________________________ vfio-users mailing list [email protected] https://www.redhat.com/mailman/listinfo/vfio-users
_______________________________________________ vfio-users mailing list [email protected] https://www.redhat.com/mailman/listinfo/vfio-users
