genirq: Flags mismatch irq 17. 00000000 (vfio-intx(0000:07:04.0)) vs. 00000000 (vfio-intx(0000:01:00.1))
I'm using the SuperMicro X10SAT, kernel 3.13-rc5, with ACS override on Intel root ports. I'm trying to pass several devices to the same guest: 01:00.0 -- [1002:68be] Radeon HD 5750 01:00.1 -- [1002:aa58] HDMI Audio (not really needed) 07:04.0 -- [13f6:8788] Xonar D1/DX sound card, behind PEX8112 09:00.0 -- [1912:0014] Renesas uPD720201 (USB 3.0) When trying to start qemu with various combinations of those devices: vfio: Error: Failed to setup INTx fd: Device or resource busy Sound card conflicts with HDMI audio: genirq: Flags mismatch irq 17. (vfio-intx(:07:04.0)) vs. (vfio-intx(:01:00.1)) USB controller conflicts with video card: genirq: Flags mismatch irq 16. (vfio-intx(:09:00.0)) vs. (vfio-intx(:01:00.0)) On the ArchLinux forums, I was told that this means each device wants the interrupt line to itself. Oddly, the host locks up if I launch Xorg with 'radeon' and 'snd-virtuoso' both loaded. (Primary video is the Intel graphics.) These devices all work fine together using pci-assign, but pci-assign requires ejecting the Radeon before VM shutdown. VFIO allows me to start the VM if I forward only the sound card and the video card, but I really need the USB controller, as well. What can I do to forward those three devices via VFIO? -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: genirq: Flags mismatch irq 17. 00000000 (vfio-intx(0000:07:04.0)) vs. 00000000 (vfio-intx(0000:01:00.1))
On 01/10/2014 09:33 AM, Alex Williamson wrote: On Fri, 2014-01-10 at 08:23 -0800, Dana Goyette wrote: I'm using the SuperMicro X10SAT, kernel 3.13-rc5, with ACS override on Intel root ports. I'm trying to pass several devices to the same guest: 01:00.0 -- [1002:68be] Radeon HD 5750 01:00.1 -- [1002:aa58] HDMI Audio (not really needed) 07:04.0 -- [13f6:8788] Xonar D1/DX sound card, behind PEX8112 09:00.0 -- [1912:0014] Renesas uPD720201 (USB 3.0) When trying to start qemu with various combinations of those devices: vfio: Error: Failed to setup INTx fd: Device or resource busy Sound card conflicts with HDMI audio: genirq: Flags mismatch irq 17. (vfio-intx(:07:04.0)) vs. (vfio-intx(:01:00.1)) USB controller conflicts with video card: genirq: Flags mismatch irq 16. (vfio-intx(:09:00.0)) vs. (vfio-intx(:01:00.0)) On the ArchLinux forums, I was told that this means each device wants the interrupt line to itself. Oddly, the host locks up if I launch Xorg with 'radeon' and 'snd-virtuoso' both loaded. (Primary video is the Intel graphics.) These devices all work fine together using pci-assign, but pci-assign requires ejecting the Radeon before VM shutdown. VFIO allows me to start the VM if I forward only the sound card and the video card, but I really need the USB controller, as well. What can I do to forward those three devices via VFIO? Change slots until the interrupts don't conflict, maybe there's an IO-APIC option you can enable in the BIOS, or get devices that properly support PCI2.3 INTx disable. pci-assign tries to work around this by using MSI to emulate INTx, but I'm not convinced that this doesn't create more problems than it solves or I'd try to do the same with vfio. The Xorg problem might also suggest there's an existing interrupt routing problem on the system. It is surprising that you have so many devices that doesn't support INTx disable, can you try running this script on them: #!/bin/sh # Usage $0 PCI device, ex: 9:00.0 INTX=$(( 0x400 )) ORIG=$(( 0x$(setpci -s $1 4.w) )) if [ $(( $INTX $ORIG )) -ne 0 ]; then echo INTx disable supported and enabled on $1 exit 0 fi NEW=$(printf %04x $(( $INTX | $ORIG ))) setpci -s $1 4.w=$NEW NEW=$(( 0x$(setpci -s $1 4.w) )) if [ $(( $INTX $NEW )) -ne 0 ]; then echo INTx disable support available on $1 else echo INTx disable support NOT available on $1 fi NEW=$(printf %04x $ORIG) setpci -s $1 4.w=$NEW -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html After cold boot, without any attempts to run VM: Video card: INTx disable support available on 01:00.0 HDMI Audio: INTx disable supported and enabled on 01:00.1 USB: INTx disable supported and enabled on 09:00.0 PEX8112 (sound card sits behind this): INTx disable support available on 06:00.0) Sound card: INTx disable support NOT available on 07:04.0 If the sound card is the only one lacking this feature, then it seems odd that the USB controller is the one I have to remove for the VM to boot. The lspci -nnvv listing for my current sound card is quite sparse: 07:04.0 Multimedia audio controller [0401]: C-Media Electronics Inc CMI8788 [Oxygen HD Audio] [13f6:8788] Subsystem: ASUSTeK Computer Inc. Virtuoso 100 (Xonar D1) [1043:834f] Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium TAbort- TAbort- MAbort- SERR- PERR- INTx- Interrupt: pin A routed to IRQ 10 Region 0: I/O ports at c000 [disabled] [size=256] Capabilities: [c0] Power Management version 2 Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- I'm willing to purchase a different sound card if that will fix the problem. Does lspci (-v or -vv) tell you if INTx Disable is available? -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: genirq: Flags mismatch irq 17. 00000000 (vfio-intx(0000:07:04.0)) vs. 00000000 (vfio-intx(0000:01:00.1))
On 01/10/2014 01:12 PM, Alex Williamson wrote: On Fri, 2014-01-10 at 12:18 -0800, Dana Goyette wrote: On 01/10/2014 09:33 AM, Alex Williamson wrote: On Fri, 2014-01-10 at 08:23 -0800, Dana Goyette wrote: I'm using the SuperMicro X10SAT, kernel 3.13-rc5, with ACS override on Intel root ports. I'm trying to pass several devices to the same guest: 01:00.0 -- [1002:68be] Radeon HD 5750 01:00.1 -- [1002:aa58] HDMI Audio (not really needed) 07:04.0 -- [13f6:8788] Xonar D1/DX sound card, behind PEX8112 09:00.0 -- [1912:0014] Renesas uPD720201 (USB 3.0) When trying to start qemu with various combinations of those devices: vfio: Error: Failed to setup INTx fd: Device or resource busy Sound card conflicts with HDMI audio: genirq: Flags mismatch irq 17. (vfio-intx(:07:04.0)) vs. (vfio-intx(:01:00.1)) USB controller conflicts with video card: genirq: Flags mismatch irq 16. (vfio-intx(:09:00.0)) vs. (vfio-intx(:01:00.0)) On the ArchLinux forums, I was told that this means each device wants the interrupt line to itself. Oddly, the host locks up if I launch Xorg with 'radeon' and 'snd-virtuoso' both loaded. (Primary video is the Intel graphics.) These devices all work fine together using pci-assign, but pci-assign requires ejecting the Radeon before VM shutdown. VFIO allows me to start the VM if I forward only the sound card and the video card, but I really need the USB controller, as well. What can I do to forward those three devices via VFIO? Change slots until the interrupts don't conflict, maybe there's an IO-APIC option you can enable in the BIOS, or get devices that properly support PCI2.3 INTx disable. pci-assign tries to work around this by using MSI to emulate INTx, but I'm not convinced that this doesn't create more problems than it solves or I'd try to do the same with vfio. The Xorg problem might also suggest there's an existing interrupt routing problem on the system. It is surprising that you have so many devices that doesn't support INTx disable, can you try running this script on them: #!/bin/sh # Usage $0 PCI device, ex: 9:00.0 INTX=$(( 0x400 )) ORIG=$(( 0x$(setpci -s $1 4.w) )) if [ $(( $INTX $ORIG )) -ne 0 ]; then echo INTx disable supported and enabled on $1 exit 0 fi NEW=$(printf %04x $(( $INTX | $ORIG ))) setpci -s $1 4.w=$NEW NEW=$(( 0x$(setpci -s $1 4.w) )) if [ $(( $INTX $NEW )) -ne 0 ]; then echo INTx disable support available on $1 else echo INTx disable support NOT available on $1 fi NEW=$(printf %04x $ORIG) setpci -s $1 4.w=$NEW After cold boot, without any attempts to run VM: Video card: INTx disable support available on 01:00.0 HDMI Audio: INTx disable supported and enabled on 01:00.1 USB: INTx disable supported and enabled on 09:00.0 PEX8112 (sound card sits behind this): INTx disable support available on 06:00.0) Sound card: INTx disable support NOT available on 07:04.0 If the sound card is the only one lacking this feature, then it seems odd that the USB controller is the one I have to remove for the VM to boot. Please report the output of: $ cat /sys/module/vfio_pci/parameters/nointxmask I don't understand why your flags are 0 for all the devices, only the sound card should have that. The lspci -nnvv listing for my current sound card is quite sparse: 07:04.0 Multimedia audio controller [0401]: C-Media Electronics Inc CMI8788 [Oxygen HD Audio] [13f6:8788] Subsystem: ASUSTeK Computer Inc. Virtuoso 100 (Xonar D1) [1043:834f] Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium TAbort- TAbort- MAbort- SERR- PERR- INTx- Interrupt: pin A routed to IRQ 10 Region 0: I/O ports at c000 [disabled] [size=256] Capabilities: [c0] Power Management version 2 Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- I'm willing to purchase a different sound card if that will fix the problem. Does lspci (-v or -vv) tell you if INTx Disable is available? No, lspci only reports the state of the DisINTx bit, not whether it's supported. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Looks like I had 'nointxmask' set to 1 for some reason (in /etc/modprobe.d/custom.conf). With nointxmask NOT set, the conflict messages change: genirq: Flags mismatch irq 17. (vfio-intx(:07:04.0)) vs. 0080 (vfio-intx(:01:00.1)) The latter is the HDMI audio. Removing it from the VM allows the VM to start. It would still be nice for the message to say something more descriptive. The flag meanings are not obvious, and when both sides are the same, it doesn't seem like a mismatch
Re: IOMMU groups ... PEX8606 switch?
On 01/03/2014 04:03 PM, Alex Williamson wrote: On Mon, 2013-12-30 at 16:13 -0800, Dana Goyette wrote: On 12/29/2013 08:16 PM, Alex Williamson wrote: On Sat, 2013-12-28 at 23:32 -0800, Dana Goyette wrote: On 12/28/2013 7:23 PM, Alex Williamson wrote: On Sat, 2013-12-28 at 18:31 -0800, Dana Goyette wrote: I have purchased both a SuperMicro X10SAE and an X10SAT, and I need to soon decide which one to keep. The SuperMicro X10SAT has all the PCIe x1 slots hidden behind a PLX PEX8066 switch, which claims to support ACS. I'd expect the devices downstream of the PLX switch to be in separate groups. With Linux 3.13-rc5 and enable overrides for missing ACS capabilities applied and set for the Intel root ports, the devices behind the switch remain stuck in the same group. In terms of passing devices to different VMs, which is better: all devices on different root ports, or all devices behind the one ACS-supporting switch? Can you provide lspci -vvv info? If you're getting that for groups either the switch has ACS capabilities, but doesn't support the features we need or we're doing something wrong. Thanks, I initially tried attaching the output as a .txt file, but it's too large. Anyway, here's the output of lspci -nnvvv (you may notice that I moved the Radeon to a different slot). Well, something seems amiss since the downstream switch ports all seem to support and enable the correct set of ACS capabilities. I'm tending to suspect something wrong with the ACS override patch or how it's being used since your IOMMU group is still based at the root port. Each root port is isolated from the other root ports though, so something is happening with the override patch. Can you provide the kernel command line you use to enable ACS overrides and the override patch you're using, as it applies to 3.13-rc5? Thanks, Alex -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html I'm using the original acs-override patch from this post: https://lkml.org/lkml/2013/5/30/513 Kernel parameter is: pcie_acs_override=id:8086:8c10,id:8086:8c12,id:8086:8c16,id:8086:8c18 When booting a kernel without the override patch, the following devices are all in the same group: Intel Root Ports 1, 2, 4, 5; ASMedia SATA controller; PLX PEX8606 switch; Renesas USB controller; TI Firewire controller; Intel I210 Ethernet controller. Could you please try the patch below and send dmesg for the system once booted. This applies directly to upstream and includes the acs override patch. Thanks, (removed patch from quote.) Here's the complete dmesg, with pcie_acs_override still set: http://pastebin.com/YHuKnrTb Most relevant section: [0.524362] DMAR: No ATSR found [0.524386] IOMMU 1 0xfed91000: using Queued invalidation [0.524389] IOMMU: Setting RMRR: [0.524398] IOMMU: Setting identity map for device :00:1d.0 [0x7bea1000 - 0x7bea] [0.524423] IOMMU: Setting identity map for device :00:1a.0 [0x7bea1000 - 0x7bea] [0.524441] IOMMU: Setting identity map for device :00:14.0 [0x7bea1000 - 0x7bea] [0.524454] IOMMU: Prepare 0-16MiB unity mapping for LPC [0.524461] IOMMU: Setting identity map for device :00:1f.0 [0x0 - 0xff] [0.524548] PCI-DMA: Intel(R) Virtualization Technology for Directed I/O [0.524551] intel_iommu_add_device(:00:00.0) [0.524552] dma_pdev #1: :00:00.0 [0.524553] dma_pdev #2: :00:00.0 [0.524554] dma_pdev #3: :00:00.0 [0.524554] dma_pdev #4: :00:00.0 [0.524565] intel_iommu_add_device(:00:01.0) [0.524566] dma_pdev #1: :00:01.0 [0.524567] dma_pdev #2: :00:01.0 [0.524569] pci_acs_enabled(:00:01.0, 001d) [0.524572] pci_acs_flags_enabled no ACS capability on :00:01.0 [0.524573] pci_acs_flags_enabled(:00:01.0, 001d) - false [0.524574] - false [0.524575] pci_acs_enabled(:00:01.0, 001d) [0.524577] pci_acs_flags_enabled no ACS capability on :00:01.0 [0.524578] pci_acs_flags_enabled(:00:01.0, 001d) - false [0.524579] - false [0.524580] dma_pdev #3: :00:01.0 [0.524581] dma_pdev #4: :00:01.0 [0.524584] intel_iommu_add_device(:00:01.1) [0.524586] dma_pdev #1: :00:01.1 [0.524586] dma_pdev #2: :00:01.1 [0.524587] pci_acs_enabled(:00:01.1, 001d) [0.524589] pci_acs_flags_enabled no ACS capability on :00:01.1 [0.524590] pci_acs_flags_enabled(:00:01.1, 001d) - false [0.524591] - false [0.524592] pci_acs_enabled(:00:01.0, 001d) [0.524593] pci_acs_flags_enabled no ACS capability on :00:01.0 [0.524595] pci_acs_flags_enabled(:00:01.0, 001d) - false [0.524596] - false [0.524596] dma_pdev #3: :00:01.0 [0.524597] dma_pdev #4: :00:01.0 [0.524599] intel_iommu_add_device(:00:02.0) [0.524601] dma_pdev #1:
Re: IOMMU groups ... PEX8606 switch?
On 01/04/2014 12:22 PM, Alex Williamson wrote: On Sat, 2014-01-04 at 11:26 -0800, Dana Goyette wrote: On 01/03/2014 04:03 PM, Alex Williamson wrote: On Mon, 2013-12-30 at 16:13 -0800, Dana Goyette wrote: On 12/29/2013 08:16 PM, Alex Williamson wrote: On Sat, 2013-12-28 at 23:32 -0800, Dana Goyette wrote: On 12/28/2013 7:23 PM, Alex Williamson wrote: On Sat, 2013-12-28 at 18:31 -0800, Dana Goyette wrote: I have purchased both a SuperMicro X10SAE and an X10SAT, and I need to soon decide which one to keep. The SuperMicro X10SAT has all the PCIe x1 slots hidden behind a PLX PEX8066 switch, which claims to support ACS. I'd expect the devices downstream of the PLX switch to be in separate groups. With Linux 3.13-rc5 and enable overrides for missing ACS capabilities applied and set for the Intel root ports, the devices behind the switch remain stuck in the same group. In terms of passing devices to different VMs, which is better: all devices on different root ports, or all devices behind the one ACS-supporting switch? Can you provide lspci -vvv info? If you're getting that for groups either the switch has ACS capabilities, but doesn't support the features we need or we're doing something wrong. Thanks, I initially tried attaching the output as a .txt file, but it's too large. Anyway, here's the output of lspci -nnvvv (you may notice that I moved the Radeon to a different slot). Well, something seems amiss since the downstream switch ports all seem to support and enable the correct set of ACS capabilities. I'm tending to suspect something wrong with the ACS override patch or how it's being used since your IOMMU group is still based at the root port. Each root port is isolated from the other root ports though, so something is happening with the override patch. Can you provide the kernel command line you use to enable ACS overrides and the override patch you're using, as it applies to 3.13-rc5? Thanks, Alex -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html I'm using the original acs-override patch from this post: https://lkml.org/lkml/2013/5/30/513 Kernel parameter is: pcie_acs_override=id:8086:8c10,id:8086:8c12,id:8086:8c16,id:8086:8c18 Actually, you're not: pcie_acs_override=id:8086:8c10,id:8086:8c16,id:8086:8c18,id:8086:ac1a,id:8086:8c1c,id:8086:8c1e,id:10b5:8606 And we register all of them: [0.00] PCIe ACS bypass added for 8086:8c10 [0.00] PCIe ACS bypass added for 8086:8c16 [0.00] PCIe ACS bypass added for 8086:8c18 [0.00] PCIe ACS bypass added for 8086:ac1a [0.00] PCIe ACS bypass added for 8086:8c1c [0.00] PCIe ACS bypass added for 8086:8c1e [0.00] PCIe ACS bypass added for 10b5:8606 However, note that the root port causing you trouble is 8086:8c12, which isn't provided as an override, therefore the code is doing the right thing and grouping all devices behind that root port together. Thanks for catching that -- I certainly missed it! I've added the override for that root port and removed the override for the PLX switch; now all the ports are indeed in separate groups. Do we yet know if it'll be possible to properly isolate the Intel root ports, without this ACS override? -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: IOMMU groups ... PEX8606 switch?
On 01/02/2014 11:36 AM, Alex Williamson wrote: On Mon, 2013-12-30 at 16:13 -0800, Dana Goyette wrote: On 12/29/2013 08:16 PM, Alex Williamson wrote: On Sat, 2013-12-28 at 23:32 -0800, Dana Goyette wrote: On 12/28/2013 7:23 PM, Alex Williamson wrote: On Sat, 2013-12-28 at 18:31 -0800, Dana Goyette wrote: I have purchased both a SuperMicro X10SAE and an X10SAT, and I need to soon decide which one to keep. The SuperMicro X10SAT has all the PCIe x1 slots hidden behind a PLX PEX8066 switch, which claims to support ACS. I'd expect the devices downstream of the PLX switch to be in separate groups. With Linux 3.13-rc5 and enable overrides for missing ACS capabilities applied and set for the Intel root ports, the devices behind the switch remain stuck in the same group. In terms of passing devices to different VMs, which is better: all devices on different root ports, or all devices behind the one ACS-supporting switch? Can you provide lspci -vvv info? If you're getting that for groups either the switch has ACS capabilities, but doesn't support the features we need or we're doing something wrong. Thanks, I initially tried attaching the output as a .txt file, but it's too large. Anyway, here's the output of lspci -nnvvv (you may notice that I moved the Radeon to a different slot). Well, something seems amiss since the downstream switch ports all seem to support and enable the correct set of ACS capabilities. I'm tending to suspect something wrong with the ACS override patch or how it's being used since your IOMMU group is still based at the root port. Each root port is isolated from the other root ports though, so something is happening with the override patch. Can you provide the kernel command line you use to enable ACS overrides and the override patch you're using, as it applies to 3.13-rc5? Thanks, Alex -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html I'm using the original acs-override patch from this post: https://lkml.org/lkml/2013/5/30/513 Kernel parameter is: pcie_acs_override=id:8086:8c10,id:8086:8c12,id:8086:8c16,id:8086:8c18 When booting a kernel without the override patch, the following devices are all in the same group: Intel Root Ports 1, 2, 4, 5; ASMedia SATA controller; PLX PEX8606 switch; Renesas USB controller; TI Firewire controller; Intel I210 Ethernet controller. Ok, here's my shot in the dark; we must be detecting something about the upstream switch port to make it fail the ACS test and the only thing I can find that might do this is if the PCI config header on the upstream switch reported itself as a multifunction device. Multifunction upstream switch ports do need ACS capabilities to make sure that traffic isn't routed back through other functions. Single function devices do not. To test that theory, please provide 'lspci -vxs 4:00.0'. We're looking to see whether the byte at 0xe has the MSB set. If it does, it lies that it's a multifunction device. If it doesn't I'll have to get the dart board back out. FWIW, you should be able to work around this by adding id:10b5:8606 to your list of overrides. Long term, if this is the problem, we'll want to add a quirk to sanitize the multifunction device flag. Thanks, Alex 04:00.0 is the ASMedia SATA controller; I'll assume you meant the upstream port of the PLX switch. 05:00.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) (prog-if 00 [Normal decode]) Flags: bus master, fast devsel, latency 0 Memory at eeb0 (32-bit, non-prefetchable) [size=128K] Bus: primary=05, secondary=06, subordinate=0c, sec-latency=0 Memory behind bridge: ee80-eeaf Capabilities: [40] Power Management version 3 Capabilities: [48] MSI: Enable+ Count=1/4 Maskable+ 64bit+ Capabilities: [68] Express Upstream Port, MSI 00 Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch Capabilities: [100] Device Serial Number ba-86-01-10-b5-df-0e-00 Capabilities: [fb4] Advanced Error Reporting Capabilities: [138] Power Budgeting ? Capabilities: [148] Virtual Channel Capabilities: [448] Vendor Specific Information: ID= Rev=0 Len=0cc ? Capabilities: [950] Vendor Specific Information: ID=0001 Rev=0 Len=010 ? Kernel driver in use: pcieport 00: b5 10 06 86 47 05 10 40 ba 00 04 06 10 00 01 00 10: 00 00 b0 ee 00 00 00 00 05 06 0c 00 f1 01 00 00 20: 80 ee a0 ee f1 ff 01 00 00 00 00 00 00 00 00 00 30: 00 00 00 00 40 00 00 00 00 00 00 00 0a 01 13 00 -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: IOMMU groups ... PEX8606 switch?
On 01/02/2014 01:01 PM, Dana Goyette wrote: On 01/02/2014 11:36 AM, Alex Williamson wrote: On Mon, 2013-12-30 at 16:13 -0800, Dana Goyette wrote: On 12/29/2013 08:16 PM, Alex Williamson wrote: On Sat, 2013-12-28 at 23:32 -0800, Dana Goyette wrote: On 12/28/2013 7:23 PM, Alex Williamson wrote: On Sat, 2013-12-28 at 18:31 -0800, Dana Goyette wrote: I have purchased both a SuperMicro X10SAE and an X10SAT, and I need to soon decide which one to keep. The SuperMicro X10SAT has all the PCIe x1 slots hidden behind a PLX PEX8066 switch, which claims to support ACS. I'd expect the devices downstream of the PLX switch to be in separate groups. With Linux 3.13-rc5 and enable overrides for missing ACS capabilities applied and set for the Intel root ports, the devices behind the switch remain stuck in the same group. In terms of passing devices to different VMs, which is better: all devices on different root ports, or all devices behind the one ACS-supporting switch? Can you provide lspci -vvv info? If you're getting that for groups either the switch has ACS capabilities, but doesn't support the features we need or we're doing something wrong. Thanks, I initially tried attaching the output as a .txt file, but it's too large. Anyway, here's the output of lspci -nnvvv (you may notice that I moved the Radeon to a different slot). Well, something seems amiss since the downstream switch ports all seem to support and enable the correct set of ACS capabilities. I'm tending to suspect something wrong with the ACS override patch or how it's being used since your IOMMU group is still based at the root port. Each root port is isolated from the other root ports though, so something is happening with the override patch. Can you provide the kernel command line you use to enable ACS overrides and the override patch you're using, as it applies to 3.13-rc5? Thanks, Alex -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html I'm using the original acs-override patch from this post: https://lkml.org/lkml/2013/5/30/513 Kernel parameter is: pcie_acs_override=id:8086:8c10,id:8086:8c12,id:8086:8c16,id:8086:8c18 When booting a kernel without the override patch, the following devices are all in the same group: Intel Root Ports 1, 2, 4, 5; ASMedia SATA controller; PLX PEX8606 switch; Renesas USB controller; TI Firewire controller; Intel I210 Ethernet controller. Ok, here's my shot in the dark; we must be detecting something about the upstream switch port to make it fail the ACS test and the only thing I can find that might do this is if the PCI config header on the upstream switch reported itself as a multifunction device. Multifunction upstream switch ports do need ACS capabilities to make sure that traffic isn't routed back through other functions. Single function devices do not. To test that theory, please provide 'lspci -vxs 4:00.0'. We're looking to see whether the byte at 0xe has the MSB set. If it does, it lies that it's a multifunction device. If it doesn't I'll have to get the dart board back out. FWIW, you should be able to work around this by adding id:10b5:8606 to your list of overrides. Long term, if this is the problem, we'll want to add a quirk to sanitize the multifunction device flag. Thanks, Alex 04:00.0 is the ASMedia SATA controller; I'll assume you meant the upstream port of the PLX switch. 05:00.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) (prog-if 00 [Normal decode]) Flags: bus master, fast devsel, latency 0 Memory at eeb0 (32-bit, non-prefetchable) [size=128K] Bus: primary=05, secondary=06, subordinate=0c, sec-latency=0 Memory behind bridge: ee80-eeaf Capabilities: [40] Power Management version 3 Capabilities: [48] MSI: Enable+ Count=1/4 Maskable+ 64bit+ Capabilities: [68] Express Upstream Port, MSI 00 Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch Capabilities: [100] Device Serial Number ba-86-01-10-b5-df-0e-00 Capabilities: [fb4] Advanced Error Reporting Capabilities: [138] Power Budgeting ? Capabilities: [148] Virtual Channel Capabilities: [448] Vendor Specific Information: ID= Rev=0 Len=0cc ? Capabilities: [950] Vendor Specific Information: ID=0001 Rev=0 Len=010 ? Kernel driver in use: pcieport 00: b5 10 06 86 47 05 10 40 ba 00 04 06 10 00 01 00 10: 00 00 b0 ee 00 00 00 00 05 06 0c 00 f1 01 00 00 20: 80 ee a0 ee f1 ff 01 00 00 00 00 00 00 00 00 00 30: 00 00 00 00 40 00 00 00 00 00 00 00 0a 01 13 00 -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Just in case: The root port: 00:1c.1 PCI bridge
Re: IOMMU groups ... PEX8606 switch?
On 12/29/2013 08:16 PM, Alex Williamson wrote: On Sat, 2013-12-28 at 23:32 -0800, Dana Goyette wrote: On 12/28/2013 7:23 PM, Alex Williamson wrote: On Sat, 2013-12-28 at 18:31 -0800, Dana Goyette wrote: I have purchased both a SuperMicro X10SAE and an X10SAT, and I need to soon decide which one to keep. The SuperMicro X10SAT has all the PCIe x1 slots hidden behind a PLX PEX8066 switch, which claims to support ACS. I'd expect the devices downstream of the PLX switch to be in separate groups. With Linux 3.13-rc5 and enable overrides for missing ACS capabilities applied and set for the Intel root ports, the devices behind the switch remain stuck in the same group. In terms of passing devices to different VMs, which is better: all devices on different root ports, or all devices behind the one ACS-supporting switch? Can you provide lspci -vvv info? If you're getting that for groups either the switch has ACS capabilities, but doesn't support the features we need or we're doing something wrong. Thanks, I initially tried attaching the output as a .txt file, but it's too large. Anyway, here's the output of lspci -nnvvv (you may notice that I moved the Radeon to a different slot). Well, something seems amiss since the downstream switch ports all seem to support and enable the correct set of ACS capabilities. I'm tending to suspect something wrong with the ACS override patch or how it's being used since your IOMMU group is still based at the root port. Each root port is isolated from the other root ports though, so something is happening with the override patch. Can you provide the kernel command line you use to enable ACS overrides and the override patch you're using, as it applies to 3.13-rc5? Thanks, Alex -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html I'm using the original acs-override patch from this post: https://lkml.org/lkml/2013/5/30/513 Kernel parameter is: pcie_acs_override=id:8086:8c10,id:8086:8c12,id:8086:8c16,id:8086:8c18 When booting a kernel without the override patch, the following devices are all in the same group: Intel Root Ports 1, 2, 4, 5; ASMedia SATA controller; PLX PEX8606 switch; Renesas USB controller; TI Firewire controller; Intel I210 Ethernet controller. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
IOMMU groups: better with Intel root ports, or with PEX8606 switch?
I have purchased both a SuperMicro X10SAE and an X10SAT, and I need to soon decide which one to keep. The SuperMicro X10SAT has all the PCIe x1 slots hidden behind a PLX PEX8066 switch, which claims to support ACS. I'd expect the devices downstream of the PLX switch to be in separate groups. With Linux 3.13-rc5 and enable overrides for missing ACS capabilities applied and set for the Intel root ports, the devices behind the switch remain stuck in the same group. In terms of passing devices to different VMs, which is better: all devices on different root ports, or all devices behind the one ACS-supporting switch? X10SAT IOMMU groups: ### Group 0 ### 00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v3 Processor DRAM Controller (rev 06) ### Group 1 ### 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06) 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Juniper PRO [Radeon HD 5750] 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Juniper HDMI Audio [Radeon HD 5700 Series] ### Group 2 ### 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3 Processor Integrated Graphics Controller (rev 06) ### Group 3 ### 00:03.0 Audio device: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller (rev 06) ### Group 4 ### 00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 05) ### Group 5 ### 00:16.0 Communication controller: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 (rev 04) 00:16.3 Serial controller: Intel Corporation 8 Series/C220 Series Chipset Family KT Controller (rev 04) ### Group 6 ### 00:19.0 Ethernet controller: Intel Corporation Ethernet Connection I217-LM (rev 05) ### Group 7 ### 00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 (rev 05) ### Group 8 ### 00:1b.0 Audio device: Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller (rev 05) ### Group 9 ### 00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d5) ### Group 10 ### 00:1c.1 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #2 (rev d5) 03:00.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) 04:01.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) 04:04.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) 04:05.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) 04:07.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) 04:09.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) 08:00.0 USB controller: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller (rev 03) 09:00.0 PCI bridge: Texas Instruments XIO2213A/B/XIO2221 PCI Express to PCI Bridge [Cheetah Express] (rev 01) 0a:00.0 FireWire (IEEE 1394): Texas Instruments XIO2213A/B/XIO2221 IEEE-1394b OHCI Controller [Cheetah Express] (rev 01) ### Group 11 ### 00:1c.3 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 (rev d5) ### Group 12 ### 00:1c.4 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #5 (rev d5) ### Group 13 ### 00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 (rev 05) ### Group 14 ### 00:1f.0 ISA bridge: Intel Corporation C226 Series Chipset Family Server Advanced SKU LPC Controller (rev 05) 00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05) 00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller (rev 05) 00:1f.6 Signal processing controller: Intel Corporation 8 Series Chipset Family Thermal Management Controller (rev 05) ### Group 15 ### 02:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01) ### Group 16 ### 0b:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03) -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
