Re: [kvm] [PATCH 13/16] kvm: enable MSI-X capabilty for assigned device
Hi Sheng, On Wed, 2009-04-08 at 10:26 +0800, Sheng Yang wrote: On Wednesday 08 April 2009 00:38:10 Alex Williamson wrote: On Tue, 2009-04-07 at 14:09 +0800, Sheng Yang wrote: Could you enable DEVICE_ASSSIGNMENT_DEBUG=1 in qemu/hw/device-assignment.c and post the output? Yup, see below. The error comes after I 'ifdown eth0; ifup eth0' in the guest. Note bnx2 appears to only turn on MSIX for SMP systems. Thanks, Alex Seems your ifdown/ifup script reload the module? No, the bnx2 module isn't unloaded on ifdown. Oh god, I found one bug after checked the spec: System software reads this field to determine the MSI-X Table Size *N*, which is encoded as *N-1*. For example, a returned value of “011” indicates a table size of 4. But it seems still can't explain the problem...(OK, it may affect the guest in a unknown way as well...) I would post a fix for it soon. [snip] The writing to MMIO have been intercepted, but code fail to count it? Strange... Could you try this debug? I added the debug printfs, plus the MSI-X table size patch, and printed the value of msg_ctrl as we loop through. Output below. This is what made me think the MSI-X state isn't getting cleared when the driver closes the interface. Let me know what you think. Thanks, Alex init_assigned_device: Registering real physical device 03:00.0 (bus=3 dev=0 func=0) get_real_device: region 0 size 33554432 start 0xf400 type 512 resource_fd 19 assigned_dev_pci_read_config: (4.0): address= val=0x14e4 len=2 assigned_dev_pci_read_config: (4.0): address=0002 val=0x1639 len=2 assigned_dev_pci_read_config: (4.0): address= val=0x14e4 len=2 assigned_dev_pci_read_config: (4.0): address=0002 val=0x1639 len=2 assigned_dev_pci_read_config: (4.0): address= val=0x14e4 len=2 assigned_dev_pci_read_config: (4.0): address=0002 val=0x1639 len=2 assigned_dev_pci_read_config: (4.0): address=000a val=0x0200 len=2 assigned_dev_pci_read_config: (4.0): address= val=0x14e4 len=2 assigned_dev_pci_read_config: (4.0): address=0002 val=0x1639 len=2 assigned_dev_pci_write_config: (4.0): address=0010 val=0x len=4 assigned_dev_pci_read_config: (4.0): address=0010 val=0xfe00 len=4 assigned_dev_pci_read_config: (4.0): address=0010 val=0xfe00 len=4 assigned_dev_pci_write_config: (4.0): address=0010 val=0xf400 len=4 assigned_dev_iomem_map: e_phys=f400 r_virt=0x7f4fd9bfc000 type=0 len=0200 region_num=0 assigned_dev_iomem_map: munmap done, virt_base 0x0x7f4fd9c08000 assigned_dev_pci_read_config: (4.0): address=0004 val=0x0442 len=2 assigned_dev_pci_write_config: (4.0): address=0004 val=0x0442 len=2 assigned_dev_pci_write_config: NON BAR (4.0): address=0004 val=0x0442 len=2 assigned_dev_pci_write_config: (4.0): address=0014 val=0x len=4 assigned_dev_pci_read_config: (4.0): address=0014 val=0x len=4 assigned_dev_pci_write_config: (4.0): address=0018 val=0x len=4 assigned_dev_pci_read_config: (4.0): address=0018 val=0x len=4 assigned_dev_pci_write_config: (4.0): address=001c val=0x len=4 assigned_dev_pci_read_config: (4.0): address=001c val=0x len=4 assigned_dev_pci_write_config: (4.0): address=0020 val=0x len=4 assigned_dev_pci_read_config: (4.0): address=0020 val=0x len=4 assigned_dev_pci_write_config: (4.0): address=0024 val=0x len=4 assigned_dev_pci_read_config: (4.0): address=0024 val=0x len=4 assigned_dev_pci_write_config: (4.0): address=0030 val=0x len=4 assigned_dev_pci_write_config: NON BAR (4.0): address=0030 val=0x len=4 assigned_dev_pci_read_config: (4.0): address=0030 val=0x0001 len=4 assigned_dev_pci_read_config: (4.0): address=0030 val=0x0001 len=4 assigned_dev_pci_write_config: (4.0): address=0030 val=0x0001 len=4 assigned_dev_pci_write_config: NON BAR (4.0): address=0030 val=0x0001 len=4 assigned_dev_pci_read_config: (4.0): address=0004 val=0x0442 len=2 assigned_dev_pci_write_config: (4.0): address=0004 val=0x0442 len=2 assigned_dev_pci_write_config: NON BAR (4.0): address=0004 val=0x0442 len=2 assigned_dev_pci_read_config: (4.0): address=003d val=0x0001 len=1 assigned_dev_pci_write_config: (4.0): address=003c val=0x000b len=1 assigned_dev_pci_read_config: (4.0): address=000a val=0x0200 len=2 assigned_dev_pci_read_config: (4.0): address= val=0x14e4 len=2 assigned_dev_pci_read_config: (4.0): address=0002 val=0x1639 len=2 assigned_dev_pci_read_config: (4.0): address=000e val=0x0080 len=1 assigned_dev_pci_read_config: (4.0): address= val=0x163914e4 len=4 assigned_dev_pci_read_config: (4.0): address=000e val=0x0080 len=1 assigned_dev_pci_read_config: (4.0): address=0006 val=0x0010 len=2 assigned_dev_pci_read_config: (4.0): address=0034 val=0x0040 len=1 assigned_dev_pci_read_config: (4.0): address=0040 val=0x0005 len=1 assigned_dev_pci_read_config: (4.0):
Re: [kvm] [PATCH 13/16] kvm: enable MSI-X capabilty for assigned device
On Thursday 09 April 2009 00:13:56 Alex Williamson wrote: Hi Sheng, On Wed, 2009-04-08 at 10:26 +0800, Sheng Yang wrote: On Wednesday 08 April 2009 00:38:10 Alex Williamson wrote: On Tue, 2009-04-07 at 14:09 +0800, Sheng Yang wrote: Could you enable DEVICE_ASSSIGNMENT_DEBUG=1 in qemu/hw/device-assignment.c and post the output? Yup, see below. The error comes after I 'ifdown eth0; ifup eth0' in the guest. Note bnx2 appears to only turn on MSIX for SMP systems. Thanks, Alex Seems your ifdown/ifup script reload the module? No, the bnx2 module isn't unloaded on ifdown. Oh god, I found one bug after checked the spec: System software reads this field to determine the MSI-X Table Size *N*, which is encoded as *N-1*. For example, a returned value of “011” indicates a table size of 4. But it seems still can't explain the problem...(OK, it may affect the guest in a unknown way as well...) I would post a fix for it soon. [snip] The writing to MMIO have been intercepted, but code fail to count it? Strange... Could you try this debug? I added the debug printfs, plus the MSI-X table size patch, and printed the value of msg_ctrl as we loop through. Output below. This is what made me think the MSI-X state isn't getting cleared when the driver closes the interface. Let me know what you think. Thanks, Thanks Alex, now I know where the problem is. Part of functional haven't been implemented... address=0052 val=0x8008 len=2 the MSIX capabilty position is 0x50 the MSIX entries_max_nr is 0x9 0: msg_ctrl: 0001 1: msg_ctrl: 0001 2: msg_ctrl: 0001 3: msg_ctrl: 0001 4: msg_ctrl: 0001 5: msg_ctrl: 0001 6: msg_ctrl: 0001 7: msg_ctrl: 0001 8: msg_ctrl: 0001 MSI-X entry number is zero! assigned_dev_update_msix_mmio: No such device or address Driver write to the vectors at first, then enable MSI-X, msix_mmio_writel: write to MSI-X entry table mmio offset 0xc, val 0x0 msix_mmio_writel: write to MSI-X entry table mmio offset 0x1c, val 0x0 msix_mmio_writel: write to MSI-X entry table mmio offset 0x2c, val 0x0 msix_mmio_writel: write to MSI-X entry table mmio offset 0x3c, val 0x0 msix_mmio_writel: write to MSI-X entry table mmio offset 0x4c, val 0x0 msix_mmio_writel: write to MSI-X entry table mmio offset 0x5c, val 0x0 msix_mmio_writel: write to MSI-X entry table mmio offset 0x6c, val 0x0 msix_mmio_writel: write to MSI-X entry table mmio offset 0x7c, val 0x0 And finally clear the mask bit. For current we didn't implement mask capability in MSI-X vectors, so it won't work... OK. I'd like to remove the check of mask bit and only ignored unused vector when msg data is zero now(hope it won't cause more problems). And we would add support for per-vector mask later. Thanks for help to debug! -- regards Yang, Sheng -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [kvm] [PATCH 13/16] kvm: enable MSI-X capabilty for assigned device
On Saturday 04 April 2009 05:27:43 Alex Williamson wrote:
On Tue, 2009-03-17 at 11:50 +0800, Sheng Yang wrote:
+if (*ctrl_word PCI_MSIX_ENABLE) {
+if (assigned_dev_update_msix_mmio(pci_dev) 0) {
+perror(assigned_dev_update_msix_mmio);
+return;
+}
+if (kvm_assign_irq(kvm_context, assigned_irq_data) 0) {
+perror(assigned_dev_enable_msix: assign irq);
+return;
+}
+assigned_dev-irq_requested_type = assigned_irq_data.flags;
+}
+}
Do we need some disable logic here? If I toggle a bnx2 NIC in a guest,
I get the following when it attempts to come back up:
MSI-X entry number is zero!
assigned_dev_update_msix_mmio: No such device or address
It seems that driver didn't fill the MMIO with any correct MSIX information,
or the program fail to intercept it after driver set enable bit of MSIX. It's
strange... (Have it got something to do with PM and some EXP feature you
mentioned?)
Could you enable DEVICE_ASSSIGNMENT_DEBUG=1 in qemu/hw/device-assignment.c and
post the output?
Thanks!
--
regards
Yang, Sheng
Thanks,
Alex
--
To unsubscribe from this list: send the line unsubscribe kvm in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line unsubscribe kvm in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [kvm] [PATCH 13/16] kvm: enable MSI-X capabilty for assigned device
On Tue, 2009-04-07 at 14:09 +0800, Sheng Yang wrote: On Saturday 04 April 2009 05:27:43 Alex Williamson wrote: Do we need some disable logic here? If I toggle a bnx2 NIC in a guest, I get the following when it attempts to come back up: MSI-X entry number is zero! assigned_dev_update_msix_mmio: No such device or address It seems that driver didn't fill the MMIO with any correct MSIX information, or the program fail to intercept it after driver set enable bit of MSIX. It's strange... (Have it got something to do with PM and some EXP feature you mentioned?) My guess was that it filled in the MSIX info, but then can't find a free slot to reload the MSIX data when it tries to re-enable MSIX. I hacked the bnx2 driver to not rely on PM and EXP capabilities for this test, it seems to work, but it's possible that I broke something. My host also locks up the second time I try to export this device to a guest, maybe a problem with my bnx2 hacks, MSIX not getting reset, or prototype hardware. I'll see if I can find another MSIX capable device to export to a guest. Could you enable DEVICE_ASSSIGNMENT_DEBUG=1 in qemu/hw/device-assignment.c and post the output? Yup, see below. The error comes after I 'ifdown eth0; ifup eth0' in the guest. Note bnx2 appears to only turn on MSIX for SMP systems. Thanks, Alex $ sudo qemu-system-x86_64 -hda kvm-vtd.img -m 4096 -net none -vnc :1 -pcidevice host=03:00.0 -boot c -smp 8 init_assigned_device: Registering real physical device 03:00.0 (bus=3 dev=0 func=0) get_real_device: region 0 size 33554432 start 0xf400 type 512 resource_fd 19 assigned_dev_pci_read_config: (4.0): address= val=0x14e4 len=2 assigned_dev_pci_read_config: (4.0): address=0002 val=0x1639 len=2 assigned_dev_pci_read_config: (4.0): address= val=0x14e4 len=2 assigned_dev_pci_read_config: (4.0): address=0002 val=0x1639 len=2 assigned_dev_pci_read_config: (4.0): address= val=0x14e4 len=2 assigned_dev_pci_read_config: (4.0): address=0002 val=0x1639 len=2 assigned_dev_pci_read_config: (4.0): address=000a val=0x0200 len=2 assigned_dev_pci_read_config: (4.0): address= val=0x14e4 len=2 assigned_dev_pci_read_config: (4.0): address=0002 val=0x1639 len=2 assigned_dev_pci_write_config: (4.0): address=0010 val=0x len=4 assigned_dev_pci_read_config: (4.0): address=0010 val=0xfe00 len=4 assigned_dev_pci_read_config: (4.0): address=0010 val=0xfe00 len=4 assigned_dev_pci_write_config: (4.0): address=0010 val=0xf400 len=4 assigned_dev_iomem_map: e_phys=f400 r_virt=0x7f14f4ba type=0 len=0200 region_num=0 assigned_dev_iomem_map: munmap done, virt_base 0x0x7f14f4bac000 assigned_dev_pci_read_config: (4.0): address=0004 val=0x0442 len=2 assigned_dev_pci_write_config: (4.0): address=0004 val=0x0442 len=2 assigned_dev_pci_write_config: NON BAR (4.0): address=0004 val=0x0442 len=2 assigned_dev_pci_write_config: (4.0): address=0014 val=0x len=4 assigned_dev_pci_read_config: (4.0): address=0014 val=0x len=4 assigned_dev_pci_write_config: (4.0): address=0018 val=0x len=4 assigned_dev_pci_read_config: (4.0): address=0018 val=0x len=4 assigned_dev_pci_write_config: (4.0): address=001c val=0x len=4 assigned_dev_pci_read_config: (4.0): address=001c val=0x len=4 assigned_dev_pci_write_config: (4.0): address=0020 val=0x len=4 assigned_dev_pci_read_config: (4.0): address=0020 val=0x len=4 assigned_dev_pci_write_config: (4.0): address=0024 val=0x len=4 assigned_dev_pci_read_config: (4.0): address=0024 val=0x len=4 assigned_dev_pci_write_config: (4.0): address=0030 val=0x len=4 assigned_dev_pci_write_config: NON BAR (4.0): address=0030 val=0x len=4 assigned_dev_pci_read_config: (4.0): address=0030 val=0x0001 len=4 assigned_dev_pci_read_config: (4.0): address=0030 val=0x0001 len=4 assigned_dev_pci_write_config: (4.0): address=0030 val=0x0001 len=4 assigned_dev_pci_write_config: NON BAR (4.0): address=0030 val=0x0001 len=4 assigned_dev_pci_read_config: (4.0): address=0004 val=0x0442 len=2 assigned_dev_pci_write_config: (4.0): address=0004 val=0x0442 len=2 assigned_dev_pci_write_config: NON BAR (4.0): address=0004 val=0x0442 len=2 assigned_dev_pci_read_config: (4.0): address=003d val=0x0001 len=1 assigned_dev_pci_write_config: (4.0): address=003c val=0x000b len=1 assigned_dev_pci_read_config: (4.0): address=000a val=0x0200 len=2 assigned_dev_pci_read_config: (4.0): address= val=0x14e4 len=2 assigned_dev_pci_read_config: (4.0): address=0002 val=0x1639 len=2 assigned_dev_pci_read_config: (4.0): address=000e val=0x0080 len=1 assigned_dev_pci_read_config: (4.0): address= val=0x163914e4 len=4 assigned_dev_pci_read_config: (4.0): address=000e val=0x0080 len=1 assigned_dev_pci_read_config: (4.0): address=0006 val=0x0010 len=2 assigned_dev_pci_read_config: (4.0):
Re: [kvm] [PATCH 13/16] kvm: enable MSI-X capabilty for assigned device
On Wednesday 08 April 2009 00:38:10 Alex Williamson wrote: On Tue, 2009-04-07 at 14:09 +0800, Sheng Yang wrote: On Saturday 04 April 2009 05:27:43 Alex Williamson wrote: Do we need some disable logic here? If I toggle a bnx2 NIC in a guest, I get the following when it attempts to come back up: MSI-X entry number is zero! assigned_dev_update_msix_mmio: No such device or address It seems that driver didn't fill the MMIO with any correct MSIX information, or the program fail to intercept it after driver set enable bit of MSIX. It's strange... (Have it got something to do with PM and some EXP feature you mentioned?) My guess was that it filled in the MSIX info, but then can't find a free slot to reload the MSIX data when it tries to re-enable MSIX. I hacked the bnx2 driver to not rely on PM and EXP capabilities for this test, it seems to work, but it's possible that I broke something. My host also locks up the second time I try to export this device to a guest, maybe a problem with my bnx2 hacks, MSIX not getting reset, or prototype hardware. I'll see if I can find another MSIX capable device to export to a guest. Could you enable DEVICE_ASSSIGNMENT_DEBUG=1 in qemu/hw/device-assignment.c and post the output? Yup, see below. The error comes after I 'ifdown eth0; ifup eth0' in the guest. Note bnx2 appears to only turn on MSIX for SMP systems. Thanks, Alex Seems your ifdown/ifup script reload the module? Oh god, I found one bug after checked the spec: System software reads this field to determine the MSI-X Table Size *N*, which is encoded as *N-1*. For example, a returned value of “011” indicates a table size of 4. But it seems still can't explain the problem...(OK, it may affect the guest in a unknown way as well...) I would post a fix for it soon. val=0x0008 len=2 assigned_dev_pci_write_config: (4.0): address=0052 val=0x0008 len=2 assigned_dev_pci_read_config: (4.0): address=0006 val=0x0010 len=2 assigned_dev_pci_read_config: (4.0): address=0034 val=0x0040 len=1 assigned_dev_pci_read_config: (4.0): address=0040 val=0x0005 len=1 assigned_dev_pci_read_config: (4.0): address=0041 val=0x0050 len=1 assigned_dev_pci_read_config: (4.0): address=0050 val=0x0011 len=1 assigned_dev_pci_read_config: (4.0): address=0052 val=0x0008 len=2 assigned_dev_pci_read_config: (4.0): address=0054 val=0xc000 len=4 msix_mmio_writel: write to MSI-X entry table mmio offset 0x0, val 0xfeeff00c msix_mmio_writel: write to MSI-X entry table mmio offset 0x4, val 0x0 msix_mmio_writel: write to MSI-X entry table mmio offset 0x8, val 0x4191 msix_mmio_writel: write to MSI-X entry table mmio offset 0x10, val 0xfeeff00c msix_mmio_writel: write to MSI-X entry table mmio offset 0x14, val 0x0 msix_mmio_writel: write to MSI-X entry table mmio offset 0x18, val 0x4199 msix_mmio_writel: write to MSI-X entry table mmio offset 0x20, val 0xfeeff00c msix_mmio_writel: write to MSI-X entry table mmio offset 0x24, val 0x0 msix_mmio_writel: write to MSI-X entry table mmio offset 0x28, val 0x41a1 msix_mmio_writel: write to MSI-X entry table mmio offset 0x30, val 0xfeeff00c msix_mmio_writel: write to MSI-X entry table mmio offset 0x34, val 0x0 msix_mmio_writel: write to MSI-X entry table mmio offset 0x38, val 0x41a9 msix_mmio_writel: write to MSI-X entry table mmio offset 0x40, val 0xfeeff00c msix_mmio_writel: write to MSI-X entry table mmio offset 0x44, val 0x0 msix_mmio_writel: write to MSI-X entry table mmio offset 0x48, val 0x41b1 msix_mmio_writel: write to MSI-X entry table mmio offset 0x50, val 0xfeeff00c msix_mmio_writel: write to MSI-X entry table mmio offset 0x54, val 0x0 msix_mmio_writel: write to MSI-X entry table mmio offset 0x58, val 0x41b9 msix_mmio_writel: write to MSI-X entry table mmio offset 0x60, val 0xfeeff00c msix_mmio_writel: write to MSI-X entry table mmio offset 0x64, val 0x0 msix_mmio_writel: write to MSI-X entry table mmio offset 0x68, val 0x41c1 msix_mmio_writel: write to MSI-X entry table mmio offset 0x70, val 0xfeeff00c msix_mmio_writel: write to MSI-X entry table mmio offset 0x74, val 0x0 msix_mmio_writel: write to MSI-X entry table mmio offset 0x78, val 0x41c9 msix_mmio_writel: write to MSI-X entry table mmio offset 0x80, val 0xfeeff00c msix_mmio_writel: write to MSI-X entry table mmio offset 0x84, val 0x0 msix_mmio_writel: write to MSI-X entry table mmio offset 0x88, val 0x41d1 assigned_dev_pci_read_config: (4.0): address=0004 val=0x0046 len=2 assigned_dev_pci_write_config: (4.0): address=0004 val=0x0446 len=2 assigned_dev_pci_write_config: NON BAR (4.0): The writing to MMIO have been intercepted, but code fail to count it? Strange... Could you try this debug? diff --git a/qemu/hw/device-assignment.c b/qemu/hw/device-assignment.c index 09e54ae..ba31bed 100644 --- a/qemu/hw/device-assignment.c +++ b/qemu/hw/device-assignment.c @@ -45,7 +45,7 @@ #define IORESOURCE_DMA
Re: [kvm] [PATCH 13/16] kvm: enable MSI-X capabilty for assigned device
On Tue, 2009-03-17 at 11:50 +0800, Sheng Yang wrote:
+if (*ctrl_word PCI_MSIX_ENABLE) {
+if (assigned_dev_update_msix_mmio(pci_dev) 0) {
+perror(assigned_dev_update_msix_mmio);
+return;
+}
+if (kvm_assign_irq(kvm_context, assigned_irq_data) 0) {
+perror(assigned_dev_enable_msix: assign irq);
+return;
+}
+assigned_dev-irq_requested_type = assigned_irq_data.flags;
+}
+}
Do we need some disable logic here? If I toggle a bnx2 NIC in a guest,
I get the following when it attempts to come back up:
MSI-X entry number is zero!
assigned_dev_update_msix_mmio: No such device or address
Thanks,
Alex
--
To unsubscribe from this list: send the line unsubscribe kvm in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
