64-bit hypervisor, 32-bit Dom0 kernel with customizing patches in the areas of
md, nbd and loop.
Also running with these relevant patches:
https://lkml.org/lkml/2011/9/1/100, "[PATCH v2] xen: x86_32: do not enable
iterrupts when returning from exception in interrupt context"
http://xen.1045712.n5.nabble.com/PATCH-mm-sync-vmalloc-address-space-page-tables-in-alloc-vm-area-td4757995.html
"[PATCH] mm: sync vmalloc address space page tables in alloc_vm_area()" (this
patch was reverted for 3.1 but this is 3.0.4)
and an additional 2048 NR_IRQS to support (as I understand it) all the virtual
devices we might support with 50 guests.
# uname -r
3.0.4-33.xen0
Neal
From: Dave, Tushar N [mailto:[email protected]]
Sent: Monday, December 05, 2011 2:25 PM
To: Taylor, Neal E; [email protected]; Brandeburg, Jesse
Cc: Quach, Minh T
Subject: RE: No usable DMA configuration: 82541GI
Are you running 64 bit kernel? uname -r ?
-Tushar
From: Taylor, Neal E [mailto:[email protected]]
Sent: Monday, December 05, 2011 7:24 AM
To: Dave, Tushar N; [email protected]; Brandeburg, Jesse
Cc: Quach, Minh T
Subject: RE: No usable DMA configuration: 82541GI
The only mods I have there are debug statements. That section of code looks
like:
/*
* there is a workaround being applied below that limits
* 64-bit DMA addresses to 64-bit hardware. There are some
* 32-bit adapters that Tx hang when given 64-bit DMA addresses
*/
pci_using_dac = 0;
printk(KERN_ERR "NT_DBG: hw->bus_type:%d, e1000_bus_type_pcix: %d\n",
hw->bus_type, e1000_bus_type_pcix);
if ((hw->bus_type == e1000_bus_type_pcix) &&
!dma_set_mask(&pdev->dev, DMA_BIT_MASK(64))) {
/*
* according to DMA-API-HOWTO, coherent calls will always
* succeed if the set call did
*/
dma_set_coherent_mask(&pdev->dev, DMA_BIT_MASK(64));
pci_using_dac = 1;
} else {
err = dma_set_mask(&pdev->dev, DMA_BIT_MASK(32));
if (err) {
pr_err("No usable DMA config, aborting\n");
goto err_dma;
}
dma_set_coherent_mask(&pdev->dev, DMA_BIT_MASK(32));
}
From: Dave, Tushar N [mailto:[email protected]]
Sent: Sunday, December 04, 2011 1:17 PM
To: Taylor, Neal E; [email protected]; Brandeburg, Jesse
Cc: Quach, Minh T
Subject: RE: No usable DMA configuration: 82541GI
Neal,
Just to be sure, you have in-kernel e1000 driver without any modifications,
right?
I asked because e1000 driver have bus specific dma APIs (e.g pci_dma*) already
replaced with dma API.
e.g driver uses dma_set_mask() instead of pci_set_dma_mask()
-Tushar
From: Taylor, Neal E [mailto:[email protected]]
Sent: Friday, December 02, 2011 4:47 PM
To: [email protected]; Dave, Tushar N; Brandeburg, Jesse
Cc: Taylor, Neal E; Quach, Minh T
Subject: No usable DMA configuration: 82541GI
I'm working on a system with a pair of 82541GI cards where, on bare metal it
works but under Xen 4.1.1 I get "e1000: No usable DMA config, aborting."
ata_piix and pata_acpi drivers are also unable to use interrupts but have
viable fallbacks.
One of the relevant sections of code was created with this patch:
From: Jesse Brandeburg
<[email protected]<mailto:[email protected]>>
commit e508be174ad36b0cf9b324cd04978c2b13c21502 upstream.
Several users report issues with 32-bit adapters when plugged
into PCI slots in machines with >= 4GB ram.
<snip>
+ /*
+ * there is a workaround being applied below that limits
+ * 64-bit DMA addresses to 64-bit hardware. There are some
+ * 32-bit adapters that Tx hang when given 64-bit DMA addresses
+ */
+ pci_using_dac = 0;
+ if ((hw->bus_type == e1000_bus_type_pcix) &&
+ !pci_set_dma_mask(pdev, DMA_BIT_MASK(64))) {
+ /*
+ * according to DMA-API-HOWTO, coherent calls will always
+ * succeed if the set call did
+ */
+ pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(64));
+ pci_using_dac = 1;
+ } else if (!pci_set_dma_mask(pdev, DMA_BIT_MASK(32))) {
+ pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(32));
+ } else {
+ E1000_ERR("No usable DMA configuration, aborting\n");
+ goto err_dma;
+ }
The problem seems to be that DMA_BIT_MASK(32) is selected. Within
pci_set_dma_mask, it compares that mask to "xen_virt_to_bus(xen_io_tlb_end -
1)" which returns the value "0x 1,20fd,feff." If I hardcode the selection to a
DMA_BIT_MASK(64), it works.
When running on bare metal, "ops->dma_supported" is zero, so this code is never
exercised.
I'm working on Linux 3.0.4 but have tested with Linux 3.1 and see the same
problem.
config file attached.
lspci below.
/var/log/messages snippet below.
What's the correct solution: refining the "workaround" to recognize specific
adapters that will work (or will fail)? Something that will allow interrupts to
work for other drivers as well?
Neal Taylor
Principal Software Engineer
CA Technologies
# cat /tmp/lspci
00:00.0 Host bridge: Intel Corporation E7520 Memory Controller Hub (rev 09)
00:02.0 PCI bridge: Intel Corporation E7525/E7520/E7320 PCI Express Port A (rev
09)
00:04.0 PCI bridge: Intel Corporation E7525/E7520 PCI Express Port B (rev 09)
00:05.0 PCI bridge: Intel Corporation E7520 PCI Express Port B1 (rev 09)
00:06.0 PCI bridge: Intel Corporation E7520 PCI Express Port C (rev 09)
00:1d.0 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) USB UHCI
Controller #1 (rev 02)
00:1d.1 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) USB UHCI
Controller #2 (rev 02)
00:1d.2 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) USB UHCI
Controller #3 (rev 02)
00:1d.7 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) USB2 EHCI
Controller (rev 02)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev c2)
00:1f.0 ISA bridge: Intel Corporation 82801EB/ER (ICH5/ICH5R) LPC Interface
Bridge (rev 02)
00:1f.1 IDE interface: Intel Corporation 82801EB/ER (ICH5/ICH5R) IDE Controller
(rev 02)
01:00.0 PCI bridge: Intel Corporation 80332 [Dobson] I/O processor (A-Segment
Bridge) (rev 06)
01:00.2 PCI bridge: Intel Corporation 80332 [Dobson] I/O processor (B-Segment
Bridge) (rev 06)
02:05.0 SCSI storage controller: LSI Logic / Symbios Logic 53c1030 PCI-X
Fusion-MPT Dual Ultra320 SCSI (rev 08)
05:00.0 PCI bridge: Intel Corporation 6700PXH PCI Express-to-PCI Bridge A (rev
09)
05:00.2 PCI bridge: Intel Corporation 6700PXH PCI Express-to-PCI Bridge B (rev
09)
06:07.0 Ethernet controller: Intel Corporation 82541GI Gigabit Ethernet
Controller (rev 05)
07:08.0 Ethernet controller: Intel Corporation 82541GI Gigabit Ethernet
Controller (rev 05)
09:05.0 Class ff00: Dell Remote Access Card 4 Daughter Card
09:05.1 Class ff00: Dell Remote Access Card 4 Daughter Card Virtual UART
09:05.2 Class ff00: Dell Remote Access Card 4 Daughter Card SMIC interface
09:06.0 IDE interface: Silicon Image, Inc. PCI0680 Ultra ATA-133 Host
Controller (rev 02)
09:0d.0 VGA compatible controller: ATI Technologies Inc Radeon RV100 QY [Radeon
7000/VE]
# cat /tmp/lspci-n
00:00.0 0600: 8086:3590 (rev 09)
00:02.0 0604: 8086:3595 (rev 09)
00:04.0 0604: 8086:3597 (rev 09)
00:05.0 0604: 8086:3598 (rev 09)
00:06.0 0604: 8086:3599 (rev 09)
00:1d.0 0c03: 8086:24d2 (rev 02)
00:1d.1 0c03: 8086:24d4 (rev 02)
00:1d.2 0c03: 8086:24d7 (rev 02)
00:1d.7 0c03: 8086:24dd (rev 02)
00:1e.0 0604: 8086:244e (rev c2)
00:1f.0 0601: 8086:24d0 (rev 02)
00:1f.1 0101: 8086:24db (rev 02)
01:00.0 0604: 8086:0330 (rev 06)
01:00.2 0604: 8086:0332 (rev 06)
02:05.0 0100: 1000:0030 (rev 08)
05:00.0 0604: 8086:0329 (rev 09)
05:00.2 0604: 8086:032a (rev 09)
06:07.0 0200: 8086:1076 (rev 05)
07:08.0 0200: 8086:1076 (rev 05)
09:05.0 ff00: 1028:0011
09:05.1 ff00: 1028:0012
09:05.2 ff00: 1028:0014
09:06.0 0101: 1095:0680 (rev 02)
09:0d.0 0300: 1002:5159
/var/log/messages:
.
.
.
Dec 1 15:55:38 Gridos_Node kernel: [ 23.289513] e1000: Copyright (c)
1999-2006 Intel Corporation.
Dec 1 15:55:38 Gridos_Node kernel: [ 23.289679] e1000 0000:06:07.0: PCI INT
A -> GSI 64 (level, low) -> IRQ 64
Dec 1 15:55:38 Gridos_Node kernel: [ 23.290203] NT_DBG:
e1000_get_bus_info:bus_info status: 52099
Dec 1 15:55:38 Gridos_Node kernel: [ 23.290213] NT_DBG: hw->but_type:1,
e1000_bus_type_pcix: 2
Dec 1 15:55:38 Gridos_Node kernel: [ 23.290222] Area 2a mask:
00000000ffffffff; ops->dma_supported: (null).
Dec 1 15:55:38 Gridos_Node kernel: [ 23.290230] Area2a: dma_supported:
c11528ba.
Dec 1 15:55:38 Gridos_Node kernel: [ 23.290240] Area 2c char *: 4,
phys_addr_t: 8, dma_addr_t: 8.
Dec 1 15:55:38 Gridos_Node kernel: [ 23.290249] Area 2c xen_io_tlb_end:
dc32cf00, converted to 'phys': 1c32cf00, then 'bus': 120fdfeff for answer 1.
Dec 1 15:55:38 Gridos_Node kernel: [ 23.290257] Area 2b returning 0 (or is
that 0.)
Dec 1 15:55:38 Gridos_Node kernel: [ 23.290265] e1000: No usable DMA config,
aborting
Dec 1 15:55:38 Gridos_Node kernel: [ 23.290701] e1000 0000:06:07.0: PCI INT
A disabled
Dec 1 15:55:38 Gridos_Node kernel: [ 23.290727] e1000: probe of 0000:06:07.0
failed with error -5
Dec 1 15:55:38 Gridos_Node kernel: [ 23.290812] e1000 0000:07:08.0: PCI INT
A -> GSI 65 (level, low) -> IRQ 65
Dec 1 15:55:38 Gridos_Node kernel: [ 23.291226] NT_DBG:
e1000_get_bus_info:bus_info status: 52097
Dec 1 15:55:38 Gridos_Node kernel: [ 23.291237] NT_DBG: hw->but_type:1,
e1000_bus_type_pcix: 2
Dec 1 15:55:38 Gridos_Node kernel: [ 23.291246] Area 2a mask:
00000000ffffffff; ops->dma_supported: (null).
Dec 1 15:55:38 Gridos_Node kernel: [ 23.291253] Area2a: dma_supported:
c11528ba.
Dec 1 15:55:38 Gridos_Node kernel: [ 23.291261] Area 2c char *: 4,
phys_addr_t: 8, dma_addr_t: 8.
Dec 1 15:55:38 Gridos_Node kernel: [ 23.291270] Area 2c xen_io_tlb_end:
dc32cf00, converted to 'phys': 1c32cf00, then 'bus': 120fdfeff for answer 1.
Dec 1 15:55:38 Gridos_Node kernel: [ 23.291278] Area 2b returning 0 (or is
that 0.)
Dec 1 15:55:38 Gridos_Node kernel: [ 23.291284] e1000: No usable DMA config,
aborting
Dec 1 15:55:38 Gridos_Node kernel: [ 23.291716] e1000 0000:07:08.0: PCI INT
A disabled
Dec 1 15:55:38 Gridos_Node kernel: [ 23.291734] e1000: probe of 0000:07:08.0
failed with error -5
.
.
.
------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure
contains a definitive record of customers, application performance,
security threats, fraudulent activity, and more. Splunk takes this
data and makes sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-novd2d
_______________________________________________
E1000-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel® Ethernet, visit
http://communities.intel.com/community/wired
