Donald I have been reported by another PF_RING user of the following problem (Dell 710 and Intel 82576):
Wed Sep 14 2011 06:00:11 An OEM diagnostic event has occurred. Critical 0.000009Wed Sep 14 2011 06:00:11 A bus fatal error was detected on a component at bus 0 device 6 function 0. Critical 0.000008Wed Sep 14 2011 06:00:11 A bus fatal error was detected on a component at slot 1. Normal 0.000007Wed Sep 14 2011 06:00:11 An OEM diagnostic event has occurred. Critical 0.000006Wed Sep 14 2011 06:00:11 A bus fatal error was detected on a component at bus 0 device 5 function 0. Critical 0.000005Wed Sep 14 2011 06:00:10 A bus fatal error was detected on a component at slot 2. Normal 0.000004Wed Sep 14 2011 06:00:08 An OEM diagnostic event has occurred. Critical 0.000003Wed Sep 14 2011 06:00:08 A bus fatal error was detected on a component at bus 0 device 6 function 0. Critical 0.000002Wed Sep 14 2011 06:00:08 A bus fatal error was detected on a component at slot 1. Normal 0.000001Wed Sep 14 2011 06:00:08 An OEM diagnostic event has occurred. Additionally, we captured the following logs as well: alloc kstat_irqs on node -1 pcieport 0000:00:09.0: irq 62 for MSI/MSI-X pcieport 0000:00:09.0: setting latency timer to 64 aer 0000:00:01.0:pcie02: PCIe errors handled by platform firmware. aer 0000:00:03.0:pcie02: PCIe errors handled by platform firmware. aer 0000:00:04.0:pcie02: PCIe errors handled by platform firmware. aer 0000:00:05.0:pcie02: PCIe errors handled by platform firmware. aer 0000:00:06.0:pcie02: PCIe errors handled by platform firmware. aer 0000:00:07.0:pcie02: PCIe errors handled by platform firmware. aer 0000:00:09.0:pcie02: PCIe errors handled by platform firmware. I believe there's a BIOS issue on Dell's. What do you think? Regards Luca On Sep 4, 2011, at 1:25 PM, Luca Deri wrote: > Donald > thanks for the reply. I don't think this is a PF_RING issue (even using the > vanilla ixgbe driver we observe the same behavior) but rather a Dell/Intel > issue. From what I see on dmesg, it seems that DCA is disabled and we have no > way to enable it. I'm not sure if this is due to BIOS limitations. What I can > tell you is that a low-end core2duo is much faster than this multiprocessor > machine, and this is an indication that there's something wrong on this setup. > > Regards Luca > > On Sep 3, 2011, at 2:33 AM, Skidmore, Donald C wrote: > >>> -----Original Message----- >>> From: andrew_leh...@agilent.com [mailto:andrew_leh...@agilent.com] >>> Sent: Thursday, September 01, 2011 2:17 AM >>> To: e1000-devel@lists.sourceforge.net >>> Cc: d...@ntop.org >>> Subject: [E1000-devel] Problems with Dell R810. >>> >>> Hi, >>> >>> I recently purchased as Dell R810 for use with Luca Deri's PF_RING >>> networking driver for the 10 Gigabit PCI Express Network Driver and the >>> Silicom 10Gig card that uses the 82599EB chipset, machine is running >>> Fedora Core 14. >>> >>> Luca's driver is described here: >>> http://www.ntop.org/blog/pf_ring/introducing-the-10-gbit-pf_ring-dna- >>> driver/ >>> >>> Only the machine doesn't seem to want to play ball. We have tried a >>> number of things and so eventually Luca suggested this mailing list, I >>> do hope someone can help? >>> >>> The machine spec is as follows. >>> >>> 2x Intel Xeon L7555 Processor (1.86GHz, 8C, 24M Cache, 5.86 GT/s QPI, >>> 95W TDP, Turbo, HT), DDR3-980MHz >>> 128GB Memory for 2/4CPU (16x8GB Quad Rank LV RDIMMs) 1066MHz >>> Additional 2x Intel Xeon L7555 Processor (1.86GHz, 8C, 24M Cache, 5.86 >>> GT/s QPI, 95W TDP, Turbo, HT), Upgrade to 4CPU >>> 2 600GB SAS 6Gbps 10k 2.5" HD >>> Silicom 82599EB 10 Gigabit Ethernet NIC. >>> >>> According to Luca's experiments on his test machine, not an R810 >>> (actually quite a low spec machine by comparison) we should be getting >>> the following results, unfortunately, the R810 performance is very poor; >>> it struggles at less than 8% capacity of a 10 Gig link on one core; >>> Luca's test application (byte and packet counts only) and his machine >>> can process a 100% of a 10 Gig Link on one core. >>> >>> http://www.ntop.org/blog/pf_ring/how-to-sendreceive-26mpps-using- >>> pf_ring-on-commodity-hardware/ >>> >>> Importantly, Luca also seems to be getting excellent CPU usage figures, >>> see the bottom of the page, indicating that both DCA and IOATDMA are >>> operating correctly. My problem is that even on light network loads my >>> CPU hits 100% and packets are dropped, indicating, to me, that >>> DCA/IOATDMA isn't working. >>> >>> I have switched on IOATDMA in the Dell's BIOS, it's off by default, and >>> discovered the following site where it talks about configuring a machine >>> to use DCA and IOATDMA etc. I even found a chap who reported similar >>> performance problems but with a Dell R710 and how he fixed it. I tried >>> all this but still no improvement! >>> >>> http://www.mail-archive.com/ntop-misc@listgateway.unipi.it/msg01185.html >>> >>> The R810 seems to use a 7500 chipset. >>> >>> http://www.dell.com/downloads/global/products/pedge/pedge_r810_specsheet >>> _en.pdf >>> >>> So, I think this is the R810 chipset reference http://www- >>> techdoc.intel.com/content/dam/doc/datasheet/7500-chipset-datasheet.pdf, >>> see page 453 >>> >>> The program sets bits (0x8C @ bit 0) but it doesn't seem to stay set, >>> so consecutive calls to "dca_probe" seem to always say "DCA disabled, >>> enabling now." >>> >>> I commented out some of the defines in the original code as they are >>> already set in the Linux kernel and, of course, changed the registers to >>> point to the ones on page 453 - I hope they are correct. >>> >>> Still no luck the CPU usage is way too high. >>> >>> #define _XOPEN_SOURCE 500 >>> >>> #include <stdio.h> >>> #include <stdlib.h> >>> #include <pci/pci.h> >>> #include <sys/io.h> >>> #include <fcntl.h> >>> #include <sys/stat.h> >>> #include <sys/types.h> >>> #include <unistd.h> >>> >>> #define INTEL_BRIDGE_DCAEN_OFFSET 0x8c >>> #define INTEL_BRIDGE_DCAEN_BIT 0 >>> /*#define PCI_HEADER_TYPE_BRIDGE 1 */ >>> /*#define PCI_VENDOR_ID_INTEL 0x8086 *//* lol @ intel */ >>> /*#define PCI_HEADER_TYPE 0x0e */ >>> #define MSR_P6_DCA_CAP 0x000001f8 >>> #define NUM_CPUS 64 >>> >>> void check_dca(struct pci_dev *dev) >>> { >>> u32 dca = pci_read_long(dev, INTEL_BRIDGE_DCAEN_OFFSET); >>> printf("DCA old value %d.\n", dca); >>> if (!(dca & (1 << INTEL_BRIDGE_DCAEN_BIT))) { >>> printf("DCA disabled, enabling now.\n"); >>> dca |= 1 << INTEL_BRIDGE_DCAEN_BIT; >>> printf("DCA new value %d.\n", dca); >>> pci_write_long(dev, INTEL_BRIDGE_DCAEN_OFFSET, dca); >>> } else { >>> printf("DCA already enabled!\n"); >>> } >>> } >>> >>> void msr_dca_enable(void) >>> { >>> char msr_file_name[64]; >>> int fd = 0, i = 0; >>> u64 data; >>> >>> for (;i < NUM_CPUS; i++) { >>> sprintf(msr_file_name, "/dev/cpu/%d/msr", i); >>> fd = open(msr_file_name, O_RDWR); >>> if (fd < 0) { >>> perror("open failed!"); >>> exit(1); >>> } >>> if (pread(fd, &data, sizeof(data), MSR_P6_DCA_CAP) != >>> sizeof(data)) { >>> perror("reading msr failed!"); >>> exit(1); >>> } >>> >>> printf("got msr value: %*llx\n", 1, (unsigned long >>> long)data); >>> if (!(data & 1)) { >>> data |= 1; >>> if (pwrite(fd, &data, sizeof(data), MSR_P6_DCA_CAP) != >>> sizeof(data)) { >>> perror("writing msr failed!"); >>> exit(1); >>> } >>> } else { >>> printf("msr already enabled for CPU %d\n", i); >>> } >>> } >>> } >>> >>> int main(void) >>> { >>> struct pci_access *pacc; >>> struct pci_dev *dev; >>> u8 type; >>> >>> pacc = pci_alloc(); >>> pci_init(pacc); >>> >>> pci_scan_bus(pacc); >>> for (dev = pacc->devices; dev; dev=dev->next) { >>> pci_fill_info(dev, PCI_FILL_IDENT | PCI_FILL_BASES); >>> if (dev->vendor_id == PCI_VENDOR_ID_INTEL) { >>> type = pci_read_byte(dev, PCI_HEADER_TYPE); >>> if (type == PCI_HEADER_TYPE_BRIDGE) { >>> check_dca(dev); >>> } >>> } >>> } >>> >>> msr_dca_enable(); >>> return 0; >>> } >>> >>> As you can see ixgbe, dca and ioatdma modules are loaded. >>> >>> # lsmod >>> >>> Module Size Used by >>> ixgbe 200547 0 >>> pf_ring 327754 4 >>> tcp_lp 2111 0 >>> fuse 61934 3 >>> sunrpc 201569 1 >>> ip6t_REJECT 4263 2 >>> nf_conntrack_ipv6 18078 4 >>> ip6table_filter 1687 1 >>> ip6_tables 17497 1 ip6table_filter >>> ipv6 286505 184 ip6t_REJECT,nf_conntrack_ipv6 >>> uinput 7368 0 >>> ioatdma 51376 72 >>> i7core_edac 16210 0 >>> dca 5590 2 ixgbe,ioatdma >>> bnx2 65569 0 >>> mdio 3934 0 >>> ses 6319 0 >>> dcdbas 8540 0 >>> edac_core 41336 1 i7core_edac >>> iTCO_wdt 11256 0 >>> iTCO_vendor_support 2610 1 iTCO_wdt >>> power_meter 9545 0 >>> hed 2206 0 >>> serio_raw 4640 0 >>> microcode 18662 0 >>> enclosure 7518 1 ses >>> megaraid_sas 37653 2 >>> >>> # uname -a >>> Linux test 2.6.35.14-95.fc14.x86_64 #1 SMP Tue Aug 16 21:01:58 UTC 2011 >>> x86_64 x86_64 x86_64 GNU/Linux >>> >>> Thanks, >>> >>> Andrew >> >> Hey Andrew, >> >> Sorry you're having issues with the 28599 and ixgbe. I haven't done much >> with the PF_RING networking driver but maybe we can see what is going on >> with the ixgbe driver. It would help to know a little be more information >> like: >> >> - What there any interesting system log messages of note? >> >> - How are your interrupt being divided among your queue's (cat >> /proc/interrupts)? I know your testing with just one CPU are you also just >> using one queue or affinizing one to that CPU? >> >> - Could you provide the lspic -vvv output. To verify you NIC is getting a >> PCIe x8 connection. >> >> - What kind of cpu usage are you seeing if you don't use just the base >> driver running at line rate with something like netperf/iperf? >> >> - Have you attempted this without DCA? Like I said above I don't have much >> experience with PF_RING so I may be missing some fundamental advantage it is >> suppose to gain from operation with DCA in this mode. >> >> These are just off the top of my head if I think of anything else I'll let >> you know. >> >> Thanks, >> -Don Skidmore <donald.c.skidm...@intel.com> > > --- > > "Debugging is twice as hard as writing the code in the first place. > Therefore, if you write the code as cleverly as possible, you are, by > definition, not smart enough to debug it. - Brian W. Kernighan > ------------------------------------------------------------------------------ Doing More with Less: The Next Generation Virtual Desktop What are the key obstacles that have prevented many mid-market businesses from deploying virtual desktops? How do next-generation virtual desktops provide companies an easier-to-deploy, easier-to-manage and more affordable virtual desktop model.http://www.accelacomm.com/jaw/sfnl/114/51426474/ _______________________________________________ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired
