Hi Donald and Luca, I have managed to obtain the loan a R710 and using the Silicom card and Luca's code I can send in excess of 14Million packets per sec, so whatever the problem with the R710 Luca has reported it is not the same as my issues with the R810! Of course, unless my R810 has suffered the same fault as the R710 listed below and both are now broken in the same way. Does a reboot clear your other user's problem Luca or is it permanent?
Luca here's the details... ./pfsend -i dna:eth4 -g 1 -l 60 -n 0 -r 10 TX rate: [current 14'238'148.23 pps/9.57 Gbps][average 14'223'555.75 pps/9.56 Gbps][total 2'147'799'248.00 pkts] TX rate: [current 14'240'502.43 pps/9.57 Gbps][average 14'223'667.24 pps/9.56 Gbps][total 2'162'040'021.00 pkts] TX rate: [current 14'239'155.21 pps/9.57 Gbps][average 14'223'768.47 pps/9.56 Gbps][total 2'176'279'461.00 pkts] TX rate: [current 14'238'531.22 pps/9.57 Gbps][average 14'223'864.33 pps/9.56 Gbps][total 2'190'518'277.00 pkts] Thanks Andrew -----Original Message----- From: Luca Deri [mailto:d...@ntop.org] Sent: Thursday, September 15, 2011 3:05 PM To: Skidmore, Donald C Cc: LEHANE,ANDREW (A-Scotland,ex1); e1000-devel@lists.sourceforge.net Subject: Re: Problems with Dell R810. Donald I have been reported by another PF_RING user of the following problem (Dell 710 and Intel 82576): Wed Sep 14 2011 06:00:11 An OEM diagnostic event has occurred. Critical 0.000009Wed Sep 14 2011 06:00:11 A bus fatal error was detected on a component at bus 0 device 6 function 0. Critical 0.000008Wed Sep 14 2011 06:00:11 A bus fatal error was detected on a component at slot 1. Normal 0.000007Wed Sep 14 2011 06:00:11 An OEM diagnostic event has occurred. Critical 0.000006Wed Sep 14 2011 06:00:11 A bus fatal error was detected on a component at bus 0 device 5 function 0. Critical 0.000005Wed Sep 14 2011 06:00:10 A bus fatal error was detected on a component at slot 2. Normal 0.000004Wed Sep 14 2011 06:00:08 An OEM diagnostic event has occurred. Critical 0.000003Wed Sep 14 2011 06:00:08 A bus fatal error was detected on a component at bus 0 device 6 function 0. Critical 0.000002Wed Sep 14 2011 06:00:08 A bus fatal error was detected on a component at slot 1. Normal 0.000001Wed Sep 14 2011 06:00:08 An OEM diagnostic event has occurred. Additionally, we captured the following logs as well: alloc kstat_irqs on node -1 pcieport 0000:00:09.0: irq 62 for MSI/MSI-X pcieport 0000:00:09.0: setting latency timer to 64 aer 0000:00:01.0:pcie02: PCIe errors handled by platform firmware. aer 0000:00:03.0:pcie02: PCIe errors handled by platform firmware. aer 0000:00:04.0:pcie02: PCIe errors handled by platform firmware. aer 0000:00:05.0:pcie02: PCIe errors handled by platform firmware. aer 0000:00:06.0:pcie02: PCIe errors handled by platform firmware. aer 0000:00:07.0:pcie02: PCIe errors handled by platform firmware. aer 0000:00:09.0:pcie02: PCIe errors handled by platform firmware. I believe there's a BIOS issue on Dell's. What do you think? Regards Luca On Sep 4, 2011, at 1:25 PM, Luca Deri wrote: > Donald > thanks for the reply. I don't think this is a PF_RING issue (even using the > vanilla ixgbe driver we observe the same behavior) but rather a Dell/Intel > issue. From what I see on dmesg, it seems that DCA is disabled and we have no > way to enable it. I'm not sure if this is due to BIOS limitations. What I can > tell you is that a low-end core2duo is much faster than this multiprocessor > machine, and this is an indication that there's something wrong on this setup. > > Regards Luca > > On Sep 3, 2011, at 2:33 AM, Skidmore, Donald C wrote: > >>> -----Original Message----- >>> From: andrew_leh...@agilent.com [mailto:andrew_leh...@agilent.com] >>> Sent: Thursday, September 01, 2011 2:17 AM >>> To: e1000-devel@lists.sourceforge.net >>> Cc: d...@ntop.org >>> Subject: [E1000-devel] Problems with Dell R810. >>> >>> Hi, >>> >>> I recently purchased as Dell R810 for use with Luca Deri's PF_RING >>> networking driver for the 10 Gigabit PCI Express Network Driver and >>> the Silicom 10Gig card that uses the 82599EB chipset, machine is >>> running Fedora Core 14. >>> >>> Luca's driver is described here: >>> http://www.ntop.org/blog/pf_ring/introducing-the-10-gbit-pf_ring-dna >>> - >>> driver/ >>> >>> Only the machine doesn't seem to want to play ball. We have tried a >>> number of things and so eventually Luca suggested this mailing list, >>> I do hope someone can help? >>> >>> The machine spec is as follows. >>> >>> 2x Intel Xeon L7555 Processor (1.86GHz, 8C, 24M Cache, 5.86 GT/s >>> QPI, 95W TDP, Turbo, HT), DDR3-980MHz 128GB Memory for 2/4CPU >>> (16x8GB Quad Rank LV RDIMMs) 1066MHz Additional 2x Intel Xeon L7555 >>> Processor (1.86GHz, 8C, 24M Cache, 5.86 GT/s QPI, 95W TDP, Turbo, >>> HT), Upgrade to 4CPU >>> 2 600GB SAS 6Gbps 10k 2.5" HD >>> Silicom 82599EB 10 Gigabit Ethernet NIC. >>> >>> According to Luca's experiments on his test machine, not an R810 >>> (actually quite a low spec machine by comparison) we should be >>> getting the following results, unfortunately, the R810 performance >>> is very poor; it struggles at less than 8% capacity of a 10 Gig link >>> on one core; Luca's test application (byte and packet counts only) >>> and his machine can process a 100% of a 10 Gig Link on one core. >>> >>> http://www.ntop.org/blog/pf_ring/how-to-sendreceive-26mpps-using- >>> pf_ring-on-commodity-hardware/ >>> >>> Importantly, Luca also seems to be getting excellent CPU usage >>> figures, see the bottom of the page, indicating that both DCA and >>> IOATDMA are operating correctly. My problem is that even on light >>> network loads my CPU hits 100% and packets are dropped, indicating, >>> to me, that DCA/IOATDMA isn't working. >>> >>> I have switched on IOATDMA in the Dell's BIOS, it's off by default, >>> and discovered the following site where it talks about configuring a >>> machine to use DCA and IOATDMA etc. I even found a chap who reported >>> similar performance problems but with a Dell R710 and how he fixed >>> it. I tried all this but still no improvement! >>> >>> http://www.mail-archive.com/ntop-misc@listgateway.unipi.it/msg01185. >>> html >>> >>> The R810 seems to use a 7500 chipset. >>> >>> http://www.dell.com/downloads/global/products/pedge/pedge_r810_specs >>> heet >>> _en.pdf >>> >>> So, I think this is the R810 chipset reference http://www- >>> techdoc.intel.com/content/dam/doc/datasheet/7500-chipset-datasheet.p >>> df, >>> see page 453 >>> >>> The program sets bits (0x8C @ bit 0) but it doesn't seem to stay >>> set, so consecutive calls to "dca_probe" seem to always say "DCA >>> disabled, enabling now." >>> >>> I commented out some of the defines in the original code as they are >>> already set in the Linux kernel and, of course, changed the >>> registers to point to the ones on page 453 - I hope they are correct. >>> >>> Still no luck the CPU usage is way too high. >>> >>> #define _XOPEN_SOURCE 500 >>> >>> #include <stdio.h> >>> #include <stdlib.h> >>> #include <pci/pci.h> >>> #include <sys/io.h> >>> #include <fcntl.h> >>> #include <sys/stat.h> >>> #include <sys/types.h> >>> #include <unistd.h> >>> >>> #define INTEL_BRIDGE_DCAEN_OFFSET 0x8c >>> #define INTEL_BRIDGE_DCAEN_BIT 0 >>> /*#define PCI_HEADER_TYPE_BRIDGE 1 */ >>> /*#define PCI_VENDOR_ID_INTEL 0x8086 *//* lol @ intel */ >>> /*#define PCI_HEADER_TYPE 0x0e */ >>> #define MSR_P6_DCA_CAP 0x000001f8 >>> #define NUM_CPUS 64 >>> >>> void check_dca(struct pci_dev *dev) >>> { >>> u32 dca = pci_read_long(dev, INTEL_BRIDGE_DCAEN_OFFSET); >>> printf("DCA old value %d.\n", dca); >>> if (!(dca & (1 << INTEL_BRIDGE_DCAEN_BIT))) { >>> printf("DCA disabled, enabling now.\n"); >>> dca |= 1 << INTEL_BRIDGE_DCAEN_BIT; >>> printf("DCA new value %d.\n", dca); >>> pci_write_long(dev, INTEL_BRIDGE_DCAEN_OFFSET, dca); >>> } else { >>> printf("DCA already enabled!\n"); >>> } >>> } >>> >>> void msr_dca_enable(void) >>> { >>> char msr_file_name[64]; >>> int fd = 0, i = 0; >>> u64 data; >>> >>> for (;i < NUM_CPUS; i++) { >>> sprintf(msr_file_name, "/dev/cpu/%d/msr", i); >>> fd = open(msr_file_name, O_RDWR); >>> if (fd < 0) { >>> perror("open failed!"); >>> exit(1); >>> } >>> if (pread(fd, &data, sizeof(data), MSR_P6_DCA_CAP) != >>> sizeof(data)) { >>> perror("reading msr failed!"); >>> exit(1); >>> } >>> >>> printf("got msr value: %*llx\n", 1, (unsigned long >>> long)data); >>> if (!(data & 1)) { >>> data |= 1; >>> if (pwrite(fd, &data, sizeof(data), MSR_P6_DCA_CAP) != >>> sizeof(data)) { >>> perror("writing msr failed!"); >>> exit(1); >>> } >>> } else { >>> printf("msr already enabled for CPU %d\n", i); >>> } >>> } >>> } >>> >>> int main(void) >>> { >>> struct pci_access *pacc; >>> struct pci_dev *dev; >>> u8 type; >>> >>> pacc = pci_alloc(); >>> pci_init(pacc); >>> >>> pci_scan_bus(pacc); >>> for (dev = pacc->devices; dev; dev=dev->next) { >>> pci_fill_info(dev, PCI_FILL_IDENT | PCI_FILL_BASES); >>> if (dev->vendor_id == PCI_VENDOR_ID_INTEL) { >>> type = pci_read_byte(dev, PCI_HEADER_TYPE); >>> if (type == PCI_HEADER_TYPE_BRIDGE) { >>> check_dca(dev); >>> } >>> } >>> } >>> >>> msr_dca_enable(); >>> return 0; >>> } >>> >>> As you can see ixgbe, dca and ioatdma modules are loaded. >>> >>> # lsmod >>> >>> Module Size Used by >>> ixgbe 200547 0 >>> pf_ring 327754 4 >>> tcp_lp 2111 0 >>> fuse 61934 3 >>> sunrpc 201569 1 >>> ip6t_REJECT 4263 2 >>> nf_conntrack_ipv6 18078 4 >>> ip6table_filter 1687 1 >>> ip6_tables 17497 1 ip6table_filter >>> ipv6 286505 184 ip6t_REJECT,nf_conntrack_ipv6 >>> uinput 7368 0 >>> ioatdma 51376 72 >>> i7core_edac 16210 0 >>> dca 5590 2 ixgbe,ioatdma >>> bnx2 65569 0 >>> mdio 3934 0 >>> ses 6319 0 >>> dcdbas 8540 0 >>> edac_core 41336 1 i7core_edac >>> iTCO_wdt 11256 0 >>> iTCO_vendor_support 2610 1 iTCO_wdt >>> power_meter 9545 0 >>> hed 2206 0 >>> serio_raw 4640 0 >>> microcode 18662 0 >>> enclosure 7518 1 ses >>> megaraid_sas 37653 2 >>> >>> # uname -a >>> Linux test 2.6.35.14-95.fc14.x86_64 #1 SMP Tue Aug 16 21:01:58 UTC >>> 2011 >>> x86_64 x86_64 x86_64 GNU/Linux >>> >>> Thanks, >>> >>> Andrew >> >> Hey Andrew, >> >> Sorry you're having issues with the 28599 and ixgbe. I haven't done much >> with the PF_RING networking driver but maybe we can see what is going on >> with the ixgbe driver. It would help to know a little be more information >> like: >> >> - What there any interesting system log messages of note? >> >> - How are your interrupt being divided among your queue's (cat >> /proc/interrupts)? I know your testing with just one CPU are you also just >> using one queue or affinizing one to that CPU? >> >> - Could you provide the lspic -vvv output. To verify you NIC is getting a >> PCIe x8 connection. >> >> - What kind of cpu usage are you seeing if you don't use just the base >> driver running at line rate with something like netperf/iperf? >> >> - Have you attempted this without DCA? Like I said above I don't have much >> experience with PF_RING so I may be missing some fundamental advantage it is >> suppose to gain from operation with DCA in this mode. >> >> These are just off the top of my head if I think of anything else I'll let >> you know. >> >> Thanks, >> -Don Skidmore <donald.c.skidm...@intel.com> > > --- > > "Debugging is twice as hard as writing the code in the first place. > Therefore, if you write the code as cleverly as possible, you are, by > definition, not smart enough to debug it. - Brian W. Kernighan > ------------------------------------------------------------------------------ Doing More with Less: The Next Generation Virtual Desktop What are the key obstacles that have prevented many mid-market businesses from deploying virtual desktops? How do next-generation virtual desktops provide companies an easier-to-deploy, easier-to-manage and more affordable virtual desktop model.http://www.accelacomm.com/jaw/sfnl/114/51426474/ _______________________________________________ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired
