Hi, any News?
Stefan Excuse my typo sent from my mobile phone. > Am 26.08.2015 um 16:37 schrieb Rose, Gregory V <gregory.v.r...@intel.com>: > > > >> -----Original Message----- >> From: Stefan Priebe - Profihost AG [mailto:s.pri...@profihost.ag] >> Sent: Wednesday, August 26, 2015 5:34 AM >> To: Rose, Gregory V >> Cc: e1000-devel@lists.sourceforge.net >> Subject: Re: [E1000-devel] dropped rx with i40e >> >> I have a reproducer!! It happens under cache pressure. >> >> I can reproduce this with the following: >> >> 1.) start iperf -s on the system wird X710 >> 2.) Start sending a lot of data to it i used 20 other servers using iperf >> -c TARGETIP -d -t 600 >> 3.) read a lot of big files (i used find /DIRWITHBIGFILES ... -exec dd >> if=FILE of=/dev/zero bs=4M) >> >> I can trigger the drops with this code 100%. > > Great, I'll try to set up a similar environment today. Finding 20 other > servers won't work for me but I think I can certainly emulate a similar > traffic profile. > > I'll let you know how it goes. > > Thanks, > > - Greg > >> >> Mit freundlichen Grüßen >> Stefan Priebe >> Bachelor of Science in Computer Science (BSCS) Vorstand (CTO) >> >> ------------------------------- >> Profihost AG >> Expo Plaza 1 >> 30539 Hannover >> Deutschland >> >> Tel.: +49 (511) 5151 8181 | Fax.: +49 (511) 5151 8282 >> URL: http://www.profihost.com | E-Mail: i...@profihost.com >> >> Sitz der Gesellschaft: Hannover, USt-IdNr. DE813460827 >> Registergericht: Amtsgericht Hannover, Register-Nr.: HRB 202350 >> Vorstand: Cristoph Bluhm, Sebastian Bluhm, Stefan Priebe >> Aufsichtsrat: Prof. Dr. iur. Winfried Huck (Vorsitzender) >> >>> Am 24.08.2015 um 19:40 schrieb Rose, Gregory V: >>> So if you swap one of the newer cards with one of the older cards do >>> the packet drops continue to follow the older card around? >>> >>> >>> >>> If so, then please get me the PBA of one of the newer cards and the >>> PBA of one of the older ones that fails. >>> >>> >>> >>> Thanks, >>> >>> >>> >>> - Greg >>> >>> >>> >>> *From:*Stefan Priebe - Profihost AG [mailto:s.pri...@profihost.ag] >>> *Sent:* Saturday, August 22, 2015 12:14 AM >>> *To:* Rose, Gregory V >>> *Cc:* e1000-devel@lists.sourceforge.net >>> *Subject:* Re: [E1000-devel] dropped rx with i40e >>> >>> >>> >>> I still have those ugly drops. Generally it seems it takes 20-48 hours >>> before they occour but than it doesn't stop. >>> >>> >>> >>> The strange thing I've 8 newer boxes using same os and hw where I've >>> never seen those drops. Are there different xl 710 hw revisions? I >>> already checked hw revision of mainboard. >>> >>> Stefan >>> >>> >>> >>> Excuse my typo sent from my mobile phone. >>> >>> >>> Am 21.08.2015 um 02:51 schrieb Rose, Gregory V >>> <gregory.v.r...@intel.com >>> <mailto:gregory.v.r...@intel.com>>: >>> >>> Stefan, >>> >>> Late update on this. I'm being told know that there are additional >>> parameters that may need to be changed. Please hold until I get >>> this updated information. >>> >>> Thanks and regards, >>> >>> - Greg >>> >>> >>> -----Original Message----- >>> >>> From: Rose, Gregory V [mailto:gregory.v.r...@intel.com] >>> >>> Sent: Thursday, August 20, 2015 4:26 PM >>> >>> To: Stefan Priebe >>> >>> Cc: e1000-devel@lists.sourceforge.net >>> <mailto:e1000-devel@lists.sourceforge.net> >>> >>> Subject: Re: [E1000-devel] dropped rx with i40e >>> >>> >>> >>> Stefan, >>> >>> >>> >>> You will need to run the update utility again but this time >>> edit the >>> >>> nvmupdate.cfg file to change this: >>> >>> >>> >>> SKIP OROM: TRUE >>> >>> >>> >>> to this: >>> >>> >>> >>> SKIP OROM: FALSE >>> >>> >>> >>> That should then also update the option ROM for the PXE utility. >>> >>> >>> >>> Regards, >>> >>> >>> >>> - Greg >>> >>> >>> >>> -----Original Message----- >>> >>> From: Stefan Priebe [mailto:s.pri...@profihost.ag] >>> >>> Sent: Wednesday, August 19, 2015 10:40 AM >>> >>> To: Rose, Gregory V >>> >>> Cc: e1000-devel@lists.sourceforge.net >>> <mailto:e1000-devel@lists.sourceforge.net> >>> >>> Subject: Re: [E1000-devel] dropped rx with i40e >>> >>> >>> >>> Correct message: >>> >>> PXE-m10: The application for the device detected a newer >>> version of >>> >>> the nvm image than expexted. >>> >>> >>> >>> So how do i update the application code of the XL710? >>> >>> >>> >>> Stefan >>> >>> Am 19.08.2015 um 19:04 schrieb Stefan Priebe - Profihost AG: >>> >>> An additional question since I've updated to latest >>> xl710 firmware >>> >>> released 17 August the cards show at boot time (bios >>> init/ post) >>> >>> something like application code detected a newer nvm >>> image than >>> >>> expected please update application software. But I can't >>> find an >>> >>> application code update for the xl710 cards. >>> >>> >>> >>> Stefan >>> >>> >>> >>> Excuse my typo sent from my mobile phone. >>> >>> >>> >>> Am 19.08.2015 um 18:20 schrieb Rose, Gregory V >>> >>> <gregory.v.r...@intel.com >>> <mailto:gregory.v.r...@intel.com> >>> >>> <mailto:gregory.v.r...@intel.com>>: >>> >>> >>> >>> >>> >>> -----Original Message----- >>> >>> From: Stefan Priebe - Profihost AG >>> [mailto:s.pri...@profihost.ag] >>> >>> Sent: Wednesday, August 19, 2015 12:01 AM >>> >>> To: Rose, Gregory V; >>> e1000-devel@lists.sourceforge.net >>> <mailto:e1000-devel@lists.sourceforge.net> >>> >>> <mailto:e1000-devel@lists.sourceforge.net> >>> >>> Subject: Re: [E1000-devel] dropped rx with >>> i40e >>> >>> >>> >>> Hi, >>> >>> >>> >>> sad you are not able to reproduce. The good >>> thing i can't >>> >>> reproduce myself either ;-( >>> >>> >>> >>> it just happens out of nothing on the nodes. My >>> current expection >>> >>> is that it happens when spikes of packets occur >>> after being idle >>> >>> for some >>> >>> time. >>> >>> >>> >>> >>> >>> OK, my traffic generation was fairly constant. Let >>> me modify it to >>> >>> send bursts of traffic after long idle periods. >>> Maybe that will >>> >>> help to reproduce. >>> >>> >>> >>> Something you might try is examining your systems' >>> BIOS settings >>> >>> for sleep states and make sure the machines don't go >>> too deep into >>> >>> a sleep state when they're idle. The time it takes >>> to come from >>> >>> sleep state to handling traffic bursts can cause >>> some dropped >>> >>> packets. >>> >>> >>> >>> Thanks, >>> >>> >>> >>> - Greg >>> >>> >>> >>> The good news is after upgrading to the latest >>> intel fw released >>> >>> two days ago and to the latest 1.3.38 driver - >>> it works on 10 out >>> >>> of my >>> >>> 18 testing hosts. >>> >>> >>> >>> Currently i've no idea why it does not on those >> 8. >>> >>> >>> >>> Stefan >>> >>> Am 19.08.2015 um 00:24 schrieb Rose, Gregory V: >>> >>> >>> >>> >>> >>> -----Original Message----- >>> >>> From: Stefan Priebe - Profihost AG >>> >>> [mailto:s.pri...@profihost.ag] >>> >>> Sent: Tuesday, August 18, 2015 12:28 >>> AM >>> >>> To: Rose, Gregory V; >>> e1000-devel@lists.sourceforge.net >>> >>> <mailto:e1000-devel@lists.sourceforge.net> >>> >>> >>> <mailto:e1000-devel@lists.sourceforge.net> >>> >>> Subject: Re: [E1000-devel] dropped rx >>> with i40e >>> >>> >>> >>> Hi Greg, >>> >>> >>> >>> could you tell me the output of ethtool >>> -i and ethtool -a and >>> >>> ethtool -c and ethtool -k? >>> >>> >>> >>> OK, I pasted it in below. I ran traffic >>> overnight and there were >>> >>> no >>> >>> dropped packets or other errors. Everything >>> seemed fine. >>> >>> >>> >>> - Greg >>> >>> >>> >>> [root@paelab-gvrose ~]# ethtool -i bond0 >>> >>> driver: bonding >>> >>> version: 3.7.1 >>> >>> firmware-version: 2 >>> >>> bus-info: >>> >>> supports-statistics: no >>> >>> supports-test: no >>> >>> supports-eeprom-access: no >>> >>> supports-register-dump: no >>> >>> supports-priv-flags: no >>> >>> [root@paelab-gvrose ~]# ethtool -c bond0 >>> Coalesce parameters for >>> >>> bond0: >>> >>> Cannot get device coalesce settings: >>> Operation not supported >>> >>> [root@paelab-gvrose ~]# ethtool -k bond0 >>> Features for bond0: >>> >>> rx-checksumming: off [fixed] >>> >>> tx-checksumming: on >>> >>> tx-checksum-ipv4: off [fixed] >>> >>> tx-checksum-ip-generic: on >>> >>> tx-checksum-ipv6: off [fixed] >>> >>> tx-checksum-fcoe-crc: off [fixed] >>> >>> tx-checksum-sctp: off [fixed] >>> >>> scatter-gather: on >>> >>> tx-scatter-gather: on >>> >>> tx-scatter-gather-fraglist: off >>> [requested on] >>> >>> tcp-segmentation-offload: on >>> >>> tx-tcp-segmentation: on >>> >>> tx-tcp-ecn-segmentation: on >>> >>> tx-tcp6-segmentation: on >>> >>> udp-fragmentation-offload: off [fixed] >>> >>> generic-segmentation-offload: on >>> >>> generic-receive-offload: on >>> >>> large-receive-offload: off >>> >>> rx-vlan-offload: on >>> >>> tx-vlan-offload: on >>> >>> ntuple-filters: off [fixed] >>> >>> receive-hashing: off [fixed] >>> >>> highdma: on >>> >>> rx-vlan-filter: on >>> >>> vlan-challenged: off [fixed] >>> >>> tx-lockless: on [fixed] >>> >>> netns-local: on [fixed] >>> >>> tx-gso-robust: off [fixed] >>> >>> tx-fcoe-segmentation: off [fixed] >>> >>> tx-gre-segmentation: off [fixed] >>> >>> tx-ipip-segmentation: off [fixed] >>> >>> tx-sit-segmentation: off [fixed] >>> >>> tx-udp_tnl-segmentation: on >>> >>> tx-mpls-segmentation: off [fixed] >>> >>> fcoe-mtu: off [fixed] >>> >>> tx-nocache-copy: off >>> >>> loopback: off [fixed] >>> >>> rx-fcs: off [fixed] >>> >>> rx-all: off [fixed] >>> >>> tx-vlan-stag-hw-insert: off [fixed] >>> >>> rx-vlan-stag-hw-parse: off [fixed] >>> >>> rx-vlan-stag-filter: off [fixed] >>> >>> l2-fwd-offload: off [fixed] >>> >>> busy-poll: off [fixed] >>> >>> [root@paelab-gvrose ~]# ethtool -i p4p1 >>> >>> driver: i40e >>> >>> version: 1.2.47 >>> >>> firmware-version: f4.40.35115 a1.4 n4.53 >>> e1ce7 >>> >>> bus-info: 0000:82:00.0 >>> >>> supports-statistics: yes >>> >>> supports-test: yes >>> >>> supports-eeprom-access: yes >>> >>> supports-register-dump: yes >>> >>> supports-priv-flags: yes >>> >>> [root@paelab-gvrose ~]# ethtool -c p4p1 >>> Coalesce parameters for >>> >>> p4p1: >>> >>> Adaptive RX: on TX: on >>> >>> stats-block-usecs: 0 >>> >>> sample-interval: 0 >>> >>> pkt-rate-low: 0 >>> >>> pkt-rate-high: 0 >>> >>> >>> >>> rx-usecs: 62 >>> >>> rx-frames: 0 >>> >>> rx-usecs-irq: 0 >>> >>> rx-frames-irq: 256 >>> >>> >>> >>> tx-usecs: 122 >>> >>> tx-frames: 0 >>> >>> tx-usecs-irq: 0 >>> >>> tx-frames-irq: 256 >>> >>> >>> >>> rx-usecs-low: 0 >>> >>> rx-frame-low: 0 >>> >>> tx-usecs-low: 0 >>> >>> tx-frame-low: 0 >>> >>> >>> >>> rx-usecs-high: 0 >>> >>> rx-frame-high: 0 >>> >>> tx-usecs-high: 0 >>> >>> tx-frame-high: 0 >>> >>> >>> >>> [root@paelab-gvrose ~]# ethtool -k p4p1 >>> Features for p4p1: >>> >>> rx-checksumming: on >>> >>> tx-checksumming: on >>> >>> tx-checksum-ipv4: on >>> >>> tx-checksum-ip-generic: off [fixed] >>> >>> tx-checksum-ipv6: on >>> >>> tx-checksum-fcoe-crc: off [fixed] >>> >>> tx-checksum-sctp: on >>> >>> scatter-gather: on >>> >>> tx-scatter-gather: on >>> >>> tx-scatter-gather-fraglist: off >>> [fixed] >>> >>> tcp-segmentation-offload: on >>> >>> tx-tcp-segmentation: on >>> >>> tx-tcp-ecn-segmentation: on >>> >>> tx-tcp6-segmentation: on >>> >>> udp-fragmentation-offload: off [fixed] >>> >>> generic-segmentation-offload: on >>> >>> generic-receive-offload: on >>> >>> large-receive-offload: off [fixed] >>> >>> rx-vlan-offload: on >>> >>> tx-vlan-offload: on >>> >>> ntuple-filters: on >>> >>> receive-hashing: on >>> >>> highdma: on >>> >>> rx-vlan-filter: on >>> >>> vlan-challenged: off [fixed] >>> >>> tx-lockless: off [fixed] >>> >>> netns-local: off [fixed] >>> >>> tx-gso-robust: off [fixed] >>> >>> tx-fcoe-segmentation: off [fixed] >>> >>> tx-gre-segmentation: off [fixed] >>> >>> tx-ipip-segmentation: off [fixed] >>> >>> tx-sit-segmentation: off [fixed] >>> >>> tx-udp_tnl-segmentation: on >>> >>> tx-mpls-segmentation: off [fixed] >>> >>> fcoe-mtu: off [fixed] >>> >>> tx-nocache-copy: off >>> >>> loopback: off [fixed] >>> >>> rx-fcs: off [fixed] >>> >>> rx-all: off [fixed] >>> >>> tx-vlan-stag-hw-insert: off [fixed] >>> >>> rx-vlan-stag-hw-parse: off [fixed] >>> >>> rx-vlan-stag-filter: off [fixed] >>> >>> l2-fwd-offload: off [fixed] >>> >>> busy-poll: off [fixed] >>> >>> [root@paelab-gvrose ~]# ethtool -i p4p4 >>> >>> driver: i40e >>> >>> version: 1.2.47 >>> >>> firmware-version: f4.40.35115 a1.4 n4.53 >>> e1ce7 >>> >>> bus-info: 0000:82:00.3 >>> >>> supports-statistics: yes >>> >>> supports-test: yes >>> >>> supports-eeprom-access: yes >>> >>> supports-register-dump: yes >>> >>> supports-priv-flags: yes >>> >>> [root@paelab-gvrose ~]# ethtool -c p4p4 >>> Coalesce parameters for >>> >>> p4p4: >>> >>> Adaptive RX: on TX: on >>> >>> stats-block-usecs: 0 >>> >>> sample-interval: 0 >>> >>> pkt-rate-low: 0 >>> >>> pkt-rate-high: 0 >>> >>> >>> >>> rx-usecs: 62 >>> >>> rx-frames: 0 >>> >>> rx-usecs-irq: 0 >>> >>> rx-frames-irq: 256 >>> >>> >>> >>> tx-usecs: 122 >>> >>> tx-frames: 0 >>> >>> tx-usecs-irq: 0 >>> >>> tx-frames-irq: 256 >>> >>> >>> >>> rx-usecs-low: 0 >>> >>> rx-frame-low: 0 >>> >>> tx-usecs-low: 0 >>> >>> tx-frame-low: 0 >>> >>> >>> >>> rx-usecs-high: 0 >>> >>> rx-frame-high: 0 >>> >>> tx-usecs-high: 0 >>> >>> tx-frame-high: 0 >>> >>> >>> >>> [root@paelab-gvrose ~]# ethtool -k p4p4 >>> Features for p4p4: >>> >>> rx-checksumming: on >>> >>> tx-checksumming: on >>> >>> tx-checksum-ipv4: on >>> >>> tx-checksum-ip-generic: off [fixed] >>> >>> tx-checksum-ipv6: on >>> >>> tx-checksum-fcoe-crc: off [fixed] >>> >>> tx-checksum-sctp: on >>> >>> scatter-gather: on >>> >>> tx-scatter-gather: on >>> >>> tx-scatter-gather-fraglist: off >>> [fixed] >>> >>> tcp-segmentation-offload: on >>> >>> tx-tcp-segmentation: on >>> >>> tx-tcp-ecn-segmentation: on >>> >>> tx-tcp6-segmentation: on >>> >>> udp-fragmentation-offload: off [fixed] >>> >>> generic-segmentation-offload: on >>> >>> generic-receive-offload: on >>> >>> large-receive-offload: off [fixed] >>> >>> rx-vlan-offload: on >>> >>> tx-vlan-offload: on >>> >>> ntuple-filters: on >>> >>> receive-hashing: on >>> >>> highdma: on >>> >>> rx-vlan-filter: on >>> >>> vlan-challenged: off [fixed] >>> >>> tx-lockless: off [fixed] >>> >>> netns-local: off [fixed] >>> >>> tx-gso-robust: off [fixed] >>> >>> tx-fcoe-segmentation: off [fixed] >>> >>> tx-gre-segmentation: off [fixed] >>> >>> tx-ipip-segmentation: off [fixed] >>> >>> tx-sit-segmentation: off [fixed] >>> >>> tx-udp_tnl-segmentation: on >>> >>> tx-mpls-segmentation: off [fixed] >>> >>> fcoe-mtu: off [fixed] >>> >>> tx-nocache-copy: off >>> >>> loopback: off [fixed] >>> >>> rx-fcs: off [fixed] >>> >>> rx-all: off [fixed] >>> >>> tx-vlan-stag-hw-insert: off [fixed] >>> >>> rx-vlan-stag-hw-parse: off [fixed] >>> >>> rx-vlan-stag-filter: off [fixed] >>> >>> l2-fwd-offload: off [fixed] >>> >>> busy-poll: off [fixed] >>> >>> >>> >>> >>> >>> Another difference to the ixgbe is that >>> large-receive-offload is >>> >>> fixed to off in ethtool -k. >>> >>> >>> >>> Stefan >>> >>> Am 17.08.2015 um 23:46 schrieb Rose, >>> Gregory V: >>> >>> >>> >>> -----Original >>> Message----- >>> >>> From: Stefan Priebe >>> >>> [mailto:s.pri...@profihost.ag] >>> >>> Sent: Thursday, August >>> 13, 2015 12:02 PM >>> >>> To: Rose, Gregory V; >>> e1000- >> de...@lists.sourceforge.net >>> >>> <mailto:e1000-devel@lists.sourceforge.net> >>> >>> >>> <mailto:e1000-devel@lists.sourceforge.net> >>> >>> Subject: Re: >>> [E1000-devel] dropped rx >>> with i40e >>> >>> >>> >>> Am 13.08.2015 um 20:59 >>> schrieb Rose, Gregory V: >>> >>> Thanks Stefan, I'll >>> set up a test to >>> replicate your >>> traffic >>> >>> profile as >>> >>> closely as possible and >>> let it run overnight to >>> see if I can >>> >>> repro and then update >>> you tomorrow. >>> >>> >>> >>> It does seem that it >>> has nothing to do >>> with load so that >>> >>> makes it even >>> >>> more curious. >>> >>> >>> >>> May it be related to >>> jumbo frames? >>> >>> >>> >>> Stefan, >>> >>> >>> >>> I've got the test up and running >>> now. Here's the interface >>> >>> config: >>> >>> >>> >>> 6: bond0: >> <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> >>> mtu 9000 >>> >>> qdisc >>> >>> noqueue state UP >>> >>> link/ether 68:05:ca:2f:83:10 brd >>> ff:ff:ff:ff:ff:ff >>> >>> inet 200.0.0.10/24 brd 200.0.0.255 >>> scope global bond0 >>> >>> valid_lft forever preferred_lft >>> forever >>> >>> inet6 fe80::6a05:caff:fe2f:8310/64 >>> scope link >>> >>> valid_lft forever preferred_lft >>> forever >>> >>> 9: p4p1: >> <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> >>> mtu 9000 qdisc >>> >>> mq >>> >>> master bond0 state UP qlen 1000 >>> >>> link/ether 68:05:ca:2f:83:10 brd >>> ff:ff:ff:ff:ff:ff >>> >>> inet6 fe80::6a05:caff:fe2f:8310/64 >>> scope link >>> >>> valid_lft forever preferred_lft >>> forever >>> >>> 12: p4p4: >> <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> >>> mtu 9000 >>> >>> qdisc mq >>> >>> master bond0 state UP qlen 1000 >>> >>> link/ether 68:05:ca:2f:83:10 brd >>> ff:ff:ff:ff:ff:ff >>> >>> inet6 fe80::6a05:caff:fe2f:8310/64 >>> scope link >>> >>> valid_lft forever preferred_lft >>> forever >>> >>> >>> >>> p4p1 and p4p4 are the i40e >>> interfaces bonded to the bond0 >>> LACP >>> >>> interface. >>> >>> >>> >>> Here's the traffic results from a >>> transmitter: >>> >>> >>> >>> Interim result: 9899.84 10^6bits/s >>> over 5.001 seconds ending at >>> >>> 1439847706.406 Interim result: >>> 9900.15 10^6bits/s over 5.001 >>> >>> seconds ending at 1439847711.407 >>> >>> >>> >>> I have a script watching the dropped >>> packets for the two slaved >>> >>> interfaces p4p1 and p4p4: >>> >>> >>> >>> Every 1.0s: ./t1 >> Mon >>> Aug >>> >>> 17 >>> >>> 14:43:31 2015 >>> >>> >>> >>> rx_dropped: 0 >>> >>> tx_dropped: 0 >>> >>> rx_fcoe_dropped: 0 >>> >>> port.rx_dropped: 0 >>> >>> port.tx_dropped_link_down: 0 >>> >>> rx_dropped: 0 >>> >>> tx_dropped: 0 >>> >>> rx_fcoe_dropped: 0 >>> >>> port.rx_dropped: 0 >>> >>> port.tx_dropped_link_down: 0 >>> >>> >>> >>> I'll let this run overnight and get >>> back to you with results >>> >>> tomorrow. >>> >>> >>> >>> Regards, >>> >>> >>> >>> - Greg >>> >>> >>> >>> >>> ---------------------------------------------------------------------- >>> ---- >>> >>> ---- >>> >>> _______________________________________________ >>> >>> E1000-devel mailing list >>> >>> E1000-devel@lists.sourceforge.net >>> <mailto:E1000-devel@lists.sourceforge.net> >>> >>> https://lists.sourceforge.net/lists/listinfo/e1000-devel >>> >>> To learn more about Intel® Ethernet, visit >>> >>> http://communities.intel.com/community/wired >>>
------------------------------------------------------------------------------
_______________________________________________ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired
