I have a reproducer!! It happens under cache pressure. I can reproduce this with the following:
1.) start iperf -s on the system wird X710 2.) Start sending a lot of data to it i used 20 other servers using iperf -c TARGETIP -d -t 600 3.) read a lot of big files (i used find /DIRWITHBIGFILES ... -exec dd if=FILE of=/dev/zero bs=4M) I can trigger the drops with this code 100%. Mit freundlichen Grüßen Stefan Priebe Bachelor of Science in Computer Science (BSCS) Vorstand (CTO) ------------------------------- Profihost AG Expo Plaza 1 30539 Hannover Deutschland Tel.: +49 (511) 5151 8181 | Fax.: +49 (511) 5151 8282 URL: http://www.profihost.com | E-Mail: i...@profihost.com Sitz der Gesellschaft: Hannover, USt-IdNr. DE813460827 Registergericht: Amtsgericht Hannover, Register-Nr.: HRB 202350 Vorstand: Cristoph Bluhm, Sebastian Bluhm, Stefan Priebe Aufsichtsrat: Prof. Dr. iur. Winfried Huck (Vorsitzender) Am 24.08.2015 um 19:40 schrieb Rose, Gregory V: > So if you swap one of the newer cards with one of the older cards do the > packet drops continue to follow the older card around? > > > > If so, then please get me the PBA of one of the newer cards and the PBA > of one of the older ones that fails. > > > > Thanks, > > > > - Greg > > > > *From:*Stefan Priebe - Profihost AG [mailto:s.pri...@profihost.ag] > *Sent:* Saturday, August 22, 2015 12:14 AM > *To:* Rose, Gregory V > *Cc:* e1000-devel@lists.sourceforge.net > *Subject:* Re: [E1000-devel] dropped rx with i40e > > > > I still have those ugly drops. Generally it seems it takes 20-48 hours > before they occour but than it doesn't stop. > > > > The strange thing I've 8 newer boxes using same os and hw where I've > never seen those drops. Are there different xl 710 hw revisions? I > already checked hw revision of mainboard. > > Stefan > > > > Excuse my typo sent from my mobile phone. > > > Am 21.08.2015 um 02:51 schrieb Rose, Gregory V <gregory.v.r...@intel.com > <mailto:gregory.v.r...@intel.com>>: > > Stefan, > > Late update on this. I'm being told know that there are additional > parameters that may need to be changed. Please hold until I get > this updated information. > > Thanks and regards, > > - Greg > > > -----Original Message----- > > From: Rose, Gregory V [mailto:gregory.v.r...@intel.com] > > Sent: Thursday, August 20, 2015 4:26 PM > > To: Stefan Priebe > > Cc: e1000-devel@lists.sourceforge.net > <mailto:e1000-devel@lists.sourceforge.net> > > Subject: Re: [E1000-devel] dropped rx with i40e > > > > Stefan, > > > > You will need to run the update utility again but this time edit the > > nvmupdate.cfg file to change this: > > > > SKIP OROM: TRUE > > > > to this: > > > > SKIP OROM: FALSE > > > > That should then also update the option ROM for the PXE utility. > > > > Regards, > > > > - Greg > > > > -----Original Message----- > > From: Stefan Priebe [mailto:s.pri...@profihost.ag] > > Sent: Wednesday, August 19, 2015 10:40 AM > > To: Rose, Gregory V > > Cc: e1000-devel@lists.sourceforge.net > <mailto:e1000-devel@lists.sourceforge.net> > > Subject: Re: [E1000-devel] dropped rx with i40e > > > > Correct message: > > PXE-m10: The application for the device detected a newer > version of > > the nvm image than expexted. > > > > So how do i update the application code of the XL710? > > > > Stefan > > Am 19.08.2015 um 19:04 schrieb Stefan Priebe - Profihost AG: > > An additional question since I've updated to latest > xl710 firmware > > released 17 August the cards show at boot time (bios > init/ post) > > something like application code detected a newer nvm > image than > > expected please update application software. But I can't > find an > > application code update for the xl710 cards. > > > > Stefan > > > > Excuse my typo sent from my mobile phone. > > > > Am 19.08.2015 um 18:20 schrieb Rose, Gregory V > > <gregory.v.r...@intel.com <mailto:gregory.v.r...@intel.com> > > <mailto:gregory.v.r...@intel.com>>: > > > > > > -----Original Message----- > > From: Stefan Priebe - Profihost AG > [mailto:s.pri...@profihost.ag] > > Sent: Wednesday, August 19, 2015 12:01 AM > > To: Rose, Gregory V; > e1000-devel@lists.sourceforge.net > <mailto:e1000-devel@lists.sourceforge.net> > > <mailto:e1000-devel@lists.sourceforge.net> > > Subject: Re: [E1000-devel] dropped rx with i40e > > > > Hi, > > > > sad you are not able to reproduce. The good > thing i can't > > reproduce myself either ;-( > > > > it just happens out of nothing on the nodes. My > current expection > > is that it happens when spikes of packets occur > after being idle > > for some > > time. > > > > > > OK, my traffic generation was fairly constant. Let > me modify it to > > send bursts of traffic after long idle periods. > Maybe that will > > help to reproduce. > > > > Something you might try is examining your systems' > BIOS settings > > for sleep states and make sure the machines don't go > too deep into > > a sleep state when they're idle. The time it takes > to come from > > sleep state to handling traffic bursts can cause > some dropped > > packets. > > > > Thanks, > > > > - Greg > > > > The good news is after upgrading to the latest > intel fw released > > two days ago and to the latest 1.3.38 driver - > it works on 10 out > > of my > > 18 testing hosts. > > > > Currently i've no idea why it does not on those 8. > > > > Stefan > > Am 19.08.2015 um 00:24 schrieb Rose, Gregory V: > > > > > > -----Original Message----- > > From: Stefan Priebe - Profihost AG > > [mailto:s.pri...@profihost.ag] > > Sent: Tuesday, August 18, 2015 12:28 AM > > To: Rose, Gregory V; > e1000-devel@lists.sourceforge.net > <mailto:e1000-devel@lists.sourceforge.net> > > <mailto:e1000-devel@lists.sourceforge.net> > > Subject: Re: [E1000-devel] dropped rx > with i40e > > > > Hi Greg, > > > > could you tell me the output of ethtool > -i and ethtool -a and > > ethtool -c and ethtool -k? > > > > OK, I pasted it in below. I ran traffic > overnight and there were > > no > > dropped packets or other errors. Everything > seemed fine. > > > > - Greg > > > > [root@paelab-gvrose ~]# ethtool -i bond0 > > driver: bonding > > version: 3.7.1 > > firmware-version: 2 > > bus-info: > > supports-statistics: no > > supports-test: no > > supports-eeprom-access: no > > supports-register-dump: no > > supports-priv-flags: no > > [root@paelab-gvrose ~]# ethtool -c bond0 > Coalesce parameters for > > bond0: > > Cannot get device coalesce settings: > Operation not supported > > [root@paelab-gvrose ~]# ethtool -k bond0 > Features for bond0: > > rx-checksumming: off [fixed] > > tx-checksumming: on > > tx-checksum-ipv4: off [fixed] > > tx-checksum-ip-generic: on > > tx-checksum-ipv6: off [fixed] > > tx-checksum-fcoe-crc: off [fixed] > > tx-checksum-sctp: off [fixed] > > scatter-gather: on > > tx-scatter-gather: on > > tx-scatter-gather-fraglist: off > [requested on] > > tcp-segmentation-offload: on > > tx-tcp-segmentation: on > > tx-tcp-ecn-segmentation: on > > tx-tcp6-segmentation: on > > udp-fragmentation-offload: off [fixed] > > generic-segmentation-offload: on > > generic-receive-offload: on > > large-receive-offload: off > > rx-vlan-offload: on > > tx-vlan-offload: on > > ntuple-filters: off [fixed] > > receive-hashing: off [fixed] > > highdma: on > > rx-vlan-filter: on > > vlan-challenged: off [fixed] > > tx-lockless: on [fixed] > > netns-local: on [fixed] > > tx-gso-robust: off [fixed] > > tx-fcoe-segmentation: off [fixed] > > tx-gre-segmentation: off [fixed] > > tx-ipip-segmentation: off [fixed] > > tx-sit-segmentation: off [fixed] > > tx-udp_tnl-segmentation: on > > tx-mpls-segmentation: off [fixed] > > fcoe-mtu: off [fixed] > > tx-nocache-copy: off > > loopback: off [fixed] > > rx-fcs: off [fixed] > > rx-all: off [fixed] > > tx-vlan-stag-hw-insert: off [fixed] > > rx-vlan-stag-hw-parse: off [fixed] > > rx-vlan-stag-filter: off [fixed] > > l2-fwd-offload: off [fixed] > > busy-poll: off [fixed] > > [root@paelab-gvrose ~]# ethtool -i p4p1 > > driver: i40e > > version: 1.2.47 > > firmware-version: f4.40.35115 a1.4 n4.53 e1ce7 > > bus-info: 0000:82:00.0 > > supports-statistics: yes > > supports-test: yes > > supports-eeprom-access: yes > > supports-register-dump: yes > > supports-priv-flags: yes > > [root@paelab-gvrose ~]# ethtool -c p4p1 > Coalesce parameters for > > p4p1: > > Adaptive RX: on TX: on > > stats-block-usecs: 0 > > sample-interval: 0 > > pkt-rate-low: 0 > > pkt-rate-high: 0 > > > > rx-usecs: 62 > > rx-frames: 0 > > rx-usecs-irq: 0 > > rx-frames-irq: 256 > > > > tx-usecs: 122 > > tx-frames: 0 > > tx-usecs-irq: 0 > > tx-frames-irq: 256 > > > > rx-usecs-low: 0 > > rx-frame-low: 0 > > tx-usecs-low: 0 > > tx-frame-low: 0 > > > > rx-usecs-high: 0 > > rx-frame-high: 0 > > tx-usecs-high: 0 > > tx-frame-high: 0 > > > > [root@paelab-gvrose ~]# ethtool -k p4p1 > Features for p4p1: > > rx-checksumming: on > > tx-checksumming: on > > tx-checksum-ipv4: on > > tx-checksum-ip-generic: off [fixed] > > tx-checksum-ipv6: on > > tx-checksum-fcoe-crc: off [fixed] > > tx-checksum-sctp: on > > scatter-gather: on > > tx-scatter-gather: on > > tx-scatter-gather-fraglist: off [fixed] > > tcp-segmentation-offload: on > > tx-tcp-segmentation: on > > tx-tcp-ecn-segmentation: on > > tx-tcp6-segmentation: on > > udp-fragmentation-offload: off [fixed] > > generic-segmentation-offload: on > > generic-receive-offload: on > > large-receive-offload: off [fixed] > > rx-vlan-offload: on > > tx-vlan-offload: on > > ntuple-filters: on > > receive-hashing: on > > highdma: on > > rx-vlan-filter: on > > vlan-challenged: off [fixed] > > tx-lockless: off [fixed] > > netns-local: off [fixed] > > tx-gso-robust: off [fixed] > > tx-fcoe-segmentation: off [fixed] > > tx-gre-segmentation: off [fixed] > > tx-ipip-segmentation: off [fixed] > > tx-sit-segmentation: off [fixed] > > tx-udp_tnl-segmentation: on > > tx-mpls-segmentation: off [fixed] > > fcoe-mtu: off [fixed] > > tx-nocache-copy: off > > loopback: off [fixed] > > rx-fcs: off [fixed] > > rx-all: off [fixed] > > tx-vlan-stag-hw-insert: off [fixed] > > rx-vlan-stag-hw-parse: off [fixed] > > rx-vlan-stag-filter: off [fixed] > > l2-fwd-offload: off [fixed] > > busy-poll: off [fixed] > > [root@paelab-gvrose ~]# ethtool -i p4p4 > > driver: i40e > > version: 1.2.47 > > firmware-version: f4.40.35115 a1.4 n4.53 e1ce7 > > bus-info: 0000:82:00.3 > > supports-statistics: yes > > supports-test: yes > > supports-eeprom-access: yes > > supports-register-dump: yes > > supports-priv-flags: yes > > [root@paelab-gvrose ~]# ethtool -c p4p4 > Coalesce parameters for > > p4p4: > > Adaptive RX: on TX: on > > stats-block-usecs: 0 > > sample-interval: 0 > > pkt-rate-low: 0 > > pkt-rate-high: 0 > > > > rx-usecs: 62 > > rx-frames: 0 > > rx-usecs-irq: 0 > > rx-frames-irq: 256 > > > > tx-usecs: 122 > > tx-frames: 0 > > tx-usecs-irq: 0 > > tx-frames-irq: 256 > > > > rx-usecs-low: 0 > > rx-frame-low: 0 > > tx-usecs-low: 0 > > tx-frame-low: 0 > > > > rx-usecs-high: 0 > > rx-frame-high: 0 > > tx-usecs-high: 0 > > tx-frame-high: 0 > > > > [root@paelab-gvrose ~]# ethtool -k p4p4 > Features for p4p4: > > rx-checksumming: on > > tx-checksumming: on > > tx-checksum-ipv4: on > > tx-checksum-ip-generic: off [fixed] > > tx-checksum-ipv6: on > > tx-checksum-fcoe-crc: off [fixed] > > tx-checksum-sctp: on > > scatter-gather: on > > tx-scatter-gather: on > > tx-scatter-gather-fraglist: off [fixed] > > tcp-segmentation-offload: on > > tx-tcp-segmentation: on > > tx-tcp-ecn-segmentation: on > > tx-tcp6-segmentation: on > > udp-fragmentation-offload: off [fixed] > > generic-segmentation-offload: on > > generic-receive-offload: on > > large-receive-offload: off [fixed] > > rx-vlan-offload: on > > tx-vlan-offload: on > > ntuple-filters: on > > receive-hashing: on > > highdma: on > > rx-vlan-filter: on > > vlan-challenged: off [fixed] > > tx-lockless: off [fixed] > > netns-local: off [fixed] > > tx-gso-robust: off [fixed] > > tx-fcoe-segmentation: off [fixed] > > tx-gre-segmentation: off [fixed] > > tx-ipip-segmentation: off [fixed] > > tx-sit-segmentation: off [fixed] > > tx-udp_tnl-segmentation: on > > tx-mpls-segmentation: off [fixed] > > fcoe-mtu: off [fixed] > > tx-nocache-copy: off > > loopback: off [fixed] > > rx-fcs: off [fixed] > > rx-all: off [fixed] > > tx-vlan-stag-hw-insert: off [fixed] > > rx-vlan-stag-hw-parse: off [fixed] > > rx-vlan-stag-filter: off [fixed] > > l2-fwd-offload: off [fixed] > > busy-poll: off [fixed] > > > > > > Another difference to the ixgbe is that > large-receive-offload is > > fixed to off in ethtool -k. > > > > Stefan > > Am 17.08.2015 um 23:46 schrieb Rose, > Gregory V: > > > > -----Original Message----- > > From: Stefan Priebe > [mailto:s.pri...@profihost.ag] > > Sent: Thursday, August > 13, 2015 12:02 PM > > To: Rose, Gregory V; > > e1000-devel@lists.sourceforge.net > > <mailto:e1000-devel@lists.sourceforge.net> > > > <mailto:e1000-devel@lists.sourceforge.net> > > Subject: Re: > [E1000-devel] dropped rx > with i40e > > > > Am 13.08.2015 um 20:59 > schrieb Rose, Gregory V: > > Thanks Stefan, I'll > set up a test to > replicate your traffic > > profile as > > closely as possible and > let it run overnight to > see if I can > > repro and then update > you tomorrow. > > > > It does seem that it > has nothing to do > with load so that > > makes it even > > more curious. > > > > May it be related to > jumbo frames? > > > > Stefan, > > > > I've got the test up and running > now. Here's the interface > > config: > > > > 6: bond0: > <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> > mtu 9000 > > qdisc > > noqueue state UP > > link/ether 68:05:ca:2f:83:10 brd > ff:ff:ff:ff:ff:ff > > inet 200.0.0.10/24 brd 200.0.0.255 > scope global bond0 > > valid_lft forever preferred_lft > forever > > inet6 fe80::6a05:caff:fe2f:8310/64 > scope link > > valid_lft forever preferred_lft > forever > > 9: p4p1: > <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> > mtu 9000 qdisc > > mq > > master bond0 state UP qlen 1000 > > link/ether 68:05:ca:2f:83:10 brd > ff:ff:ff:ff:ff:ff > > inet6 fe80::6a05:caff:fe2f:8310/64 > scope link > > valid_lft forever preferred_lft > forever > > 12: p4p4: > <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> > mtu 9000 > > qdisc mq > > master bond0 state UP qlen 1000 > > link/ether 68:05:ca:2f:83:10 brd > ff:ff:ff:ff:ff:ff > > inet6 fe80::6a05:caff:fe2f:8310/64 > scope link > > valid_lft forever preferred_lft > forever > > > > p4p1 and p4p4 are the i40e > interfaces bonded to the bond0 LACP > > interface. > > > > Here's the traffic results from a > transmitter: > > > > Interim result: 9899.84 10^6bits/s > over 5.001 seconds ending at > > 1439847706.406 Interim result: > 9900.15 10^6bits/s over 5.001 > > seconds ending at 1439847711.407 > > > > I have a script watching the dropped > packets for the two slaved > > interfaces p4p1 and p4p4: > > > > Every 1.0s: ./t1 > Mon > Aug > > 17 > > 14:43:31 2015 > > > > rx_dropped: 0 > > tx_dropped: 0 > > rx_fcoe_dropped: 0 > > port.rx_dropped: 0 > > port.tx_dropped_link_down: 0 > > rx_dropped: 0 > > tx_dropped: 0 > > rx_fcoe_dropped: 0 > > port.rx_dropped: 0 > > port.tx_dropped_link_down: 0 > > > > I'll let this run overnight and get > back to you with results > > tomorrow. > > > > Regards, > > > > - Greg > > > > > -------------------------------------------------------------------------- > > ---- > > _______________________________________________ > > E1000-devel mailing list > > E1000-devel@lists.sourceforge.net > <mailto:E1000-devel@lists.sourceforge.net> > > https://lists.sourceforge.net/lists/listinfo/e1000-devel > > To learn more about Intel® Ethernet, visit > > http://communities.intel.com/community/wired > ------------------------------------------------------------------------------ _______________________________________________ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired
