> -----Original Message----- > From: Stefan Priebe - Profihost AG [mailto:s.pri...@profihost.ag] > Sent: Wednesday, August 26, 2015 5:34 AM > To: Rose, Gregory V > Cc: e1000-devel@lists.sourceforge.net > Subject: Re: [E1000-devel] dropped rx with i40e > > I have a reproducer!! It happens under cache pressure. > > I can reproduce this with the following: > > 1.) start iperf -s on the system wird X710 > 2.) Start sending a lot of data to it i used 20 other servers using iperf > -c TARGETIP -d -t 600 > 3.) read a lot of big files (i used find /DIRWITHBIGFILES ... -exec dd > if=FILE of=/dev/zero bs=4M) > > I can trigger the drops with this code 100%.
Great, I'll try to set up a similar environment today. Finding 20 other servers won't work for me but I think I can certainly emulate a similar traffic profile. I'll let you know how it goes. Thanks, - Greg > > Mit freundlichen Grüßen > Stefan Priebe > Bachelor of Science in Computer Science (BSCS) Vorstand (CTO) > > ------------------------------- > Profihost AG > Expo Plaza 1 > 30539 Hannover > Deutschland > > Tel.: +49 (511) 5151 8181 | Fax.: +49 (511) 5151 8282 > URL: http://www.profihost.com | E-Mail: i...@profihost.com > > Sitz der Gesellschaft: Hannover, USt-IdNr. DE813460827 > Registergericht: Amtsgericht Hannover, Register-Nr.: HRB 202350 > Vorstand: Cristoph Bluhm, Sebastian Bluhm, Stefan Priebe > Aufsichtsrat: Prof. Dr. iur. Winfried Huck (Vorsitzender) > > Am 24.08.2015 um 19:40 schrieb Rose, Gregory V: > > So if you swap one of the newer cards with one of the older cards do > > the packet drops continue to follow the older card around? > > > > > > > > If so, then please get me the PBA of one of the newer cards and the > > PBA of one of the older ones that fails. > > > > > > > > Thanks, > > > > > > > > - Greg > > > > > > > > *From:*Stefan Priebe - Profihost AG [mailto:s.pri...@profihost.ag] > > *Sent:* Saturday, August 22, 2015 12:14 AM > > *To:* Rose, Gregory V > > *Cc:* e1000-devel@lists.sourceforge.net > > *Subject:* Re: [E1000-devel] dropped rx with i40e > > > > > > > > I still have those ugly drops. Generally it seems it takes 20-48 hours > > before they occour but than it doesn't stop. > > > > > > > > The strange thing I've 8 newer boxes using same os and hw where I've > > never seen those drops. Are there different xl 710 hw revisions? I > > already checked hw revision of mainboard. > > > > Stefan > > > > > > > > Excuse my typo sent from my mobile phone. > > > > > > Am 21.08.2015 um 02:51 schrieb Rose, Gregory V > > <gregory.v.r...@intel.com > > <mailto:gregory.v.r...@intel.com>>: > > > > Stefan, > > > > Late update on this. I'm being told know that there are additional > > parameters that may need to be changed. Please hold until I get > > this updated information. > > > > Thanks and regards, > > > > - Greg > > > > > > -----Original Message----- > > > > From: Rose, Gregory V [mailto:gregory.v.r...@intel.com] > > > > Sent: Thursday, August 20, 2015 4:26 PM > > > > To: Stefan Priebe > > > > Cc: e1000-devel@lists.sourceforge.net > > <mailto:e1000-devel@lists.sourceforge.net> > > > > Subject: Re: [E1000-devel] dropped rx with i40e > > > > > > > > Stefan, > > > > > > > > You will need to run the update utility again but this time > > edit the > > > > nvmupdate.cfg file to change this: > > > > > > > > SKIP OROM: TRUE > > > > > > > > to this: > > > > > > > > SKIP OROM: FALSE > > > > > > > > That should then also update the option ROM for the PXE utility. > > > > > > > > Regards, > > > > > > > > - Greg > > > > > > > > -----Original Message----- > > > > From: Stefan Priebe [mailto:s.pri...@profihost.ag] > > > > Sent: Wednesday, August 19, 2015 10:40 AM > > > > To: Rose, Gregory V > > > > Cc: e1000-devel@lists.sourceforge.net > > <mailto:e1000-devel@lists.sourceforge.net> > > > > Subject: Re: [E1000-devel] dropped rx with i40e > > > > > > > > Correct message: > > > > PXE-m10: The application for the device detected a newer > > version of > > > > the nvm image than expexted. > > > > > > > > So how do i update the application code of the XL710? > > > > > > > > Stefan > > > > Am 19.08.2015 um 19:04 schrieb Stefan Priebe - Profihost AG: > > > > An additional question since I've updated to latest > > xl710 firmware > > > > released 17 August the cards show at boot time (bios > > init/ post) > > > > something like application code detected a newer nvm > > image than > > > > expected please update application software. But I can't > > find an > > > > application code update for the xl710 cards. > > > > > > > > Stefan > > > > > > > > Excuse my typo sent from my mobile phone. > > > > > > > > Am 19.08.2015 um 18:20 schrieb Rose, Gregory V > > > > <gregory.v.r...@intel.com > > <mailto:gregory.v.r...@intel.com> > > > > <mailto:gregory.v.r...@intel.com>>: > > > > > > > > > > > > -----Original Message----- > > > > From: Stefan Priebe - Profihost AG > > [mailto:s.pri...@profihost.ag] > > > > Sent: Wednesday, August 19, 2015 12:01 AM > > > > To: Rose, Gregory V; > > e1000-devel@lists.sourceforge.net > > <mailto:e1000-devel@lists.sourceforge.net> > > > > <mailto:e1000-devel@lists.sourceforge.net> > > > > Subject: Re: [E1000-devel] dropped rx with > > i40e > > > > > > > > Hi, > > > > > > > > sad you are not able to reproduce. The good > > thing i can't > > > > reproduce myself either ;-( > > > > > > > > it just happens out of nothing on the nodes. My > > current expection > > > > is that it happens when spikes of packets occur > > after being idle > > > > for some > > > > time. > > > > > > > > > > > > OK, my traffic generation was fairly constant. Let > > me modify it to > > > > send bursts of traffic after long idle periods. > > Maybe that will > > > > help to reproduce. > > > > > > > > Something you might try is examining your systems' > > BIOS settings > > > > for sleep states and make sure the machines don't go > > too deep into > > > > a sleep state when they're idle. The time it takes > > to come from > > > > sleep state to handling traffic bursts can cause > > some dropped > > > > packets. > > > > > > > > Thanks, > > > > > > > > - Greg > > > > > > > > The good news is after upgrading to the latest > > intel fw released > > > > two days ago and to the latest 1.3.38 driver - > > it works on 10 out > > > > of my > > > > 18 testing hosts. > > > > > > > > Currently i've no idea why it does not on those > 8. > > > > > > > > Stefan > > > > Am 19.08.2015 um 00:24 schrieb Rose, Gregory V: > > > > > > > > > > > > -----Original Message----- > > > > From: Stefan Priebe - Profihost AG > > > > [mailto:s.pri...@profihost.ag] > > > > Sent: Tuesday, August 18, 2015 12:28 > > AM > > > > To: Rose, Gregory V; > > e1000-devel@lists.sourceforge.net > > > > <mailto:e1000-devel@lists.sourceforge.net> > > > > > > <mailto:e1000-devel@lists.sourceforge.net> > > > > Subject: Re: [E1000-devel] dropped rx > > with i40e > > > > > > > > Hi Greg, > > > > > > > > could you tell me the output of ethtool > > -i and ethtool -a and > > > > ethtool -c and ethtool -k? > > > > > > > > OK, I pasted it in below. I ran traffic > > overnight and there were > > > > no > > > > dropped packets or other errors. Everything > > seemed fine. > > > > > > > > - Greg > > > > > > > > [root@paelab-gvrose ~]# ethtool -i bond0 > > > > driver: bonding > > > > version: 3.7.1 > > > > firmware-version: 2 > > > > bus-info: > > > > supports-statistics: no > > > > supports-test: no > > > > supports-eeprom-access: no > > > > supports-register-dump: no > > > > supports-priv-flags: no > > > > [root@paelab-gvrose ~]# ethtool -c bond0 > > Coalesce parameters for > > > > bond0: > > > > Cannot get device coalesce settings: > > Operation not supported > > > > [root@paelab-gvrose ~]# ethtool -k bond0 > > Features for bond0: > > > > rx-checksumming: off [fixed] > > > > tx-checksumming: on > > > > tx-checksum-ipv4: off [fixed] > > > > tx-checksum-ip-generic: on > > > > tx-checksum-ipv6: off [fixed] > > > > tx-checksum-fcoe-crc: off [fixed] > > > > tx-checksum-sctp: off [fixed] > > > > scatter-gather: on > > > > tx-scatter-gather: on > > > > tx-scatter-gather-fraglist: off > > [requested on] > > > > tcp-segmentation-offload: on > > > > tx-tcp-segmentation: on > > > > tx-tcp-ecn-segmentation: on > > > > tx-tcp6-segmentation: on > > > > udp-fragmentation-offload: off [fixed] > > > > generic-segmentation-offload: on > > > > generic-receive-offload: on > > > > large-receive-offload: off > > > > rx-vlan-offload: on > > > > tx-vlan-offload: on > > > > ntuple-filters: off [fixed] > > > > receive-hashing: off [fixed] > > > > highdma: on > > > > rx-vlan-filter: on > > > > vlan-challenged: off [fixed] > > > > tx-lockless: on [fixed] > > > > netns-local: on [fixed] > > > > tx-gso-robust: off [fixed] > > > > tx-fcoe-segmentation: off [fixed] > > > > tx-gre-segmentation: off [fixed] > > > > tx-ipip-segmentation: off [fixed] > > > > tx-sit-segmentation: off [fixed] > > > > tx-udp_tnl-segmentation: on > > > > tx-mpls-segmentation: off [fixed] > > > > fcoe-mtu: off [fixed] > > > > tx-nocache-copy: off > > > > loopback: off [fixed] > > > > rx-fcs: off [fixed] > > > > rx-all: off [fixed] > > > > tx-vlan-stag-hw-insert: off [fixed] > > > > rx-vlan-stag-hw-parse: off [fixed] > > > > rx-vlan-stag-filter: off [fixed] > > > > l2-fwd-offload: off [fixed] > > > > busy-poll: off [fixed] > > > > [root@paelab-gvrose ~]# ethtool -i p4p1 > > > > driver: i40e > > > > version: 1.2.47 > > > > firmware-version: f4.40.35115 a1.4 n4.53 > > e1ce7 > > > > bus-info: 0000:82:00.0 > > > > supports-statistics: yes > > > > supports-test: yes > > > > supports-eeprom-access: yes > > > > supports-register-dump: yes > > > > supports-priv-flags: yes > > > > [root@paelab-gvrose ~]# ethtool -c p4p1 > > Coalesce parameters for > > > > p4p1: > > > > Adaptive RX: on TX: on > > > > stats-block-usecs: 0 > > > > sample-interval: 0 > > > > pkt-rate-low: 0 > > > > pkt-rate-high: 0 > > > > > > > > rx-usecs: 62 > > > > rx-frames: 0 > > > > rx-usecs-irq: 0 > > > > rx-frames-irq: 256 > > > > > > > > tx-usecs: 122 > > > > tx-frames: 0 > > > > tx-usecs-irq: 0 > > > > tx-frames-irq: 256 > > > > > > > > rx-usecs-low: 0 > > > > rx-frame-low: 0 > > > > tx-usecs-low: 0 > > > > tx-frame-low: 0 > > > > > > > > rx-usecs-high: 0 > > > > rx-frame-high: 0 > > > > tx-usecs-high: 0 > > > > tx-frame-high: 0 > > > > > > > > [root@paelab-gvrose ~]# ethtool -k p4p1 > > Features for p4p1: > > > > rx-checksumming: on > > > > tx-checksumming: on > > > > tx-checksum-ipv4: on > > > > tx-checksum-ip-generic: off [fixed] > > > > tx-checksum-ipv6: on > > > > tx-checksum-fcoe-crc: off [fixed] > > > > tx-checksum-sctp: on > > > > scatter-gather: on > > > > tx-scatter-gather: on > > > > tx-scatter-gather-fraglist: off > > [fixed] > > > > tcp-segmentation-offload: on > > > > tx-tcp-segmentation: on > > > > tx-tcp-ecn-segmentation: on > > > > tx-tcp6-segmentation: on > > > > udp-fragmentation-offload: off [fixed] > > > > generic-segmentation-offload: on > > > > generic-receive-offload: on > > > > large-receive-offload: off [fixed] > > > > rx-vlan-offload: on > > > > tx-vlan-offload: on > > > > ntuple-filters: on > > > > receive-hashing: on > > > > highdma: on > > > > rx-vlan-filter: on > > > > vlan-challenged: off [fixed] > > > > tx-lockless: off [fixed] > > > > netns-local: off [fixed] > > > > tx-gso-robust: off [fixed] > > > > tx-fcoe-segmentation: off [fixed] > > > > tx-gre-segmentation: off [fixed] > > > > tx-ipip-segmentation: off [fixed] > > > > tx-sit-segmentation: off [fixed] > > > > tx-udp_tnl-segmentation: on > > > > tx-mpls-segmentation: off [fixed] > > > > fcoe-mtu: off [fixed] > > > > tx-nocache-copy: off > > > > loopback: off [fixed] > > > > rx-fcs: off [fixed] > > > > rx-all: off [fixed] > > > > tx-vlan-stag-hw-insert: off [fixed] > > > > rx-vlan-stag-hw-parse: off [fixed] > > > > rx-vlan-stag-filter: off [fixed] > > > > l2-fwd-offload: off [fixed] > > > > busy-poll: off [fixed] > > > > [root@paelab-gvrose ~]# ethtool -i p4p4 > > > > driver: i40e > > > > version: 1.2.47 > > > > firmware-version: f4.40.35115 a1.4 n4.53 > > e1ce7 > > > > bus-info: 0000:82:00.3 > > > > supports-statistics: yes > > > > supports-test: yes > > > > supports-eeprom-access: yes > > > > supports-register-dump: yes > > > > supports-priv-flags: yes > > > > [root@paelab-gvrose ~]# ethtool -c p4p4 > > Coalesce parameters for > > > > p4p4: > > > > Adaptive RX: on TX: on > > > > stats-block-usecs: 0 > > > > sample-interval: 0 > > > > pkt-rate-low: 0 > > > > pkt-rate-high: 0 > > > > > > > > rx-usecs: 62 > > > > rx-frames: 0 > > > > rx-usecs-irq: 0 > > > > rx-frames-irq: 256 > > > > > > > > tx-usecs: 122 > > > > tx-frames: 0 > > > > tx-usecs-irq: 0 > > > > tx-frames-irq: 256 > > > > > > > > rx-usecs-low: 0 > > > > rx-frame-low: 0 > > > > tx-usecs-low: 0 > > > > tx-frame-low: 0 > > > > > > > > rx-usecs-high: 0 > > > > rx-frame-high: 0 > > > > tx-usecs-high: 0 > > > > tx-frame-high: 0 > > > > > > > > [root@paelab-gvrose ~]# ethtool -k p4p4 > > Features for p4p4: > > > > rx-checksumming: on > > > > tx-checksumming: on > > > > tx-checksum-ipv4: on > > > > tx-checksum-ip-generic: off [fixed] > > > > tx-checksum-ipv6: on > > > > tx-checksum-fcoe-crc: off [fixed] > > > > tx-checksum-sctp: on > > > > scatter-gather: on > > > > tx-scatter-gather: on > > > > tx-scatter-gather-fraglist: off > > [fixed] > > > > tcp-segmentation-offload: on > > > > tx-tcp-segmentation: on > > > > tx-tcp-ecn-segmentation: on > > > > tx-tcp6-segmentation: on > > > > udp-fragmentation-offload: off [fixed] > > > > generic-segmentation-offload: on > > > > generic-receive-offload: on > > > > large-receive-offload: off [fixed] > > > > rx-vlan-offload: on > > > > tx-vlan-offload: on > > > > ntuple-filters: on > > > > receive-hashing: on > > > > highdma: on > > > > rx-vlan-filter: on > > > > vlan-challenged: off [fixed] > > > > tx-lockless: off [fixed] > > > > netns-local: off [fixed] > > > > tx-gso-robust: off [fixed] > > > > tx-fcoe-segmentation: off [fixed] > > > > tx-gre-segmentation: off [fixed] > > > > tx-ipip-segmentation: off [fixed] > > > > tx-sit-segmentation: off [fixed] > > > > tx-udp_tnl-segmentation: on > > > > tx-mpls-segmentation: off [fixed] > > > > fcoe-mtu: off [fixed] > > > > tx-nocache-copy: off > > > > loopback: off [fixed] > > > > rx-fcs: off [fixed] > > > > rx-all: off [fixed] > > > > tx-vlan-stag-hw-insert: off [fixed] > > > > rx-vlan-stag-hw-parse: off [fixed] > > > > rx-vlan-stag-filter: off [fixed] > > > > l2-fwd-offload: off [fixed] > > > > busy-poll: off [fixed] > > > > > > > > > > > > Another difference to the ixgbe is that > > large-receive-offload is > > > > fixed to off in ethtool -k. > > > > > > > > Stefan > > > > Am 17.08.2015 um 23:46 schrieb Rose, > > Gregory V: > > > > > > > > -----Original > > Message----- > > > > From: Stefan Priebe > > > > [mailto:s.pri...@profihost.ag] > > > > Sent: Thursday, August > > 13, 2015 12:02 PM > > > > To: Rose, Gregory V; > > e1000- > de...@lists.sourceforge.net > > > > <mailto:e1000-devel@lists.sourceforge.net> > > > > > > <mailto:e1000-devel@lists.sourceforge.net> > > > > Subject: Re: > > [E1000-devel] dropped rx > > with i40e > > > > > > > > Am 13.08.2015 um 20:59 > > schrieb Rose, Gregory V: > > > > Thanks Stefan, I'll > > set up a test to > > replicate your > > traffic > > > > profile as > > > > closely as possible and > > let it run overnight to > > see if I can > > > > repro and then update > > you tomorrow. > > > > > > > > It does seem that it > > has nothing to do > > with load so that > > > > makes it even > > > > more curious. > > > > > > > > May it be related to > > jumbo frames? > > > > > > > > Stefan, > > > > > > > > I've got the test up and running > > now. Here's the interface > > > > config: > > > > > > > > 6: bond0: > > > <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> > > mtu 9000 > > > > qdisc > > > > noqueue state UP > > > > link/ether 68:05:ca:2f:83:10 brd > > ff:ff:ff:ff:ff:ff > > > > inet 200.0.0.10/24 brd 200.0.0.255 > > scope global bond0 > > > > valid_lft forever preferred_lft > > forever > > > > inet6 fe80::6a05:caff:fe2f:8310/64 > > scope link > > > > valid_lft forever preferred_lft > > forever > > > > 9: p4p1: > > > <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> > > mtu 9000 qdisc > > > > mq > > > > master bond0 state UP qlen 1000 > > > > link/ether 68:05:ca:2f:83:10 brd > > ff:ff:ff:ff:ff:ff > > > > inet6 fe80::6a05:caff:fe2f:8310/64 > > scope link > > > > valid_lft forever preferred_lft > > forever > > > > 12: p4p4: > > > <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> > > mtu 9000 > > > > qdisc mq > > > > master bond0 state UP qlen 1000 > > > > link/ether 68:05:ca:2f:83:10 brd > > ff:ff:ff:ff:ff:ff > > > > inet6 fe80::6a05:caff:fe2f:8310/64 > > scope link > > > > valid_lft forever preferred_lft > > forever > > > > > > > > p4p1 and p4p4 are the i40e > > interfaces bonded to the bond0 > > LACP > > > > interface. > > > > > > > > Here's the traffic results from a > > transmitter: > > > > > > > > Interim result: 9899.84 10^6bits/s > > over 5.001 seconds ending at > > > > 1439847706.406 Interim result: > > 9900.15 10^6bits/s over 5.001 > > > > seconds ending at 1439847711.407 > > > > > > > > I have a script watching the dropped > > packets for the two slaved > > > > interfaces p4p1 and p4p4: > > > > > > > > Every 1.0s: ./t1 > > > Mon > > Aug > > > > 17 > > > > 14:43:31 2015 > > > > > > > > rx_dropped: 0 > > > > tx_dropped: 0 > > > > rx_fcoe_dropped: 0 > > > > port.rx_dropped: 0 > > > > port.tx_dropped_link_down: 0 > > > > rx_dropped: 0 > > > > tx_dropped: 0 > > > > rx_fcoe_dropped: 0 > > > > port.rx_dropped: 0 > > > > port.tx_dropped_link_down: 0 > > > > > > > > I'll let this run overnight and get > > back to you with results > > > > tomorrow. > > > > > > > > Regards, > > > > > > > > - Greg > > > > > > > > > > ---------------------------------------------------------------------- > > ---- > > > > ---- > > > > _______________________________________________ > > > > E1000-devel mailing list > > > > E1000-devel@lists.sourceforge.net > > <mailto:E1000-devel@lists.sourceforge.net> > > > > https://lists.sourceforge.net/lists/listinfo/e1000-devel > > > > To learn more about Intel® Ethernet, visit > > > > http://communities.intel.com/community/wired > > ------------------------------------------------------------------------------ _______________________________________________ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired
