None yet Stefan – I’m all wrapped up in some other customer issues. While we would like to figure out why the i40e driver drops packets in scenarios where the ixgbe driver does not it is at this point a performance issue, so other issues that are more critical have to be addressed first. I do intend to investigate but I just need to find some time.
Please stay tuned. - Greg From: Stefan Priebe - Profihost AG [mailto:s.pri...@profihost.ag] Sent: Friday, August 28, 2015 12:22 AM To: Rose, Gregory V Cc: e1000-devel@lists.sourceforge.net Subject: Re: [E1000-devel] dropped rx with i40e Hi, any News? Stefan Excuse my typo sent from my mobile phone. Am 26.08.2015 um 16:37 schrieb Rose, Gregory V <gregory.v.r...@intel.com<mailto:gregory.v.r...@intel.com>>: -----Original Message----- From: Stefan Priebe - Profihost AG [mailto:s.pri...@profihost.ag] Sent: Wednesday, August 26, 2015 5:34 AM To: Rose, Gregory V Cc: e1000-devel@lists.sourceforge.net<mailto:e1000-devel@lists.sourceforge.net> Subject: Re: [E1000-devel] dropped rx with i40e I have a reproducer!! It happens under cache pressure. I can reproduce this with the following: 1.) start iperf -s on the system wird X710 2.) Start sending a lot of data to it i used 20 other servers using iperf -c TARGETIP -d -t 600 3.) read a lot of big files (i used find /DIRWITHBIGFILES ... -exec dd if=FILE of=/dev/zero bs=4M) I can trigger the drops with this code 100%. Great, I'll try to set up a similar environment today. Finding 20 other servers won't work for me but I think I can certainly emulate a similar traffic profile. I'll let you know how it goes. Thanks, - Greg Mit freundlichen Grüßen Stefan Priebe Bachelor of Science in Computer Science (BSCS) Vorstand (CTO) ------------------------------- Profihost AG Expo Plaza 1 30539 Hannover Deutschland Tel.: +49 (511) 5151 8181 | Fax.: +49 (511) 5151 8282 URL: http://www.profihost.com | E-Mail: i...@profihost.com<mailto:i...@profihost.com> Sitz der Gesellschaft: Hannover, USt-IdNr. DE813460827 Registergericht: Amtsgericht Hannover, Register-Nr.: HRB 202350 Vorstand: Cristoph Bluhm, Sebastian Bluhm, Stefan Priebe Aufsichtsrat: Prof. Dr. iur. Winfried Huck (Vorsitzender) Am 24.08.2015 um 19:40 schrieb Rose, Gregory V: So if you swap one of the newer cards with one of the older cards do the packet drops continue to follow the older card around? If so, then please get me the PBA of one of the newer cards and the PBA of one of the older ones that fails. Thanks, - Greg *From:*Stefan Priebe - Profihost AG [mailto:s.pri...@profihost.ag] *Sent:* Saturday, August 22, 2015 12:14 AM *To:* Rose, Gregory V *Cc:* e1000-devel@lists.sourceforge.net<mailto:e1000-devel@lists.sourceforge.net> *Subject:* Re: [E1000-devel] dropped rx with i40e I still have those ugly drops. Generally it seems it takes 20-48 hours before they occour but than it doesn't stop. The strange thing I've 8 newer boxes using same os and hw where I've never seen those drops. Are there different xl 710 hw revisions? I already checked hw revision of mainboard. Stefan Excuse my typo sent from my mobile phone. Am 21.08.2015 um 02:51 schrieb Rose, Gregory V <gregory.v.r...@intel.com<mailto:gregory.v.r...@intel.com> <mailto:gregory.v.r...@intel.com>>: Stefan, Late update on this. I'm being told know that there are additional parameters that may need to be changed. Please hold until I get this updated information. Thanks and regards, - Greg -----Original Message----- From: Rose, Gregory V [mailto:gregory.v.r...@intel.com] Sent: Thursday, August 20, 2015 4:26 PM To: Stefan Priebe Cc: e1000-devel@lists.sourceforge.net<mailto:e1000-devel@lists.sourceforge.net> <mailto:e1000-devel@lists.sourceforge.net> Subject: Re: [E1000-devel] dropped rx with i40e Stefan, You will need to run the update utility again but this time edit the nvmupdate.cfg file to change this: SKIP OROM: TRUE to this: SKIP OROM: FALSE That should then also update the option ROM for the PXE utility. Regards, - Greg -----Original Message----- From: Stefan Priebe [mailto:s.pri...@profihost.ag] Sent: Wednesday, August 19, 2015 10:40 AM To: Rose, Gregory V Cc: e1000-devel@lists.sourceforge.net<mailto:e1000-devel@lists.sourceforge.net> <mailto:e1000-devel@lists.sourceforge.net> Subject: Re: [E1000-devel] dropped rx with i40e Correct message: PXE-m10: The application for the device detected a newer version of the nvm image than expexted. So how do i update the application code of the XL710? Stefan Am 19.08.2015 um 19:04 schrieb Stefan Priebe - Profihost AG: An additional question since I've updated to latest xl710 firmware released 17 August the cards show at boot time (bios init/ post) something like application code detected a newer nvm image than expected please update application software. But I can't find an application code update for the xl710 cards. Stefan Excuse my typo sent from my mobile phone. Am 19.08.2015 um 18:20 schrieb Rose, Gregory V <gregory.v.r...@intel.com<mailto:gregory.v.r...@intel.com> <mailto:gregory.v.r...@intel.com> <mailto:gregory.v.r...@intel.com>>: -----Original Message----- From: Stefan Priebe - Profihost AG [mailto:s.pri...@profihost.ag] Sent: Wednesday, August 19, 2015 12:01 AM To: Rose, Gregory V; e1000-devel@lists.sourceforge.net<mailto:e1000-devel@lists.sourceforge.net> <mailto:e1000-devel@lists.sourceforge.net> <mailto:e1000-devel@lists.sourceforge.net> Subject: Re: [E1000-devel] dropped rx with i40e Hi, sad you are not able to reproduce. The good thing i can't reproduce myself either ;-( it just happens out of nothing on the nodes. My current expection is that it happens when spikes of packets occur after being idle for some time. OK, my traffic generation was fairly constant. Let me modify it to send bursts of traffic after long idle periods. Maybe that will help to reproduce. Something you might try is examining your systems' BIOS settings for sleep states and make sure the machines don't go too deep into a sleep state when they're idle. The time it takes to come from sleep state to handling traffic bursts can cause some dropped packets. Thanks, - Greg The good news is after upgrading to the latest intel fw released two days ago and to the latest 1.3.38 driver - it works on 10 out of my 18 testing hosts. Currently i've no idea why it does not on those 8. Stefan Am 19.08.2015 um 00:24 schrieb Rose, Gregory V: -----Original Message----- From: Stefan Priebe - Profihost AG [mailto:s.pri...@profihost.ag] Sent: Tuesday, August 18, 2015 12:28 AM To: Rose, Gregory V; e1000-devel@lists.sourceforge.net<mailto:e1000-devel@lists.sourceforge.net> <mailto:e1000-devel@lists.sourceforge.net> <mailto:e1000-devel@lists.sourceforge.net> Subject: Re: [E1000-devel] dropped rx with i40e Hi Greg, could you tell me the output of ethtool -i and ethtool -a and ethtool -c and ethtool -k? OK, I pasted it in below. I ran traffic overnight and there were no dropped packets or other errors. Everything seemed fine. - Greg [root@paelab-gvrose ~]# ethtool -i bond0 driver: bonding version: 3.7.1 firmware-version: 2 bus-info: supports-statistics: no supports-test: no supports-eeprom-access: no supports-register-dump: no supports-priv-flags: no [root@paelab-gvrose ~]# ethtool -c bond0 Coalesce parameters for bond0: Cannot get device coalesce settings: Operation not supported [root@paelab-gvrose ~]# ethtool -k bond0 Features for bond0: rx-checksumming: off [fixed] tx-checksumming: on tx-checksum-ipv4: off [fixed] tx-checksum-ip-generic: on tx-checksum-ipv6: off [fixed] tx-checksum-fcoe-crc: off [fixed] tx-checksum-sctp: off [fixed] scatter-gather: on tx-scatter-gather: on tx-scatter-gather-fraglist: off [requested on] tcp-segmentation-offload: on tx-tcp-segmentation: on tx-tcp-ecn-segmentation: on tx-tcp6-segmentation: on udp-fragmentation-offload: off [fixed] generic-segmentation-offload: on generic-receive-offload: on large-receive-offload: off rx-vlan-offload: on tx-vlan-offload: on ntuple-filters: off [fixed] receive-hashing: off [fixed] highdma: on rx-vlan-filter: on vlan-challenged: off [fixed] tx-lockless: on [fixed] netns-local: on [fixed] tx-gso-robust: off [fixed] tx-fcoe-segmentation: off [fixed] tx-gre-segmentation: off [fixed] tx-ipip-segmentation: off [fixed] tx-sit-segmentation: off [fixed] tx-udp_tnl-segmentation: on tx-mpls-segmentation: off [fixed] fcoe-mtu: off [fixed] tx-nocache-copy: off loopback: off [fixed] rx-fcs: off [fixed] rx-all: off [fixed] tx-vlan-stag-hw-insert: off [fixed] rx-vlan-stag-hw-parse: off [fixed] rx-vlan-stag-filter: off [fixed] l2-fwd-offload: off [fixed] busy-poll: off [fixed] [root@paelab-gvrose ~]# ethtool -i p4p1 driver: i40e version: 1.2.47 firmware-version: f4.40.35115 a1.4 n4.53 e1ce7 bus-info: 0000:82:00.0 supports-statistics: yes supports-test: yes supports-eeprom-access: yes supports-register-dump: yes supports-priv-flags: yes [root@paelab-gvrose ~]# ethtool -c p4p1 Coalesce parameters for p4p1: Adaptive RX: on TX: on stats-block-usecs: 0 sample-interval: 0 pkt-rate-low: 0 pkt-rate-high: 0 rx-usecs: 62 rx-frames: 0 rx-usecs-irq: 0 rx-frames-irq: 256 tx-usecs: 122 tx-frames: 0 tx-usecs-irq: 0 tx-frames-irq: 256 rx-usecs-low: 0 rx-frame-low: 0 tx-usecs-low: 0 tx-frame-low: 0 rx-usecs-high: 0 rx-frame-high: 0 tx-usecs-high: 0 tx-frame-high: 0 [root@paelab-gvrose ~]# ethtool -k p4p1 Features for p4p1: rx-checksumming: on tx-checksumming: on tx-checksum-ipv4: on tx-checksum-ip-generic: off [fixed] tx-checksum-ipv6: on tx-checksum-fcoe-crc: off [fixed] tx-checksum-sctp: on scatter-gather: on tx-scatter-gather: on tx-scatter-gather-fraglist: off [fixed] tcp-segmentation-offload: on tx-tcp-segmentation: on tx-tcp-ecn-segmentation: on tx-tcp6-segmentation: on udp-fragmentation-offload: off [fixed] generic-segmentation-offload: on generic-receive-offload: on large-receive-offload: off [fixed] rx-vlan-offload: on tx-vlan-offload: on ntuple-filters: on receive-hashing: on highdma: on rx-vlan-filter: on vlan-challenged: off [fixed] tx-lockless: off [fixed] netns-local: off [fixed] tx-gso-robust: off [fixed] tx-fcoe-segmentation: off [fixed] tx-gre-segmentation: off [fixed] tx-ipip-segmentation: off [fixed] tx-sit-segmentation: off [fixed] tx-udp_tnl-segmentation: on tx-mpls-segmentation: off [fixed] fcoe-mtu: off [fixed] tx-nocache-copy: off loopback: off [fixed] rx-fcs: off [fixed] rx-all: off [fixed] tx-vlan-stag-hw-insert: off [fixed] rx-vlan-stag-hw-parse: off [fixed] rx-vlan-stag-filter: off [fixed] l2-fwd-offload: off [fixed] busy-poll: off [fixed] [root@paelab-gvrose ~]# ethtool -i p4p4 driver: i40e version: 1.2.47 firmware-version: f4.40.35115 a1.4 n4.53 e1ce7 bus-info: 0000:82:00.3 supports-statistics: yes supports-test: yes supports-eeprom-access: yes supports-register-dump: yes supports-priv-flags: yes [root@paelab-gvrose ~]# ethtool -c p4p4 Coalesce parameters for p4p4: Adaptive RX: on TX: on stats-block-usecs: 0 sample-interval: 0 pkt-rate-low: 0 pkt-rate-high: 0 rx-usecs: 62 rx-frames: 0 rx-usecs-irq: 0 rx-frames-irq: 256 tx-usecs: 122 tx-frames: 0 tx-usecs-irq: 0 tx-frames-irq: 256 rx-usecs-low: 0 rx-frame-low: 0 tx-usecs-low: 0 tx-frame-low: 0 rx-usecs-high: 0 rx-frame-high: 0 tx-usecs-high: 0 tx-frame-high: 0 [root@paelab-gvrose ~]# ethtool -k p4p4 Features for p4p4: rx-checksumming: on tx-checksumming: on tx-checksum-ipv4: on tx-checksum-ip-generic: off [fixed] tx-checksum-ipv6: on tx-checksum-fcoe-crc: off [fixed] tx-checksum-sctp: on scatter-gather: on tx-scatter-gather: on tx-scatter-gather-fraglist: off [fixed] tcp-segmentation-offload: on tx-tcp-segmentation: on tx-tcp-ecn-segmentation: on tx-tcp6-segmentation: on udp-fragmentation-offload: off [fixed] generic-segmentation-offload: on generic-receive-offload: on large-receive-offload: off [fixed] rx-vlan-offload: on tx-vlan-offload: on ntuple-filters: on receive-hashing: on highdma: on rx-vlan-filter: on vlan-challenged: off [fixed] tx-lockless: off [fixed] netns-local: off [fixed] tx-gso-robust: off [fixed] tx-fcoe-segmentation: off [fixed] tx-gre-segmentation: off [fixed] tx-ipip-segmentation: off [fixed] tx-sit-segmentation: off [fixed] tx-udp_tnl-segmentation: on tx-mpls-segmentation: off [fixed] fcoe-mtu: off [fixed] tx-nocache-copy: off loopback: off [fixed] rx-fcs: off [fixed] rx-all: off [fixed] tx-vlan-stag-hw-insert: off [fixed] rx-vlan-stag-hw-parse: off [fixed] rx-vlan-stag-filter: off [fixed] l2-fwd-offload: off [fixed] busy-poll: off [fixed] Another difference to the ixgbe is that large-receive-offload is fixed to off in ethtool -k. Stefan Am 17.08.2015 um 23:46 schrieb Rose, Gregory V: -----Original Message----- From: Stefan Priebe [mailto:s.pri...@profihost.ag] Sent: Thursday, August 13, 2015 12:02 PM To: Rose, Gregory V; e1000- de...@lists.sourceforge.net<mailto:de...@lists.sourceforge.net> <mailto:e1000-devel@lists.sourceforge.net> <mailto:e1000-devel@lists.sourceforge.net> Subject: Re: [E1000-devel] dropped rx with i40e Am 13.08.2015 um 20:59 schrieb Rose, Gregory V: Thanks Stefan, I'll set up a test to replicate your traffic profile as closely as possible and let it run overnight to see if I can repro and then update you tomorrow. It does seem that it has nothing to do with load so that makes it even more curious. May it be related to jumbo frames? Stefan, I've got the test up and running now. Here's the interface config: 6: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP link/ether 68:05:ca:2f:83:10 brd ff:ff:ff:ff:ff:ff inet 200.0.0.10/24 brd 200.0.0.255 scope global bond0 valid_lft forever preferred_lft forever inet6 fe80::6a05:caff:fe2f:8310/64 scope link valid_lft forever preferred_lft forever 9: p4p1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master bond0 state UP qlen 1000 link/ether 68:05:ca:2f:83:10 brd ff:ff:ff:ff:ff:ff inet6 fe80::6a05:caff:fe2f:8310/64 scope link valid_lft forever preferred_lft forever 12: p4p4: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master bond0 state UP qlen 1000 link/ether 68:05:ca:2f:83:10 brd ff:ff:ff:ff:ff:ff inet6 fe80::6a05:caff:fe2f:8310/64 scope link valid_lft forever preferred_lft forever p4p1 and p4p4 are the i40e interfaces bonded to the bond0 LACP interface. Here's the traffic results from a transmitter: Interim result: 9899.84 10^6bits/s over 5.001 seconds ending at 1439847706.406 Interim result: 9900.15 10^6bits/s over 5.001 seconds ending at 1439847711.407 I have a script watching the dropped packets for the two slaved interfaces p4p1 and p4p4: Every 1.0s: ./t1 Mon Aug 17 14:43:31 2015 rx_dropped: 0 tx_dropped: 0 rx_fcoe_dropped: 0 port.rx_dropped: 0 port.tx_dropped_link_down: 0 rx_dropped: 0 tx_dropped: 0 rx_fcoe_dropped: 0 port.rx_dropped: 0 port.tx_dropped_link_down: 0 I'll let this run overnight and get back to you with results tomorrow. Regards, - Greg ---------------------------------------------------------------------- ---- ---- _______________________________________________ E1000-devel mailing list E1000-devel@lists.sourceforge.net<mailto:E1000-devel@lists.sourceforge.net> <mailto:E1000-devel@lists.sourceforge.net> https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired
------------------------------------------------------------------------------
_______________________________________________ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired
