Stefan, I've run into a glitch while in the process of configuring a setup for repo - we need to do some reconfig in our lab. I've put in the appropriate lab request and they're usually pretty good about quick response but I may not get the test running tonight. If not I'll get it started ASAP.
Thanks for your patience. - Greg > -----Original Message----- > From: Rose, Gregory V [mailto:gregory.v.r...@intel.com] > Sent: Thursday, August 13, 2015 1:16 PM > To: Stefan Priebe; e1000-devel@lists.sourceforge.net > Subject: Re: [E1000-devel] dropped rx with i40e > > > -----Original Message----- > > From: Stefan Priebe [mailto:s.pri...@profihost.ag] > > Sent: Thursday, August 13, 2015 12:02 PM > > To: Rose, Gregory V; e1000-devel@lists.sourceforge.net > > Subject: Re: [E1000-devel] dropped rx with i40e > > > > Am 13.08.2015 um 20:59 schrieb Rose, Gregory V: > > > Thanks Stefan, I'll set up a test to replicate your traffic profile > > > as > > closely as possible and let it run overnight to see if I can repro and > > then update you tomorrow. > > > > > > It does seem that it has nothing to do with load so that makes it > > > even > > more curious. > > > > May it be related to jumbo frames? > > Could be - I'll be looking at that angle during my testing and evaluation. > > - Greg > > > > > Stefan > > > > > > > > - Greg > > > > > >> -----Original Message----- > > >> From: Stefan Priebe [mailto:s.pri...@profihost.ag] > > >> Sent: Thursday, August 13, 2015 11:53 AM > > >> To: Rose, Gregory V; e1000-devel@lists.sourceforge.net > > >> Subject: Re: [E1000-devel] dropped rx with i40e > > >> > > >> Hi, > > >> > > >> sorry for top posting. > > >> > > >> I will try to describe the workload as good as i can. > > >> > > >> Application is ceph storage (http://ceph.com/). > > >> > > >> Workload is TCP Only, Active/Active bond on both ports of the XL710 > > >> card and jumbo frames (MTU 9000). Traffic peak was 400MBit/s - So > > >> overall speed does not seem to matter. Also i can use iperf and get > > >> a constant speed of 9.8Gb/s in both directions without any rx drops. > > >> > > >> The drops don't occur regulary they just happen at a time X and > > >> then > > stop. > > >> After some hours it happens again. > > >> Stefan > > >> > > >> Am 13.08.2015 um 17:58 schrieb Rose, Gregory V: > > >>> My apologies but I've been unable to get back to this issue. > > >>> > > >>> After reviewing the thread I don't see anything about steps to > > >>> reproduce > > >> the problem. I understand that you're seeing dropped packets with > > >> the > > >> Xl710 with various versions of the i40e driver while the X520 with > > >> the ixgbe driver does not drop packets under the same load. > > >>> > > >>> I don't' see any description of the type of traffic load that is > > >>> causing > > >> the problem. That would help me to reproduce the issue. > > >>> > > >>> Keep in mind that dropped packets in and of itself is not a bug. > > >>> It may > > >> mean that the X520 and the ixgbe driver are more mature and have > > >> had more "tuning" and thus work better under the type of traffic > > >> load you have on your network. Thus it is important that we > > >> understand the type of traffic you're seeing on your network so > > >> that we can work on making the XL710 and i40e driver performance on > > >> par with the X520 and > > the ixgbe driver. > > >>> > > >>> One other thing. Below I notice this: > > >>> > > >>>> I tested this one: > > >>>> ethtool -C eth3 adaptive-rx off adaptive-tx off rx-usecs 2 > > >>>> tx-usecs > > >>>> 0 > > >>> > > >>> I believe that you would be better off using higher values. > > >>> Really low > > >> values mean the HW interrupt will fire more often - instead you > > >> should allow the soft IRQ polling to keep processing packets. > > >>> > > >>> - Greg > > >>> > > >>>> -----Original Message----- > > >>>> From: Stefan Priebe - Profihost AG [mailto:s.pri...@profihost.ag] > > >>>> Sent: Thursday, August 13, 2015 5:41 AM > > >>>> To: Rose, Gregory V; e1000-devel@lists.sourceforge.net > > >>>> Subject: Re: [E1000-devel] dropped rx with i40e > > >>>> > > >>>> 1.3.12-k from net-next devel does not help either ;-( > > >>>> > > >>>> Should we open an intel support ticket? We really need a solution. > > >>>> > > >>>> Stefan > > >>>> > > >>>> Am 12.08.2015 um 10:29 schrieb Stefan Priebe - Profihost AG: > > >>>>> Might this be a memory allocation problem? It happens only after > > >>>>> some hours running and when the whole memory is filled with > > >>>>> linux fs > > >> cache. > > >>>>> > > >>>>> Is the i40e driver using kmalloc or vmalloc? > > >>>>> > > >>>>> Stefan > > >>>>> Am 11.08.2015 um 06:03 schrieb Stefan Priebe: > > >>>>>> One more thing to note. It mostly happens after around 8-24 > > >>>>>> hours and i could stop it again by rebooting the system/server. > > >>>>>> (can't prove > > >>>>>> it) > > >>>>>> > > >>>>>> Stefan > > >>>>>> Am 06.08.2015 um 22:59 schrieb Rose, Gregory V: > > >>>>>>> Thanks Stefan. I think for now you've given us enough data to > > >>>>>>> go on > > >>>>>>> - I've got some research to do and then I'll get back to you. > > >>>>>>> > > >>>>>>> - Greg > > >>>>>>> > > >>>>>>>> -----Original Message----- > > >>>>>>>> From: Stefan Priebe - Profihost AG > > >>>>>>>> [mailto:s.pri...@profihost.ag] > > >>>>>>>> Sent: Wednesday, August 05, 2015 11:32 PM > > >>>>>>>> To: Rose, Gregory V; e1000-devel@lists.sourceforge.net > > >>>>>>>> Subject: Re: [E1000-devel] dropped rx with i40e > > >>>>>>>> > > >>>>>>>> Am 06.08.2015 um 00:22 schrieb Rose, Gregory V: > > >>>>>>>>> Stefan, > > >>>>>>>>> > > >>>>>>>>> Could you please send me the output of 'ethtool' and > > >>>>>>>>> 'ethtool - > > i' > > >>>>>>>>> for > > >>>>>>>> each i40e interface that is experiencing the dropped packets > > issue? > > >>>>>>>> > > >>>>>>>> These are around 100 cards. So i won't post the output for > > >>>>>>>> all of > > >>>> them. > > >>>>>>>> As they're all using the same driver and the same firmware - > > >>>>>>>> we updated all of them i hope it's ok to post the output only > > >>>>>>>> from one > > >>>> of them. > > >>>>>>>> > > >>>>>>>> # ethtool eth2 > > >>>>>>>> Settings for eth2: > > >>>>>>>> Supported ports: [ FIBRE ] > > >>>>>>>> Supported link modes: 10000baseT/Full > > >>>>>>>> Supported pause frame use: Symmetric > > >>>>>>>> Supports auto-negotiation: No > > >>>>>>>> Advertised link modes: Not reported > > >>>>>>>> Advertised pause frame use: No > > >>>>>>>> Advertised auto-negotiation: No > > >>>>>>>> Speed: 10000Mb/s > > >>>>>>>> Duplex: Full > > >>>>>>>> Port: Direct Attach Copper > > >>>>>>>> PHYAD: 0 > > >>>>>>>> Transceiver: external > > >>>>>>>> Auto-negotiation: off > > >>>>>>>> Supports Wake-on: g > > >>>>>>>> Wake-on: d > > >>>>>>>> Current message level: 0x0000000f (15) > > >>>>>>>> drv probe link timer > > >>>>>>>> Link detected: yes # ethtool -i eth2 > > >>>>>>>> driver: i40e > > >>>>>>>> version: 1.3.4-k > > >>>>>>>> firmware-version: f4.33.31377 a1.2 n4.42 e191b > > >>>>>>>> bus-info: 0000:03:00.0 > > >>>>>>>> supports-statistics: yes > > >>>>>>>> supports-test: yes > > >>>>>>>> supports-eeprom-access: yes > > >>>>>>>> supports-register-dump: yes > > >>>>>>>> supports-priv-flags: yes > > >>>>>>>> > > >>>>>>>>> Also, the system log might help also - dmesg can get that. > > >>>>>>>>> That'll > > >>>>>>>> give me something to look at. > > >>>>>>>> > > >>>>>>>> As this one is pretty long. i pasted dmesg to pastebin: > > >>>>>>>> http://pastebin.com/raw.php?i=7Tjp3eDT > > >>>>>>>> > > >>>>>>>>> By the way, have you tried using ethtool to turn adaptive RX > > >>>>>>>>> and TX off > > >>>>>>>> using ethtool to see if that has any impact on the dropped > > packets? > > >>>>>>>> > > >>>>>>>> I tested this one: > > >>>>>>>> ethtool -C eth3 adaptive-rx off adaptive-tx off rx-usecs 2 > > >>>>>>>> tx-usecs > > >>>>>>>> 0 > > >>>>>>>> > > >>>>>>>> but it has not helped. Still dropped rx packets. While a 2nd > > >>>>>>>> system receiving the same load using ixgbe has no dropped > > packets. > > >>>>>>>> > > >>>>>>>>> That might be an easy test to run. > > >>>>>>>> > > >>>>>>>> Thanks! > > >>>>>>>> > > >>>>>>>> Greets, > > >>>>>>>> Stefan > > >>>>>>>> > > >>>>>>>> > > >>>>>>>>> Thanks, > > >>>>>>>>> > > >>>>>>>>> - Greg > > >>>>>>>>> > > >>>>>>>>>> -----Original Message----- > > >>>>>>>>>> From: Stefan Priebe [mailto:s.pri...@profihost.ag] > > >>>>>>>>>> Sent: Wednesday, August 05, 2015 11:14 AM > > >>>>>>>>>> To: e1000-devel@lists.sourceforge.net > > >>>>>>>>>> Subject: Re: [E1000-devel] dropped rx with i40e > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> Something i've noticed: > > >>>>>>>>>> ixgbe: > > >>>>>>>>>> Adaptive RX: off TX: off > > >>>>>>>>>> rx-usecs: 1 > > >>>>>>>>>> tx-usecs: 0 > > >>>>>>>>>> > > >>>>>>>>>> i40e: > > >>>>>>>>>> Adaptive RX: on TX: on > > >>>>>>>>>> rx-usecs: 62 > > >>>>>>>>>> tx-usecs: 122 > > >>>>>>>>>> > > >>>>>>>>>> Stefan > > >>>>>>>>>> > > >>>>>>>>>> Am 05.08.2015 um 09:02 schrieb Stefan Priebe - Profihost AG: > > >>>>>>>>>>> Hello list, > > >>>>>>>>>>> > > >>>>>>>>>>> we're using the intel X520 cards with the ixgbe driver > > >>>>>>>>>>> since a long time for our cloud infrastructure. We never > > >>>>>>>>>>> had a problem with dropped packets and everything was always > fine. > > >>>>>>>>>>> > > >>>>>>>>>>> Since a year we started switching to the X710 cards as > > >>>>>>>>>>> they're better regarding their specs (lower power > > >>>>>>>>>>> consumption, lower latency, better price). > > >>>>>>>>>>> > > >>>>>>>>>>> We've around 100 X710 cards running now and we had a lot > > >>>>>>>>>>> of trouble with them. Back in 2014 there were a firmware > > >>>>>>>>>>> bug, then there were driver problems with bonding and so on. > > >>>>>>>>>>> > > >>>>>>>>>>> Now we have detected a new problem! We're seeing a lot of > > >>>>>>>>>>> rx_dropped packets on all X710 cards while all ixgbe based > > >>>>>>>>>>> cards are working > > >>>>>>>> fine. > > >>>>>>>>>>> > > >>>>>>>>>>> I've tested the 1.2.48 driver als also the latest 1.3.4-k > > >>>>>>>>>>> driver from 4.2-rc5. > > >>>>>>>>>>> > > >>>>>>>>>>> Can anybody help? > > >>>>>>>>>>> > > >>>>>>>>>>> Greets, > > >>>>>>>>>>> Stefan > > >>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> ----------------------------------------------------------- > > >>>>>>>>>> -- > > >>>>>>>>>> -- > > >>>>>>>>>> -- > > >>>>>>>>>> ---- > > >>>>>>>>>> ----- > > >>>>>>>>>> ---- > > >>>>>>>>>> _______________________________________________ > > >>>>>>>>>> E1000-devel mailing list > > >>>>>>>>>> E1000-devel@lists.sourceforge.net > > >>>>>>>>>> https://lists.sourceforge.net/lists/listinfo/e1000-devel > > >>>>>>>>>> To learn more about Intel® Ethernet, visit > > >>>>>>>>>> http://communities.intel.com/community/wired > > -------------------------------------------------------------------------- > ---- > _______________________________________________ > E1000-devel mailing list > E1000-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/e1000-devel > To learn more about Intel® Ethernet, visit > http://communities.intel.com/community/wired ------------------------------------------------------------------------------ _______________________________________________ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired
