That's interesting! We had issues in the past with LLDP and LACP but it "only" caused bundles to flap, not some funky speed. I'm trying to re-read the spec, but I don't see any specific requirements for LLDP to be enabled for LACP to actually work. I do believe it LACP has its own negotiation/handshake setup.
On Mon, Mar 14, 2022 at 9:57 PM Johnson, Brian <brian.john...@intel.com> wrote: > The LLDP FW Agent only “eats” LLDP packets but it can confuse LACP, OVS or > other sw bridges since they expect LLDP info from the TOR. With LACP the > bond sw does not receive anything from the TOR so it might think the port > is down. The FW agent also response back with the same MAC on both ports so > that can also cause packets to get dropped or sent out of order. The > behavior and side effects can be hard to troubleshoot to figure out the > root cause. The original design was to simplify the setup of DCB for FCoE > and provide TOR information out of band to lifecycle controllers via a BMC. > Since the initial release we have been making updates and improvements X710 > (700 Series) and significant changes to the implementation on 800 Series to > address the issues. > > I have brought issue back up with our engineering and support team to work > with our OEM server and ISVs partners to help improve the communication of > the issue. > > Brian Johnson > Intel Corp > Sent from my iPhone > > > On Mar 14, 2022, at 5:02 PM, Laurent Dumont <laurentfdum...@gmail.com> > wrote: > > > > Wow! That is good to know. We had issues in the past with hardware lldp > > agent eating our OS lldp packets. > > > > But never eating the actual data packets! > > > >> On Mon, Mar 14, 2022, 7:43 PM Matthew Weiner < > mlwei...@lakelandschools.org> > >> wrote: > >> > >> The issue was with the integrated LLDP daemon. Disabling it according > to > >> Jesse Brandeburg's excellent recommendation solved the issue that both > >> Citrix and Dell couldn't figure out. Running the following on all NICs > in > >> the LACP bond brought my performance from 5 megabit to full ten gigabit > >> line speed. > >> > >> ethtool --set-priv-flags (interface) disable-fw-lldp on > >> > >> Just a heads up for my fellow Dell owners, there is a setting to turn > this > >> off in the UEFI setup, but it appears it's able to be overridden by the > >> driver so even though you thought you disabled it in the UEFI firmware > >> options, it can still be in effect until you use ethtool to disable it. > >> > >> Thanks again to Jesse Brandeburg for getting me out of this really > >> tough situation! > >> > >> > >> On Mon, Mar 14, 2022 at 5:47 PM Laurent Dumont < > laurentfdum...@gmail.com> > >> wrote: > >> > >>> That's really weird. > >>> > >>> - How are you measuring performance? iperf? > >>> - Are you able to put a Ubuntu/other OS directly on the R740 and > >>> validate the performance? > >>> > >>> We have a lot of X710 with various firmware and I have never seen > >>> something like this. > >>> > >>> On Mon, Mar 14, 2022 at 2:49 PM Matthew Weiner < > >>> mlwei...@lakelandschools.org> wrote: > >>> > >>>> I'll give that a try after my day ends, just in case it drops the > >> link. I > >>>> had a problem before where I changed some offload settings using > ethtool > >>>> and the links went hard down and the only way I could bring them back > up > >>>> was a reboot. I'll let you the results after I make the change. > >>>> > >>>> My kernel is 4.19.0+1 - it's Citrix Hypervisor (formerly XenServer) > 8.2 > >>>> LTSR. > >>>> > >>>> > >>>> > >>>> On Mon, Mar 14, 2022 at 1:21 PM Jesse Brandeburg < > >>>> jesse.brandeb...@intel.com> > >>>> wrote: > >>>> > >>>>> On 3/14/2022 8:28 AM, Kevin Bowling wrote: > >>>>>> Fortville (700) has always been a bit of a disaster > >>>>>> ( > >>>> https://cdrdv2.intel.com/v1/dl/getContent/331430?explicitVersion=true > ), > >>>>>> I'd see if you can press your Intel reps into getting you the 550s > >> or > >>>>>> the 800-series NICs for the unnecessary troubles it's a much nicer > >>>>>> design. > >>>>>> > >>>>>> It's surprising they are shipping new cards with that old of a > >>>>>> firmware, you should be on 8.50 for the driver you are running > >>>>>> ( > >>>>> > >>>> > >> > https://www.intel.com/content/www/us/en/download/18635/non-volatile-memory-nvm-update-utility-for-intel-ethernet-adapters-700-series-linux.html > >>>>> ). > >>>>>> Doing the FW update is worth a shot but most issues I've seen have > >>>>>> been driver related and you are running a pretty recent driver. > >>>>>> > >>>>>> Regards, > >>>>>> Kevin > >>>>>> > >>>>>> On Mon, Mar 14, 2022 at 7:44 AM Matthew Weiner > >>>>>> <mlwei...@lakelandschools.org> wrote: > >>>>>>> > >>>>>>> I'm at my wits end with this, Citrix is stumped, Dell is stumped, > >> and > >>>>> with > >>>>>>> the supply chain issues the way they are we can't just yank these > >>>> out in > >>>>>>> favor of X550s. The problem is we have a group of Dell R740s with > >>>> X710 > >>>>>>> dual-port NICs and the performance is, in a word, awful. Like 5-6 > >>>>> megabit > >>>>>>> upload and 250 megabit download awful. However, identical server > >>>>> hardware > >>>>>>> with any other card, be it a Broadcom or an Intel X550T, no issues. > >>>> We > >>>>> can > >>>>>>> get line rate all day long. The latest attempt was swapping the > >> X710 > >>>>> for a > >>>>>>> newer X710-T2L-t, which performed maybe 5-10 percent better. We've > >>>>> tried > >>>>>>> three different driver revisions, firmware, BIOS, all the available > >>>>>>> Hypervisor updates, it still performs the same. > >>>>>>> > >>>>>>> The servers in question have X550s on the motherboard mezzanine > >> card > >>>>> which > >>>>>>> perform fine, and a single dual-port X710 in the PCIe riser. The > >>>> X710 > >>>>> is > >>>>>>> set up with an LACP pair trunked with three VLANs tagged across it. > >>>> In > >>>>>>> this pool we also have servers with X550s on the PCIe cards, and > >>>>>>> Broadcoms. All those with an identical configuration perform > >> without > >>>>>>> issue, it's only the X710s that show this problem. > >>>>> > >>>>> Hi Matt, sorry to hear about this problem. Let's poke a bit (please > be > >>>>> patient with me) and see if we can help you. > >>>>> > >>>>> Have you followed the steps like located here: > >>>>> > >>>>> > >>>> > >> > https://www.thomas-krenn.com/en/wiki/Intel_Ethernet_700_Series_LACP_Configuration > >>>>> > >>>>> As there are definitely known problems with LACP mode and the > driver's > >>>>> default settings. > >>>>> > >>>>> You can try the above workaround and see if it helps. If that does > >> help, > >>>>> then there are ways to make the settings get applied by ethtool as > the > >>>>> system comes up. > >>>>> > >>>>> Please let us know how it goes. > >>>>> > >>>>> It would be helpful to know what kernel you're running, just for good > >>>>> measure. > >>>>> > >>>>> PS. You may want to subscribe to e1000-devel as it is currently > >> holding > >>>>> your messages because you're not a subscriber, and they have to be > >>>>> manually released. > >>>>> > >>>> > >>>> _______________________________________________ > >>>> E1000-devel mailing list > >>>> E1000-devel@lists.sourceforge.net > >>>> https://lists.sourceforge.net/lists/listinfo/e1000-devel > >>>> To learn more about Intel Ethernet, visit > >>>> https://forums.intel.com/s/topic/0TO0P00000018NbWAI/intel-ethernet > >>>> > >>> > >> > >> _______________________________________________ > >> E1000-devel mailing list > >> E1000-devel@lists.sourceforge.net > >> https://lists.sourceforge.net/lists/listinfo/e1000-devel > >> To learn more about Intel Ethernet, visit > >> https://forums.intel.com/s/topic/0TO0P00000018NbWAI/intel-ethernet > >> > > > > _______________________________________________ > > E1000-devel mailing list > > E1000-devel@lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/e1000-devel > > To learn more about Intel Ethernet, visit > https://forums.intel.com/s/topic/0TO0P00000018NbWAI/intel-ethernet > _______________________________________________ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel Ethernet, visit https://forums.intel.com/s/topic/0TO0P00000018NbWAI/intel-ethernet
