I have experience with the Solaris driver, but not Illumos, so I won't be able to assist in any debug since the implementations differ too much, but I am familiar with the interfaces and can say that in the official i40e Solaris driver we've always used VMDq2 VSIs for the non-default ring groups, not PF VSIs.
I would try changing the ring group configuration in i40e_add_vsi() to use VMDq2 VSIs and see if that makes a difference. There shouldn't be any reason Illumos would specifically need PF VSIs here. http://src.illumos.org/source/xref/illumos-gate/usr/src/uts/common/io/i40e/i40e_main.c#2126 change: ctx.flags = I40E_AQ_VSI_TYPE_PF; to: ctx.flags = I40E_AQ_VSI_TYPE_VMDQ2; Thanks, Nick > -----Original Message----- > From: Alexander Duyck <alexander.du...@gmail.com> > Sent: Friday, December 13, 2019 11:27 AM > To: Youzhong Yang <yy...@mathworks.com> > Cc: e1000-devel@lists.sourceforge.net > Subject: Re: [E1000-devel] X710: very slow rx speed (a few MiB/second) > when using non-default VSI. > > Before you start complaining about what the driver is or isn't doing you > should probably understand that both the Linux and FreeBSD drivers do have > to create a PF VSI as that is the default VSI type if I am not mistaken. Do > you > know what the goal is in creating multiple PF VSIs? I don't see the reason for > doing that versus allocating VMDq2 VSIs? > > If you take a look at the i40e Linux driver it supports a feature called ADq > which can be used to create multiple VMDq2 VSIs. > > Nobody on this list likely has any experience with Illuminos. I would > recommend getting in touch with the maintainer for the Illuminos i40e driver > and working with them as they would know how that driver is supposed to > work and why the functionality is implemented the way it is. > > Thanks. > > - Alex > > On Fri, Dec 13, 2019 at 11:04 AM Youzhong Yang <yy...@mathworks.com> > wrote: > > > > As I said, a supported OS (Linux and FreeBSD?) does not create the kind of > VSI (PF VSI) in its i40e driver, are you asking me to test nothing on your > supported OS? > > > > "Intel(r) Ethernet Controller X710/ XXV710/XL710 Datasheet" claims it > supports PF VSIs, so why would you say it is not supported? Because your > customer is not using Linux and FreeBSD? > > > > It does not make any sense. Sorry. > > > > -----Original Message----- > > From: Fujinaka, Todd <todd.fujin...@intel.com> > > Sent: Friday, December 13, 2019 1:45 PM > > To: Youzhong Yang <yy...@mathworks.com>; > > e1000-devel@lists.sourceforge.net > > Subject: RE: X710: very slow rx speed (a few MiB/second) when using non- > default VSI. > > > > You can try official channels but you're just going to get an official > > "Talk to > Illuminos" from me. > > > > As I said, you can try installing a supported OS to see if it's a hardware > > issue, > but we have no support for Illuminos. > > > > Todd Fujinaka > > Software Application Engineer > > Datacenter Engineering Group > > Intel Corporation > > todd.fujin...@intel.com > > > > -----Original Message----- > > From: Youzhong Yang <yy...@mathworks.com> > > Sent: Friday, December 13, 2019 10:11 AM > > To: Fujinaka, Todd <todd.fujin...@intel.com>; > > e1000-devel@lists.sourceforge.net > > Subject: RE: X710: very slow rx speed (a few MiB/second) when using non- > default VSI. > > > > This mailing list seems quite quiet. Any hope my issue could get some > attention? If it is a thing of 'Only Linux and FreeBSD are supported', is > there > any other communication channel through which this issue can be escalated? > Thanks a lot for your understanding. > > > > -----Original Message----- > > From: Youzhong Yang <yy...@mathworks.com> > > Sent: Tuesday, December 3, 2019 12:21 PM > > To: Fujinaka, Todd <todd.fujin...@intel.com>; > > e1000-devel@lists.sourceforge.net > > Subject: RE: X710: very slow rx speed (a few MiB/second) when using non- > default VSI. > > > > Hello Todd, > > > > Thank you very much for replying so fast. I am sorry I missed your e-mail as > it was blocked by our company e-mail filters. > > > > I have contacted illumos dev twice but unfortunately there was no reply, it > seems the i40e driver developer is no longer active. I am trying to figure out > how I can fix the issue by myself, I know how to do driver development but > this particular issue seems to be very unique, and I suspect either something > is missing when creating the VSIs, or there might be a bug (possibly in the > Ethernet card firmware) in the way non-default PF VSI is created. > > > > I have researched the Linux and FreeBSD i40e driver code, none of them > does the same kind of thing our illumos driver does. For each i40e instance, > we create 31 PF VSIs in addition to the default one initiated by the card > firmware. When receiving data through the RX queues of the default VSI, the > performance is very good, but when receiving data through the RX queues of > the non-default VSI, the performance can be very poor. > > > > The "Intel(r) Ethernet Controller X710/ XXV710/XL710 Datasheet" suggests > there's nothing wrong by having 32 PF VSIs. Please refer to its page 20: > > > > "VSI Support > > The X710/XXV710/XL710 supports a total of 384 VSIs. VSI assignment is > flexible, but the choice to support 384 VSIs is motivated by the following > usage example: > > * 256 VSIs for VFs or VMDq2 > > * 32 VSIs for PFs" > > > > We purchased a bunch of XL710 cards, hoping it can deliver better > performance than the ixgbe cards. Sadly we were hit by this performance > issue, and I've tried troubleshooting/debugging for more than 2 months, but > so far no luck. > > > > Your kind assistance would be highly appreciated. > > > > Sincerely, > > > > -Youzhong Yang > > The MathWorks Inc. > > > > -----Original Message----- > > From: Fujinaka, Todd <todd.fujin...@intel.com> > > Sent: Monday, December 2, 2019 4:39 PM > > To: Youzhong Yang <yy...@mathworks.com>; > > e1000-devel@lists.sourceforge.net > > Subject: RE: X710: very slow rx speed (a few MiB/second) when using non- > default VSI. > > > > I think you're going to have to ask Illumos about this. Can you reproduce > this on a supported OS such as Red Hat Linux or FreeBSD? > > > > Todd Fujinaka > > Software Application Engineer > > Datacenter Engineering Group > > Intel Corporation > > todd.fujin...@intel.com > > > > -----Original Message----- > > From: Youzhong Yang <yy...@mathworks.com> > > Sent: Monday, December 2, 2019 12:19 PM > > To: e1000-devel@lists.sourceforge.net > > Subject: [E1000-devel] X710: very slow rx speed (a few MiB/second) when > using non-default VSI. > > > > Hello everyone, > > > > I submitted a help request on Intel website and was redirected to this > mailing list. The request can be found here: > > > > X710: very slow rx speed (a few MiB/second) when using non-default VSI. > > > > > > I experienced very strange performance issue when sending data from a > client to a server with X710 card. Details about the investigation can be > found > here: > > > > https://github.com/youzhongyang/i40e_investigation > > > > > > > > The server runs illumos (derived from open solaris), its i40e driver can be > found here: > > > > http://src.illumos.org/source/xref/illumos-gate/usr/src/uts/common/io/ > > i40e/ > > > > > > > > So for each i40e instance, we create 32 VSIs (including the default one > created by the firmware), each VSI is assigned 8 trq pairs. When a VIF/VNIC is > created, the OS assigns a VSI to it, data receiving through the VNIC's IP > will be > done by the relevant VSI's 8 RX rings. However, data sending will always go > through the default VSI's 8 TX rings. > > > > > > > > The function for creating non-default VSI is here: > > > > http://src.illumos.org/source/xref/illumos-gate/usr/src/uts/common/io/ > > i40e/i40e_main.c#2111 > > > > > > > > I've upgraded the firmware version to the latest, still the same issue. Is > > this > a known issue? or is there anything missing when setting up the non-default > VSI? > > > > > > > > Your help/advice would be highly appreciated. > > > > Sincerely, > > -Youzhong Yang > > > > > > _______________________________________________ > > E1000-devel mailing list > > E1000-devel@lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/e1000-devel > > To learn more about Intel Ethernet, visit > > https://forums.intel.com/s/topic/0TO0P00000018NbWAI/intel-ethernet > > > > > > > > _______________________________________________ > > E1000-devel mailing list > > E1000-devel@lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/e1000-devel > > To learn more about Intel Ethernet, visit > > https://forums.intel.com/s/topic/0TO0P00000018NbWAI/intel-ethernet > > > _______________________________________________ > E1000-devel mailing list > E1000-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/e1000-devel > To learn more about Intel Ethernet, visit > https://forums.intel.com/s/topic/0TO0P00000018NbWAI/intel-ethernet _______________________________________________ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel Ethernet, visit https://forums.intel.com/s/topic/0TO0P00000018NbWAI/intel-ethernet
