Thanks Alex and Nick. It is nice that finally we come to the technical aspect of it, and I really appreciate your input.
Changing type PF to type VMDQ2 was the first thing I have tried when debugging this issue. It didn't help. Yesterday I installed Solaris 11.4, create VNICs by setting "ring-group=exclusive", ran my reproduction programs, there was no performance issue. Obviously Intel knows how to do the proper initialization/configuration of VMDq2 VSIs. I am wondering what that is. If you could shed some light on it, that would be really helpful. Thanks, -Youzhong -----Original Message----- From: Nunley, Nicholas D <nicholas.d.nun...@intel.com> Sent: Friday, December 13, 2019 4:10 PM To: Alexander Duyck <alexander.du...@gmail.com>; Youzhong Yang <yy...@mathworks.com> Cc: e1000-devel@lists.sourceforge.net Subject: RE: [E1000-devel] X710: very slow rx speed (a few MiB/second) when using non-default VSI. I have experience with the Solaris driver, but not Illumos, so I won't be able to assist in any debug since the implementations differ too much, but I am familiar with the interfaces and can say that in the official i40e Solaris driver we've always used VMDq2 VSIs for the non-default ring groups, not PF VSIs. I would try changing the ring group configuration in i40e_add_vsi() to use VMDq2 VSIs and see if that makes a difference. There shouldn't be any reason Illumos would specifically need PF VSIs here. http://src.illumos.org/source/xref/illumos-gate/usr/src/uts/common/io/i40e/i40e_main.c#2126 change: ctx.flags = I40E_AQ_VSI_TYPE_PF; to: ctx.flags = I40E_AQ_VSI_TYPE_VMDQ2; Thanks, Nick > -----Original Message----- > From: Alexander Duyck <alexander.du...@gmail.com> > Sent: Friday, December 13, 2019 11:27 AM > To: Youzhong Yang <yy...@mathworks.com> > Cc: e1000-devel@lists.sourceforge.net > Subject: Re: [E1000-devel] X710: very slow rx speed (a few MiB/second) > when using non-default VSI. > > Before you start complaining about what the driver is or isn't doing > you should probably understand that both the Linux and FreeBSD drivers > do have to create a PF VSI as that is the default VSI type if I am not > mistaken. Do you know what the goal is in creating multiple PF VSIs? I > don't see the reason for doing that versus allocating VMDq2 VSIs? > > If you take a look at the i40e Linux driver it supports a feature > called ADq which can be used to create multiple VMDq2 VSIs. > > Nobody on this list likely has any experience with Illuminos. I would > recommend getting in touch with the maintainer for the Illuminos i40e > driver and working with them as they would know how that driver is > supposed to work and why the functionality is implemented the way it is. > > Thanks. > > - Alex > > On Fri, Dec 13, 2019 at 11:04 AM Youzhong Yang <yy...@mathworks.com> > wrote: > > > > As I said, a supported OS (Linux and FreeBSD?) does not create the > > kind of > VSI (PF VSI) in its i40e driver, are you asking me to test nothing on > your supported OS? > > > > "Intel(r) Ethernet Controller X710/ XXV710/XL710 Datasheet" claims > > it > supports PF VSIs, so why would you say it is not supported? Because > your customer is not using Linux and FreeBSD? > > > > It does not make any sense. Sorry. > > > > -----Original Message----- > > From: Fujinaka, Todd <todd.fujin...@intel.com> > > Sent: Friday, December 13, 2019 1:45 PM > > To: Youzhong Yang <yy...@mathworks.com>; > > e1000-devel@lists.sourceforge.net > > Subject: RE: X710: very slow rx speed (a few MiB/second) when using > > non- > default VSI. > > > > You can try official channels but you're just going to get an > > official "Talk to > Illuminos" from me. > > > > As I said, you can try installing a supported OS to see if it's a > > hardware issue, > but we have no support for Illuminos. > > > > Todd Fujinaka > > Software Application Engineer > > Datacenter Engineering Group > > Intel Corporation > > todd.fujin...@intel.com > > > > -----Original Message----- > > From: Youzhong Yang <yy...@mathworks.com> > > Sent: Friday, December 13, 2019 10:11 AM > > To: Fujinaka, Todd <todd.fujin...@intel.com>; > > e1000-devel@lists.sourceforge.net > > Subject: RE: X710: very slow rx speed (a few MiB/second) when using > > non- > default VSI. > > > > This mailing list seems quite quiet. Any hope my issue could get > > some > attention? If it is a thing of 'Only Linux and FreeBSD are supported', > is there any other communication channel through which this issue can be > escalated? > Thanks a lot for your understanding. > > > > -----Original Message----- > > From: Youzhong Yang <yy...@mathworks.com> > > Sent: Tuesday, December 3, 2019 12:21 PM > > To: Fujinaka, Todd <todd.fujin...@intel.com>; > > e1000-devel@lists.sourceforge.net > > Subject: RE: X710: very slow rx speed (a few MiB/second) when using > > non- > default VSI. > > > > Hello Todd, > > > > Thank you very much for replying so fast. I am sorry I missed your > > e-mail as > it was blocked by our company e-mail filters. > > > > I have contacted illumos dev twice but unfortunately there was no > > reply, it > seems the i40e driver developer is no longer active. I am trying to > figure out how I can fix the issue by myself, I know how to do driver > development but this particular issue seems to be very unique, and I > suspect either something is missing when creating the VSIs, or there > might be a bug (possibly in the Ethernet card firmware) in the way > non-default PF VSI is created. > > > > I have researched the Linux and FreeBSD i40e driver code, none of > > them > does the same kind of thing our illumos driver does. For each i40e > instance, we create 31 PF VSIs in addition to the default one > initiated by the card firmware. When receiving data through the RX > queues of the default VSI, the performance is very good, but when > receiving data through the RX queues of the non-default VSI, the performance > can be very poor. > > > > The "Intel(r) Ethernet Controller X710/ XXV710/XL710 Datasheet" > > suggests > there's nothing wrong by having 32 PF VSIs. Please refer to its page 20: > > > > "VSI Support > > The X710/XXV710/XL710 supports a total of 384 VSIs. VSI assignment > > is > flexible, but the choice to support 384 VSIs is motivated by the > following usage example: > > * 256 VSIs for VFs or VMDq2 > > * 32 VSIs for PFs" > > > > We purchased a bunch of XL710 cards, hoping it can deliver better > performance than the ixgbe cards. Sadly we were hit by this > performance issue, and I've tried troubleshooting/debugging for more > than 2 months, but so far no luck. > > > > Your kind assistance would be highly appreciated. > > > > Sincerely, > > > > -Youzhong Yang > > The MathWorks Inc. > > > > -----Original Message----- > > From: Fujinaka, Todd <todd.fujin...@intel.com> > > Sent: Monday, December 2, 2019 4:39 PM > > To: Youzhong Yang <yy...@mathworks.com>; > > e1000-devel@lists.sourceforge.net > > Subject: RE: X710: very slow rx speed (a few MiB/second) when using > > non- > default VSI. > > > > I think you're going to have to ask Illumos about this. Can you > > reproduce > this on a supported OS such as Red Hat Linux or FreeBSD? > > > > Todd Fujinaka > > Software Application Engineer > > Datacenter Engineering Group > > Intel Corporation > > todd.fujin...@intel.com > > > > -----Original Message----- > > From: Youzhong Yang <yy...@mathworks.com> > > Sent: Monday, December 2, 2019 12:19 PM > > To: e1000-devel@lists.sourceforge.net > > Subject: [E1000-devel] X710: very slow rx speed (a few MiB/second) > > when > using non-default VSI. > > > > Hello everyone, > > > > I submitted a help request on Intel website and was redirected to > > this > mailing list. The request can be found here: > > > > X710: very slow rx speed (a few MiB/second) when using non-default VSI. > > > > > > I experienced very strange performance issue when sending data from > > a > client to a server with X710 card. Details about the investigation can > be found > here: > > > > https://github.com/youzhongyang/i40e_investigation > > ub.com > > > > > > > > The server runs illumos (derived from open solaris), its i40e driver > > can be > found here: > > > > http://src.illumos.org/source/xref/illumos-gate/usr/src/uts/common/io/. > > illumos.org > > i40e/ > > > > > > > > So for each i40e instance, we create 32 VSIs (including the default > > one > created by the firmware), each VSI is assigned 8 trq pairs. When a > VIF/VNIC is created, the OS assigns a VSI to it, data receiving > through the VNIC's IP will be done by the relevant VSI's 8 RX rings. > However, data sending will always go through the default VSI's 8 TX rings. > > > > > > > > The function for creating non-default VSI is here: > > > > http://src.illumos.org/source/xref/illumos-gate/usr/src/uts/common/io/. > > illumos.org > > i40e/i40e_main.c#2111 > > > > > > > > I've upgraded the firmware version to the latest, still the same > > issue. Is this > a known issue? or is there anything missing when setting up the > non-default VSI? > > > > > > > > Your help/advice would be highly appreciated. > > > > Sincerely, > > -Youzhong Yang > > > > > > _______________________________________________ > > E1000-devel mailing list > > E1000-devel@lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/e1000-devel > > s.sourceforge.net To learn more about Intel Ethernet, visit > > https://forums.intel.com/s/topic/0TO0P00000018NbWAI/intel-ethernet > > ms.intel.com > > > > > > > > _______________________________________________ > > E1000-devel mailing list > > E1000-devel@lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/e1000-devel > > s.sourceforge.net To learn more about Intel Ethernet, visit > > https://forums.intel.com/s/topic/0TO0P00000018NbWAI/intel-ethernet > > ms.intel.com > > > _______________________________________________ > E1000-devel mailing list > E1000-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/e1000-devel. > sourceforge.net To learn more about Intel Ethernet, visit > https://forums.intel.com/s/topic/0TO0P00000018NbWAI/intel-ethernet _______________________________________________ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel Ethernet, visit https://forums.intel.com/s/topic/0TO0P00000018NbWAI/intel-ethernet
