I went through the configuration in i40e_add_vsi() and it looks correct to me, assuming it's updated to use VMDq2 VSIs. If you're observing that VNICs are receiving traffic with VMDq2 VSIs (albeit very slowly) then I wonder whether the issue is even in the HW VSI context configuration. If you want to double check the correctness of the illumos VSI configuration yourself then you should look at the Linux i40e driver as Alex suggested, though, since that's the de facto guide for proven hardware configurations.
For an additional debugging datapoint you might try manually swapping around the ring-group-index-to-VSI mappings (via i40e_fill_rx_group(): infop->mgi_driver and I40E_DEF_VSI_SEID) and check whether the "good" behavior you see sticks around on the default HW VSI or the default group. I'm not sure if it works this way on illumos, but on later versions of Solaris I remember seeing a quirk where if you have multiple ring groups enabled then the GLD layer will use nonzero group indices even for the default traffic, so you may already be using the non-default VSIs in the working case. Otherwise, you could also try snooping the traffic on the VNIC to see if you notice anything noteworthy about the few packets you do receive. You might also want to verify that interrupts and/or polling are working as expected for all groups. I think the first step would be to try and see if you can isolate the configuration differences between the working and nonworking ring groups/VSIs, though, to see if that could explain any of the behavior. Unfortunately, there's not a lot we can directly help with here since illumos is not a supported OS. None of us have any experience with the illumos driver and there are simply too many ways it could be misconfiguring itself and/or the hardware to make any assumptions about which parts might be working. As others have mentioned, the only official advice we can give is to contact the illumos maintainers since they're the ones providing support for the driver. Thanks, Nick > -----Original Message----- > From: Youzhong Yang <yy...@mathworks.com> > Sent: Monday, December 16, 2019 6:06 AM > To: Nunley, Nicholas D <nicholas.d.nun...@intel.com>; Alexander Duyck > <alexander.du...@gmail.com> > Cc: e1000-devel@lists.sourceforge.net > Subject: RE: [E1000-devel] X710: very slow rx speed (a few MiB/second) > when using non-default VSI. > > Thanks Alex and Nick. It is nice that finally we come to the technical aspect > of > it, and I really appreciate your input. > > Changing type PF to type VMDQ2 was the first thing I have tried when > debugging this issue. It didn't help. > > Yesterday I installed Solaris 11.4, create VNICs by setting "ring- > group=exclusive", ran my reproduction programs, there was no performance > issue. > > Obviously Intel knows how to do the proper initialization/configuration of > VMDq2 VSIs. I am wondering what that is. If you could shed some light on it, > that would be really helpful. > > Thanks, > > -Youzhong > > -----Original Message----- > From: Nunley, Nicholas D <nicholas.d.nun...@intel.com> > Sent: Friday, December 13, 2019 4:10 PM > To: Alexander Duyck <alexander.du...@gmail.com>; Youzhong Yang > <yy...@mathworks.com> > Cc: e1000-devel@lists.sourceforge.net > Subject: RE: [E1000-devel] X710: very slow rx speed (a few MiB/second) > when using non-default VSI. > > I have experience with the Solaris driver, but not Illumos, so I won't be able > to assist in any debug since the implementations differ too much, but I am > familiar with the interfaces and can say that in the official i40e Solaris > driver > we've always used VMDq2 VSIs for the non-default ring groups, not PF VSIs. > > I would try changing the ring group configuration in i40e_add_vsi() to use > VMDq2 VSIs and see if that makes a difference. There shouldn't be any > reason Illumos would specifically need PF VSIs here. > > http://src.illumos.org/source/xref/illumos- > gate/usr/src/uts/common/io/i40e/i40e_main.c#2126 > change: > ctx.flags = I40E_AQ_VSI_TYPE_PF; > to: > ctx.flags = I40E_AQ_VSI_TYPE_VMDQ2; > > Thanks, > Nick > > > -----Original Message----- > > From: Alexander Duyck <alexander.du...@gmail.com> > > Sent: Friday, December 13, 2019 11:27 AM > > To: Youzhong Yang <yy...@mathworks.com> > > Cc: e1000-devel@lists.sourceforge.net > > Subject: Re: [E1000-devel] X710: very slow rx speed (a few MiB/second) > > when using non-default VSI. > > > > Before you start complaining about what the driver is or isn't doing > > you should probably understand that both the Linux and FreeBSD drivers > > do have to create a PF VSI as that is the default VSI type if I am not > > mistaken. Do you know what the goal is in creating multiple PF VSIs? I > > don't see the reason for doing that versus allocating VMDq2 VSIs? > > > > If you take a look at the i40e Linux driver it supports a feature > > called ADq which can be used to create multiple VMDq2 VSIs. > > > > Nobody on this list likely has any experience with Illuminos. I would > > recommend getting in touch with the maintainer for the Illuminos i40e > > driver and working with them as they would know how that driver is > > supposed to work and why the functionality is implemented the way it is. > > > > Thanks. > > > > - Alex > > > > On Fri, Dec 13, 2019 at 11:04 AM Youzhong Yang <yy...@mathworks.com> > > wrote: > > > > > > As I said, a supported OS (Linux and FreeBSD?) does not create the > > > kind of > > VSI (PF VSI) in its i40e driver, are you asking me to test nothing on > > your supported OS? > > > > > > "Intel(r) Ethernet Controller X710/ XXV710/XL710 Datasheet" claims > > > it > > supports PF VSIs, so why would you say it is not supported? Because > > your customer is not using Linux and FreeBSD? > > > > > > It does not make any sense. Sorry. > > > > > > -----Original Message----- > > > From: Fujinaka, Todd <todd.fujin...@intel.com> > > > Sent: Friday, December 13, 2019 1:45 PM > > > To: Youzhong Yang <yy...@mathworks.com>; > > > e1000-devel@lists.sourceforge.net > > > Subject: RE: X710: very slow rx speed (a few MiB/second) when using > > > non- > > default VSI. > > > > > > You can try official channels but you're just going to get an > > > official "Talk to > > Illuminos" from me. > > > > > > As I said, you can try installing a supported OS to see if it's a > > > hardware issue, > > but we have no support for Illuminos. > > > > > > Todd Fujinaka > > > Software Application Engineer > > > Datacenter Engineering Group > > > Intel Corporation > > > todd.fujin...@intel.com > > > > > > -----Original Message----- > > > From: Youzhong Yang <yy...@mathworks.com> > > > Sent: Friday, December 13, 2019 10:11 AM > > > To: Fujinaka, Todd <todd.fujin...@intel.com>; > > > e1000-devel@lists.sourceforge.net > > > Subject: RE: X710: very slow rx speed (a few MiB/second) when using > > > non- > > default VSI. > > > > > > This mailing list seems quite quiet. Any hope my issue could get > > > some > > attention? If it is a thing of 'Only Linux and FreeBSD are supported', > > is there any other communication channel through which this issue can be > escalated? > > Thanks a lot for your understanding. > > > > > > -----Original Message----- > > > From: Youzhong Yang <yy...@mathworks.com> > > > Sent: Tuesday, December 3, 2019 12:21 PM > > > To: Fujinaka, Todd <todd.fujin...@intel.com>; > > > e1000-devel@lists.sourceforge.net > > > Subject: RE: X710: very slow rx speed (a few MiB/second) when using > > > non- > > default VSI. > > > > > > Hello Todd, > > > > > > Thank you very much for replying so fast. I am sorry I missed your > > > e-mail as > > it was blocked by our company e-mail filters. > > > > > > I have contacted illumos dev twice but unfortunately there was no > > > reply, it > > seems the i40e driver developer is no longer active. I am trying to > > figure out how I can fix the issue by myself, I know how to do driver > > development but this particular issue seems to be very unique, and I > > suspect either something is missing when creating the VSIs, or there > > might be a bug (possibly in the Ethernet card firmware) in the way non- > default PF VSI is created. > > > > > > I have researched the Linux and FreeBSD i40e driver code, none of > > > them > > does the same kind of thing our illumos driver does. For each i40e > > instance, we create 31 PF VSIs in addition to the default one > > initiated by the card firmware. When receiving data through the RX > > queues of the default VSI, the performance is very good, but when > > receiving data through the RX queues of the non-default VSI, the > performance can be very poor. > > > > > > The "Intel(r) Ethernet Controller X710/ XXV710/XL710 Datasheet" > > > suggests > > there's nothing wrong by having 32 PF VSIs. Please refer to its page 20: > > > > > > "VSI Support > > > The X710/XXV710/XL710 supports a total of 384 VSIs. VSI assignment > > > is > > flexible, but the choice to support 384 VSIs is motivated by the > > following usage example: > > > * 256 VSIs for VFs or VMDq2 > > > * 32 VSIs for PFs" > > > > > > We purchased a bunch of XL710 cards, hoping it can deliver better > > performance than the ixgbe cards. Sadly we were hit by this > > performance issue, and I've tried troubleshooting/debugging for more > > than 2 months, but so far no luck. > > > > > > Your kind assistance would be highly appreciated. > > > > > > Sincerely, > > > > > > -Youzhong Yang > > > The MathWorks Inc. > > > > > > -----Original Message----- > > > From: Fujinaka, Todd <todd.fujin...@intel.com> > > > Sent: Monday, December 2, 2019 4:39 PM > > > To: Youzhong Yang <yy...@mathworks.com>; > > > e1000-devel@lists.sourceforge.net > > > Subject: RE: X710: very slow rx speed (a few MiB/second) when using > > > non- > > default VSI. > > > > > > I think you're going to have to ask Illumos about this. Can you > > > reproduce > > this on a supported OS such as Red Hat Linux or FreeBSD? > > > > > > Todd Fujinaka > > > Software Application Engineer > > > Datacenter Engineering Group > > > Intel Corporation > > > todd.fujin...@intel.com > > > > > > -----Original Message----- > > > From: Youzhong Yang <yy...@mathworks.com> > > > Sent: Monday, December 2, 2019 12:19 PM > > > To: e1000-devel@lists.sourceforge.net > > > Subject: [E1000-devel] X710: very slow rx speed (a few MiB/second) > > > when > > using non-default VSI. > > > > > > Hello everyone, > > > > > > I submitted a help request on Intel website and was redirected to > > > this > > mailing list. The request can be found here: > > > > > > X710: very slow rx speed (a few MiB/second) when using non-default > VSI. > > > > > > > > > I experienced very strange performance issue when sending data from > > > a > > client to a server with X710 card. Details about the investigation can > > be found > > here: > > > > > > https://github.com/youzhongyang/i40e_investigation > > > ub.com > > > > > > > > > > > > The server runs illumos (derived from open solaris), its i40e driver > > > can be > > found here: > > > > > > http://src.illumos.org/source/xref/illumos- > gate/usr/src/uts/common/io/. > > > illumos.org > > > i40e/ > > > > > > > > > > > > So for each i40e instance, we create 32 VSIs (including the default > > > one > > created by the firmware), each VSI is assigned 8 trq pairs. When a > > VIF/VNIC is created, the OS assigns a VSI to it, data receiving > > through the VNIC's IP will be done by the relevant VSI's 8 RX rings. > > However, data sending will always go through the default VSI's 8 TX rings. > > > > > > > > > > > > The function for creating non-default VSI is here: > > > > > > http://src.illumos.org/source/xref/illumos- > gate/usr/src/uts/common/io/. > > > illumos.org > > > i40e/i40e_main.c#2111 > > > > > > > > > > > > I've upgraded the firmware version to the latest, still the same > > > issue. Is this > > a known issue? or is there anything missing when setting up the > > non-default VSI? > > > > > > > > > > > > Your help/advice would be highly appreciated. > > > > > > Sincerely, > > > -Youzhong Yang > > > > > > > > > _______________________________________________ > > > E1000-devel mailing list > > > E1000-devel@lists.sourceforge.net > > > https://lists.sourceforge.net/lists/listinfo/e1000-devel > > > s.sourceforge.net To learn more about Intel Ethernet, visit > > > https://forums.intel.com/s/topic/0TO0P00000018NbWAI/intel-ethernet > > > ms.intel.com > > > > > > > > > > > > _______________________________________________ > > > E1000-devel mailing list > > > E1000-devel@lists.sourceforge.net > > > https://lists.sourceforge.net/lists/listinfo/e1000-devel > > > s.sourceforge.net To learn more about Intel Ethernet, visit > > > https://forums.intel.com/s/topic/0TO0P00000018NbWAI/intel-ethernet > > > ms.intel.com > > > > > > _______________________________________________ > > E1000-devel mailing list > > E1000-devel@lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/e1000-devel. > > sourceforge.net To learn more about Intel Ethernet, visit > > https://forums.intel.com/s/topic/0TO0P00000018NbWAI/intel-ethernet _______________________________________________ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel Ethernet, visit https://forums.intel.com/s/topic/0TO0P00000018NbWAI/intel-ethernet
