Hello Todd, Thank you very much for replying so fast. I am sorry I missed your e-mail as it was blocked by our company e-mail filters.
I have contacted illumos dev twice but unfortunately there was no reply, it seems the i40e driver developer is no longer active. I am trying to figure out how I can fix the issue by myself, I know how to do driver development but this particular issue seems to be very unique, and I suspect either something is missing when creating the VSIs, or there might be a bug (possibly in the Ethernet card firmware) in the way non-default PF VSI is created. I have researched the Linux and FreeBSD i40e driver code, none of them does the same kind of thing our illumos driver does. For each i40e instance, we create 31 PF VSIs in addition to the default one initiated by the card firmware. When receiving data through the RX queues of the default VSI, the performance is very good, but when receiving data through the RX queues of the non-default VSI, the performance can be very poor. The "Intel(r) Ethernet Controller X710/ XXV710/XL710 Datasheet" suggests there's nothing wrong by having 32 PF VSIs. Please refer to its page 20: "VSI Support The X710/XXV710/XL710 supports a total of 384 VSIs. VSI assignment is flexible, but the choice to support 384 VSIs is motivated by the following usage example: * 256 VSIs for VFs or VMDq2 * 32 VSIs for PFs" We purchased a bunch of XL710 cards, hoping it can deliver better performance than the ixgbe cards. Sadly we were hit by this performance issue, and I've tried troubleshooting/debugging for more than 2 months, but so far no luck. Your kind assistance would be highly appreciated. Sincerely, -Youzhong Yang The MathWorks Inc. -----Original Message----- From: Fujinaka, Todd <todd.fujin...@intel.com> Sent: Monday, December 2, 2019 4:39 PM To: Youzhong Yang <yy...@mathworks.com>; e1000-devel@lists.sourceforge.net Subject: RE: X710: very slow rx speed (a few MiB/second) when using non-default VSI. I think you're going to have to ask Illumos about this. Can you reproduce this on a supported OS such as Red Hat Linux or FreeBSD? Todd Fujinaka Software Application Engineer Datacenter Engineering Group Intel Corporation todd.fujin...@intel.com -----Original Message----- From: Youzhong Yang <yy...@mathworks.com> Sent: Monday, December 2, 2019 12:19 PM To: e1000-devel@lists.sourceforge.net Subject: [E1000-devel] X710: very slow rx speed (a few MiB/second) when using non-default VSI. Hello everyone, I submitted a help request on Intel website and was redirected to this mailing list. The request can be found here: X710: very slow rx speed (a few MiB/second) when using non-default VSI. I experienced very strange performance issue when sending data from a client to a server with X710 card. Details about the investigation can be found here: https://github.com/youzhongyang/i40e_investigation The server runs illumos (derived from open solaris), its i40e driver can be found here: http://src.illumos.org/source/xref/illumos-gate/usr/src/uts/common/io/i40e/ So for each i40e instance, we create 32 VSIs (including the default one created by the firmware), each VSI is assigned 8 trq pairs. When a VIF/VNIC is created, the OS assigns a VSI to it, data receiving through the VNIC's IP will be done by the relevant VSI's 8 RX rings. However, data sending will always go through the default VSI's 8 TX rings. The function for creating non-default VSI is here: http://src.illumos.org/source/xref/illumos-gate/usr/src/uts/common/io/i40e/i40e_main.c#2111 I've upgraded the firmware version to the latest, still the same issue. Is this a known issue? or is there anything missing when setting up the non-default VSI? Your help/advice would be highly appreciated. Sincerely, -Youzhong Yang _______________________________________________ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel Ethernet, visit https://forums.intel.com/s/topic/0TO0P00000018NbWAI/intel-ethernet _______________________________________________ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel Ethernet, visit https://forums.intel.com/s/topic/0TO0P00000018NbWAI/intel-ethernet
