So, here's what I see and the versions of the tools I'm running. Perhaps your
version of ethtool is too old?
[htfujina@htfujina-fc ~]$ sudo ethtool -K p128p1 ntuple off
[htfujina@htfujina-fc ~]$ sudo ethtool -k p128p1
Features for p128p1:
rx-checksumming: on
tx-checksumming: on
tx-checksum-ipv4: on
tx-checksum-ip-generic: off [fixed]
tx-checksum-ipv6: on
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: on
scatter-gather: on
tx-scatter-gather: on
tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
tx-tcp-segmentation: on
tx-tcp-ecn-segmentation: on
tx-tcp6-segmentation: on
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off
...
[htfujina@htfujina-fc ~]$ sudo ethtool -K p128p1 ntuple on
[htfujina@htfujina-fc ~]$ sudo ethtool -k p128p1
Features for p128p1:
rx-checksumming: on
tx-checksumming: on
tx-checksum-ipv4: on
tx-checksum-ip-generic: off [fixed]
tx-checksum-ipv6: on
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: on
scatter-gather: on
tx-scatter-gather: on
tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
tx-tcp-segmentation: on
tx-tcp-ecn-segmentation: on
tx-tcp6-segmentation: on
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: on
...
[htfujina@htfujina-fc ~]$ sudo ethtool -i p128p1
driver: i40e
version: 1.3.6-k
firmware-version: f4.33.31377 a1.2 n4.42 e1932
bus-info: 0000:04:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes
[htfujina@htfujina-fc ~]$ ethtool --version
ethtool version 3.18
[htfujina@htfujina-fc ~]$
Todd Fujinaka
Software Application Engineer
Networking Division (ND)
Intel Corporation
[email protected]
(503) 712-4565
-----Original Message-----
From: Hu, Tan Chang [mailto:[email protected]]
Sent: Wednesday, July 29, 2015 7:56 AM
To: Fujinaka, Todd; e1000-devel@lists. sourceforge. net
([email protected])
Subject: RE: [EXTERNAL] RE: TCP stack is receiving out of order packer delivery
even when connected back-to-back
[root@r445403 src]# ethtool -K i40e1 ntuple off Could not change any device
features
[root@r445403 src]# ethtool -k i40e1 | grep ntuple
ntuple-filters: on
-----Original Message-----
From: Fujinaka, Todd [mailto:[email protected]]
Sent: Wednesday, July 29, 2015 8:53 AM
To: Hu, Tan Chang; e1000-devel@lists. sourceforge. net
([email protected])
Subject: RE: [EXTERNAL] RE: TCP stack is receiving out of order packer delivery
even when connected back-to-back
What did you try to disable flow director? This may be a bug.
Todd Fujinaka
Software Application Engineer
Networking Division (ND)
Intel Corporation
[email protected]
(503) 712-4565
-----Original Message-----
From: Hu, Tan Chang [mailto:[email protected]]
Sent: Wednesday, July 29, 2015 6:55 AM
To: Fujinaka, Todd
Subject: RE: [EXTERNAL] RE: TCP stack is receiving out of order packer delivery
even when connected back-to-back
The new driver version, 1.2.48, and kernel, 2.6.32-504.30.3.el6.x86_64,
doesn't allow disabling of the flow director. The combination returns " Could
not change any device features". We will look at XPS.
-----Original Message-----
From: Fujinaka, Todd [mailto:[email protected]]
Sent: Thursday, July 23, 2015 2:45 PM
To: Hu, Tan Chang; [email protected]
Cc: Naegle, John H
Subject: RE: [EXTERNAL] RE: TCP stack is receiving out of order packer delivery
even when connected back-to-back
You only sent me the last email and, as I stated before, I do NOT have access
to the customer service database. I can only go by what you tell me in this
thread.
I was waiting for you to try the three tests I described in this thread and let
me know the results. We think this is a known issue.
You've told me that one of the tests invalid since rngd is not running. Please
try the other two.
Todd Fujinaka
Software Application Engineer
Networking Division (ND)
Intel Corporation
[email protected]
(503) 712-4565
-----Original Message-----
From: Hu, Tan Chang [mailto:[email protected]]
Sent: Thursday, July 23, 2015 1:41 PM
To: Fujinaka, Todd; [email protected]
Cc: Naegle, John H
Subject: Re: [EXTERNAL] RE: TCP stack is receiving out of order packer delivery
even when connected back-to-back
Todd,
We already stated in the email that rngd was not running. Seond, if you had
access to the ticket system you would have seen that we stated the same on a
follow up response and that email attachment wasn't the last email from
customer support.
________________________________________
From: Fujinaka, Todd <[email protected]>
Sent: Thursday, July 23, 2015 2:36 PM
To: Hu, Tan Chang; [email protected]
Cc: Naegle, John H
Subject: RE: [EXTERNAL] RE: TCP stack is receiving out of order packer delivery
even when connected back-to-back
I got the attachments directly, but keep in mind that sourceforge strips the
attachments and no one else has seen them. It appears you have send me their
last email and a bunch of ethtool output.
As far as I can see, they asked you to kill rngd and you never responded with
whether that worked or not. So we're back to them asking you to test and us
waiting for those results? I gave you two more possible tests as well.
Todd Fujinaka
Software Application Engineer
Networking Division (ND)
Intel Corporation
[email protected]
(503) 712-4565
-----Original Message-----
From: Hu, Tan Chang [mailto:[email protected]]
Sent: Thursday, July 23, 2015 12:21 PM
To: Fujinaka, Todd; [email protected]
Cc: Naegle, John H
Subject: RE: [EXTERNAL] RE: TCP stack is receiving out of order packer delivery
even when connected back-to-back
Todd,
Attached is the email we received from customer support and the ethtool
output pre and post run. Disregard the reference to Mellanox cards as they were
not in the test setup since June 15, 2015. RGND process was not running on
either system. There were no other tests we were ask to run except for watching
dropped packets. We can run the suggested fixes if you can provide the
instructions to do so.
Third paragraph, last sentence, is a dangling sentence.
As John mentioned in the last email, we are seeing performance problems with
real applications and not just iperf. How is Intel setting the NIC for best
general purpose throughput and what are those numbers?
-----Original Message-----
From: Fujinaka, Todd [mailto:[email protected]]
Sent: Thursday, July 23, 2015 11:16 AM
To: Naegle, John H; Hu, Tan Chang; [email protected]
Subject: RE: [EXTERNAL] RE: TCP stack is receiving out of order packer delivery
even when connected back-to-back
OK, so I went directly to the developers and they told me what they told
customer service. I'm going to assume you got the whole message.
First, what are the results of the tests that you were asked to try?
The problem was clearly explained to me, again, that the CPU scheduler is
kicking us off the CPU and flow director takes 20 SKBs to catch up. In that
time you can get out of order packets. There are things you should have been
asked to try to fix this. Turn off XPS in the kernel. Turn off Flow Director.
Kill the rngd process (we found that the rngd process is what usually runs to
kick us off the CPU). If you don't know how to do that, google is your friend.
We have ideas of how to mitigate this without doing
Why are you seeing problems? We set up the NIC for the best general purpose
throughput. When people start running tests on corner cases or artificial
tests, we see degradation. Turn off flow director, and you'll be happier with
your iperf numbers on both i40e and ixgbe, but I'm not sure if that's going to
help for a real-world server.
Some of our competitors don't have flow director, so they won't see this
problem.
Todd Fujinaka
Software Application Engineer
Networking Division (ND)
Intel Corporation
[email protected]
(503) 712-4565
-----Original Message-----
From: Fujinaka, Todd
Sent: Thursday, July 23, 2015 9:08 AM
To: 'Naegle, John H'; Hu, Tan Chang; [email protected]
Subject: RE: [EXTERNAL] RE: TCP stack is receiving out of order packer delivery
even when connected back-to-back
If you would like a phone conversation please contact your FAE to arrange this.
Right now we don't see how this is a widespread or critical issue, but the FAE
is the one who can determine the urgency.
Todd Fujinaka
Software Application Engineer
Networking Division (ND)
Intel Corporation
[email protected]
(503) 712-4565
-----Original Message-----
From: Naegle, John H [mailto:[email protected]]
Sent: Thursday, July 23, 2015 8:31 AM
To: Fujinaka, Todd; Hu, Tan Chang; [email protected]
Subject: RE: [EXTERNAL] RE: TCP stack is receiving out of order packer delivery
even when connected back-to-back
While it is true that iperf is an artificial test, it has been very effective
at demonstrating TCP/IP issues that also impact our data movement application
performance for us!
For example, when we discovered this same issue on your 10G NICs, we took a
guess and turned off LRO. This greatly reduced the number of Out-of-Order
events (as reported by Wireshark), and our user performance jumped by almost
10X! Users had been noticing and complaining about the performance
specifically to Intel NICs for some time. When we swapped to a Chelsio NIC,
the application performance went up even more. This was in the WAN where
latency dominates performance due to the large round trip time.
We have been trading emails about these issues now for several weeks. I'd
suggest that we have a live phone discussion. I find that it is much easier to
come to common understanding in complex problems with live discussions where
questions, ideas, and misunderstandings can be quickly clarified. Richard and
I are available today after 11:00 MT through 3:00pm MT or tomorrow. We can use
my telecon bridge number or just a direct call if that is easier. Please let
us know if you can support such a conversation.
Thanks,
John Naegle
-----Original Message-----
From: Fujinaka, Todd [mailto:[email protected]]
Sent: Thursday, July 23, 2015 9:11 AM
To: Hu, Tan Chang; [email protected]
Cc: Naegle, John H
Subject: [EXTERNAL] RE: TCP stack is receiving out of order packer delivery
even when connected back-to-back
Iperf changes queues in the middle of the test.
I remember this issue (not the ticket - I don't have direct access to this) and
some of the answers you receive late in the thread were from the development
team. I don't know if it was edited by customer support (as I said, I don't
have access to the ticket) and my opinion in hallway conversations about this
was that artificial tests (such as iperf) generate artificial results. We can
tune for good iperf performance but the tuning is counterproductive for real
traffic. I can't imagine that you're buying this hardware just to perform iperf
tests.
If there is a real issue, please let us know. The "go-to guy" in our division
has already responded to customer service on this issue.
Todd Fujinaka
Software Application Engineer
Networking Division (ND)
Intel Corporation
[email protected]
(503) 712-4565
-----Original Message-----
From: Hu, Tan Chang [mailto:[email protected]]
Sent: Thursday, July 23, 2015 7:58 AM
To: Fujinaka, Todd; [email protected]
Cc: Naegle, John H
Subject: TCP stack is receiving out of order packer delivery even when
connected back-to-back
Todd,
The history of the ticket 8001197134 gave the problem statement. The
Linux TCP/IP stack is receiving out of order packets delivery from even when
the two host with Intel XL710 40 GE are connected directly back to back. The
out of order packets triggers receiver side TCP/IP DUP ACK and cause the sender
to go into fast retransmit state and kills throughput performance (especially
as the end application of the 40GE is data transfer over 100Gbps WAN
connections). The Dell R620 servers are connected via two Intel E40GQSFPSR
transceivers with a Tripp Lite N844-10M-12-P40GBase-SR4 MPO cable back-to-back.
Intel customer support doesn't think re-transmits dramatically impact
throughput, alluding to improper socket buffering, and stated that out of order
packets are counted in different statistics. Intel customer support did not
provide any numbers as what "dramatically impact" means nor how they are
looking for socket buffering or out of order statistics. What does Intel expect
in terms of XL710 throughput performance, how do you check for Intel NIC socket
buffering settings, and what out of order statistics?
Comparing "ethtool -S i40e1" before and after the iperf3 run does indicate
multiple tx changes. Iperf3 was configured to run as a single thread, we
expected the Intel NIC to handle a single flow in a single queue and not spread
around over multiple queues. Can you explain the design why scheduling would
impact a single flow?
We are also seeing similar behavior from Intel 10GE NIC in production across
WAN links while other vendor's hardware do not exhibit this Out-Of-Order packet
deliver.
-----Original Message-----
From: Fujinaka, Todd [mailto:[email protected]]
Sent: Wednesday, July 22, 2015 11:25 AM
To: Hu, Tan Chang; [email protected]
Cc: Naegle, John H
Subject: [EXTERNAL] RE: [E1000-devel] Intel Customer Support Service Request
Management request # 8001197134
Can you restate your problem so we can start over on this issue? You may want
to just start with a new subject line as well. From what I've heard from
customer service, there was no issue and the case was closed but I haven't
heard all the details at this time.
Another way to get help is to escalate this through your factory contact
(whoever sold you whatever it is you're having problems with) and they can file
a ticket in IPS.
My suggestion is to start over here with a problem statement and we can ask you
for more details as we go. I'd also suggest filing a bug on sourceforge if you
have attachments to send (inline in email is hard to read), but sourceforge is
having issues at the moment.
Todd Fujinaka
Software Application Engineer
Networking Division (ND)
Intel Corporation
[email protected]
(503) 712-4565
-----Original Message-----
From: Hu, Tan Chang [mailto:[email protected]]
Sent: Wednesday, July 22, 2015 8:27 AM
To: [email protected]
Cc: Naegle, John H
Subject: [E1000-devel] Intel Customer Support Service Request Management
request # 8001197134
Desr sir,
As per understanding from the last email from Intel support dated July 22,
2015, the ticket 8001197234 has been passed on to the Intel driver development
team. Please let us know if additional iformation, besides the previously
submitted, are needed to resolve this issue.
Thanks
------------------------------------------------------------------------------
_______________________________________________
E1000-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel® Ethernet, visit
http://communities.intel.com/community/wired
