On 4/18/2019 1:43 PM, Ilya Maximets wrote:
On 18.04.2019 11:08, Ian Stokes wrote:
On 4/18/2019 8:55 AM, Ilya Maximets wrote:
On 11.03.2019 19:31, Ilya Maximets wrote:
This partially reverts commit bde94613e6276d48a6e0be7a592ebcf9836b4aaf.
Commit bde94613e627 was aimed to slightly ( < 1%) increase performance
in the case where EMC disabled, but it avoids RSS hash calculation and
OVS has to calculate it while executing OVS_ACTION_ATTR_HASH in order
to handle balanced-tcp bonding. At the time of executing that action
there is no parsed flow, and OVS parses the packet for the second time
to calculate the hash. This happens for all packets received from the
virtual interfaces because they have no HW RSS.
Here is the example of 'perf' output for VM --> (bonded PHY) traffic:
Samples: 401K of event 'cycles', Event count (approx.): 50964771478
Overhead Shared Object Symbol
27.50% ovs-vswitchd [.] dpcls_lookup.370382
16.30% ovs-vswitchd [.] rte_vhost_dequeue_burst.9267
14.95% ovs-vswitchd [.] miniflow_extract
7.22% ovs-vswitchd [.] flow_extract
7.10% ovs-vswitchd [.] dp_netdev_input__.371002.4826
4.01% ovs-vswitchd [.] fast_path_processing.370987.4893
We can see that packet parsed twice. First time by 'miniflow_extract'
right after receiving and the second time by 'flow_extract' while
executing actions.
In this particular case calculating RSS on receive saves > 7% of the
total CPU processing time. It varies from ~7 to ~10 % depending on
scenario/traffic types.
It's better to calculate hash each time because performance
improvements of avoiding are negligible in compare with performance
drop in case of sending packets to bonded interface.
Another solution could be to pass the parsed flow explicitly through
the datapath, but this will require big code changes and will have
additional overhead for metadata updating on packet changes.
Also, this change should have small impact since SMC works well in most
cases and will be enabled/recommended by default in the future.
CC: Antonio Fischetti <[email protected]>
Fixes: bde94613e627 ("dpif-netdev: Avoid reading RSS hash when EMC is
disabled.")
Signed-off-by: Ilya Maximets <[email protected]>
---
Hi.
Any thoughts about this patch?
This is a big performance issue on setups with OVS bonding.
Hi Ilya,
I'm just finishing up performance testing I started yesterday with this patch,
I'm inclined to agree with you.
From bench marking with master I see the following with EMC disabled with
RFC2544 (0% packet loss).
(Master - No hash when EMC disabled)
Packet Size RX Rate PPs RX Rate Mbps RX %
64 6824334.554 3494.059 45.859
128 6878122.822 7043.198 81.438
256 4528973.86 9275.338 100
512 2349613.015 9624.015 100
1024 1197312.023 9808.38 100
1500 822368.143 9868.418 100
With the revert applied I see
(Patch - hash applied when EMC disabled)
Packet Size RX Rate PPs RX Rate Mbps RX %
64 6479041.962 3317.269 43.539
128 6747525.93 6909.467 79.891
256 4528894.363 9275.176 100
512 2349615.976 9624.027 100
1024 1197313.173 9808.39 100
1500 822365.834 9868.39 100
So the gain for avoiding the hash is marginal and certainly not equal to the
gain achieved when enabled with bonding as you reported.
Even testing with scaling high number of flows there is a similar pattern.
(Master - No hash when EMC disabled)
Packet Size RX Rate PPs RX Rate Mbps RX %
64 13418452.15 6870.248 45.086
(Patch - hash applied when EMC disabled)
Packet Size RX Rate PPs RX Rate Mbps RX %
64 12958087.12 6634.541 43.539
With this in mind I'll push this to master today.
Thanks for testing and applying.
What about backporting? The issue affects all branches since 2.9.
Applies cleanly down to 2.10. I could send separate patch for 2.9
if needed.
Yes, good point, I've pushed 2.11, 2.10. and a reworked patch for 2.9.
Thanks
Ian
_______________________________________________
dev mailing list
[email protected]
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
