On Wed, Oct 28, 2020 at 02:17:06PM -0400, Mark Gray wrote:
> From: Aaron Conole <[email protected]>
>
> Currently, the channel handlers are polled globally. On some
> systems, this causes a thundering herd issue where multiple
> handler threads become active, only to do no work and immediately
> sleep.
>
> The approach here is to push the netlink socket channels to discreet
> handler threads to process, rather than polling on every thread.
> This will eliminate the need to wake multiple threads.
>
> To check:
>
> ip netns add left
> ip netns add right
> ip link add center-left type veth peer name left0 netns left
> ip link add center-right type veth peer name right0 netns right
> ip link set center-left up
> ip link set center-right up
> ip -n left ip link set left0 up
> ip -n left ip addr add 172.31.110.10/24 dev left0
> ip -n right ip link set right0 up
> ip -n right ip addr add 172.31.110.11/24 dev right0
>
> ovs-vsctl add-br br0
> ovs-vsctl add-port br0 center-right
> ovs-vsctl add-port br0 center-left
>
> # in one terminal
> perf record -e sched:sched_wakeup,irq:softirq_entry -ag
>
> # in a separate terminal
> ip netns exec left arping -I left0 -c 1 172.31.110.11
>
> # in the perf terminal after exiting
> perf script
>
> Look for the number of 'handler' threads which were made active.
>
> Suggested-by: Ben Pfaff <[email protected]>
> Suggested-by: Flavio Leitner <[email protected]>
> Co-authored-by: Mark Gray <[email protected]>
> Reported-by: David Ahern <[email protected]>
> Reported-at:
> https://mail.openvswitch.org/pipermail/ovs-dev/2019-December/365857.html
> Cc: Matteo Croce <[email protected]>
> Fixes: 69c51582f ("dpif-netlink: don't allocate per thread netlink sockets")
> Signed-off-by: Aaron Conole <[email protected]>
> Signed-off-by: Mark Gray <[email protected]>
> ---
I think the patch looks good, so I gave it a try on the lab.
I added 1000 client netns and another 1000 server netns, each using
an OVS internal port. Created a flow table such that random src ports
would drop, otherwise forward the packet to the other netns.
Then I started iperf3 in all client containers in parallel
triggering upcalls from different ports.
Although I ran few times, please consider these ball park numbers:
Current Patched
# Wakes - Thread: # Wakes - Thread
270 - handler10 400 - handler32
471 - handler11 403 - handler33
150 - handler12 383 - handler34
396 - handler35
430 - handler36
The patch distributes ports to each handler and I could see that
the wake up and the load were balanced as a consequence.
Now the second test. I used one client netns and one server netns
and did an UDP burst of packets with similar flow table. So, each
new packet would trigger an upcall on the same port. The current
code would likely trigger multiple handlers while with the patch
applied would be limited to a single thread.
The difference is expressive in this scenario:
Current Patched
Max # UDP packets[*] 62k 3k4
Max # flow installed ~60k 11k
[*] Max number of UDP packets sent in burst without increasing
upcall "lost" counter.
I tested without the patch and setting n-handler-threads to 1 and
the results are close enough to the patched version.
Well, I think the patch is working as designed to fix the thundering
herd issue. However, considering that in most use cases there is
only one NIC port attached to OVS receiving all the host traffic,
then that gap might be a problem.
Thoughts?
fbl
_______________________________________________
dev mailing list
[email protected]
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
