On Mon, Feb 14, 2022 at 6:54 PM Jan Scheurich <[email protected]> wrote: > > > > We do acknowledge the benefit of non-pinned polling of phy rx queues by > > PMD threads on all NUMA nodes. It gives the auto-load balancer much better > > options to utilize spare capacity on PMDs on all NUMA nodes. > > > > > > Our patch proposed in > > > https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-45444 > > > 5555731-c996011189a3eea8&q=1&e=0dc6a0b0-959c-493e-a3de- > > fea8f3151705&u= > > > https%3A%2F%2Fmail.openvswitch.org%2Fpipermail%2Fovs-dev%2F2021- > > June%2 > > > F384547.html indeed covers the difference between phy and vhu ports. > > > One has to explicitly enable cross-NUMA-polling for individual interfaces > > with: > > > > > > ovs-vsctl set interface <Name> other_config:cross-numa-polling=true > > > > > > This would typically only be done by static configuration for the fixed > > > set of > > physical ports. There is no code in the OpenStack's os-vif handler to apply > > such > > configuration for dynamically created vhu ports. > > > > > > I would strongly suggest that cross-num-polling be introduced as a per- > > interface option as in our patch rather than as a per-datapath option as in > > your > > patch. Why not adapt our original patch to the latest OVS code base? We can > > help you with that. > > > > > > BR, Jan > > > > > > > Hi, Jan Scheurich > > > > We can achieve the static setting of pinning a phy port by combining > > pmd-rxq- > > isolate and pmd-rxq-affinity. This setting can get the same result. And we > > have > > seen the benefits. > > The new issue is the polling of vhu on one numa. Under heavy traffic, > > polling > > vhu + phy will make the pmds reach 100% usage. While other pmds on the > > other numa with only phy port reaches 70% usage. Enabling cross-numa polling > > for a vhu port would give us more benefits in this case. Overloads of > > different > > pmds on both numa would be balanced. > > As you have mentioned, there is no code to apply this config for vhu while > > creating them. A global setting would save us from dynamically detecting the > > vhu name or any new creation. > > Hi Wan Junjie, > > We have done extensive benchmarking and found that we get better overall PMD > load balance and resulting OVS performance when we do not statically pin any > rx queues and instead let the auto-load-balancing find the optimal > distribution of phy rx queues over both NUMA nodes to balance an asymmetric > load of vhu rx queues (polled only on the local NUMA node). > > Cross-NUMA polling of vhu rx queues comes with a very high latency cost due > to cross-NUMA access to volatile virtio ring pointers in every iteration (not > only when actually copying packets). Cross-NUMA polling of phy rx queues > doesn't have a similar issue. > > BR, Jan >
Hi Jan Scheurich, Thanks for the info. Yes I am using the static pinning. It should have saved the cost for calculating the 'load'. I'm with you on that. Kevin's RFC seems to be like using dynamic calculating. As for the latency cost of cross-numa on vhu, it seems to be a concern. I would say It is a trade-off with a cost. I have no bias for using the cross-numa by default. Maybe for different traffic patterns, different settings could be a proper way. Regards, Wan Junjie _______________________________________________ dev mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-dev
