On 29/03/2022 04:26, Wan Junjie wrote:
Hi Kevin,
Thanks for your comments.
On Fri, Mar 25, 2022 at 8:17 PM Kevin Traynor <[email protected]> wrote:
Hi Wan Junjie,
Not a full code review, but comments on the feature below.
On 02/03/2022 10:59, Wan Junjie wrote:
A pmd would poll all rxqs with no weight. When a pmd has one rxq
from phy port and several from vhu port, and high loads run for
both rx and tx, then the vhu can get polled more. This will cause
the polling for rx of phy port much less than the vhu port. The
loads for tx/rx will lose balance. With traffic to both directions,
rx will be limited to a very low rate as phy port get polled less.
That's an interesting observation.
For example, originally poll list for each pmd is like below:
pmd 0 phy0_0 vhu0_0 vhu0_4 vhu0_8 vhu0_12
pmd 1 phy0_1 vhu0_1 vhu0_5 vhu0_9 vhu0_13
pmd 2 phy0_2 vhu0_2 vhu0_6 vhu0_10 vhu0_14
pmd 3 phy0_3 vhu0_3 vhu0_7 vhu0_11 vhu0_15
With traffic to both directions, rx will be limited to 2Mpps and
tx is 9Mpps.
Can you explain this a bit more.
Are you saying you have 2 independent paths ?
- an ingress path of phy->ovs->vm, 2 Mpps
- an separate egress path of vm->ovs->phy, 9 Mpps
Yes. There are two independent paths.
The pps numbers‘ ratio is nearly the same with the queues' ratio 1:4.
Don't forget if you poll the nic rxq more often and increase the
throughput for ingress, then there will be less time for the pmd core to
spend polling and processing packets from the vm rxqs, so you will
reduce the throughput for the egress.
The throughput might be more level between the interfaces, but it should
be a lot lower than the 9 Mpps.
If this was a PVP test, then egress would be limited by the ingress, so
it should balance out there too.
This patch provide an option to reinforce the phy port polling.
Add a configuration for rxq schedule, which will try to balance
the poll for phy and vhu port. It will increase the poll times
for a phy port, and interlace the phy rxq and vhu rxq in the poll
list.
scale the rxq poll list:
pmd 0 phy0_0 vhu0_0 phy0_0 vhu0_4 phy0_0 vhu0_8 phy0_0 vhu0_12
pmd 1 phy0_1 vhu0_1 phy0_1 vhu0_5 phy0_1 vhu0_9 phy0_1 vhu0_13
pmd 2 phy0_2 vhu0_2 phy0_2 vhu0_6 phy0_2 vhu0_10 phy0_2 vhu0_14
pmd 3 phy0_3 vhu0_3 phy0_3 vhu0_7 phy0_3 vhu0_11 phy0_3 vhu0_15
This looks a custom scheme that might be suited to only a very limited
config, but it is not clear what that config needs to be, or how it
should work with a different config. The scheme should be more generic
and clear what the behavior will be with any other config
(ports/rxqs/core/pinning etc), or forbidden with other configs.
I did some testing and changed a few config items and it stopped polling
several rxqs which results in no traffic being passed. My testing notes
are below [0].
You mentioned rxq weights was missing in the commit message above and it
sounded like allowing the user to set port weights would be a more
generic version of what you are proposing. However, weights would only
be relevant to the rxqs on a particular pmd, and not between rxqs on
different pmds, so i'm not sure it would work.
thanks,
Kevin.
Thanks for your detailed tests. Some tests did fail. In this patch, I did not
take more queues from phy port into account. And more than one phy port
seems not working yet. If this patch's solution is good to continue,
I'll fix it in v2.
I think it needs a fuller definition of what this feature is, the
benefit and how it scales to different configs before it is worth
spending the time to write and test a v2. At the moment it's not clear
to me.
And yes, I agree that there should be a more generic scheme.
Weights would only be effective on queues from the same pmd, I think this is
the right target for the case of one phy port. No matter how many vhu ports
or phy queues we have on one pmd, the weights setting should work as a
limitation when there are flows from both directions. If there is only
traffic from
one direction, the setting won't be a bottleneck as well.
You are assuming that the phy rxq will be distributed evenly over pmd
cores. The only limitation OVS has in assigning those phy rxqs is by
NUMA (currently) and isolated pmds.
So as you say the weight/prio would only be relative to pmd, but all
high prio rxqs could be on the same pmd core. That would mean from a
user perspective they have set weight/prio on all interfaces but it is
only being respected by OVS on a random groups or rxqs.
You might be able to implement something like what you are describing in
a limited case if you took action like manually pinning one high
weight/prio rxq per pmd core with non-isolate=true and allowing OVS to
assign other low weight/prio rxqs. High weight/prio rxqs would be on
different cores then and could be polled more often than lower prio rxqs.
Other options would be to add a new assignment method based on weight
and incorprate a scale option for high prio rxqs, but that would be at
the expenses of basing it on processing cycles required so could lead to
bottlnecks on some pmd cores.
These are just some thoughts about how you can achieve something close
to what you have implemented but you still have issues with scale in
either of those cases. However, above all I think it needs a clearer
definition of the benefit etc. before considering how it could be done.
If we have more than one phy port, we can not predict the distribution of the
loads. So I don't think this is the right solution for this case.
Maybe we should
forbid the setting in this case or change the setting as per phy port instead.
What's your opinion on this case?
I think it is a very common to have multiple DPDK phy nic ports.
HTH,
Kevin.
Regards,
Wan Junjie
to enable it, run:
'ovs-vsctl set open . other_config:pmd-rxq-schedule=scaling'
to disable it, remove the setting or set it to 'single'
And it works fairly well when n_rxq of dpdk phy port equals to
number of dpdk pmds, which means one n_rxq for one pmd
Signed-off-by: Wan Junjie <[email protected]>
Reviewed-by: He Peng <[email protected]>
---
lib/dpif-netdev.c | 133 ++++++++++++++++++++++++++++++++++++++++++----
1 file changed, 122 insertions(+), 11 deletions(-)
[0]
- enable 'scale' amd add myport dpdk phy nic and vhost port
pmd thread numa_id 0 core_id 8:
isolated : false
port: dpdkvhost0 queue-id: 0 (enabled) pmd usage: 0 %
port: dpdkvhost0 queue-id: 1 (enabled) pmd usage: 0 %
port: dpdkvhost0 queue-id: 2 (enabled) pmd usage: 0 %
port: dpdkvhost0 queue-id: 3 (enabled) pmd usage: 0 %
port: myport queue-id: 0 (rescaled) pmd usage: 0 %
dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 1 rxq dpdkvhost0 2
dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 2 rxq myport 0
dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 3 rxq dpdkvhost0 1
dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 4 rxq myport 0
dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 5 rxq dpdkvhost0 3
dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 6 rxq myport 0
dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 7 rxq dpdkvhost0 0
- Add another phy nic
pmd thread numa_id 0 core_id 8:
isolated : false
port: dpdkvhost0 queue-id: 0 (enabled) pmd usage: 0 %
port: dpdkvhost0 queue-id: 1 (enabled) pmd usage: 0 %
port: dpdkvhost0 queue-id: 2 (enabled) pmd usage: 0 %
port: dpdkvhost0 queue-id: 3 (enabled) pmd usage: 0 %
port: myport queue-id: 0 (rescaled) pmd usage: 0 %
port: urport queue-id: 0 (rescaled) pmd usage: 0 %
dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 0 rxq urport 0
dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 1 rxq dpdkvhost0 2
dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 2 rxq myport 0
dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 3 rxq dpdkvhost0 1
dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 4 rxq urport 0
dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 5 rxq dpdkvhost0 3
dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 6 rxq myport 0
dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 7 rxq dpdkvhost0 0
- change number of rxqs for myport, n_rxq=4
pmd thread numa_id 0 core_id 8:
isolated : false
port: dpdkvhost0 queue-id: 0 (enabled) pmd usage: 0 %
port: dpdkvhost0 queue-id: 1 (enabled) pmd usage: 0 %
port: dpdkvhost0 queue-id: 2 (enabled) pmd usage: 0 %
port: dpdkvhost0 queue-id: 3 (enabled) pmd usage: 0 %
port: myport queue-id: 0 (rescaled) pmd usage: 0 %
port: myport queue-id: 1 (rescaled) pmd usage: 0 %
port: myport queue-id: 2 (rescaled) pmd usage: 0 %
port: myport queue-id: 3 (rescaled) pmd usage: 0 %
port: urport queue-id: 0 (rescaled) pmd usage: 0 %
dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 0 rxq myport 1
dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 1 rxq dpdkvhost0 2
dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 2 rxq urport 0
dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 3 rxq dpdkvhost0 1
dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 4 rxq myport 2
dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 5 rxq dpdkvhost0 0
dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 6 rxq myport 0
dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 7 rxq dpdkvhost0 3
*queue 3 for myport is not polled
- change number of rxq for urport, n_rxq=4
pmd thread numa_id 0 core_id 8:
isolated : false
port: dpdkvhost0 queue-id: 0 (enabled) pmd usage: 0 %
port: dpdkvhost0 queue-id: 1 (enabled) pmd usage: 0 %
port: dpdkvhost0 queue-id: 2 (enabled) pmd usage: 0 %
port: dpdkvhost0 queue-id: 3 (enabled) pmd usage: 0 %
port: myport queue-id: 0 (rescaled) pmd usage: 0 %
port: myport queue-id: 1 (rescaled) pmd usage: 0 %
port: myport queue-id: 2 (rescaled) pmd usage: 0 %
port: myport queue-id: 3 (rescaled) pmd usage: 0 %
port: urport queue-id: 0 (rescaled) pmd usage: 0 %
port: urport queue-id: 1 (rescaled) pmd usage: 0 %
port: urport queue-id: 2 (rescaled) pmd usage: 0 %
port: urport queue-id: 3 (rescaled) pmd usage: 0 %
dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 0 rxq urport 1
dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 1 rxq dpdkvhost0 2
dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 2 rxq myport 1
dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 3 rxq dpdkvhost0 1
dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 4 rxq urport 2
dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 5 rxq dpdkvhost0 0
dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 6 rxq urport 3
dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 7 rxq dpdkvhost0 3
*myport queues 0,2,3 not polled, urport queues 0 not polled
- shutdown vm
pmd thread numa_id 0 core_id 8:
isolated : false
port: dpdkvhost0 queue-id: 0 (enabled) pmd usage: 0 %
port: myport queue-id: 0 (rescaled) pmd usage: 0 %
port: myport queue-id: 1 (rescaled) pmd usage: 0 %
port: myport queue-id: 2 (rescaled) pmd usage: 0 %
port: myport queue-id: 3 (rescaled) pmd usage: 0 %
port: urport queue-id: 0 (rescaled) pmd usage: 0 %
port: urport queue-id: 1 (rescaled) pmd usage: 0 %
port: urport queue-id: 2 (rescaled) pmd usage: 0 %
port: urport queue-id: 3 (rescaled) pmd usage: 0 %
dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 0 rxq urport 1
dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 1 rxq myport 1
dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 2 rxq urport 2
dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 3 rxq urport 3
dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 4 rxq urport 0
dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 5 rxq myport 2
dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 6 rxq myport 0
dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 7 rxq myport 3
*dpdkvhost0 no longer polled
- change vm to 1 rxq and start vm
pmd thread numa_id 0 core_id 8:
isolated : false
port: dpdkvhost0 queue-id: 0 (enabled) pmd usage: 0 %
port: dpdkvhost0 queue-id: 1 (enabled) pmd usage: 0 %
port: dpdkvhost0 queue-id: 2 (enabled) pmd usage: 0 %
port: myport queue-id: 0 (rescaled) pmd usage: 0 %
port: myport queue-id: 1 (rescaled) pmd usage: 0 %
port: myport queue-id: 2 (rescaled) pmd usage: 0 %
port: myport queue-id: 3 (rescaled) pmd usage: 0 %
port: urport queue-id: 0 (rescaled) pmd usage: 0 %
port: urport queue-id: 1 (rescaled) pmd usage: 0 %
port: urport queue-id: 2 (rescaled) pmd usage: 0 %
port: urport queue-id: 3 (rescaled) pmd usage: 0 %
no debug as same vm rxq as when vm shutdown.
*rxq for vhost not polled
- change vm to 3 rxq
pmd thread numa_id 0 core_id 8:
isolated : false
port: dpdkvhost0 queue-id: 0 (enabled) pmd usage: 0 %
port: dpdkvhost0 queue-id: 1 (enabled) pmd usage: 0 %
port: dpdkvhost0 queue-id: 2 (enabled) pmd usage: 0 %
port: myport queue-id: 0 (rescaled) pmd usage: 0 %
port: myport queue-id: 1 (rescaled) pmd usage: 0 %
port: myport queue-id: 2 (rescaled) pmd usage: 0 %
port: myport queue-id: 3 (rescaled) pmd usage: 0 %
port: urport queue-id: 0 (rescaled) pmd usage: 0 %
port: urport queue-id: 1 (rescaled) pmd usage: 0 %
port: urport queue-id: 2 (rescaled) pmd usage: 0 %
port: urport queue-id: 3 (rescaled) pmd usage: 0 %
dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 0 rxq urport 1
dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 1 rxq dpdkvhost0 2
dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 2 rxq myport 1
dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 3 rxq dpdkvhost0 1
dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 4 rxq urport 2
dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 5 rxq dpdkvhost0 0
*myport queues 0,2,3 not polled, urport queues 0,3 not polled
- Remove urport
pmd thread numa_id 0 core_id 8:
isolated : false
port: dpdkvhost0 queue-id: 0 (enabled) pmd usage: 0 %
port: myport queue-id: 0 (rescaled) pmd usage: 0 %
port: myport queue-id: 1 (rescaled) pmd usage: 0 %
port: myport queue-id: 2 (rescaled) pmd usage: 0 %
port: myport queue-id: 3 (rescaled) pmd usage: 0 %
dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 0 rxq myport 1
dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 1 rxq dpdkvhost0 0
*myport queues 0,2,3 not being polled
Maybe just debug issue? chck with gdb
(gdb) p poll_cnt
$31 = 2
(gdb) p poll_list[0]->rxq->rx->netdev->name
$32 = 0x3bfade0 "myport"
(gdb) p poll_list[0]->rxq->rx->queue_id
$33 = 1
(gdb) p poll_list[1]->rxq->rx->netdev->name
$34 = 0x3bd6ce0 "dpdkvhost0"
(gdb) p poll_list[1]->rxq->rx->queue_id
$35 = 0
_______________________________________________
dev mailing list
[email protected]
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
