Hi Kevin, Thanks for your comments.
On Fri, Mar 25, 2022 at 8:17 PM Kevin Traynor <[email protected]> wrote: > > Hi Wan Junjie, > > Not a full code review, but comments on the feature below. > > On 02/03/2022 10:59, Wan Junjie wrote: > > A pmd would poll all rxqs with no weight. When a pmd has one rxq > > from phy port and several from vhu port, and high loads run for > > both rx and tx, then the vhu can get polled more. This will cause > > the polling for rx of phy port much less than the vhu port. The > > loads for tx/rx will lose balance. With traffic to both directions, > > rx will be limited to a very low rate as phy port get polled less. > > > > That's an interesting observation. > > > For example, originally poll list for each pmd is like below: > > pmd 0 phy0_0 vhu0_0 vhu0_4 vhu0_8 vhu0_12 > > pmd 1 phy0_1 vhu0_1 vhu0_5 vhu0_9 vhu0_13 > > pmd 2 phy0_2 vhu0_2 vhu0_6 vhu0_10 vhu0_14 > > pmd 3 phy0_3 vhu0_3 vhu0_7 vhu0_11 vhu0_15 > > > > With traffic to both directions, rx will be limited to 2Mpps and > > tx is 9Mpps. > > > > Can you explain this a bit more. > > Are you saying you have 2 independent paths ? > - an ingress path of phy->ovs->vm, 2 Mpps > - an separate egress path of vm->ovs->phy, 9 Mpps > Yes. There are two independent paths. The pps numbers‘ ratio is nearly the same with the queues' ratio 1:4. > > This patch provide an option to reinforce the phy port polling. > > Add a configuration for rxq schedule, which will try to balance > > the poll for phy and vhu port. It will increase the poll times > > for a phy port, and interlace the phy rxq and vhu rxq in the poll > > list. > > > > scale the rxq poll list: > > pmd 0 phy0_0 vhu0_0 phy0_0 vhu0_4 phy0_0 vhu0_8 phy0_0 vhu0_12 > > pmd 1 phy0_1 vhu0_1 phy0_1 vhu0_5 phy0_1 vhu0_9 phy0_1 vhu0_13 > > pmd 2 phy0_2 vhu0_2 phy0_2 vhu0_6 phy0_2 vhu0_10 phy0_2 vhu0_14 > > pmd 3 phy0_3 vhu0_3 phy0_3 vhu0_7 phy0_3 vhu0_11 phy0_3 vhu0_15 > > > > This looks a custom scheme that might be suited to only a very limited > config, but it is not clear what that config needs to be, or how it > should work with a different config. The scheme should be more generic > and clear what the behavior will be with any other config > (ports/rxqs/core/pinning etc), or forbidden with other configs. > > I did some testing and changed a few config items and it stopped polling > several rxqs which results in no traffic being passed. My testing notes > are below [0]. > > You mentioned rxq weights was missing in the commit message above and it > sounded like allowing the user to set port weights would be a more > generic version of what you are proposing. However, weights would only > be relevant to the rxqs on a particular pmd, and not between rxqs on > different pmds, so i'm not sure it would work. > > thanks, > Kevin. > Thanks for your detailed tests. Some tests did fail. In this patch, I did not take more queues from phy port into account. And more than one phy port seems not working yet. If this patch's solution is good to continue, I'll fix it in v2. And yes, I agree that there should be a more generic scheme. Weights would only be effective on queues from the same pmd, I think this is the right target for the case of one phy port. No matter how many vhu ports or phy queues we have on one pmd, the weights setting should work as a limitation when there are flows from both directions. If there is only traffic from one direction, the setting won't be a bottleneck as well. If we have more than one phy port, we can not predict the distribution of the loads. So I don't think this is the right solution for this case. Maybe we should forbid the setting in this case or change the setting as per phy port instead. What's your opinion on this case? Regards, Wan Junjie > > to enable it, run: > > 'ovs-vsctl set open . other_config:pmd-rxq-schedule=scaling' > > to disable it, remove the setting or set it to 'single' > > And it works fairly well when n_rxq of dpdk phy port equals to > > number of dpdk pmds, which means one n_rxq for one pmd > > > > Signed-off-by: Wan Junjie <[email protected]> > > Reviewed-by: He Peng <[email protected]> > > --- > > lib/dpif-netdev.c | 133 ++++++++++++++++++++++++++++++++++++++++++---- > > 1 file changed, 122 insertions(+), 11 deletions(-) > > > > [0] > - enable 'scale' amd add myport dpdk phy nic and vhost port > > pmd thread numa_id 0 core_id 8: > isolated : false > port: dpdkvhost0 queue-id: 0 (enabled) pmd usage: 0 % > port: dpdkvhost0 queue-id: 1 (enabled) pmd usage: 0 % > port: dpdkvhost0 queue-id: 2 (enabled) pmd usage: 0 % > port: dpdkvhost0 queue-id: 3 (enabled) pmd usage: 0 % > port: myport queue-id: 0 (rescaled) pmd usage: 0 % > > > dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 1 rxq dpdkvhost0 2 > dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 2 rxq myport 0 > dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 3 rxq dpdkvhost0 1 > dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 4 rxq myport 0 > dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 5 rxq dpdkvhost0 3 > dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 6 rxq myport 0 > dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 7 rxq dpdkvhost0 0 > > > - Add another phy nic > > pmd thread numa_id 0 core_id 8: > isolated : false > port: dpdkvhost0 queue-id: 0 (enabled) pmd usage: 0 % > port: dpdkvhost0 queue-id: 1 (enabled) pmd usage: 0 % > port: dpdkvhost0 queue-id: 2 (enabled) pmd usage: 0 % > port: dpdkvhost0 queue-id: 3 (enabled) pmd usage: 0 % > port: myport queue-id: 0 (rescaled) pmd usage: 0 % > port: urport queue-id: 0 (rescaled) pmd usage: 0 % > > dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 0 rxq urport 0 > dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 1 rxq dpdkvhost0 2 > dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 2 rxq myport 0 > dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 3 rxq dpdkvhost0 1 > dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 4 rxq urport 0 > dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 5 rxq dpdkvhost0 3 > dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 6 rxq myport 0 > dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 7 rxq dpdkvhost0 0 > > > - change number of rxqs for myport, n_rxq=4 > > pmd thread numa_id 0 core_id 8: > isolated : false > port: dpdkvhost0 queue-id: 0 (enabled) pmd usage: 0 % > port: dpdkvhost0 queue-id: 1 (enabled) pmd usage: 0 % > port: dpdkvhost0 queue-id: 2 (enabled) pmd usage: 0 % > port: dpdkvhost0 queue-id: 3 (enabled) pmd usage: 0 % > port: myport queue-id: 0 (rescaled) pmd usage: 0 % > port: myport queue-id: 1 (rescaled) pmd usage: 0 % > port: myport queue-id: 2 (rescaled) pmd usage: 0 % > port: myport queue-id: 3 (rescaled) pmd usage: 0 % > port: urport queue-id: 0 (rescaled) pmd usage: 0 % > > > dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 0 rxq myport 1 > dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 1 rxq dpdkvhost0 2 > dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 2 rxq urport 0 > dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 3 rxq dpdkvhost0 1 > dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 4 rxq myport 2 > dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 5 rxq dpdkvhost0 0 > dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 6 rxq myport 0 > dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 7 rxq dpdkvhost0 3 > > *queue 3 for myport is not polled > > > - change number of rxq for urport, n_rxq=4 > > pmd thread numa_id 0 core_id 8: > isolated : false > port: dpdkvhost0 queue-id: 0 (enabled) pmd usage: 0 % > port: dpdkvhost0 queue-id: 1 (enabled) pmd usage: 0 % > port: dpdkvhost0 queue-id: 2 (enabled) pmd usage: 0 % > port: dpdkvhost0 queue-id: 3 (enabled) pmd usage: 0 % > port: myport queue-id: 0 (rescaled) pmd usage: 0 % > port: myport queue-id: 1 (rescaled) pmd usage: 0 % > port: myport queue-id: 2 (rescaled) pmd usage: 0 % > port: myport queue-id: 3 (rescaled) pmd usage: 0 % > port: urport queue-id: 0 (rescaled) pmd usage: 0 % > port: urport queue-id: 1 (rescaled) pmd usage: 0 % > port: urport queue-id: 2 (rescaled) pmd usage: 0 % > port: urport queue-id: 3 (rescaled) pmd usage: 0 % > > > dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 0 rxq urport 1 > dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 1 rxq dpdkvhost0 2 > dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 2 rxq myport 1 > dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 3 rxq dpdkvhost0 1 > dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 4 rxq urport 2 > dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 5 rxq dpdkvhost0 0 > dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 6 rxq urport 3 > dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 7 rxq dpdkvhost0 3 > > *myport queues 0,2,3 not polled, urport queues 0 not polled > > > - shutdown vm > > pmd thread numa_id 0 core_id 8: > isolated : false > port: dpdkvhost0 queue-id: 0 (enabled) pmd usage: 0 % > port: myport queue-id: 0 (rescaled) pmd usage: 0 % > port: myport queue-id: 1 (rescaled) pmd usage: 0 % > port: myport queue-id: 2 (rescaled) pmd usage: 0 % > port: myport queue-id: 3 (rescaled) pmd usage: 0 % > port: urport queue-id: 0 (rescaled) pmd usage: 0 % > port: urport queue-id: 1 (rescaled) pmd usage: 0 % > port: urport queue-id: 2 (rescaled) pmd usage: 0 % > port: urport queue-id: 3 (rescaled) pmd usage: 0 % > > > dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 0 rxq urport 1 > dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 1 rxq myport 1 > dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 2 rxq urport 2 > dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 3 rxq urport 3 > dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 4 rxq urport 0 > dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 5 rxq myport 2 > dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 6 rxq myport 0 > dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 7 rxq myport 3 > > *dpdkvhost0 no longer polled > > - change vm to 1 rxq and start vm > > pmd thread numa_id 0 core_id 8: > isolated : false > port: dpdkvhost0 queue-id: 0 (enabled) pmd usage: 0 % > port: dpdkvhost0 queue-id: 1 (enabled) pmd usage: 0 % > port: dpdkvhost0 queue-id: 2 (enabled) pmd usage: 0 % > port: myport queue-id: 0 (rescaled) pmd usage: 0 % > port: myport queue-id: 1 (rescaled) pmd usage: 0 % > port: myport queue-id: 2 (rescaled) pmd usage: 0 % > port: myport queue-id: 3 (rescaled) pmd usage: 0 % > port: urport queue-id: 0 (rescaled) pmd usage: 0 % > port: urport queue-id: 1 (rescaled) pmd usage: 0 % > port: urport queue-id: 2 (rescaled) pmd usage: 0 % > port: urport queue-id: 3 (rescaled) pmd usage: 0 % > > > no debug as same vm rxq as when vm shutdown. > > *rxq for vhost not polled > > > - change vm to 3 rxq > > pmd thread numa_id 0 core_id 8: > isolated : false > port: dpdkvhost0 queue-id: 0 (enabled) pmd usage: 0 % > port: dpdkvhost0 queue-id: 1 (enabled) pmd usage: 0 % > port: dpdkvhost0 queue-id: 2 (enabled) pmd usage: 0 % > port: myport queue-id: 0 (rescaled) pmd usage: 0 % > port: myport queue-id: 1 (rescaled) pmd usage: 0 % > port: myport queue-id: 2 (rescaled) pmd usage: 0 % > port: myport queue-id: 3 (rescaled) pmd usage: 0 % > port: urport queue-id: 0 (rescaled) pmd usage: 0 % > port: urport queue-id: 1 (rescaled) pmd usage: 0 % > port: urport queue-id: 2 (rescaled) pmd usage: 0 % > port: urport queue-id: 3 (rescaled) pmd usage: 0 % > > > dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 0 rxq urport 1 > dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 1 rxq dpdkvhost0 2 > dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 2 rxq myport 1 > dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 3 rxq dpdkvhost0 1 > dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 4 rxq urport 2 > dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 5 rxq dpdkvhost0 0 > > > *myport queues 0,2,3 not polled, urport queues 0,3 not polled > > - Remove urport > > pmd thread numa_id 0 core_id 8: > isolated : false > port: dpdkvhost0 queue-id: 0 (enabled) pmd usage: 0 % > port: myport queue-id: 0 (rescaled) pmd usage: 0 % > port: myport queue-id: 1 (rescaled) pmd usage: 0 % > port: myport queue-id: 2 (rescaled) pmd usage: 0 % > port: myport queue-id: 3 (rescaled) pmd usage: 0 % > > > dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 0 rxq myport 1 > dpif_netdev(pmd-c08/id:8)|DBG|PMD 8: 1 rxq dpdkvhost0 0 > > *myport queues 0,2,3 not being polled > > Maybe just debug issue? chck with gdb > > (gdb) p poll_cnt > $31 = 2 > (gdb) p poll_list[0]->rxq->rx->netdev->name > $32 = 0x3bfade0 "myport" > (gdb) p poll_list[0]->rxq->rx->queue_id > $33 = 1 > (gdb) p poll_list[1]->rxq->rx->netdev->name > $34 = 0x3bd6ce0 "dpdkvhost0" > (gdb) p poll_list[1]->rxq->rx->queue_id > $35 = 0 > _______________________________________________ dev mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-dev
