On 05/06/2017 02:01 AM, Greg Rose wrote: > On Fri, May 5, 2017 at 9:34 AM, Kevin Traynor <[email protected]> wrote: >> Rxqs are scheduled to be handled across available pmds in round robin >> order with no weight or priority. >> >> It can happen that some very busy queues are handled by one pmd which >> does not have enough cycles to prevent packets being dropped on them. >> While at the same time another pmd which handles queues with no traffic >> on them, is essentially idling. >> >> Rxq scheduling happens as a result of a number of events and when it does, >> the same unweighted round robin approach is applied each time. > > I've seen and heard of HW that can be configured with a weighted > round robin approach in which some queues or pools of queues will be > given a higher priority in which they have more 'credits'. > Specifically I know of some Intel HW that is capable of that. What I > do not know is if any of that ever made it into the Linux kernel or if > it is something that might cause your statement to be inaccurate in > all cases. > > My worry is that this assertion may not be correct. > > I'm copying John Fastabend, one of my old co-workers at Intel to catch > is take. He could probably point us in the right direction. > > - Greg
Hi Greg, Thanks for your reply. I'm thinking I was too vague when I said "Rxq scheduling", and we may be talking about slightly different things. Let me elaborate on what I mean and why I used that term. When using the userspace datapath and dpdk type ports (e.g. type dpdk, dpdkvhostuser) the rxqs from those ports are polled for packets by pmd thread(s) in OVS. If there is just one pmd thread, it will poll all the dpdk ports rxqs. However, if there are more than one pmd threads, the OVS userspace code will distribute the polling of rxqs across the pmds. The function that decides the assignment of rxqs to pmds is called rxq_scheduling() and that is the code I was referring to (see patch 6/6), with the patchset changing the method of assignment and hoping to improve the results. Either as part of this or separate, it might be worth changing the name of the function to something like rxq_pmd_assign() to be more specific about what it does. Let me know if this clears things up, or I misinterpreted. thanks, Kevin. > >> >> This patchset proposes to augment the round robin nature of rxq scheduling >> by counting the processing cycles used by the rxqs during their operation >> and incorporate it into the rxq scheduling. >> >> Before distributing in a round robin manner, the rxqs will be sorted in >> order of the processing cycles they have been consuming. Assuming multiple >> pmds, this ensures that the measured rxqs using most processing cycles will >> be distributed to different cores. >> >> To try out: >> This patchset requires the updated pmd counting patch applied as a >> prerequisite. https://patchwork.ozlabs.org/patch/729970/ >> >> Alternatively the series with dependencies can be cloned from here: >> https://github.com/kevintraynor/ovs-rxq.git >> >> Simple way to test is add some dpdk ports, add multiple pmds, vary traffic >> rates and rxqs on ports and trigger reschedules e.g. by changing rxqs or >> the pmd-cpu-mask. >> >> Check rxq distribution with ovs-appctl dpif-netdev/pmd-rxq-show and see >> if it matches expected. >> >> todo: >> -possibly add a dedicated reschedule trigger command >> -use consistent type names >> -update docs >> -more testing, especially for dual numa >> >> thanks, >> Kevin. >> >> Kevin Traynor (6): >> dpif-netdev: Add rxq processing cycle counters. >> dpif-netdev: Update rxq processing cycles from >> cycles_count_intermediate. >> dpif-netdev: Change polled_queue to use dp_netdev_rxq. >> dpif-netdev: Make dpcls optimization interval more generic. >> dpif-netdev: Count the rxq processing cycles for an rxq. >> dpif-netdev: Change rxq_scheduling to use rxq processing cycles. >> >> lib/dpif-netdev.c | 163 >> ++++++++++++++++++++++++++++++++++++++++++++---------- >> 1 file changed, 133 insertions(+), 30 deletions(-) >> >> -- >> 1.8.3.1 >> >> _______________________________________________ >> dev mailing list >> [email protected] >> https://mail.openvswitch.org/mailman/listinfo/ovs-dev _______________________________________________ dev mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-dev
