On Mon, 2017-05-08 at 11:49 +0100, Kevin Traynor wrote: > On 05/06/2017 02:01 AM, Greg Rose wrote: > > On Fri, May 5, 2017 at 9:34 AM, Kevin Traynor <[email protected]> wrote: > >> Rxqs are scheduled to be handled across available pmds in round robin > >> order with no weight or priority. > >> > >> It can happen that some very busy queues are handled by one pmd which > >> does not have enough cycles to prevent packets being dropped on them. > >> While at the same time another pmd which handles queues with no traffic > >> on them, is essentially idling. > >> > >> Rxq scheduling happens as a result of a number of events and when it does, > >> the same unweighted round robin approach is applied each time. > > > > I've seen and heard of HW that can be configured with a weighted > > round robin approach in which some queues or pools of queues will be > > given a higher priority in which they have more 'credits'. > > Specifically I know of some Intel HW that is capable of that. What I > > do not know is if any of that ever made it into the Linux kernel or if > > it is something that might cause your statement to be inaccurate in > > all cases. > > > > My worry is that this assertion may not be correct. > > > > I'm copying John Fastabend, one of my old co-workers at Intel to catch > > is take. He could probably point us in the right direction. > > > > - Greg > > Hi Greg, > > Thanks for your reply. I'm thinking I was too vague when I said "Rxq > scheduling", and we may be talking about slightly different things. Let > me elaborate on what I mean and why I used that term. > > When using the userspace datapath and dpdk type ports (e.g. type dpdk, > dpdkvhostuser) the rxqs from those ports are polled for packets by pmd > thread(s) in OVS. If there is just one pmd thread, it will poll all the > dpdk ports rxqs. However, if there are more than one pmd threads, the > OVS userspace code will distribute the polling of rxqs across the pmds. > The function that decides the assignment of rxqs to pmds is called > rxq_scheduling() and that is the code I was referring to (see patch > 6/6), with the patchset changing the method of assignment and hoping to > improve the results. > > Either as part of this or separate, it might be worth changing the name > of the function to something like rxq_pmd_assign() to be more specific > about what it does. > > Let me know if this clears things up, or I misinterpreted. > > thanks, > Kevin.
That does indeed clear things up. Thank you! - Greg > > > > >> > >> This patchset proposes to augment the round robin nature of rxq scheduling > >> by counting the processing cycles used by the rxqs during their operation > >> and incorporate it into the rxq scheduling. > >> > >> Before distributing in a round robin manner, the rxqs will be sorted in > >> order of the processing cycles they have been consuming. Assuming multiple > >> pmds, this ensures that the measured rxqs using most processing cycles will > >> be distributed to different cores. > >> > >> To try out: > >> This patchset requires the updated pmd counting patch applied as a > >> prerequisite. https://patchwork.ozlabs.org/patch/729970/ > >> > >> Alternatively the series with dependencies can be cloned from here: > >> https://github.com/kevintraynor/ovs-rxq.git > >> > >> Simple way to test is add some dpdk ports, add multiple pmds, vary traffic > >> rates and rxqs on ports and trigger reschedules e.g. by changing rxqs or > >> the pmd-cpu-mask. > >> > >> Check rxq distribution with ovs-appctl dpif-netdev/pmd-rxq-show and see > >> if it matches expected. > >> > >> todo: > >> -possibly add a dedicated reschedule trigger command > >> -use consistent type names > >> -update docs > >> -more testing, especially for dual numa > >> > >> thanks, > >> Kevin. > >> > >> Kevin Traynor (6): > >> dpif-netdev: Add rxq processing cycle counters. > >> dpif-netdev: Update rxq processing cycles from > >> cycles_count_intermediate. > >> dpif-netdev: Change polled_queue to use dp_netdev_rxq. > >> dpif-netdev: Make dpcls optimization interval more generic. > >> dpif-netdev: Count the rxq processing cycles for an rxq. > >> dpif-netdev: Change rxq_scheduling to use rxq processing cycles. > >> > >> lib/dpif-netdev.c | 163 > >> ++++++++++++++++++++++++++++++++++++++++++++---------- > >> 1 file changed, 133 insertions(+), 30 deletions(-) > >> > >> -- > >> 1.8.3.1 > >> > >> _______________________________________________ > >> dev mailing list > >> [email protected] > >> https://mail.openvswitch.org/mailman/listinfo/ovs-dev > _______________________________________________ dev mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-dev
