This is the easiest thing to try out // #define QUEUES_PER_PRIO 4 #define QUEUES_PER_PRIO 1
The default value expects that ODP is running at least 4 cores. When you run cores less than that, unfairness starts to appear. However, unfairness and overload are two different things. Even the default value should eventually schedule all queue if you give it a chance. Input packet rate (+ anything app circulates back to scheduler) must be in average less than 100% CPU throughput. Another algorithm would be more fair for a bit longer time, but eventually would also fail under constant overload (queues build up until a pool runs out of packets). Anyway, the above define could be changed to a variable that is set during init (based on max thread counts, etc). -Petri > -----Original Message----- > From: ext Nicolas Morey-Chaisemartin [mailto:[email protected]] > Sent: Tuesday, September 08, 2015 4:57 PM > To: Ola Liljedahl; Bill Fischofer > Cc: LNG ODP Mailman List; Savolainen, Petri (Nokia - FI/Espoo) > Subject: Re: [lng-odp] Scheduler, QUEUES_PER_PRIO and fairness > > > > On 09/08/2015 03:33 PM, Ola Liljedahl wrote: > > Sorry I missed this discussion. It is really interesting. IMO the > linux-generic scheduler is too simplistic to be used as is or its > behaviour copied. We have seen some undesirable behaviour in our > internal work where we use ODP. Very simplified, we essentially have > two levels of processing, both fed by the scheduler to the same shared > set of CPU's (sometimes only one CPU). The first level is fed using one > queue and the second level is fed using N queues. Both levels are fed > the same amount of packets so the first level queue transfers N packets > while the second level queues transfer 1 packet each (on the average). > The linux-generic scheduler does round robin over queues, trying to > give *equal* attention to each queue. But in our design, all queues are > not equal and the first level queue should get more attention. > > > > One idea was to specify weights with the queues and do some kind of > weighted queueing. But I don't like the idea of trying to know the > optimal weights and then configure how the scheduler should work. > Possibly these weights are not constant. I think the scheduler needs to > dynamically adapt to the current workload, I think this will be much > more robust and it also requires less configuration by the application > (which is always difficult). > Agreed. > > > > I think the scheduler needs to give queues attention corresponding to > how many events they (currently) contain. One potential way of doing > that is to keep scheduling from a queue until it either is empty or you > move on to the next queue with probability 2^-n (n being the number of > events on the queue). The more events on a queue, the longer you serve > that queue but with some increasing probability (as the queue drains) > you move on to the next eligible (non-empty) queue. Over time this > should give some kind of "fair" attention to all queues without > stalling indefinitely on some queue that refuses to drain. > > > The issue with this kind of algorithm is that it can be quite difficult > to have an efficient (and fair) parallel version of it. > If you want to be fair, you need to synchronize your threads so that > your probability not only depends on what your thread saw in its queues > but what the others saw too. > It often means adding a big lock or something of sort. > > I'm not saying it's impossible but being both scalable (with more > cores) and fair is not an easy problem. > > > The changes to a scheduler that does unconditional round robin > shouldn't be too complicated. > > A very easy change that would fix the starvation issue would be to > start iterating from the previous offset (+1?) instead of our thread > id. > This way we can ensure that all thread will go around all the queues. > It's not good for cache though. > > I can prepare a patch but I will need someone to evaluate the > performance impact on x86. I don't have a clean setup available for > benchmarking. > > Nicolas _______________________________________________ lng-odp mailing list [email protected] https://lists.linaro.org/mailman/listinfo/lng-odp
