On 07/15/10 10:30, Andrej Podzimek wrote:
Well, this is exactly why I am having problems. The dispatched tasks are so short that dispatching can easily take (shooting in the dark) 20 times longer than the tasks themselves. Consequently, the task queue threads need to hold their bucket mutexes locked for 95% of their effective running time. This is not (yet) disastrous, since there are per CPU buckets. However, once all buckets get full, most threads (and especially the task producers) start to compete for the global task queue mutex, which is what I observe. They spin all the time and it is just a matter of chance that those 100 tasks per second eventually get dispatched.
So this is like driving to the store for a case of beer of sugar, but on each trip buying only a single bottle :-).
What I originally tested was a workload with big bursts (== millions) of small tasks that take just a few instructions in most cases, but some of them (perhaps one in a thousand) *might* sleep. Batching tasks works fine (as already mentioned), but then none of the tasks can sleep. So I tried the task queues, but they are obviously not designed for this type of workload. As you say, a task would have to take much more time than the dispatching overhead to keep mutex contention acceptable.
This is a classic threading problem. If you make the partitioning too coarse, a single CPU being busy with something else will greatly increase overall time to complete the task at hand... but if you make it too fine, you spend far more time in task dispatch than actually doing work. The behavior of overly contended kernel adaptive mutexes combined with the system scheduling class also can add chaos when threads are descheduled. As rules of thumb: For maximum parallelism break up work to do in N chunks, where N is at least from 10 to 100 times the number of CPUs/threads, but also 10 to 100 times the cost of dispatching a operation. - Bart -- Bart Smaalders Solaris Kernel Performance bart.smaald...@oracle.com http://blogs.sun.com/barts "You will contribute more with mercurial than with thunderbird." _______________________________________________ on-discuss mailing list on-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/on-discuss
