On Sat, May 9, 2015 at 12:31 AM, Michael Welle <mwe012...@gmx.net> wrote: >> As a general rule, queues need to have both ends operating >> simultaneously, otherwise you're likely to have them blocking. In >> theory, your code should all work with ridiculously low queue sizes; >> the only cost will be concurrency (since you'd forever be waiting for >> the queue, so your tasks will all be taking turns). I tested this by >> changing the Queue() calls to Queue(1), and the code took about twice >> as long to complete. :) > ;) I know, as you might guess it's not a real world example. It's just > to explore the multiprocessing module.
Sure, but even in a real-world example, it shouldn't ever be necessary to create huge queues. Larger queues allow for inconsistent performance of producer and/or consumer (eg if your consumer takes 1s to do each of 500 jobs, then 500s to do one job, it's capable of coping with a producer that puts one job on the queue every 2s, but only if the queue can handle ~250 jobs), but otherwise, the only effect of shrinking the queues is to force the processes into lock-step. Bigger queues mean that an over-performing producer will run you out of memory rather than get blocked. At very least, it should be a safe way to debug your logic - cut the queues down to just 2-3 elements each, and keep on printing out what's in the queue. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
