On Thu, Feb 25, 2010 at 7:26 PM, Jeffrey Yasskin <jyass...@gmail.com> wrote: > On Thu, Feb 25, 2010 at 7:10 PM, Brian Quinlan <br...@sweetapp.com> wrote: >> On Feb 26, 2010, at 4:27 AM, Jeffrey Yasskin wrote: >>> Heh. If you're going to put that in the pep, at least make it correct >>> (sleeping is not synchronization): >> >> I can't tell if you are joking or not. Was my demonstration of a possible >> deadlock scenario really unclear? > > It's clear; it's just wrong code, even if the futures weren't a cycle. > Waiting using sleep in any decently-sized system is guaranteed to > cause problems. Yes, this example will work nearly every time > (although if you get your load high enough, you'll still see > NameErrors), but it's not the kind of thing we should be showing > users. (For that matter, communicating between futures using globals > is also a bad use of them, but it's not outright broken.) > >>> Thanks. I still disagree, and think users are much more likely to be >>> surprised by occasional deadlocks due to cycles of executors than they are >>> about guaranteed deadlocks from cycles of futures, but I don't want to be >>> the only one holding up the PEP by insisting on this. >> >> Cycles of futures are not guaranteed to deadlock. Remove the sleeps from my >> example and it will deadlock a small percentage of the time. > > It only fails to deadlock when it fails to create a cycle of futures. > > > > It sounds like Antoine also wants you to either have the threaded > futures steal work or detect executor cycles and raise an exception.
FWIW, the other way to fix these deadlocks is to write a smarter thread pool. If the thread pool can notice that it's not using as many CPUs as it's been told to use, it can start a new thread, which runs the queued task and resolves the deadlock. It's actually a better solution in the long run since it also solves the problem with wait-for-one deadlocking or behaving badly. The problem is that this is surprisingly hard to get right. Measuring current CPU use is tricky and non-portable; if you start new threads too aggressively, you can run out of memory or start thrashing; and if you don't start threads aggressively enough you hurt performance. _______________________________________________ stdlib-sig mailing list stdlib-sig@python.org http://mail.python.org/mailman/listinfo/stdlib-sig
