One thing I think is worth raising here: If my experience with concurrent.futures is far from universal, my expectations for this change may not be helpful, so let me lay that all out explicitly.
In my experience, concurrent.futures has two very good, but very different, uses. First, there’s trivial concurrency. The last stage of my processing is an obviously parallelizable pure function? Just toss it in an executor and I’m done. My asyncio code needs to call something blocking that doesn’t have an async equivalent? Toss it in an executor and I’m done. On the other extreme, there are cases where your data flow is complicated, and organizing it all around the composability of futures helps rein in that complexity. In between those two extremes, it’s often easier to use something different—e.g., a multiprocessing.Pool has a lot more options, and a lot more convenience functions for different ways of mapping; I’ve never missed Java’s thread-per-task executor because it’s easier to just use a Thread directly; you wouldn’t want a timer scheduling queue that hid all the time information under the covers; etc. I’m expect this change will mostly be helpful for the trivial case. The last stage of my processing is obviously parallelizable? Just stick it in an executor and … wait, it’s still too slow, so the caller is wasting too many resources in the queue? Just stick it in a _bounded_ executor and I’m done. In the really complicated cases, the backpressure is usually going to be in the future waiting, not in the task queuing, even if that’s more complicated to set up. Because otherwise the flow isn’t 100% composable anymore, which is the whole reason I was using concurrent.futures. Anything in the moderately complicated range, and I‘m probably going to want an explicitly-managed queue rather than one hidden behind an abstraction, or a separate semaphore, or whatever. But then in those cases, I’m probably not even using concurrent.futures in the first place. So, max_queue_len with blocking submit (and maybe also with raising submit_nowait) handles virtually all of the simple cases that I can think of or that anyone else has suggested, and I don’t think it matters much how many of the moderate or extremely complicated cases it handles. _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/CE56FZQODIWEP3PUSK4I2R2S4LESEJLB/ Code of Conduct: http://python.org/psf/codeofconduct/
