[Python-ideas] Re: SerialExecutor for concurrent.futures + Convenience constructor

A protocol that other Future implementations would be great. The Dask distributed library has an API compatible with concurrent.futures, but would never be appropriate for inclusion in the standard library. It'd be perfect if Dask's Future objects would work well with

[Python-ideas] Re: SerialExecutor for concurrent.futures + Convenience constructor

> Anyway, I think the spelled-out “Synchronous” may be a better name, to avoid the (very likely) case of people mistakenly reading “Sync” as short for “Synchronized”. It’s no longer than “ProcessPool”, and, although it is easy to typo, tab-completion or copy-paste helps, and how many times do you

[Python-ideas] Re: SerialExecutor for concurrent.futures + Convenience constructor

On Feb 17, 2020, at 15:41, Jonathan Crall wrote: > > FWIW I found the term "SyncExecutor" really confusing when I was reading this > thread. I thought it was short for Synchonized, but I just realized its > actually short for Synchronous, which makes much more sense. While >

[Python-ideas] Re: SerialExecutor for concurrent.futures + Convenience constructor

Based on the conversation so far, I agree with @Kyle Stanley's breakdown of the proposal. I think shelving the "*Add a new way to create and specify executor*" and focusing on "*Add a SerialExecutor, which does not use threads or processes*" is the best way forward. For context, I'm a machine

[Python-ideas] Re: SerialExecutor for concurrent.futures + Convenience constructor

> I'm much more lukewarm on set_state(). How hard is it to reimplement > one's own Future if someone wants a different implementation? By > allowing people to change the future's internal state, we're also > giving them a (small) gun to shoot themselves with. Yeah, I don't feel quite as

[Python-ideas] Re: SerialExecutor for concurrent.futures + Convenience constructor

On Mon, 17 Feb 2020 12:19:59 -0800 Guido van Rossum wrote: > It's actually really hard to implement your own Future class that works > well with concurrent.futures.as_completed() -- this is basically what > complicated the OP's implementation. Maybe it would be useful to look into > a protocol to

[Python-ideas] Re: SerialExecutor for concurrent.futures + Convenience constructor

It's actually really hard to implement your own Future class that works well with concurrent.futures.as_completed() -- this is basically what complicated the OP's implementation. Maybe it would be useful to look into a protocol to allow alternative Future implementations to hook into that? On

[Python-ideas] Re: SerialExecutor for concurrent.futures + Convenience constructor

On Sun, 16 Feb 2020 19:46:13 -0500 Kyle Stanley wrote: > > Based on the proposal in the OP, I had considered that it might also be > needed to be able to manually set the state of the future through something > like a `Future.set_state()`, which would have a parameter for accessing it > safely

[Python-ideas] Re: SerialExecutor for concurrent.futures + Convenience constructor

> Hm, but doesn't the OP's example require *synchronously* reading and writing the state? Correct. But in the OP's example, they wanted to use their own "FakeCondition" for reading and writing the state, rather than the executor's internal condition (which is bypassed when you directly access or

[Python-ideas] Re: SerialExecutor for concurrent.futures + Convenience constructor

Hm, but doesn't the OP's example require *synchronously* reading and writing the state? On Sun, Feb 16, 2020 at 4:47 PM Kyle Stanley wrote: > > That sounds useful to me indeed. I assume you mean something like a > > state() method? We already have Queue.qsize() which works a bit like > > this

[Python-ideas] Re: SerialExecutor for concurrent.futures + Convenience constructor

> That sounds useful to me indeed. I assume you mean something like a > state() method? We already have Queue.qsize() which works a bit like > this (unlocked and advisory). Yep, a `Future.state()` method is exactly what I had in mind! I hadn't considered that `Queue.qsize()` was analogous, but

[Python-ideas] Re: SerialExecutor for concurrent.futures + Convenience constructor

On Sun, 16 Feb 2020 17:41:36 -0500 Kyle Stanley wrote: > > As a side note, are we still interested in expanding the public API for the > Future class? Particularly for a public means of accessing the state. The > primary motivation for it was this topic, but I could easily the same > issues

[Python-ideas] Re: SerialExecutor for concurrent.futures + Convenience constructor

> I don't think we need to be dogmatic here. If someone wants to provide > it on PyPI, then be it. But if they'd rather contribute it to the > stdlib, we should examine the relevant PR at face value. > Asking it to be exercised first on PyPI is worthwhile if the domain > space is complex or

[Python-ideas] Re: SerialExecutor for concurrent.futures + Convenience constructor

I'm happy to defer to Antoine, who is the subject expert here (and Brian Quinlan, the original author). On Sun, Feb 16, 2020 at 6:48 AM Antoine Pitrou wrote: > On Sun, 16 Feb 2020 09:29:36 -0500 > Kyle Stanley wrote: > > > > After Andrew explained his own use case for it with isolating bugs to

[Python-ideas] Re: SerialExecutor for concurrent.futures + Convenience constructor

On Sun, 16 Feb 2020 09:29:36 -0500 Kyle Stanley wrote: > > After Andrew explained his own use case for it with isolating bugs to > ensure that the issue wasn't occurring as a result of parallelism, threads, > processes, etc; I certainly can see how it would be useful. I could also > see a use

[Python-ideas] Re: SerialExecutor for concurrent.futures + Convenience constructor

> FWIW, I agree with Andrew here. Being able to swap a > ThreadPoolExecutor or ProcessPoolExecutor with a serial version using > the same API can have benefits in various situations. One is > easier debugging (in case the problem you have to debug isn't a race > condition, of course :-)).

[Python-ideas] Re: SerialExecutor for concurrent.futures + Convenience constructor

On Sat, 15 Feb 2020 14:16:39 -0800 Andrew Barnert via Python-ideas wrote: > > On Feb 15, 2020, at 13:36, Jonathan Crall wrote: > > > > Also, there is no duck-typed class that behaves like an executor, but does > > its processing in serial. Often times a develop will want to run a task in > >

[Python-ideas] Re: SerialExecutor for concurrent.futures + Convenience constructor

I opened a bpo issue to expand upon the public API for cf.Future: https://bugs.python.org/issue39645. Any feedback would be greatly appreciated. (: On Sun, Feb 16, 2020 at 12:22 AM Guido van Rossum wrote: > > > On Sat, Feb 15, 2020 at 21:00 Kyle Stanley wrote: > >> > I've never felt the need

[Python-ideas] Re: SerialExecutor for concurrent.futures + Convenience constructor

> No; the point of launch policies is that you can (without needing an executor object[1]) tell the task to run “async” (on its own thread[2]), “deferred” (serially[3] on first demand), or “immediate” (serially right now)[4]. You can even or together multiple policies to let the implementation

[Python-ideas] Re: SerialExecutor for concurrent.futures + Convenience constructor

On Feb 15, 2020, at 20:29, Kyle Stanley wrote: > Add a SerialExecutor, which does not use threads or processes > > Andrew Barnert wrote: > > e.g., in C++, you only use executors via the std::async function, and you > > can just pass a launch option instead of an executor to run synchronously >

[Python-ideas] Re: SerialExecutor for concurrent.futures + Convenience constructor

On Sat, Feb 15, 2020 at 21:00 Kyle Stanley wrote: > > I've never felt the need for either of these myself, nor have I observed > it in others I worked with. In general I feel the difference between > processes and threads is so large that I can't believe a realistic > application would work with

[Python-ideas] Re: SerialExecutor for concurrent.futures + Convenience constructor

> I've never felt the need for either of these myself, nor have I observed it in others I worked with. In general I feel the difference between processes and threads is so large that I can't believe a realistic application would work with either. Also, ThreadPoolExecutor and ProcessPoolExecutor

[Python-ideas] Re: SerialExecutor for concurrent.futures + Convenience constructor

This seems to be two separate proposals: 1) Add a new way to create and specify executor 2) Add a SerialExecutor, which does not use threads or processes So, I'll respond to each one separately. *Add a new way to create and specify executor* Jonathan Crall wrote: > The library's

[Python-ideas] Re: SerialExecutor for concurrent.futures + Convenience constructor

Having tried my hand at a simpler version for about 15 minutes, I see the reason for the fiddly subclass of Future -- it seems over-engineered because concurrent.future is complicated. I've never felt the need for either of these myself, nor have I observed it in others I worked with. In general

[Python-ideas] Re: SerialExecutor for concurrent.futures + Convenience constructor

This implementation is a proof-of-concept that I've been using for awhile . Its certain that any version that made it into the stdlib would have to be more carefully designed than the implementation I threw

[Python-ideas] Re: SerialExecutor for concurrent.futures + Convenience constructor

> On Feb 15, 2020, at 13:36, Jonathan Crall wrote: > > Also, there is no duck-typed class that behaves like an executor, but does > its processing in serial. Often times a develop will want to run a task in > parallel, but depending on the environment they may want to disable threading > or