@Guido As an aside, my understanding was libraries that fall back to c (Numpy as an example) release the GIL for load heavy operations. But I believe the explanation would hold in the general case if you replace thread with process using a ProcessPoolExecutor, that it would be good to be able to submit a callback function back to the executor.
Sent from my iPhone > On Jan 26, 2018, at 12:10 PM, Daniel Collins <dancollin...@gmail.com> wrote: > > @Guido: I agree, that’s a much cleaner solution to pass the executor. > However, I think the last line should be future.add_done_callback(callback) > return newf > > not executor.submit. > > I’ll rewrite it like this and resubmit tonight for discussion. > > Sent from my iPhone > >> On Jan 26, 2018, at 11:59 AM, Guido van Rossum <gu...@python.org> wrote: >> >> @Bar: I don't know about exposing _chain_future(). Certainly it's overkill >> for what the OP wants -- their PR only cares about chaining >> concurrent.future.Future. >> >> @Daniel: I present the following simpler solution -- it requires you to >> explicitly pass the executor, but since 'fn' is being submitted to an >> executor, EIBTI. >> >> def then(executor, future, fn): >> newf = concurrent.futures.Future() >> def callback(fut): >> f = executor.submit(fn, fut) >> try: >> newf.set_result(f.result()) >> except CancelledError: >> newf.cancel() >> except Exception as err: >> newf.set_exception(err) >> return executor.submit(callback) >> >> I can't quite follow your reasoning about worker threads (and did you >> realize that because of the GIL, Python doesn't actually use multiple >> cores?). But I suppose it doesn't matter whether I understand that -- your >> point is that you want the 'fn' function submitted to the executor, not run >> as a "done callback". And that's reasonable. But modifying so much code just >> so the Future can know which to executor it belongs so you can make then() a >> method seems overkill. >> >>> On Fri, Jan 26, 2018 at 8:54 AM, Daniel Collins <dancollin...@gmail.com> >>> wrote: >>> So, just going point by point: >>> >>> Yes, absolutely put this off for 3.8. I didn’t know the freeze was so close >>> or I would have put the 3.8 tag on originally. >>> >>> Yes, absolutely it is only meant for concurrent.futures futures, it only >>> changes async where async uses concurrent.futures futures. >>> >>> Here’s a more fleshed out description of the use case: >>> >>> Assume you have two functions. Function a(x: str)->AResult fetches an >>> AResult object from a web resource, function b(y: AResult) performs some >>> computationally heavy work on AResult. >>> >>> Assume you’re calling a 10 times with a threadpoolexecutor with 2 worker >>> theads. If you were to schedule a as future using submit, and b as a >>> callback, the executions would look like this: >>> >>> ExecutorThread: b*10 >>> Worker1: a*5 >>> Worker2: a*5 >>> >>> This only gets worse as more work (b) is scheduled as a callback for the >>> result from a. >>> >>> Now you could resolve this by, instead of submitting b as a callback, >>> submitting the following lambda: >>> >>> lambda x: executor.submit(b, x) >>> >>> But then you wouldn’t have easy access to this new future. You would have >>> to build a lot of boilerplate code to collect that future into some >>> external collection, and this would only get worse the deeper the nesting >>> goes. >>> >>> With this syntax on the other hand, if you run a 10 times using submit, but >>> then run a_fut.then(b) for each future, execution instead looks like this: >>> >>> ExecutorThread: >>> Worker1: a*5 b*5 >>> Worker2: a*5 b*5 >>> >>> You can also do additional depth easily. Suppose you want to run 3 c >>> operations (processes the output of b) for each b operation. Then you could >>> call this like >>> >>> b_fut = a_fut.then(b) >>> >>> for i in range(3): >>> b_fut.then(c) >>> >>> And the execution would look like this: >>> >>> ExecutorThread: >>> Worker1: a*5 b*5 c*15 >>> Worker2: a*5 b*5 c*15 >>> >>> Which would be very difficult to do otherwise, and distributes the load >>> across the workers, while having direct access to the outputs of the calls >>> to c. >>> >>> -dancollins34 >>> >>> Sent from my iPhone >>> >>>> On Jan 26, 2018, at 1:07 AM, Guido van Rossum <gu...@python.org> wrote: >>>> >>>> I really don't want to distract Yury with this. Let's consider this (or >>>> something that addresses the same need) for 3.8. >>>> >>>> To be clear this is meant as a feature for concurrent.futures.Future, not >>>> for asyncio.Future. (It's a bit confusing since you also change asyncio.) >>>> >>>> Also to be honest I don't understand the use case *or* the semantics very >>>> well. You have some explaining to do... >>>> >>>> (Also, full links: https://bugs.python.org/issue32672; >>>> https://github.com/python/cpython/pull/5335) >>>> >>>>> On Thu, Jan 25, 2018 at 8:38 PM, Daniel Collins <dancollin...@gmail.com> >>>>> wrote: >>>>> Hello all, >>>>> >>>>> So, first time posting here. I’ve been bothered for a while about the >>>>> lack of the ability to chain futures in python, such that the next future >>>>> will execute upon the first’s completion. So I submitted a pr to do >>>>> this. This would add the .then(self, fn) method to >>>>> concurrent.futures.Future. Thoughts? >>>>> >>>>> -dancollins34 >>>>> >>>>> Github PR #5335 >>>>> bugs.python.org issue #32672 >>>>> >>>>> _______________________________________________ >>>>> Python-ideas mailing list >>>>> Python-ideas@python.org >>>>> https://mail.python.org/mailman/listinfo/python-ideas >>>>> Code of Conduct: http://python.org/psf/codeofconduct/ >>>>> >>>> >>>> >>>> >>>> -- >>>> --Guido van Rossum (python.org/~guido) >> >> >> >> -- >> --Guido van Rossum (python.org/~guido)
_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
