I recommend adding "import pdb; pdb.set_trace()" before the yield-from that apparently doesn't return and stepping through things (perhaps using Emacs gud).
On Sat, Feb 15, 2014 at 8:54 PM, David Foster <[email protected]> wrote: > > I could imagine that the database connection is tied to a thread but not > to a loop > > Yes. For the UI and DB threads, my presumption was that there was a > dedicated loop for each that wraps around a single thread. > > - The UI loop for wxWidgets could be implemented on top of the > wx.CallAfter primitive which schedules on the native UI loop. > - A dedicated DB thread is needed by the sqlite interface, which > requires that all calls made relative to a single database be made on the > same thread. > > > So you can probably implement the @loop_affinity() decorator, but I > don't see how you could implement switch_to_loop() using the published > Tulip API -- you'd have to modify Task._step() to add another special case > to the block starting with "if isinstance(result, futures.Future):" and > then use a similar strategy as I mentioned above. > > Indeed. I prototyped a modification to Task._step() that recognized a > SwitchToLoop(...) command that was yielded by the switch_to_loop function. > > > Fortunately, a proper multi-event-loop solution should be possible using > call_soon_threadsafe(). [...] > > Here's a first stab: > > def loop_affinity(other_loop): > def decorator(coro): > @asyncio.coroutine > def decorated(*args, **kwargs): > run_coro_result = Future() > > def run_coro(): > coro_result = Task(coro(*args, **kwargs)) > > def coro_is_done(coro_result): > try: > run_coro_result.set_result(coro_result.result()) > except Exception as e: > run_coro_result.set_exception(e) > except BaseException as e: > run_coro_result.set_exception(e) > raise e > coro_result.add_done_callback(coro_is_done) > other_loop.call_soon_threadsafe(run_coro) > > print('run_coro_result: waiting') > yield from run_coro_result > print('run_coro_result: done') # FIXME: Never get here :( > > return decorated > return decorator > > I must be getting some detail slightly off since the line marked as FIXME > is never reached. Strange, since set_result() is getting called properly. > > -- > David Foster > http://dafoster.net/ > > On Saturday, February 15, 2014 7:03:51 PM UTC-8, Guido van Rossum wrote: > >> Thinking about this a little more, I suppose an executor wouldn't be the >> right solution, because what you want to do in the other loop/thread is >> schedule (and wait for) a coroutine on that thread's event loop, not just >> run a function in a thread. At least that's how you've written your example. >> >> (I could imagine that the database connection is tied to a thread but not >> to a loop, and then things are a little bit simpler, but not much -- you >> would have to implement a new executor whose "thread pool" is a single, >> pre-existing thread.) >> >> Fortunately, a proper multi-event-loop solution should be possible using >> call_soon_threadsafe(). What you have to do is in the current loop use the >> other loop's call_soon_threadsafe() to schedule a helper function that >> starts your coroutine (it'll have to wrap it in a Task created in the >> context of the other loop). In order to be able to wait for that in the >> original loop, you have to create a Future in the original loop, and add a >> callback to the Task that uses the original loop's call_soon_threadsafe() >> to pass the result to this Future. >> >> This sounds complicated, and it is, but you can wrap it all up in a >> helper function (or a decorator with the signature you showed in your >> example), and it's the price you pay for a complex scenario (the >> call_soon_threadsafe() calls are needed whenever control transfers between >> threads). >> >> So you can probably implement the @loop_affinity() decorator, but I don't >> see how you could implement switch_to_loop() using the published Tulip API >> -- you'd have to modify Task._step() to add another special case to the >> block starting with "if isinstance(result, futures.Future):" and then use a >> similar strategy as I mentioned above. >> >> Now, that doesn't make the use case any less interesting, but I do >> recommend that you try to write the decorator version first, and use it for >> a while, before you decide that you need to switch loops in the middle. (I >> somehow expect that the helper-function/decorator version will take care of >> most of your needs, if you just structure your code appropriately.) >> >> >> >> >> On Sat, Feb 15, 2014 at 3:36 PM, David Foster <[email protected]> wrote: >> >>> *(Migrating this thread from the python-ideas list)* >>> >>> Tulip does not appear to address a scenario of mine where I need >>> different parts of the same coroutine to run on specific threads or event >>> loops. Consider the following code which needs different parts executed on >>> a UI thread, database thread, and background thread: >>> >>> def tree_node_clicked(tree_node): >>> print('Clicked: ' + repr(tree_node)) >>> def db_task(): >>> db_node = fetch_node_from_db(tree_node.url) >>> >>> def process_node(db_node): >>> tree_node.set_contents(db_node.contents) >>> # (done) >>> >>> if db_node is not None: >>> def bg_task(): >>> db_node = fetch_node_from_internet(tree_node.url) >>> def db_task(): >>> insert_into_db(db_node) >>> ui_call_soon(process_node, db_node) >>> db_call_soon(db_task) >>> bg_call_soon(bg_task) >>> else: >>> ui_call_soon(process_node, db_node) >>> db_call_soon(db_task) >>> >>> Imagine if you could write this function like the following: >>> >>> ui_loop = ... >>> db_loop = ... >>> bg_loop = ... >>> >>> @asyncio.coroutine >>> def tree_node_clicked(tree_node): >>> print('Clicked: ' + repr(tree_node)) >>> >>> yield from switch_to_loop(db_loop) >>> db_node = fetch_node_from_db(tree_node) >>> if db_node is not None: >>> yield from switch_to_loop(bg_loop) >>> db_node = fetch_node_from_internet(tree_node.url) >>> >>> yield from switch_to_loop(db_loop) >>> insert_into_db(db_node) >>> >>> yield from switch_to_loop(ui_loop) >>> tree_node.set_contents(db_node.contents) >>> >>> (The preceding relies on a hypothetical *switch_to_loop* primitive.) >>> >>> Or even better: If you created a decorator like *@loop_affinity* that >>> automatically called switch_to_loop(...) if the current event loop wasn't >>> correct, you could even write the following even-more simplified version: >>> >>> @asyncio.coroutine >>> def tree_node_clicked(tree_node): >>> print('Clicked: ' + repr(tree_node)) >>> db_node = yield from fetch_node_from_db(tree_node) >>> if db_node is not None: >>> db_node = yield from fetch_node_from_internet(tree_node.url) >>> yield from insert_into_db(db_node) >>> yield from tree_node.set_contents(db_node.contents) >>> >>> @loop_affinity(db_loop) >>> def fetch_node_from_db(...): ... >>> >>> @loop_affinity(bg_loop) >>> def fetch_node_from_internet(...): ... >>> >>> @loop_affinity(db_loop) >>> def insert_into_db(...): ... >>> >>> @loop_affinity(ui_loop) >>> def set_contents(...): ... >>> >>> Wouldn't that be just grand? >>> >>> Is there a way to implement a primitive similar to switch_to_loop(...) >>> on top of existing primitives in Tulip? Or would that be considered a new >>> feature? >>> >>> Jonathan Slenders and Guido suggested on python-ideas that I would be >>> able to implement this idea (although perhaps not switch_to_loop) using >>> run_in_executor. Regarding specifically the implementation >>> of switch_to_loop, it is not clear to me how I could get access to the next >>> part of the current coroutine which I would need to pass to run_in_executor >>> (or something similar). >>> >> >> >> >> -- >> --Guido van Rossum (python.org/~guido) >> > -- --Guido van Rossum (python.org/~guido)
