On Sun, Jan 25, 2015 at 6:11 PM, Guido van Rossum <[email protected]> wrote:
> On Sun, Jan 25, 2015 at 2:52 PM, Ben Darnell <[email protected]> wrote: > >> On Sun, Jan 25, 2015 at 5:31 PM, Guido van Rossum <[email protected]> >> wrote: >> >>> I'm probably a bit dense today, but without browsing the code I'm not >>> entirely sure how your thing works. >>> >> >> Sorry, the implementation in Tornado is in this series of commits: >> https://github.com/tornadoweb/tornado/compare/05c3073ce363...841b2d4de32b >> > > OK, not light Sunday afternoon reading. :-) > It's not as bad as it looks; there's some refactoring and backwards-compatibililty weirdness in there too. The asyncio patch will be easier to follow. > > >> What type of thing is the single-dispatch dispatching on? >>> >> >> It dispaches on the type of the thing that was yielded. So to handle >> Deferred, I have this: >> >> + @gen.convert_yielded.register(Deferred) >> + def _(d): >> + f = Future() >> + def errback(failure): >> + try: >> + failure.raiseException() >> + # Should never happen, but just in case >> + raise Exception("errback called without error") >> + except: >> + f.set_exc_info(sys.exc_info()) >> + d.addCallbacks(f.set_result, errback) >> + return f >> > > OK, I think I get this part. I'm so glad I invested a week of my summer > vacation two years ago in understanding Deferred > <https://groups.google.com/d/msg/python-tulip/ut4vTG-08k8/PWZzUXX9HYIJ>. > :-) > > >> I then call tornado.gen.convert_yielded() on the result of g.send() (or >> g.throw()) before doing anything else with it. >> >> >>> How would the asyncio and Tornado versions cooperate? >>> >> >> They don't have to cooperate; they can be completely unaware of each >> other. I was able to implement this for Tornado without asyncio or Twisted >> having anything comparable. >> > > Ah, so we each just register things that handle the other's kind of Future > by converting it to one of our own kind, and then the framework of the > caller (the generator that calls yield) gets to keep control. Right? I'm a > little worried (again not yet having fully internalized the whole process) > that there might be some cases where frameworks keep passing the ball > without making progress, but I can also easily imagine that that's not how > it works. > Right. The coroutine runner (tornado.gen.coroutine, asyncio.Task, twisted.internet.defer.inlineCallbacks) is a constant; a Tornado coroutine doesn't magically become an asyncio.Task. It's just the Futures that get wrapped. There's no risk of getting stuck in a loop - convert_yielded() is only called once and if it doesn't return a value of the right Future type then it's an error. > > >> >> >>> >>> Perhaps you can submit a patch for asyncio to clarify the proposal? >>> >> >> Sure. Is the tulip project on code.google.com still the place for that >> or is new development happening in the cpython repo? >> > > It doesn't really matter -- we merge back and forth these days. You may > find the asyncio repo easier to deal with. > Indeed. Here's a patch: https://codereview.appspot.com/193540043/ There are two subtleties here: * asyncio requires the use of 'yield from' everywhere, but third-party future-like objects may not be iterable, so they must be allowed with 'yield'. I disabled a check for Future._blocking when convert_yielded returned anything but the original object * asyncio.Future requires an event loop to be set for the current thread at construction time. This required me to call asyncio.set_event_loop in the test for this feature when most comparable tests in test_tasks.py do not have this. -Ben > >> -Ben >> >> >>> >>> On Sun, Jan 25, 2015 at 9:42 AM, Ben Darnell <[email protected]> wrote: >>> >>>> In Tornado 4.1 [1], I've added a functools.singledispatch-based >>>> registry for objects that can be yielded in coroutines. This allows for >>>> Future-like objects from different frameworks to be mixed seamlessly in the >>>> same coroutine (We currently have support for tornado.concurrent.Future, >>>> concurrent.futures.Future, asyncio.Future[2], and >>>> twisted.internet.defer.Deferred) >>>> >>>> I'd like to suggest that asyncio add something similar to facilitate >>>> the use of libraries from different frameworks in the same coroutine, not >>>> just sharing the same event loop. It would be a pretty simple change: just >>>> call a function decorated with @singledispatch (whose default >>>> implementation is the identity function) before handling the result in >>>> Task._step. >>>> >>>> I've included a simple example of a hybrid tornado/asyncio coroutine >>>> below. >>>> >>>> -Ben >>>> >>>> [1] Currently in beta, installable with `pip install >>>> https://github.com/tornadoweb/tornado/archive/v4.1.0b1.zip` >>>> <https://github.com/tornadoweb/tornado/archive/v4.1.0b1.zip> >>>> [2] The future objects are all similar enough that they don't really >>>> need this level of abstraction, but they serve as a proof of concept. >>>> >>>> # requirements: >>>> # python3.4 (for 3.3, add asyncio and singledispatch) >>>> # aiohttp >>>> # https://github.com/tornadoweb/tornado/archive/v4.1.0b1.zip >>>> import aiohttp >>>> import asyncio >>>> import tornado.gen >>>> import tornado.httpclient >>>> import tornado.ioloop >>>> import tornado.platform.asyncio >>>> >>>> @tornado.gen.coroutine >>>> def main(): >>>> t_client = tornado.httpclient.AsyncHTTPClient() >>>> t_response = yield t_client.fetch('http://www.google.com') >>>> print('tornado: read %d bytes with status code %d' % >>>> (len(t_response.body), t_response.code)) >>>> >>>> a_response = yield from aiohttp.request('GET', ' >>>> http://www.google.com') >>>> a_body = yield from a_response.read() >>>> print('aiohttp: read %d bytes with status code %d' % >>>> (len(a_body), a_response.status)) >>>> >>>> # Alternate python2-compatible syntax >>>> a_response2 = yield asyncio.async(aiohttp.request('GET', ' >>>> http://www.google.com')) >>>> a_body2 = yield asyncio.async(a_response2.read()) >>>> print('aiohttp2: read %d bytes with status code %d' % >>>> (len(a_body2), a_response2.status)) >>>> >>>> if __name__ == '__main__': >>>> >>>> tornado.ioloop.IOLoop.configure(tornado.platform.asyncio.AsyncIOMainLoop) >>>> tornado.ioloop.IOLoop.current().run_sync(main) >>>> >>>> >>> >>> >>> -- >>> --Guido van Rossum (python.org/~guido) >>> >> >> > > > -- > --Guido van Rossum (python.org/~guido) >
