On Tue, Jan 27, 2015 at 11:37 AM, Guido van Rossum <[email protected]> wrote:
> Thanks for the patch. I think I understand most of it now. Have you
> written and tested the Tornado conversion yet? What does it look like?
>
See below.
>
> Now let's go back to the Deferred converter you showed. The asyncio
> converter would be very similar, just calling f.set_exception() instead of
> f.set_exc_info(). So support I have an asyncio coroutine and I yield a
> Deferred. I'm probably not calling Deferred() myself -- I'm calling some
> Twisted thing that happens to return a Deferred (which means that the
> result isn't ready yet -- but at some point it will be ready and then the
> deferred will fire). So I yield this Deferred, and the asyncio event loop
> continues.
>
Correct.
>
> But there seems a piece missing here: we need the Twisted event loop to
> run in order for the Deferred to make progress. (Presumably the Deferred
> waits for I/O completion on some socket -- and that socket is registered
> with the Twisted event loop, not with the Tulip event loop.) I assume
> there's a way this is done in Tornado and an analogous way should work for
> Tulip too. But what is it? Is this part in your Tornado patch too?
>
This is where the event-loop level integration comes in. We already have an
implementation of the Tornado IOLoop that wraps the asyncio loop. (Twisted
has a bunch of similar adapters, although I don't see an asyncio one in
their docs) You'd just need to configure it by doing something like
`tornado.platform.asyncio.AsyncIOMainLoop().install()` at the start of the
program.
Here's the same demo from my original email going in the other direction:
import asyncio
import tornado.concurrent
import tornado.httpclient
import tornado.platform.asyncio
@asyncio.convert_yielded.register(tornado.concurrent.Future)
def _(t_future):
a_future = asyncio.Future()
def cb(t_future):
if t_future.exception() is not None:
a_future.set_exception(t_future.exception())
else:
a_future.set_result(t_future.result())
t_future.add_done_callback(cb)
return a_future
@asyncio.coroutine
def main():
client = tornado.httpclient.AsyncHTTPClient()
resp = yield client.fetch('http://www.google.com')
print('fetched %d bytes' % len(resp.body))
if __name__ == '__main__':
tornado.platform.asyncio.AsyncIOMainLoop().install()
asyncio.get_event_loop().run_until_complete(main())
>
> On Sun, Jan 25, 2015 at 5:19 PM, Ben Darnell <[email protected]> wrote:
>
>> On Sun, Jan 25, 2015 at 6:11 PM, Guido van Rossum <[email protected]>
>> wrote:
>>
>>> On Sun, Jan 25, 2015 at 2:52 PM, Ben Darnell <[email protected]> wrote:
>>>
>>>> On Sun, Jan 25, 2015 at 5:31 PM, Guido van Rossum <[email protected]>
>>>> wrote:
>>>>
>>>>> I'm probably a bit dense today, but without browsing the code I'm not
>>>>> entirely sure how your thing works.
>>>>>
>>>>
>>>> Sorry, the implementation in Tornado is in this series of commits:
>>>>
>>>> https://github.com/tornadoweb/tornado/compare/05c3073ce363...841b2d4de32b
>>>>
>>>
>>> OK, not light Sunday afternoon reading. :-)
>>>
>>
>> It's not as bad as it looks; there's some refactoring and
>> backwards-compatibililty weirdness in there too. The asyncio patch will be
>> easier to follow.
>>
>>
>>>
>>>
>>>> What type of thing is the single-dispatch dispatching on?
>>>>>
>>>>
>>>> It dispaches on the type of the thing that was yielded. So to handle
>>>> Deferred, I have this:
>>>>
>>>> + @gen.convert_yielded.register(Deferred)
>>>> + def _(d):
>>>> + f = Future()
>>>> + def errback(failure):
>>>> + try:
>>>> + failure.raiseException()
>>>> + # Should never happen, but just in case
>>>> + raise Exception("errback called without error")
>>>> + except:
>>>> + f.set_exc_info(sys.exc_info())
>>>> + d.addCallbacks(f.set_result, errback)
>>>> + return f
>>>>
>>>
>>> OK, I think I get this part. I'm so glad I invested a week of my summer
>>> vacation two years ago in understanding Deferred
>>> <https://groups.google.com/d/msg/python-tulip/ut4vTG-08k8/PWZzUXX9HYIJ>.
>>> :-)
>>>
>>>
>>>> I then call tornado.gen.convert_yielded() on the result of g.send() (or
>>>> g.throw()) before doing anything else with it.
>>>>
>>>>
>>>>> How would the asyncio and Tornado versions cooperate?
>>>>>
>>>>
>>>> They don't have to cooperate; they can be completely unaware of each
>>>> other. I was able to implement this for Tornado without asyncio or Twisted
>>>> having anything comparable.
>>>>
>>>
>>> Ah, so we each just register things that handle the other's kind of
>>> Future by converting it to one of our own kind, and then the framework of
>>> the caller (the generator that calls yield) gets to keep control. Right?
>>> I'm a little worried (again not yet having fully internalized the whole
>>> process) that there might be some cases where frameworks keep passing the
>>> ball without making progress, but I can also easily imagine that that's not
>>> how it works.
>>>
>>
>> Right. The coroutine runner (tornado.gen.coroutine, asyncio.Task,
>> twisted.internet.defer.inlineCallbacks) is a constant; a Tornado coroutine
>> doesn't magically become an asyncio.Task. It's just the Futures that get
>> wrapped. There's no risk of getting stuck in a loop - convert_yielded() is
>> only called once and if it doesn't return a value of the right Future type
>> then it's an error.
>>
>>
>>>
>>>
>>>>
>>>>
>>>>>
>>>>> Perhaps you can submit a patch for asyncio to clarify the proposal?
>>>>>
>>>>
>>>> Sure. Is the tulip project on code.google.com still the place for that
>>>> or is new development happening in the cpython repo?
>>>>
>>>
>>> It doesn't really matter -- we merge back and forth these days. You may
>>> find the asyncio repo easier to deal with.
>>>
>>
>> Indeed. Here's a patch: https://codereview.appspot.com/193540043/
>>
>> There are two subtleties here:
>> * asyncio requires the use of 'yield from' everywhere, but third-party
>> future-like objects may not be iterable, so they must be allowed with
>> 'yield'. I disabled a check for Future._blocking when convert_yielded
>> returned anything but the original object
>> * asyncio.Future requires an event loop to be set for the current thread
>> at construction time. This required me to call asyncio.set_event_loop in
>> the test for this feature when most comparable tests in test_tasks.py do
>> not have this.
>>
>> -Ben
>>
>>
>>>
>>>> -Ben
>>>>
>>>>
>>>>>
>>>>> On Sun, Jan 25, 2015 at 9:42 AM, Ben Darnell <[email protected]>
>>>>> wrote:
>>>>>
>>>>>> In Tornado 4.1 [1], I've added a functools.singledispatch-based
>>>>>> registry for objects that can be yielded in coroutines. This allows for
>>>>>> Future-like objects from different frameworks to be mixed seamlessly in
>>>>>> the
>>>>>> same coroutine (We currently have support for tornado.concurrent.Future,
>>>>>> concurrent.futures.Future, asyncio.Future[2], and
>>>>>> twisted.internet.defer.Deferred)
>>>>>>
>>>>>> I'd like to suggest that asyncio add something similar to facilitate
>>>>>> the use of libraries from different frameworks in the same coroutine, not
>>>>>> just sharing the same event loop. It would be a pretty simple change:
>>>>>> just
>>>>>> call a function decorated with @singledispatch (whose default
>>>>>> implementation is the identity function) before handling the result in
>>>>>> Task._step.
>>>>>>
>>>>>> I've included a simple example of a hybrid tornado/asyncio coroutine
>>>>>> below.
>>>>>>
>>>>>> -Ben
>>>>>>
>>>>>> [1] Currently in beta, installable with `pip install
>>>>>> https://github.com/tornadoweb/tornado/archive/v4.1.0b1.zip`
>>>>>> <https://github.com/tornadoweb/tornado/archive/v4.1.0b1.zip>
>>>>>> [2] The future objects are all similar enough that they don't really
>>>>>> need this level of abstraction, but they serve as a proof of concept.
>>>>>>
>>>>>> # requirements:
>>>>>> # python3.4 (for 3.3, add asyncio and singledispatch)
>>>>>> # aiohttp
>>>>>> # https://github.com/tornadoweb/tornado/archive/v4.1.0b1.zip
>>>>>> import aiohttp
>>>>>> import asyncio
>>>>>> import tornado.gen
>>>>>> import tornado.httpclient
>>>>>> import tornado.ioloop
>>>>>> import tornado.platform.asyncio
>>>>>>
>>>>>> @tornado.gen.coroutine
>>>>>> def main():
>>>>>> t_client = tornado.httpclient.AsyncHTTPClient()
>>>>>> t_response = yield t_client.fetch('http://www.google.com')
>>>>>> print('tornado: read %d bytes with status code %d' %
>>>>>> (len(t_response.body), t_response.code))
>>>>>>
>>>>>> a_response = yield from aiohttp.request('GET', '
>>>>>> http://www.google.com')
>>>>>> a_body = yield from a_response.read()
>>>>>> print('aiohttp: read %d bytes with status code %d' %
>>>>>> (len(a_body), a_response.status))
>>>>>>
>>>>>> # Alternate python2-compatible syntax
>>>>>> a_response2 = yield asyncio.async(aiohttp.request('GET', '
>>>>>> http://www.google.com'))
>>>>>> a_body2 = yield asyncio.async(a_response2.read())
>>>>>> print('aiohttp2: read %d bytes with status code %d' %
>>>>>> (len(a_body2), a_response2.status))
>>>>>>
>>>>>> if __name__ == '__main__':
>>>>>>
>>>>>> tornado.ioloop.IOLoop.configure(tornado.platform.asyncio.AsyncIOMainLoop)
>>>>>> tornado.ioloop.IOLoop.current().run_sync(main)
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> --Guido van Rossum (python.org/~guido)
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> --Guido van Rossum (python.org/~guido)
>>>
>>
>>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>
