On Mon, Nov 9, 2015 at 11:14 PM, Guido van Rossum <[email protected]> wrote:
> The intention of this code is to make it possible (and legal) to decorate a
> non-generator function with `@asyncio.coroutine`, and to make that
> effectively equivalent to the same decorated function with an unreachable
> yield inside it (which makes it a generator). IOW these two should be
> equivalent:
>
> @asyncio.coroutine
> def foo():
> return 42
>
> @asyncio.coroutine
> def foo():
> return 42
> yield # dummy to make it a coroutine
>
> You should be able to write in some other coroutine
>
> x = yield from foo()
> assert x == 42
>
> regardless of which definition of foo() you've got.
>
> Note that for the first version the decorator always matters, while for the
> second version it only matters in debug mode.
>
> But it looks like the code has some subtle bug. :-(
>
> I think one of the problems is that we don't always test with debug mode.
>
> There are probably also some cases that the code is trying to support, e.g.
> for generators in general (outside the context of asyncio) there is usually
> no difference between these two examples:
>
> def foo():
> yield 1
> yield 2
> def bar():
> return foo()
>
> vs.
>
> def bar():
> yield 1
> yield 2
>
> -- in either case, calling bar() returns a generator that when iterated over
> generates the sequence [1, 2]. But what you've discovered is that if we
> decorate both functions with @asyncio.coroutine, there *is* a difference.
> And I think the code is making a half-hearted attempt to paper over the
> difference, but failing due to debug mode.
Thanks for the explanation. Reading asyncio documentation I thought about
asyncio-coroutines as some special kind functions (implemented using
generators) that can be run as coroutines by the event loop.
Almost always I develop asyncio-based programs with enabled
PYTHONASYNCIODEBUG, which in some cases doesn't allow to mix
usual functions and asyncio.coroutine-functions due to the discussed bug ---
this is one of the reason I thought that asyncio-coroutines can't be used as
simple generator functions.
It would be nice if asyncio has better documentation for this case, or at
least link on a good tutorial.
Other bug I experienced in Python 3.4 about different behaviour when
PYTHONASYNCIODEBUG is enabled is when wrapped with
asyncio.coroutine function doesn't have special attributes like
"__name__". In Python 3.4.3 with enabled asyncio debug a function is
wrapped using following code:
@functools.wraps(func)
def wrapper(*args, **kwds):
w = CoroWrapper(coro(*args, **kwds), func)
if w._source_traceback:
del w._source_traceback[-1]
w.__name__ = func.__name__
if hasattr(func, '__qualname__'):
w.__qualname__ = func.__qualname__
w.__doc__ = func.__doc__
return w
note the unconditional access to "__name__" and "__doc__" attributes,
which function can not have in some conditions.
As this use case looks strange and unrealistic I met him in a real
application that used mocking in tests. The code there was something like
following:
def f():
return "f result"
mocked_f = Mock(wraps=f)
coro_func = asyncio.coroutine(mocked_f)
print(loop.run_until_complete(coro_func()))
mocked_f.assert_called_once_with()
which failed only in the debug mode:
$ python3 asyncio_debug_attr_access.py
f result
$ PYTHONASYNCIODEBUG=X python3 asyncio_debug_attr_access.py
Traceback (most recent call last):
File "asyncio_debug_attr_access.py", line 21, in <module>
print(loop.run_until_complete(coro_func()))
File "/usr/lib/python3.4/asyncio/coroutines.py", line 154, in wrapper
w.__name__ = func.__name__
File "/usr/lib/python3.4/unittest/mock.py", line 570, in __getattr__
raise AttributeError(name)
AttributeError: __name__
Exception ignored in: $
Here is complete example:
https://gist.github.com/rutsky/65cee7728135b05d49c3
It can be bug in the unittest.mock library, that mock doesn't have special
attributes, but still it's unexpected to have such different behaviour in debug
and not debug modes.
This issue is not reproduced in Python 3.5, since it has more accurate
special attribute access:
@functools.wraps(func)
def wrapper(*args, **kwds):
w = CoroWrapper(coro(*args, **kwds), func=func)
if w._source_traceback:
del w._source_traceback[-1]
# Python < 3.5 does not implement __qualname__
# on generator objects, so we set it manually.
# We use getattr as some callables (such as
# functools.partial may lack __qualname__).
w.__name__ = getattr(func, '__name__', None)
w.__qualname__ = getattr(func, '__qualname__', None)
return w
I workarounded both of the issues using monkeypatching in my project.
Should I file bug reports against Python 3.4 about them on the Python bug
tracker?
Regards,
Vladimir Rutsky
>
> On Mon, Nov 9, 2015 at 10:25 AM, Vladimir Rutsky <[email protected]>
> wrote:
>>
>> Hello!
>>
>> Can anybody explain to me this part of asyncio.coroutine code:
>>
>> <https://github.com/python/asyncio/blob/3b6a64a9fb6ec4ad0c984532aa776b130067c901/asyncio/coroutines.py#L201>
>>
>> # asyncio/coroutines.py:201:
>> def coroutine(func):
>> """Decorator to mark coroutines.
>> If the coroutine is not yielded from before it is destroyed,
>> an error message is logged.
>> """
>> if _inspect_iscoroutinefunction(func):
>> # In Python 3.5 that's all we need to do for coroutines
>> # defiend with "async def".
>> # Wrapping in CoroWrapper will happen via
>> # 'sys.set_coroutine_wrapper' function.
>> return func
>>
>> if inspect.isgeneratorfunction(func):
>> coro = func
>> else:
>> @functools.wraps(func)
>> def coro(*args, **kw):
>> res = func(*args, **kw)
>> if isinstance(res, futures.Future) or
>> inspect.isgenerator(res): # <--- This part
>> res = yield from res
>> ...
>>
>> What does this code: if wrapped by asyncio.coroutine function is not
>> generator and returns generator, Future or awaitable object,
>> then this coroutine function will yield from returned value.
>>
>> E.g. following coroutine function will actually return result of the
>> future, not future itself:
>>
>> @asyncio.coroutine
>> def return_future():
>> fut = asyncio.Future()
>> fut.set_result("return_future")
>>
>> return fut
>>
>> I can't find where such behaviour is documented? Is this a desired
>> behavior?
>>
>> According to
>> <https://docs.python.org/3/library/asyncio-task.html?highlight=iscoroutine#coroutines>:
>> > Coroutines used with asyncio may be implemented using the async def
>> > statement, or by using generators.
>> > ...
>> > Generator-based coroutines should be decorated with @asyncio.coroutine,
>> > although this is not strictly enforced.
>>
>> So technically, according to this part of documentation, it's not
>> explicitly allowed to use asyncio.coroutine with non-generator functions.
>>
>> Special behaviour when asyncio.coroutine-wrapped non-generator function
>> returns generator or Future object
>> leads to inconsistent behaviour on Python 3.4 when asyncio debugging is
>> enabled and not enabled.
>>
>> Consider following function:
>>
>> @asyncio.coroutine
>> def return_coroutine_object():
>> @asyncio.coroutine
>> def g():
>> yield from asyncio.sleep(0.01)
>> return "return_coroutine_object"
>>
>> return g()
>>
>> When debugging is disabled following code will work:
>>
>> assert loop.run_until_complete(return_coroutine_object()) ==
>> "return_coroutine_object"
>>
>> When debugging is enabled same code will not work on Python 3.4,
>> because result of "g()" is not a generator or Future --- it's debugging
>> CoroWrapper.
>> Outer asyncio.coroutine will not yield from CoroWrapper and will return
>> inner "g()" CoroWrapper.
>>
>> I made complete example demonstrating this issue:
>> <https://gist.github.com/rutsky/c72be2edeb1c8256d680>
>>
>> This issue is not reproduced in Python 3.5, since CoroWrapper is awaitable
>> in Python 3.5,
>> but still I can't find is this is a desired behaviour and why?
>>
>>
>> Regards,
>>
>> Vladimir Rutsky
>>
>
>
>
> --
> --Guido van Rossum (python.org/~guido)
