Vladimir, I would appreciate it if you filed a 3.4 bug -- if you have a patch that would be even better! Thanks for reporting this.
On Thu, Nov 12, 2015 at 2:01 AM, Vladimir Rutsky <[email protected]> wrote: > On Mon, Nov 9, 2015 at 11:14 PM, Guido van Rossum <[email protected]> wrote: >> The intention of this code is to make it possible (and legal) to decorate a >> non-generator function with `@asyncio.coroutine`, and to make that >> effectively equivalent to the same decorated function with an unreachable >> yield inside it (which makes it a generator). IOW these two should be >> equivalent: >> >> @asyncio.coroutine >> def foo(): >> return 42 >> >> @asyncio.coroutine >> def foo(): >> return 42 >> yield # dummy to make it a coroutine >> >> You should be able to write in some other coroutine >> >> x = yield from foo() >> assert x == 42 >> >> regardless of which definition of foo() you've got. >> >> Note that for the first version the decorator always matters, while for the >> second version it only matters in debug mode. >> >> But it looks like the code has some subtle bug. :-( >> >> I think one of the problems is that we don't always test with debug mode. >> >> There are probably also some cases that the code is trying to support, e.g. >> for generators in general (outside the context of asyncio) there is usually >> no difference between these two examples: >> >> def foo(): >> yield 1 >> yield 2 >> def bar(): >> return foo() >> >> vs. >> >> def bar(): >> yield 1 >> yield 2 >> >> -- in either case, calling bar() returns a generator that when iterated over >> generates the sequence [1, 2]. But what you've discovered is that if we >> decorate both functions with @asyncio.coroutine, there *is* a difference. >> And I think the code is making a half-hearted attempt to paper over the >> difference, but failing due to debug mode. > > Thanks for the explanation. Reading asyncio documentation I thought about > asyncio-coroutines as some special kind functions (implemented using > generators) that can be run as coroutines by the event loop. > Almost always I develop asyncio-based programs with enabled > PYTHONASYNCIODEBUG, which in some cases doesn't allow to mix > usual functions and asyncio.coroutine-functions due to the discussed bug --- > this is one of the reason I thought that asyncio-coroutines can't be used as > simple generator functions. > > It would be nice if asyncio has better documentation for this case, or at > least link on a good tutorial. > > Other bug I experienced in Python 3.4 about different behaviour when > PYTHONASYNCIODEBUG is enabled is when wrapped with > asyncio.coroutine function doesn't have special attributes like > "__name__". In Python 3.4.3 with enabled asyncio debug a function is > wrapped using following code: > > @functools.wraps(func) > def wrapper(*args, **kwds): > w = CoroWrapper(coro(*args, **kwds), func) > if w._source_traceback: > del w._source_traceback[-1] > w.__name__ = func.__name__ > if hasattr(func, '__qualname__'): > w.__qualname__ = func.__qualname__ > w.__doc__ = func.__doc__ > return w > > note the unconditional access to "__name__" and "__doc__" attributes, > which function can not have in some conditions. > > As this use case looks strange and unrealistic I met him in a real > application that used mocking in tests. The code there was something like > following: > > def f(): > return "f result" > > mocked_f = Mock(wraps=f) > coro_func = asyncio.coroutine(mocked_f) > print(loop.run_until_complete(coro_func())) > mocked_f.assert_called_once_with() > > which failed only in the debug mode: > > $ python3 asyncio_debug_attr_access.py > f result > $ PYTHONASYNCIODEBUG=X python3 asyncio_debug_attr_access.py > Traceback (most recent call last): > File "asyncio_debug_attr_access.py", line 21, in <module> > print(loop.run_until_complete(coro_func())) > File "/usr/lib/python3.4/asyncio/coroutines.py", line 154, in wrapper > w.__name__ = func.__name__ > File "/usr/lib/python3.4/unittest/mock.py", line 570, in __getattr__ > raise AttributeError(name) > AttributeError: __name__ > Exception ignored in: $ > > Here is complete example: > https://gist.github.com/rutsky/65cee7728135b05d49c3 > > It can be bug in the unittest.mock library, that mock doesn't have special > attributes, but still it's unexpected to have such different behaviour in > debug > and not debug modes. > > This issue is not reproduced in Python 3.5, since it has more accurate > special attribute access: > > @functools.wraps(func) > def wrapper(*args, **kwds): > w = CoroWrapper(coro(*args, **kwds), func=func) > if w._source_traceback: > del w._source_traceback[-1] > # Python < 3.5 does not implement __qualname__ > # on generator objects, so we set it manually. > # We use getattr as some callables (such as > # functools.partial may lack __qualname__). > w.__name__ = getattr(func, '__name__', None) > w.__qualname__ = getattr(func, '__qualname__', None) > return w > > I workarounded both of the issues using monkeypatching in my project. > > Should I file bug reports against Python 3.4 about them on the Python bug > tracker? > > > Regards, > > Vladimir Rutsky > >> >> On Mon, Nov 9, 2015 at 10:25 AM, Vladimir Rutsky <[email protected]> >> wrote: >>> >>> Hello! >>> >>> Can anybody explain to me this part of asyncio.coroutine code: >>> >>> <https://github.com/python/asyncio/blob/3b6a64a9fb6ec4ad0c984532aa776b130067c901/asyncio/coroutines.py#L201> >>> >>> # asyncio/coroutines.py:201: >>> def coroutine(func): >>> """Decorator to mark coroutines. >>> If the coroutine is not yielded from before it is destroyed, >>> an error message is logged. >>> """ >>> if _inspect_iscoroutinefunction(func): >>> # In Python 3.5 that's all we need to do for coroutines >>> # defiend with "async def". >>> # Wrapping in CoroWrapper will happen via >>> # 'sys.set_coroutine_wrapper' function. >>> return func >>> >>> if inspect.isgeneratorfunction(func): >>> coro = func >>> else: >>> @functools.wraps(func) >>> def coro(*args, **kw): >>> res = func(*args, **kw) >>> if isinstance(res, futures.Future) or >>> inspect.isgenerator(res): # <--- This part >>> res = yield from res >>> ... >>> >>> What does this code: if wrapped by asyncio.coroutine function is not >>> generator and returns generator, Future or awaitable object, >>> then this coroutine function will yield from returned value. >>> >>> E.g. following coroutine function will actually return result of the >>> future, not future itself: >>> >>> @asyncio.coroutine >>> def return_future(): >>> fut = asyncio.Future() >>> fut.set_result("return_future") >>> >>> return fut >>> >>> I can't find where such behaviour is documented? Is this a desired >>> behavior? >>> >>> According to >>> <https://docs.python.org/3/library/asyncio-task.html?highlight=iscoroutine#coroutines>: >>> > Coroutines used with asyncio may be implemented using the async def >>> > statement, or by using generators. >>> > ... >>> > Generator-based coroutines should be decorated with @asyncio.coroutine, >>> > although this is not strictly enforced. >>> >>> So technically, according to this part of documentation, it's not >>> explicitly allowed to use asyncio.coroutine with non-generator functions. >>> >>> Special behaviour when asyncio.coroutine-wrapped non-generator function >>> returns generator or Future object >>> leads to inconsistent behaviour on Python 3.4 when asyncio debugging is >>> enabled and not enabled. >>> >>> Consider following function: >>> >>> @asyncio.coroutine >>> def return_coroutine_object(): >>> @asyncio.coroutine >>> def g(): >>> yield from asyncio.sleep(0.01) >>> return "return_coroutine_object" >>> >>> return g() >>> >>> When debugging is disabled following code will work: >>> >>> assert loop.run_until_complete(return_coroutine_object()) == >>> "return_coroutine_object" >>> >>> When debugging is enabled same code will not work on Python 3.4, >>> because result of "g()" is not a generator or Future --- it's debugging >>> CoroWrapper. >>> Outer asyncio.coroutine will not yield from CoroWrapper and will return >>> inner "g()" CoroWrapper. >>> >>> I made complete example demonstrating this issue: >>> <https://gist.github.com/rutsky/c72be2edeb1c8256d680> >>> >>> This issue is not reproduced in Python 3.5, since CoroWrapper is awaitable >>> in Python 3.5, >>> but still I can't find is this is a desired behaviour and why? >>> >>> >>> Regards, >>> >>> Vladimir Rutsky >>> >> >> >> >> -- >> --Guido van Rossum (python.org/~guido) -- --Guido van Rossum (python.org/~guido)
