[issue27243] __aiter__ should return async iterator instead of awaitable

[Roundup Robot]

Roundup Robot added the comment:

New changeset a0d50aad7b02 by Yury Selivanov in branch '3.6':
Issue #27243: Change PendingDeprecationWarning -> DeprecationWarning.
https://hg.python.org/cpython/rev/a0d50aad7b02

New changeset 9550f0d22d27 by Yury Selivanov in branch 'default':
Merge 3.6 (issue #27243)
https://hg.python.org/cpython/rev/9550f0d22d27

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27243] __aiter__ should return async iterator instead of awaitable

[Yury Selivanov]

Yury Selivanov added the comment:

> I didn’t realize, sorry for the noise

Actually thanks for reporting this, Martin.  I didn't realize that 
sys.version_info was 3.5.1 in 3.5 branch.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27243] __aiter__ should return async iterator instead of awaitable

[Martin Panter]

Martin Panter added the comment:

I didn’t realize, sorry for the noise

--
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27243] __aiter__ should return async iterator instead of awaitable

[Yury Selivanov]

Yury Selivanov added the comment:

This is because sys.version_info is 3.5.1 (not 3.5.2 yet) in the "3.5" branch.  
Once 3.5.2 RC is tagged the warning will disappear.

https://github.com/python/cpython/blob/master/Lib/asyncio/streams.py#L693

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27243] __aiter__ should return async iterator instead of awaitable

[Martin Panter]

Martin Panter added the comment:

Test suite emits a new warning, and fails under python -Werror:

==
ERROR: test_readline (test.test_asyncio.test_pep492.StreamReaderTests)
--
Traceback (most recent call last):
  File 
"/media/disk/home/proj/python/cpython/Lib/test/test_asyncio/test_pep492.py", 
line 89, in test_readline
data = self.loop.run_until_complete(reader())
  File "/media/disk/home/proj/python/cpython/Lib/asyncio/base_events.py", line 
387, in run_until_complete
return future.result()
  File "/media/disk/home/proj/python/cpython/Lib/asyncio/futures.py", line 274, 
in result
raise self._exception
  File "/media/disk/home/proj/python/cpython/Lib/asyncio/tasks.py", line 239, 
in _step
result = coro.send(None)
  File 
"/media/disk/home/proj/python/cpython/Lib/test/test_asyncio/test_pep492.py", 
line 85, in reader
async for line in stream:
PendingDeprecationWarning: 'StreamReader' implements legacy __aiter__ protocol; 
__aiter__ should return an asynchronous iterator, not awaitable

--
nosy: +martin.panter
status: closed -> open

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27243] __aiter__ should return async iterator instead of awaitable

[Yury Selivanov]

Yury Selivanov added the comment:

I've also updated PEP 492: https://hg.python.org/peps/rev/fef4b9969b9d  

Please feel free to post to this issue if you think that I should have covered 
it differently or in more detail.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27243] __aiter__ should return async iterator instead of awaitable

[Yury Selivanov]

Yury Selivanov added the comment:

Thanks a lot, Nick!  I've merged the patch.

--
resolution:  -> fixed
stage:  -> resolved
status: open -> closed
type:  -> enhancement

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27243] __aiter__ should return async iterator instead of awaitable

[Roundup Robot]

Roundup Robot added the comment:

New changeset 93ad47d63b87 by Yury Selivanov in branch '3.5':
Issue #27243: Fix __aiter__ protocol
https://hg.python.org/cpython/rev/93ad47d63b87

New changeset 9ff95c30a38e by Yury Selivanov in branch 'default':
Merge 3.5 (issue #27243)
https://hg.python.org/cpython/rev/9ff95c30a38e

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27243] __aiter__ should return async iterator instead of awaitable

[Nick Coghlan]

Nick Coghlan added the comment:

+1 from me - my only comments were on the docs updates and one of the 
explanatory comments in the code.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27243] __aiter__ should return async iterator instead of awaitable

[Yury Selivanov]

Yury Selivanov added the comment:

Nick,

Please see the updated patch.  Do you think it's ready to be merged?  I want 
buildbots to have some time with it before RC.

--
Added file: http://bugs.python.org/file43307/fix_aiter3.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27243] __aiter__ should return async iterator instead of awaitable

[Nick Coghlan]

Nick Coghlan added the comment:

Yury's proposal sounds good to me - I'll have time to do a proper review 
tomorrow (at a quick glance, my one suggestion is to move the if statement 
outside the example decorator, so the decorator is defined differently based on 
the Python version, rather than checking the version when called)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27243] __aiter__ should return async iterator instead of awaitable

[Yury Selivanov]

Yury Selivanov added the comment:

Updated patch (fix_aiter2.patch)

--
Added file: http://bugs.python.org/file43268/fix_aiter2.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27243] __aiter__ should return async iterator instead of awaitable

[STINNER Victor]

Changes by STINNER Victor :


--
nosy:  -haypo

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27243] __aiter__ should return async iterator instead of awaitable

[Yury Selivanov]

Yury Selivanov added the comment:

> Can you guys get this solid and checked in before then?

Will do my best.  I'll update the patch soon with some code comments and docs.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27243] __aiter__ should return async iterator instead of awaitable

[Larry Hastings]

Larry Hastings added the comment:

Okay.

I'm hoping to not delay 3.5.2 RC1, and the schedule calls for me to tag the 
release Saturday afternoon.  Can you guys get this solid and checked in before 
then?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27243] __aiter__ should return async iterator instead of awaitable

[Guido van Rossum]

Guido van Rossum added the comment:

Unless Nick disagrees (or unless we can't figure out how to implement it) I
think Yury's proposal makes sense. So if Larry is asking for my fiat, this
is it.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27243] __aiter__ should return async iterator instead of awaitable

[Yury Selivanov]

Yury Selivanov added the comment:

It's a rather long & conservative process, but it will be easier for people to 
migrate their code.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27243] __aiter__ should return async iterator instead of awaitable

[Yury Selivanov]

Yury Selivanov added the comment:

> While I agree this needs to be fixed, one key piece of documentation needed 
> will be to cover how to write an __aiter__ method that does the right thing 
> on both 3.5.1 and 3.5.2+ (and also avoids the deprecation warning in the 
> latter case).

Oh, this is tough.

How about we do this:

1. In 3.5.2 we start to support new behaviour. We raise 
PendingDeprecationWarning.  We update documentation and PEP 492.  The 
recommended way to write 3.5 code is to keep returning awaitables from 
__aiter__.  We can add a snippet of code for a compatibility decorator to the 
docs.

2. In 3.6 we start to raise DeprecationWarning.  The recommended way to write 
code for 3.6 & 3.7 is to return async iterators from __aiter__.

3. In 3.7 we remove support of the old behaviour.

It's a rather long & conservative process, but it will be

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27243] __aiter__ should return async iterator instead of awaitable

[Larry Hastings]

Larry Hastings added the comment:

As RM my default position is naturally "don't change behavior in point 
releases".  I'm willing to be overruled by the BDFL, less so by anybody else.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27243] __aiter__ should return async iterator instead of awaitable

[Nick Coghlan]

Nick Coghlan added the comment:

Since the old behaviour is only deprecated with Yury's changes, rather than 
disallowed entirely, I think it makes sense to provide a 3.5.x version that 
also supports the new behaviour.

In addition to my docs comment above, the other thing I noticed in reviewing 
the patch is that the proposed tests don't currently check that we *don't* emit 
a deprecation warning for the newly permitted forward compatible cases that 
return an asynchronous iterator (rather than an awaitable) directly from 
__aiter__.

There are definitely some tests that exercise that path, so we could rely on 
the general principle of "the test suite shouldn't emit deprecation warnings", 
but we could also add some tests that specifically check that no warning is 
emitted in that case.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27243] __aiter__ should return async iterator instead of awaitable

[Guido van Rossum]

Guido van Rossum added the comment:

Would it be easier to handle for everyone if this did not vary between
3.5.0/1 and 3.5.2, and instead was an incompatibility in 3.6? (That would
still be allowed given 492's provisional status.)

--
nosy: +ned.deily

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27243] __aiter__ should return async iterator instead of awaitable

[Nick Coghlan]

Nick Coghlan added the comment:

While I agree this needs to be fixed, one key piece of documentation needed 
will be to cover how to write an __aiter__ method that does the right thing on 
both 3.5.1 and 3.5.2+ (and also avoids the deprecation warning in the latter 
case).

(I've also added Larry to the cc list as release manager)

--
nosy: +larry

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27243] __aiter__ should return async iterator instead of awaitable

[Yury Selivanov]

New submission from Yury Selivanov:

There is a small flaw in PEP 492 design -- __aiter__ should not return an 
awaitable object that resolves to an asynchronous iterator. It should return an 
asynchronous iterator directly.

Let me explain this by showing some examples.

I've discovered this while working on a new asynchronous generators PEP.  Let's 
pretend that we have them already: if we have a 'yield' expression in an 'async 
def' function, the function becomes an "asynchronous generator function":

   async def foo():
  await bar()
  yield 1
  await baz()
  yield 2

   # foo -- is an `asynchronous generator function`
   # foo() -- is an `asynchronous generator`

If we iterate through "foo()", it will await on "bar()", yield "1", await on 
"baz()", and yield "2":

   >>> async for el in foo():
   ... print(el)
   1
   2

If we decide to have a class with an __aiter__ that is an async generator, we'd 
write something like this:

   class Foo:
  async def __aiter__(self):
  await bar()
  yield 1
  await baz()
  yield 2

However, with the current PEP 492 design, the above code would be invalid!  The 
interpreter expects __aiter__ to return a coroutine, not an async generator.

I'm still working on the PEP for async generators, targeting CPython 3.6.  And 
once it is ready, it might still be rejected or deferred.  But in any case, 
this PEP 492 flaw has to be fixed now, in 3.5.2 (since PEP 492 is provisional).

The attached patch fixes the __aiter__ in a backwards compatible way:

1. ceval/GET_AITER opcode calls the __aiter__ method.

2. If the returned object has an '__anext__' method, GET_AITER silently wraps 
it in an awaitable, which is equivalent to the following coroutine:

async def wrapper(aiter_result):
return aiter_result

3. If the returned object does not have an '__anext__' method, a 
DeprecationWarning is raised.

--
assignee: yselivanov
components: Interpreter Core
files: fix_aiter.patch
keywords: patch
messages: 267544
nosy: brett.cannon, gvanrossum, haypo, lukasz.langa, ncoghlan, yselivanov
priority: release blocker
severity: normal
status: open
title: __aiter__ should return async iterator instead of awaitable
versions: Python 3.5, Python 3.6
Added file: http://bugs.python.org/file43264/fix_aiter.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com