[issue33041] Issues with "async for"

[STINNER Victor]

STINNER Victor  added the comment:

I didn't check if the new warning, fixed by my PR-6595 in the master branch, 
exists on Windows in 3.6 and 3.7. If it does, you might want to request a 
backport of my PR. Usually, I only fix compiler warnings in the master branch.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33041] Issues with "async for"

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 078c4e3519deeef8014541925da057bb064eb5a8 by Victor Stinner in 
branch 'master':
bpo-33041: Fix downcast warning on Windows (#6595)
https://github.com/python/cpython/commit/078c4e3519deeef8014541925da057bb064eb5a8


--
nosy: +vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33041] Issues with "async for"

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +6293

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33041] Issues with "async for"

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33041] Issues with "async for"

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:


New changeset 18d7edf32e6832a818621cb8cb3d144928eca232 by Serhiy Storchaka in 
branch '3.6':
[3.6] bpo-33041: Fixed jumping if the function contains an "async for" loop. 
(GH-6154). (GH-6199)
https://github.com/python/cpython/commit/18d7edf32e6832a818621cb8cb3d144928eca232


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33041] Issues with "async for"

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
pull_requests: +5946

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33041] Issues with "async for"

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:


New changeset b9744e924ca07ba7db977e5958b91cd8db565632 by Serhiy Storchaka in 
branch '3.7':
bpo-33041: Fixed jumping if the function contains an "async for" loop. (GH-6154)
https://github.com/python/cpython/commit/b9744e924ca07ba7db977e5958b91cd8db565632


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33041] Issues with "async for"

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:


New changeset 702f8f3611bc49b73772cce2b9b041bd11ff9b35 by Serhiy Storchaka in 
branch 'master':
bpo-33041: Rework compiling an "async for" loop. (#6142)
https://github.com/python/cpython/commit/702f8f3611bc49b73772cce2b9b041bd11ff9b35


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33041] Issues with "async for"

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

I'm surprised, but seems PR 6154 fixes the whole issue 3 in 3.7.

It keeps only one POP_BLOCK corresponding to SETUP_LOOP. It also make the 
generated bytecode a tiny bit more efficient (less jumps).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33041] Issues with "async for"

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
pull_requests: +5912

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33041] Issues with "async for"

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

PR 6142 fixes issues 2 and 4. It adds a new opcode END_ASYNC_FOR and therefore 
can't be backported. END_ASYNC_FOR combines a number of other opcodes and 
guaranties using the true StopAsyncIteration. The new opcode is neccessary also 
for detecting an "async for" loop when jump. Besides introducing the new opcode 
the code of an "async for" loop was changed in a way that allowed to disallow 
jumping into an "async for" loop (this is the only part that can be 
backported). The final bytecode is much simpler. The compiler has been cleaned 
up and its code is now much simpler too. I expect also a performance boost, but 
don't know how to benchmark this.

Perhaps only the half of issue 4 (disallowing jumps into an "async for" loop) 
can be solved in 3.7 and 3.6.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33041] Issues with "async for"

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
pull_requests: +5899

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33041] Issues with "async for"

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:


New changeset 193760f00aacb122698674ed15327dba412653a5 by Serhiy Storchaka in 
branch '3.6':
[3.6] bpo-33041: Add tests for jumps in/out of 'async with' blocks. (GH-6110). 
(GH-6141)
https://github.com/python/cpython/commit/193760f00aacb122698674ed15327dba412653a5


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33041] Issues with "async for"

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:


New changeset 773573e9ac654d4b5c6682e70360f75864acd80e by Serhiy Storchaka in 
branch '3.7':
[3.7] bpo-33041: Add tests for jumps in/out of 'async with' blocks. (GH-6110). 
(GH-6140)
https://github.com/python/cpython/commit/773573e9ac654d4b5c6682e70360f75864acd80e


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33041] Issues with "async for"

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
pull_requests: +5898

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33041] Issues with "async for"

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
pull_requests: +5897

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33041] Issues with "async for"

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:


New changeset bc300ce205f99acb1ef92c37de06dc76147e073b by Serhiy Storchaka in 
branch 'master':
bpo-33041: Add tests for jumps in/out of 'async with' blocks. (#6110)
https://github.com/python/cpython/commit/bc300ce205f99acb1ef92c37de06dc76147e073b


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33041] Issues with "async for"

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

PR 6110 adds tests for jumping in/out of 'async with' blocks. It isn't directly 
related to 'async for', but it adds helpers that will be used also in tests for 
'async for'.

I'm not sure this is a correct and good way of writing such tests.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33041] Issues with "async for"

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
pull_requests: +5872

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33041] Issues with "async for"

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
dependencies: +Fix jumping out of "with" block, cannot jump from a 'return' or 
'exception' trace event

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33041] Issues with "async for"

[miss-islington]

miss-islington  added the comment:


New changeset d0826340d96e0953793b86d0b8475d2f43a280b6 by Miss Islington (bot) 
in branch '3.6':
[3.7] bpo-33041: Add missed error checks when compile "async for" (GH-6053) 
(GH-6060)
https://github.com/python/cpython/commit/d0826340d96e0953793b86d0b8475d2f43a280b6


--
nosy: +miss-islington

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33041] Issues with "async for"

[miss-islington]

Change by miss-islington :


--
pull_requests: +5824

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33041] Issues with "async for"

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:


New changeset 9e94c0d3c78d1bc582c865240ed9353fe9689b2a by Serhiy Storchaka in 
branch '3.7':
[3.7] bpo-33041: Add missed error checks when compile "async for" (GH-6053) 
(GH-6060)
https://github.com/python/cpython/commit/9e94c0d3c78d1bc582c865240ed9353fe9689b2a


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33041] Issues with "async for"

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
pull_requests: +5822

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33041] Issues with "async for"

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:


New changeset 67ee07795bcd84b679c000780212d4d81a1490a3 by Serhiy Storchaka in 
branch 'master':
bpo-33041: Add missed error checks when compile "async for" (#6053)
https://github.com/python/cpython/commit/67ee07795bcd84b679c000780212d4d81a1490a3


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33041] Issues with "async for"

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:


New changeset 24d3201eb7f0b39a7eaf2a5b2a2ceca10ad1f8eb by Serhiy Storchaka in 
branch 'master':
bpo-33041: Fixed bytecode generation for "async for" with a complex target. 
(#6052)
https://github.com/python/cpython/commit/24d3201eb7f0b39a7eaf2a5b2a2ceca10ad1f8eb


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33041] Issues with "async for"

[Yury Selivanov]

Yury Selivanov  added the comment:

Thanks so much for looking into this, Serhiy!

> 2. StopAsyncIteration is dynamically looked up in globals. If set the global 
> StopAsyncIteration or delete it from builtins (for example at the shutdown 
> stage), this will break any "async for".

IIRC I adapted the approach from some other place in compile.c. Quick looking 
at it reveals that the `assert` statement is compiled in a similar way w.r.t. 
how AssertionError is looked up at runtime.  You might want to check if there 
are other places in compile.c that need to be fixed.

> PR 6052 fixes issue 1. I don't know what is the best place for tests. There 
> are two files with tests for "async for": test_coroutines.py and 
> test_asyncgen.py. I'm not sure that new tests use the simplest way for 
> testing this behavior. Could you please look at them Yury?

I think the new tests are fine.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33041] Issues with "async for"

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

PR 6052 fixes issue 1. I don't know what is the best place for tests. There are 
two files with tests for "async for": test_coroutines.py and test_asyncgen.py. 
I'm not sure that new tests use the simplest way for testing this behavior. 
Could you please look at them Yury?

PR 6053 fixes issue 4. Adds missed error checks, removes unused variables and 
removes generating redundant bytecode.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33041] Issues with "async for"

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
pull_requests: +5814

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33041] Issues with "async for"

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
keywords: +patch
pull_requests: +5813
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33041] Issues with "async for"

[Serhiy Storchaka]

New submission from Serhiy Storchaka :

There is a number of issues with "async for".

1. When assigning to the target raises StopAsyncIteration (in custom 
__setitem__, __setattr__ or __iter__) it will be silenced and will cause to 
stop iteration.

2. StopAsyncIteration is dynamically looked up in globals. If set the global 
StopAsyncIteration or delete it from builtins (for example at the shutdown 
stage), this will break any "async for".

3. The f_lineno setter doesn't handle jumping into or out of the "async for" 
block. Jumping into is not forbidden, and jumping out doesn't update the stack 
correctly. This can cause a crash or incorrect behavior (like iterating wrong 
loop).

4. The compiler doesn't check all errors when creating new blocks. Some blocks 
are not used. And the resulting bytecode is suboptimal.

I'll create a series of pull request for fixing all this issue. Some of them 
can be backported. Others require changes in bytecode or are too hard for 
implementing in 3.7 and earlier versions (the related code was changed in 
issue17611). Some of them depend on other PRs or other issues (like issue33026) 
and need to wait until their be merged.

--
assignee: serhiy.storchaka
components: Interpreter Core
messages: 313526
nosy: serhiy.storchaka, yselivanov
priority: normal
severity: normal
status: open
title: Issues with "async for"
type: crash
versions: Python 3.6, Python 3.7, Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com