[issue42762] infinite loop resulted by "yield"

[Xinmeng Xia]

Xinmeng Xia  added the comment:

I see,Thank you!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42762] infinite loop resulted by "yield"

[Steve Stagg]

Steve Stagg  added the comment:

I don't believe this is a bug.

You've discovered a nasty corner-case, but I think it's expected behaviour.  
There is a PEP open to make this behaviour a bit nicer:  
https://www.python.org/dev/peps/pep-0533/

The fact that older Python 3.5/6 versions don't get the infinite loop could be 
considered a bug, but this was fixed a while ago (bpo-25612), and it seems 
unlikely that this fix would be back-ported.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42762] infinite loop resulted by "yield"

[Xinmeng Xia]

Xinmeng Xia  added the comment:

I get a little confused. So is it a bug in Python 3.5 and 3.6?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42762] infinite loop resulted by "yield"

[Steve Stagg]

Steve Stagg  added the comment:

(sorry for spam!)

So, this is a retained reference issue.
If I change the script to be this:


---
import gc

DEPTH = 100

def foo():
global DEPTH
try:
yield
except BaseException as e:
DEPTH -= 1
if DEPTH < 1:
return
gc.collect()
yield from foo()


def x():
for m in foo():
print(i)

try:
x()
except:
pass


ges = [o for o in gc.get_objects() if isinstance(o, GeneratorExit)]
if ges:
ge, = ges
print(gc.get_referrers(ge))
---

Then there's a reference to the GeneratorExit being retained (I guess from the 
exception frames, althought I thought exception frames were cleared up these 
days?):

[[GeneratorExit()], {'__name__': '__main__', '__doc__': None, '__package__': 
None, '__loader__': <_frozen_importlib_external.SourceFileLoader object at 
0x7fac8899ceb0>, '__spec__': None, '__annotations__': {}, '__builtins__': 
, '__file__': '/home/sstagg/tmp/fuzztest/tx.py', 
'__cached__': None, 'gc': , 'DEPTH': 99, 'foo': 
, 'x': , 'ges': 
[GeneratorExit()], 'ge': GeneratorExit()}, ]
Exception ignored in: 
RuntimeError: generator ignored GeneratorExit.

Given the infinite loop happens during the finalization of the generator, I 
think this reference is stopping the loop from going forever.

I tried removing the "as e" from the above script, and no references are 
retained.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42762] infinite loop resulted by "yield"

[Steve Stagg]

Steve Stagg  added the comment:

That /is/ weird.  I tried a few variations, and it looks like something is 
being stored on the exception that is stopping the loop behaviour.

"except BaseException:"  <- infinite loop
"except BaseException as e:" <- NO loop
"except BaseException as e:
   del e
   yield from foo()" <- infinite loop

So is this a reference being retained somewhere?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42762] infinite loop resulted by "yield"

[Raymond Hettinger]

Raymond Hettinger  added the comment:

Even weirder, the old behavior returns if "except:" is replaced by "except 
BaseException as e:".


def foo():
try:
yield
except BaseException as e:
yield from foo()

for m in foo():
print(i)

--
nosy: +Mark.Shannon, rhettinger

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42762] infinite loop resulted by "yield"

[Steve Stagg]

Steve Stagg  added the comment:

Ok, so I now understand a bit more, and think it's not a bug!  But explaining 
it involves some fairly deep thinking about generators.

I'll try to explain my reasoning.

Let's take a simple example:

---

def foo():
try:
yield
except:
print("ERROR")

for x in foo():
print(1)
---


It's convenient to think of it as python adding an implicit throw 
StopIteration() at the end of the generator function, I'll also rewrite the 
code to be roughly equivalent:

---
 1. def foo():
 2. try:
 3. yield None
 4. except:
 5. print("ERROR")
 6. #throw StopIteration():
 7. 
 8. foo_gen = foo()
 9. while True:
10. try:
11. x = next(foo_gen)
12.except StopIteration:
13. break
14.print(1)
15. del foo_gen
---

Now, if we step through how python runs the code starting at line 8.:

8. Create foo() <- this just creates a generator object
9. Enter while loop (not interesting for now)
10. try:
11. call next() on our generator to get the next value
2. try: <- we're running the generator until we get a yield now
3. yield None <- yield None to parent code
11. x = None <- assign yielded value to x
12. except StopIteration <- no exception raised so this doesn't get triggered
14. print(1) <- Print 1 to the output
9. Reached end of while loop, return to the start
10. try:
11. Re-enter the generator to get the next value, starting from where we left 
it..
4. except: <- there was no exception raised, so this doesn't get triggered
6. (implicit) throw StopIteration because generator finished
12. `except StopIteration` <- this is triggered because generator threw 
StopIteration
13. break <- break out of while loop
15. remove foo_gen variable, and clean up generator.

<- Deleting `foo_gen` causes `.close()` to be called on the generator which 
causes a GeneratorExit exception to be raised in the generator, BUT generator 
has already finished, so the GeneratorExit does nothing.

--

This is basically how the for-loop in your example works, and you can see that 
there's no generator exception, BUT if we change the print(1) to print(i) and 
try again:

8. Create foo() <- this just creates a generator object
9. Enter while loop (not interesting for now)
10. try:
11. call next() on our generator to get the next value
2. try: <- we're running the generator until we get a yield now
3. yield None <- yield None to parent code
11. x = None <- assign yielded value to x
12. except StopIteration <- no exception raised so this doesn't get triggered
*** CHANGED BIT ***
14. print(i) <- i doesn't exist, so throw NameError
14. The exception made us exit the current stack frame, so start cleaning 
up/deleting local variables

<- Deleting `foo_gen` causes `.close()` to be called on the generator which 
causes GeneratorExit() to be raised within the generator, but the generator is 
currently paused on line 3. so raise exception as-if we're currently running 
line 3:

4. except: <- this broad except catches the GeneratorExit exception because it 
appears to have happened on line 3.
5. print("ERROR") <- We only get here if the above steps happened.

---

So, if you don't let a generator naturally finish itself, but stop consuming 
the generator before it's raised its final StopIteration, then when the 
variable goes out-of-scope, a GeneratorExit will be raised at the point of the 
last yield that it ran.

If you then catch that GeneratorExit, and enter a new un-consumed loop (as in 
your `yield from foo()` line), then that line will also create the same 
situation again in a loop..

I understand that this used to "work" in previous python versions, but 
actually, having dug into things a lot more, I think the current behaviour is 
correct, and previous behaviors were not correct.

The "bug" here is in the example code that is catching GeneratorExit and then 
creating a new generator in the except:, rather than anything in Python

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42762] infinite loop resulted by "yield"

[Steve Stagg]

Steve Stagg  added the comment:

I'm sorry, I did get a bit confused earlier, I'd mentally switched to context 
managers.

I agree this is a bug, and a kinda weird one!

I've narrowed it down to this:

If an exception causes flow to exit a for-loop that's powered by a generator, 
then when the generator object is deleted, GeneratorExit() is incorrectly 
raised in the generator.

This can be shown with the following example (easier to debug without the 
infinite loop):

---
def foo():
try:
yield
except:
print("!!! WE SHOULDN'T BE HERE!!!")

x = foo()
try:
for _ in x:
   print(i)
except NameError:
pass 

print("LOOP DONE")
del x   # <--- We shouldn't be here printed on this line.
print("FINAL")
---

As you discovered, if you change print(i) to print(1), then the "shouldn't be 
here" line is NOT printed, but if you leave it as print(i) then the exception 
is printed.

You can see that the error doesn't happen until after LOOP DONE, which is 
because `del x` is finalizing the generator object, and the invalid exception 
logic happens then.

I'm trying to get more info here, if I don't by the time you come online, I'd 
recommend creating a *new* issue, with the non-loop example above, and 
explanation because I think on this issue, I've caused a lot of noise (sorry 
again!).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42762] infinite loop resulted by "yield"

[Xinmeng Xia]

Xinmeng Xia  added the comment:

Thanks for your kind explanation! My description is a little confusing. Sorry 
about that. The problem here is that the program should stop when the exception 
is caught whatever the exception is. (I think bpo-25612 (#1773) fixed exception 
selection problems and right exception can be caught) The fact is this program 
will fall into an infinite loop. Error messages are printed in a dead loop. The 
program doesn't stop. This is not normal. There will be no loops in Python 3.5 
and 3.6 (Attached output in Python 3.5 and 3.6). 

 

Output on Python 3.5,3.6 (all errors are printed without any loop )
-
Exception ignored in: 
RuntimeError: generator ignored GeneratorExit
Traceback (most recent call last):
  File "/home/xxm/Desktop/nameChanging/report/test1.py", line 248, in 
print(i)
NameError: name 'i' is not defined
Exception ignored in: 
RuntimeError: generator ignored GeneratorExit
Exception ignored in: 
RuntimeError: generator ignored GeneratorExit


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42762] infinite loop resulted by "yield"

[Steve Stagg]

Steve Stagg  added the comment:

Behaviour was changed in this commit:
---
commit ae3087c6382011c47db82fea4d05f8bbf514265d
Author: Mark Shannon 
Date:   Sun Oct 22 22:41:51 2017 +0100

Move exc state to generator. Fixes bpo-25612 (#1773)

Move exception state information from frame objects to coroutine 
(generator/thread) object where it belongs.

---

I'm honestly not sure that the older behaviour was 'better' than current.  But 
I don't know the defined behaviours well enough to be certain

--
nosy: +stestagg

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42762] infinite loop resulted by "yield"

[Xinmeng Xia]

New submission from Xinmeng Xia :

Let's see the following program:


def foo():
try:
yield
except:
yield from foo()

for m in foo():
print(i)
===

Expected output:
On line"print(i)",  NameError: name 'i' is not defined


However, the program will fall into infinite loops when running it on Python 
3.7-3.10 with the error messages like the following.(no infinite loop on Python 
3.5 and Python 3.6)
--
Traceback (most recent call last):
  File "/home/xxm/Desktop/nameChanging/report/test1.py", line 160, in 
print(i)
RuntimeError: generator ignored GeneratorExit
Exception ignored in: 
Traceback (most recent call last):
  File "/home/xxm/Desktop/nameChanging/report/test1.py", line 160, in 
print(i)
RuntimeError: generator ignored GeneratorExit
Exception ignored in: 
Traceback (most recent call last):
  File "/home/xxm/Desktop/nameChanging/report/test1.py", line 160, in 
print(i)
RuntimeError: generator ignored GeneratorExit
Exception ignored in: 
Traceback (most recent call last):
  File "/home/xxm/Desktop/nameChanging/report/test1.py", line 160, in 
print(i)
..
--

--
components: Interpreter Core
messages: 383882
nosy: xxm
priority: normal
severity: normal
status: open
title: infinite loop resulted by "yield"
type: crash
versions: Python 3.10, Python 3.7, Python 3.8, Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com