[issue23353] generator bug with exception: tstate->exc_value is not cleared after an except block

[STINNER Victor]

STINNER Victor added the comment:

Thanks Antoine!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23353] generator bug with exception: tstate->exc_value is not cleared after an except block

[Antoine Pitrou]

Changes by Antoine Pitrou :


--
resolution:  -> fixed
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23353] generator bug with exception: tstate->exc_value is not cleared after an except block

[Roundup Robot]

Roundup Robot added the comment:

New changeset ac43268da908 by Antoine Pitrou in branch '3.4':
Issue #23353: improve exceptions tests for generators
https://hg.python.org/cpython/rev/ac43268da908

New changeset b29342f53174 by Antoine Pitrou in branch 'default':
Issue #23353: improve exceptions tests for generators
https://hg.python.org/cpython/rev/b29342f53174

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23353] generator bug with exception: tstate->exc_value is not cleared after an except block

[STINNER Victor]

STINNER Victor added the comment:

@Antoine: exctests.patch looks good to me, can you commit it?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23353] generator bug with exception: tstate->exc_value is not cleared after an except block

[Arfrever Frehtes Taifersar Arahesis]

Changes by Arfrever Frehtes Taifersar Arahesis :


--
nosy: +Arfrever

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23353] generator bug with exception: tstate->exc_value is not cleared after an except block

[Roundup Robot]

Roundup Robot added the comment:

New changeset 2cd6621a9fbc by Victor Stinner in branch '3.4':
Issue #23353, asyncio: Workaround CPython bug #23353
https://hg.python.org/cpython/rev/2cd6621a9fbc

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23353] generator bug with exception: tstate->exc_value is not cleared after an except block

[STINNER Victor]

STINNER Victor added the comment:

By the way: buildbots are green again, cool.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23353] generator bug with exception: tstate->exc_value is not cleared after an except block

[STINNER Victor]

STINNER Victor added the comment:

> It would have been nice to wait for a review. Generator tests are already in 
> test_exceptions.py.

Sorry, I wanted to quickly push your fix to fix buildbots. I dislike being the 
responsible of turning all buildbots to red...

Before working on this issue, I didn't know test_generators. Well, I didn't 
know test_exceptions neither :-) test_exceptions.py sounds like a better name 
for checks on the currently handled exception :-)

I saw that test_generators.py is mostly written with doctests. At the 
beginning, doctests were shiny and fun. Now I consider that it's worse than 
classic unit tests and I plan to rewrite doctests to unittest.TestCase classes. 
I will open a new issue for that.

> I think your patch for 2.7 is wrong as was your patch for 3.x. You shouldn't 
> change the behaviour of sys.exc_info() in the nominal case.

I now agree that gen_exc_value.patch was wrong. gen_exc_value_py27.patch was 
just a backport of my patch to Python 2.7.

(Oh I see that I uploaded gen_exc_value_py27.patch twice, it's a mistake.)

In my previous message, I asked myself if it would be possible to backport your 
patch (gen_exc_state_restore.patch) to Python 2.7.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23353] generator bug with exception: tstate->exc_value is not cleared after an except block

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

If you need non-normalized exception for testing, a NameError generated by 
interpreter is not normalized.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23353] generator bug with exception: tstate->exc_value is not cleared after an except block

[Antoine Pitrou]

Antoine Pitrou added the comment:

Additional simple tests for test_exceptions.py.

--
Added file: http://bugs.python.org/file37943/exctests.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23353] generator bug with exception: tstate->exc_value is not cleared after an except block

[Antoine Pitrou]

Antoine Pitrou added the comment:

It would have been nice to wait for a review. Generator tests are already in 
test_exceptions.py.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23353] generator bug with exception: tstate->exc_value is not cleared after an except block

[Antoine Pitrou]

Antoine Pitrou added the comment:

I think your patch for 2.7 is wrong as was your patch for 3.x. You shouldn't 
change the behaviour of sys.exc_info() in the nominal case.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23353] generator bug with exception: tstate->exc_value is not cleared after an except block

[STINNER Victor]

STINNER Victor added the comment:

I agree that it's better to not change the behaviour of generators, backward 
compatibility matters :-)

I wrote tests using my examples and I combined them with  
gen_exc_state_restore.patch. I commited the changeset in Python 3.4 and 3.5.


Backporting the fix to Python 2.7 looks more complex because the EXCEPT_HANDLE 
try block type and the POP_EXCEPT instruction are new in Python 3.0: introduced 
by 212a1fee6bf9 from the issue #3021.

What do you think? Is it worth to fix this issue in Python 2.7?

I plan to workaround this bug in Tulip to support Python 3.3. I will also 
workaround it in Trollius to support Python 2.6 and newer. So for me, it's ok 
to live with this known bug.

It's just yet another generator bug. asyncio/trollius already work around a 
yield-from bug (issue #21209) ;-)

> Note the patch also fixes the reference leak in test_asyncio.

Yes, as I explained in msg235072, this bug caused strange "memory leaks".

--
Added file: http://bugs.python.org/file37937/gen_exc_value_py27.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23353] generator bug with exception: tstate->exc_value is not cleared after an except block

[Roundup Robot]

Roundup Robot added the comment:

New changeset 4555da8c3091 by Victor Stinner in branch '3.4':
Issue #23353: Fix the exception handling of generators in PyEval_EvalFrameEx().
https://hg.python.org/cpython/rev/4555da8c3091

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23353] generator bug with exception: tstate->exc_value is not cleared after an except block

[Antoine Pitrou]

Antoine Pitrou added the comment:

Note the patch also fixes the reference leak in test_asyncio.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23353] generator bug with exception: tstate->exc_value is not cleared after an except block

[Antoine Pitrou]

Antoine Pitrou added the comment:

Attached patch fixes the test script and doesn't break any test.

--
Added file: http://bugs.python.org/file37933/gen_exc_state_restore.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23353] generator bug with exception: tstate->exc_value is not cleared after an except block

[STINNER Victor]

STINNER Victor added the comment:

> > Currently, a generator inherits the currently handled exception from
> > the "caller"
>
> This is expected, since this is how normal functions behave.

Do you see how to fix the issue without changing the behaviour?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23353] generator bug with exception: tstate->exc_value is not cleared after an except block

[STINNER Victor]

STINNER Victor added the comment:

Attached gen_exc_value_py27.patch: Patch for Python 2.7. No unit test yet.

The full test suite of trollius pass on the patched Python 2.7 and on the 
patched Python 3.5. The full test suite of asyncio also pass on the patched 
Python 3.5.

--
Added file: http://bugs.python.org/file37932/gen_exc_value_py27.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23353] generator bug with exception: tstate->exc_value is not cleared after an except block

[Antoine Pitrou]

Antoine Pitrou added the comment:

> Currently, a generator inherits the currently handled exception from
> the "caller" 

This is expected, since this is how normal functions behave.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23353] generator bug with exception: tstate->exc_value is not cleared after an except block

[STINNER Victor]

STINNER Victor added the comment:

Oh, by the way: keeping the exception after the except block is also a tricky 
reference leak. In Python 3, since exceptions store their traceback, this issue 
may keep a lot of objects alive too long, whereas they are expected to be 
destroyed much earlier.

When I started to investigate this issue, it took me 2 hours to begin to 
understand why so many objects were kept alive. It looks like a reference 
cycle, a reference leak, or other kind of complex memory leak. Clearing 
manually local variables (ex: "self = None" in methods) is not enough.

Python 2 has a sys.exc_clear() method which can be used to workaround this 
issue. It cannot be used in Python 3 since the function was removed in Python 3.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23353] generator bug with exception: tstate->exc_value is not cleared after an except block

[STINNER Victor]

STINNER Victor added the comment:

See also:

* PEP 3134: Exception Chaining and Embedded Tracebacks (Python 3.0)
* Issue #3021: Lexical exception handlers (Python 3.0) -- thread: 
https://mail.python.org/pipermail/python-3000/2008-May/013740.html
* PEP 380: Syntax for Delegating to a Subgenerator (Python 3.3)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23353] generator bug with exception: tstate->exc_value is not cleared after an except block

[STINNER Victor]

Changes by STINNER Victor :


--
nosy: +pitrou, serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23353] generator bug with exception: tstate->exc_value is not cleared after an except block

[STINNER Victor]

STINNER Victor added the comment:

Attached gen_exc_value.patch changes how generators handle the currently 
handled exception (tstate->exc_value). The patch probably lacks tests to test 
the exact behaviour of sys.exc_info(). The 3 examples below can be used to 
write such tests. But before investing time on an implemenation, I would like 
to ensure that I correctly understood the bug and discuss how it should be 
fixed.

Currently, a generator inherits the currently handled exception from the 
"caller" (function calling next(), gen.send() or gen.throw()). With the patch, 
the generator and its caller don't share the exception anymore. The generator 
still remembers the current exception when it is interrupted by yield.

It's still possible to pass the exception from the caller to the generator 
using the throw() method of the generator. My patch doesn't change the 
behaviour of throw(): see the example 3 below.

The patch also fixes the initial issue: "./python -m test test_asyncio 
test_functools" doesn't fail anymore.

Python 2.7 is also affected by the bug. I don't know yet if it's safe to change 
the behaviour of currently handled exception in Python 2 generators. It may 
break backward compatibility. We should probably check applications which 
heavily depends on generators. For example, the Trollius and Twisted projects 
use generators for coroutines in asynchronous programming.

The backward compatibility also matters in Python 3.4, so same question: should 
we change the behaviour of Python 3.4? Can it break applications?

In Python 3, the currently handled exception is more important than in Python 
2, because it is used to automatically fill the __context__ attribute of raised 
exceptions.

I didn't test the behaviour of yield-from yet, I don't know how my patch 
changes its behaviour.


Example 1:
---
import sys

def gen():
print(sys.exc_info())
yield

g = gen()
try:
raise ValueError
except Exception:
next(g)
---

Original output:
---
(, ValueError(), )
---

With the patch:
---
(None, None, None)
---


Example 2:
---
import sys

def gen():
try:
yield
raise TypeError()
except:
print(sys.exc_info())
print(sys.exc_info())
yield

g = gen()
next(g)
try:
raise ValueError
except Exception:
next(g)
---

Original output:
---
(, TypeError(), )
(, ValueError(), )
---

With the patch:
---
(, TypeError(), )
(None, None, None)
---


Example 3:
---
import sys

def gen():
try:
try:
yield
except:
print(sys.exc_info())
raise TypeError()
except Exception as exc:
print("TypeError context:", repr(exc.__context__))
yield

g = gen()
next(g)
try:
raise ValueError
except Exception as exc:
g.throw(exc)
---

Original output:
---
(, ValueError(), )
TypeError context: ValueError()
(, ValueError(), )
---

With the patch (unchanged):
---
(, ValueError(), )
TypeError context: ValueError()
(None, None, None)
---

--
keywords: +patch
versions: +Python 2.7
Added file: http://bugs.python.org/file37931/gen_exc_value.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23353] generator bug with exception: tstate->exc_value is not cleared after an except block

[STINNER Victor]

STINNER Victor added the comment:

Last major change related to generators in Python/ceval.c:
---
changeset:   47594:212a1fee6bf9
parent:  47585:b0ef00187a7e
user:Benjamin Peterson 
date:Wed Jun 11 15:59:43 2008 +
files:   Doc/library/dis.rst Doc/library/inspect.rst Doc/library/sys.rst 
Doc/reference/datamodel.rst Include/frameobject.h Include/opcode.h 
Lib/doctest.py Li
description:
#3021: Antoine Pitrou's Lexical exception handlers
---

This change introduced SWAP_EXC_STATE() and SAVE_EXC_STATE().

--
title: gnerator bug with exception: tstate->exc_value is not cleared after an 
except block -> generator bug with exception: tstate->exc_value is not cleared 
after an except block

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com