[issue23996] _PyGen_FetchStopIterationValue() crashes on unnormalised exceptions

Serhiy Storchaka added the comment: I think that's all with this issue. -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker

[issue23996] _PyGen_FetchStopIterationValue() crashes on unnormalised exceptions

Roundup Robot added the comment: New changeset bce18f5c0bc4 by Serhiy Storchaka in branch '3.5': Issue #23996: Added _PyGen_SetStopIterationValue for safe raising https://hg.python.org/cpython/rev/bce18f5c0bc4 New changeset a2c9f06ada28 by Serhiy Storchaka in branch '3.6': Issue #23996: Added

[issue23996] _PyGen_FetchStopIterationValue() crashes on unnormalised exceptions

Changes by Guido van Rossum : -- nosy: -gvanrossum ___ Python tracker ___ ___

[issue23996] _PyGen_FetchStopIterationValue() crashes on unnormalised exceptions

Serhiy Storchaka added the comment: Added comments. -- Added file: http://bugs.python.org/file45355/gen_set_stopiteration_value_2.patch ___ Python tracker

[issue23996] _PyGen_FetchStopIterationValue() crashes on unnormalised exceptions

Serhiy Storchaka added the comment: Yet one special case -- if asynchronous iterator in aiter_wrapper is an instance of StopIteration. Proposed patch adds the function _PyGen_SetStopIterationValue() that raises StopIteration with correctly wrapped value (exception is normalized only if

[issue23996] _PyGen_FetchStopIterationValue() crashes on unnormalised exceptions

Yury Selivanov added the comment: > No, this cannot be tested from the Python level. Stefan, could you please upload a C program that showcases the bug you're trying to fix? -- ___ Python tracker

[issue23996] _PyGen_FetchStopIterationValue() crashes on unnormalised exceptions

Changes by Serhiy Storchaka : Added file: http://bugs.python.org/file45347/test_stopiteration_tuple_value.patch ___ Python tracker

[issue23996] _PyGen_FetchStopIterationValue() crashes on unnormalised exceptions

Yury Selivanov added the comment: Serhiy, I think you forgot to attach the patch. aiter_wrapper shouldn't ever receive tuples, so it should be fine with PyErr_SetObject. -- ___ Python tracker

[issue23996] _PyGen_FetchStopIterationValue() crashes on unnormalised exceptions

Serhiy Storchaka added the comment: Here is a test that passed with current code but will fail with the patch. I don't know whether it make much sense. If yes, then perhaps aiter_wrapper_iternext needs the same workaround as other invocations of PyErr_SetObject(PyExc_StopIteration, ...).

[issue23996] _PyGen_FetchStopIterationValue() crashes on unnormalised exceptions

Yury Selivanov added the comment: > Looks like I forgot about this. My final fix still hasn't been applied, so > the code in Py3.4+ is incorrect now. Left a question in code review -- ___ Python tracker

[issue23996] _PyGen_FetchStopIterationValue() crashes on unnormalised exceptions

Changes by Serhiy Storchaka : -- type: crash -> behavior versions: +Python 3.7 ___ Python tracker ___

[issue23996] _PyGen_FetchStopIterationValue() crashes on unnormalised exceptions

Stefan Behnel added the comment: Looks like I forgot about this. My final fix still hasn't been applied, so the code in Py3.4+ is incorrect now. No, this cannot be tested from the Python level. -- ___ Python tracker

[issue23996] _PyGen_FetchStopIterationValue() crashes on unnormalised exceptions

Guido van Rossum added the comment: Please try to make sure this is fixed before 3.5 rc 1. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23996 ___

[issue23996] _PyGen_FetchStopIterationValue() crashes on unnormalised exceptions

Serhiy Storchaka added the comment: Could you provide tests covering all branches (normalized exception, unnormalized exception, absent value, non-tuple value, empty tuple value, non-empty tuple value...) Stefan? -- nosy: +serhiy.storchaka stage: patch review - test needed

[issue23996] _PyGen_FetchStopIterationValue() crashes on unnormalised exceptions

Stefan Behnel added the comment: The fix wasn't applied yet, so the current code in 3.4 and later branches is still incorrect. Any of the last two patches (*_value) will fix it, with my preference on the last one. -- versions: +Python 3.6 ___

[issue23996] _PyGen_FetchStopIterationValue() crashes on unnormalised exceptions

Changes by Antoine Pitrou pit...@free.fr: -- nosy: +giampaolo.rodola, gvanrossum, haypo ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23996 ___

[issue23996] _PyGen_FetchStopIterationValue() crashes on unnormalised exceptions

Yury Selivanov added the comment: Any of the last two patches (*_value) will fix it, with my preference on the last one. Stefan, the last patch looks good to me. Do you think we can have a unittest for this? -- ___ Python tracker

[issue23996] _PyGen_FetchStopIterationValue() crashes on unnormalised exceptions

Stefan Behnel added the comment: Regarding tests, it looks like iteration isn't currently tested at the C level at all. At least, the xx test modules don't have any types that use it. I can write one up next week, or add it to one of the existing types (Xxo_Type?). Unlikely that I'll make the

[issue23996] _PyGen_FetchStopIterationValue() crashes on unnormalised exceptions

Serhiy Storchaka added the comment: Is it possible to test from Python level? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23996 ___ ___

[issue23996] _PyGen_FetchStopIterationValue() crashes on unnormalised exceptions

Changes by Yury Selivanov yseliva...@gmail.com: -- nosy: +yselivanov resolution: fixed - stage: resolved - patch review ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23996 ___

[issue23996] _PyGen_FetchStopIterationValue() crashes on unnormalised exceptions

Antoine Pitrou added the comment: Have you tried benchmarking the slow solution? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23996 ___ ___

[issue23996] _PyGen_FetchStopIterationValue() crashes on unnormalised exceptions

Changes by Stefan Behnel sco...@users.sourceforge.net: Added file: http://bugs.python.org/file39692/fix_stopiteration_value.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23996 ___

[issue23996] _PyGen_FetchStopIterationValue() crashes on unnormalised exceptions

Stefan Behnel added the comment: Here are two patches that fix this case, one with special casing, one without. Please choose and apply one. -- Added file: http://bugs.python.org/file39691/fix_stopiteration_value_slow.patch ___ Python tracker

[issue23996] _PyGen_FetchStopIterationValue() crashes on unnormalised exceptions

Stefan Behnel added the comment: I noticed that my patch isn't entirely correct. If the exception value is a tuple, both PyErr_SetObject() and PyErr_NormalizeException() use it directly as *argument tuple* for the exception instantiation call, i.e. they essentially unpack it into separate

[issue23996] _PyGen_FetchStopIterationValue() crashes on unnormalised exceptions

Roundup Robot added the comment: New changeset 15c80f63ea1c by Antoine Pitrou in branch '3.4': Issue #23996: Avoid a crash when a delegated generator raises an unnormalized StopIteration exception. Patch by Stefan Behnel. https://hg.python.org/cpython/rev/15c80f63ea1c New changeset

[issue23996] _PyGen_FetchStopIterationValue() crashes on unnormalised exceptions

Antoine Pitrou added the comment: Thanks for the patch! -- resolution: - fixed stage: - resolved status: open - closed versions: -Python 3.3 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23996

[issue23996] _PyGen_FetchStopIterationValue() crashes on unnormalised exceptions

Changes by Stefan Behnel sco...@users.sourceforge.net: -- nosy: +pitrou ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23996 ___ ___

[issue23996] _PyGen_FetchStopIterationValue() crashes on unnormalised exceptions

Changes by Stefan Behnel sco...@users.sourceforge.net: -- nosy: +ncoghlan ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23996 ___ ___

[issue23996] _PyGen_FetchStopIterationValue() crashes on unnormalised exceptions

Stefan Behnel added the comment: And in fact, fixing it in ceval.c would not be enough, since gen_throw() also calls the function. So this is really the right place to fix it. -- ___ Python tracker rep...@bugs.python.org

[issue23996] _PyGen_FetchStopIterationValue() crashes on unnormalised exceptions

Stefan Behnel added the comment: Here's a better patch that avoids exception normalisation in all normal cases. -- Added file: http://bugs.python.org/file39116/fix_stopiteration_crash.patch ___ Python tracker rep...@bugs.python.org

[issue23996] _PyGen_FetchStopIterationValue() crashes on unnormalised exceptions

Stefan Behnel added the comment: And another patch update that should avoid any potential performance regressions due to the additional type check. -- Added file: http://bugs.python.org/file39119/fix_stopiteration_crash.patch ___ Python tracker

[issue23996] _PyGen_FetchStopIterationValue() crashes on unnormalised exceptions

New submission from Stefan Behnel: The yield-from implementation calls _PyGen_FetchStopIterationValue() to get the exception value. If the StopIteration exception is not normalised, e.g. because it was set by PyErr_SetObject() in a C extension, then _PyGen_FetchStopIterationValue() will cast