[issue29942] Stack overflow in itertools.chain.from_iterable.

[Thomas Wouters]

Changes by Thomas Wouters :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29942] Stack overflow in itertools.chain.from_iterable.

[Thomas Wouters]

Thomas Wouters added the comment:


New changeset d694a06206fc09b76b4507aacde5e69a248f434f by T. Wouters in branch 
'2.7':
bpo-29942: Fix the use of recursion in itertools.chain.from_iterable. (#913)
https://github.com/python/cpython/commit/d694a06206fc09b76b4507aacde5e69a248f434f


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29942] Stack overflow in itertools.chain.from_iterable.

[Thomas Wouters]

Thomas Wouters added the comment:


New changeset 9273dfe1800fc7241d69f4d523d748ebd35b3801 by T. Wouters in branch 
'3.5':
bpo-29942: Fix the use of recursion in itertools.chain.from_iterable. (#912)
https://github.com/python/cpython/commit/9273dfe1800fc7241d69f4d523d748ebd35b3801


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29942] Stack overflow in itertools.chain.from_iterable.

[Thomas Wouters]

Thomas Wouters added the comment:


New changeset 599bb181036f724629a515317f0f39520950d51c by T. Wouters in branch 
'3.6':
bpo-29942: Fix the use of recursion in itertools.chain.from_iterable. (#911)
https://github.com/python/cpython/commit/599bb181036f724629a515317f0f39520950d51c


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29942] Stack overflow in itertools.chain.from_iterable.

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Possible workaround: use chain.from_iterable(filter(None, iterables)) instead 
of chain.from_iterable(iterables). But this works only when iterables are 
collections, not iterators.

--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29942] Stack overflow in itertools.chain.from_iterable.

[Thomas Wouters]

Changes by Thomas Wouters :


--
pull_requests: +811

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29942] Stack overflow in itertools.chain.from_iterable.

[Thomas Wouters]

Changes by Thomas Wouters :


--
pull_requests: +812

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29942] Stack overflow in itertools.chain.from_iterable.

[Thomas Wouters]

Changes by Thomas Wouters :


--
pull_requests: +813

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29942] Stack overflow in itertools.chain.from_iterable.

[Thomas Wouters]

Thomas Wouters added the comment:

FWIW, we ran into this in real-world cases (Youtube, I think), when we switched 
from using a pre-built Python interpreter to one built from source using the 
same optimisation and debug levels as we use for all other C/C++ code. Even so, 
the accompanying test really does fail in pydebug mode ;-P

I'll backport to 3.6, 3.5 and 2.7.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29942] Stack overflow in itertools.chain.from_iterable.

[Thomas Wouters]

Thomas Wouters added the comment:


New changeset 5466d4af5fe76ec0a5fbc8a05675287d9e8e9d14 by T. Wouters in branch 
'master':
bpo-29942: Fix the use of recursion in itertools.chain.from_iterable. (#889)
https://github.com/python/cpython/commit/5466d4af5fe76ec0a5fbc8a05675287d9e8e9d14


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29942] Stack overflow in itertools.chain.from_iterable.

[Mark Dickinson]

Mark Dickinson added the comment:

> I would have guessed that the C compiler would have automatically removed the 
> tail recursion

I think it probably does, unless optimisation is turned off: I'm unable to 
reproduce except in debug builds of Python.

--
nosy: +mark.dickinson

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29942] Stack overflow in itertools.chain.from_iterable.

[Raymond Hettinger]

Raymond Hettinger added the comment:

This looks fine.  Feel free to apply and to backport this to earlier versions.

I would have guessed that the C compiler would have automatically removed the 
tail recursion, but your experience would indicate otherwise.

--
assignee:  -> twouters

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29942] Stack overflow in itertools.chain.from_iterable.

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
nosy: +rhettinger
stage:  -> patch review
versions: +Python 2.7, Python 3.5, Python 3.6, Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29942] Stack overflow in itertools.chain.from_iterable.

[Thomas Wouters]

New submission from Thomas Wouters:

itertools.chain.from_iterable (somewhat ironically) uses recursion to resolve 
the next iterator, which means it can run out of the C stack when there's a 
long run of empty iterables. This is most obvious when building with low 
optimisation modes, or with Py_DEBUG enabled:

Python 3.7.0a0 (heads/master:c431854a09, Mar 29 2017, 10:03:50) 
[GCC 4.8.4] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import itertools
>>> next(itertools.chain.from_iterable(() for unused in range(1000)))
Segmentation fault (core dumped)

--
messages: 290787
nosy: gregory.p.smith, twouters
priority: normal
pull_requests: 791
severity: normal
status: open
title: Stack overflow in itertools.chain.from_iterable.

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29942] Stack overflow in itertools.chain.from_iterable.

[Thomas Wouters]

Changes by Thomas Wouters :


--
type:  -> crash

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com