[issue34410] itertools.tee not thread-safe; can segfault interpreter when wrapped iterator releases GIL

Serhiy Storchaka added the comment: It would be nice to add a multi-thread supporting version of tee() or add multi-thread support in tee(), but this is different issue. -- stage: patch review -> resolved status: pending -> closed ___ Python

[issue34410] itertools.tee not thread-safe; can segfault interpreter when wrapped iterator releases GIL

Irit Katriel added the comment: The script (3.py) now gives the RuntimeError (as of 3.9) so I think the discussion about back port is over and there is nothing more to do on this issue. If nobody objects I will close this. -- status: open -> pending

[issue34410] itertools.tee not thread-safe; can segfault interpreter when wrapped iterator releases GIL

Irit Katriel added the comment: This seems resolved, can it be closed? -- nosy: +iritkatriel ___ Python tracker ___ ___

[issue34410] itertools.tee not thread-safe; can segfault interpreter when wrapped iterator releases GIL

Serhiy Storchaka added the comment: New changeset 2fb6921ab296f933caf361a662e6471e143abefc by Serhiy Storchaka in branch '2.7': [2.7] bpo-34410: Fix a crash in the tee iterator when re-enter it. (GH-15625) (GH-15740)

[issue34410] itertools.tee not thread-safe; can segfault interpreter when wrapped iterator releases GIL

miss-islington added the comment: New changeset 5190b7193c184268d5c8a9440b3a5a8bcd84a23e by Miss Islington (bot) in branch '3.7': bpo-34410: Fix a crash in the tee iterator when re-enter it. (GH-15625) https://github.com/python/cpython/commit/5190b7193c184268d5c8a9440b3a5a8bcd84a23e

[issue34410] itertools.tee not thread-safe; can segfault interpreter when wrapped iterator releases GIL

miss-islington added the comment: New changeset 6e3809c7ce9fbee11c3a3f89dd7e89829b7581ac by Miss Islington (bot) in branch '3.8': bpo-34410: Fix a crash in the tee iterator when re-enter it. (GH-15625) https://github.com/python/cpython/commit/6e3809c7ce9fbee11c3a3f89dd7e89829b7581ac

[issue34410] itertools.tee not thread-safe; can segfault interpreter when wrapped iterator releases GIL

Change by Serhiy Storchaka : -- pull_requests: +15394 pull_request: https://github.com/python/cpython/pull/15740 ___ Python tracker ___

[issue34410] itertools.tee not thread-safe; can segfault interpreter when wrapped iterator releases GIL

Change by miss-islington : -- pull_requests: +15392 pull_request: https://github.com/python/cpython/pull/15737 ___ Python tracker ___

[issue34410] itertools.tee not thread-safe; can segfault interpreter when wrapped iterator releases GIL

Serhiy Storchaka added the comment: New changeset 526a01467b3277f9fcf7f91e66c23321caa1245d by Serhiy Storchaka in branch 'master': bpo-34410: Fix a crash in the tee iterator when re-enter it. (GH-15625) https://github.com/python/cpython/commit/526a01467b3277f9fcf7f91e66c23321caa1245d

[issue34410] itertools.tee not thread-safe; can segfault interpreter when wrapped iterator releases GIL

Change by miss-islington : -- pull_requests: +15393 pull_request: https://github.com/python/cpython/pull/15738 ___ Python tracker ___

[issue34410] itertools.tee not thread-safe; can segfault interpreter when wrapped iterator releases GIL

Change by Serhiy Storchaka : -- pull_requests: +15391 pull_request: https://github.com/python/cpython/pull/15736 ___ Python tracker ___

[issue34410] itertools.tee not thread-safe; can segfault interpreter when wrapped iterator releases GIL

Xiang Zhang added the comment: >It may be better to apply it even to the developed version. There is nothing >wrong with creating the tee iterator in one thread and using it the other >thread. Or using the tee iterators with external locking. I afraid that PR >15567 can break a legitimate

[issue34410] itertools.tee not thread-safe; can segfault interpreter when wrapped iterator releases GIL

hongweipeng added the comment: As far as I am concerned, I prefer that like PR 9254 can be merged into old versions. Because it does not break the legitimate code and can prevent the program from crashing. It can be used as a compatible solution with old versions. --

[issue34410] itertools.tee not thread-safe; can segfault interpreter when wrapped iterator releases GIL

Change by Serhiy Storchaka : -- pull_requests: +15293 stage: needs patch -> patch review pull_request: https://github.com/python/cpython/pull/15625 ___ Python tracker ___

[issue34410] itertools.tee not thread-safe; can segfault interpreter when wrapped iterator releases GIL

Raymond Hettinger added the comment: Go ahead and take this in any direction you want. -- ___ Python tracker ___ ___

[issue34410] itertools.tee not thread-safe; can segfault interpreter when wrapped iterator releases GIL

Serhiy Storchaka added the comment: The documentation changes should be backported. And I think we need a change like PR 9254, but with raising a RuntimeError instead of adding the value to the queue, to be applied in older versions. It may be better to apply it even to the developed

[issue34410] itertools.tee not thread-safe; can segfault interpreter when wrapped iterator releases GIL

Raymond Hettinger added the comment: I've applied the patch to "master". I considered backporting but am thinking that would be risky at this stage in the 3.8 release. -- resolution: -> fixed stage: patch review -> resolved status: open -> closed versions: +Python 3.9 -Python 2.7,

[issue34410] itertools.tee not thread-safe; can segfault interpreter when wrapped iterator releases GIL

Change by hongweipeng : -- pull_requests: +15243 pull_request: https://github.com/python/cpython/pull/15567 ___ Python tracker ___

[issue34410] itertools.tee not thread-safe; can segfault interpreter when wrapped iterator releases GIL

Raymond Hettinger added the comment: Thanks for enumerating the options. I think 1 and 2 are the best combination. It is a reasonable restriction to not tee across threads. If someone still does, then detecting it, raising an exception, and not crashing seem like a reasonable response.

[issue34410] itertools.tee not thread-safe; can segfault interpreter when wrapped iterator releases GIL

Serhiy Storchaka added the comment: I tried to solve this issue myself, and figured out that it is not so simple. It is possible to make tee() nor crashing, but it is not possible to guarantee the order of the output without using a lock in tee(). It you can get a sequence 1, 2, 4, 3, ...

[issue34410] itertools.tee not thread-safe; can segfault interpreter when wrapped iterator releases GIL

Change by hongweipeng : -- pull_requests: +8685 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue34410] itertools.tee not thread-safe; can segfault interpreter when wrapped iterator releases GIL

Serhiy Storchaka added the comment: I take this. -- assignee: -> serhiy.storchaka nosy: +serhiy.storchaka ___ Python tracker ___

[issue34410] itertools.tee not thread-safe; can segfault interpreter when wrapped iterator releases GIL

hongweipeng added the comment: Multi-process need uses multiprocessing.Manager to share, the current problem should be tee-objcet thread safety issue.As Xiang Zhang said,`PyIter_Next` in `teedataobject_getitem` releases GIL.So the thread lock is necessary,and only lead iterator uses it when

[issue34410] itertools.tee not thread-safe; can segfault interpreter when wrapped iterator releases GIL

Change by hongweipeng : -- keywords: +patch pull_requests: +8533 stage: needs patch -> patch review ___ Python tracker ___ ___

[issue34410] itertools.tee not thread-safe; can segfault interpreter when wrapped iterator releases GIL

Karthikeyan Singaravelan added the comment: This might be redundant but I googled 'itertools tee thread safe' and came across a detailed SO answer from 2017 that explains the issue along with a similar example that causes segfault in the compiler but it was not reported here it seems. It

[issue34410] itertools.tee not thread-safe; can segfault interpreter when wrapped iterator releases GIL

Xiang Zhang added the comment: I could. But currently I don't have a good idea how to fix it elegantly. If anyone else makes a PR, I'm willing to review it. -- stage: -> needs patch versions: +Python 3.8 ___ Python tracker

[issue34410] itertools.tee not thread-safe; can segfault interpreter when wrapped iterator releases GIL

Raymond Hettinger added the comment: Xiang Zhang, would you like to submit a patch? -- ___ Python tracker ___ ___ Python-bugs-list

[issue34410] itertools.tee not thread-safe; can segfault interpreter when wrapped iterator releases GIL

Josh Rosenberg added the comment: Carlo: The point of Xiang's post is that this is only tangentially related to multiprocessing; the real problem is that tee-ing an iterator implemented in Python (of which pool.imap_unordered is just one example) and using the resulting tee-ed iterators in