[issue34410] Segfault/TimeoutError: itertools.tee of multiprocessing.pool.imap_unordered

Carlo Rosati added the comment: If what you've said is correct, would it make the most sense to create a Manager method which returns a Proxy to a tee'd iterator? -- ___ Python tracker

[issue34410] Segfault/TimeoutError: itertools.tee of multiprocessing.pool.imap_unordered

Xiang Zhang added the comment: Ahh, the infinite iterator could also releases GIL in `PyIter_Next`. -- ___ Python tracker ___ ___

[issue34410] Segfault/TimeoutError: itertools.tee of multiprocessing.pool.imap_unordered

Xiang Zhang added the comment: It seems to me the problem is tee objects might encounter race conditions while `PyIter_Next` in `teedataobject_getitem` releases GIL. Other threads then might get into the same branch since `tdo->numread` haven't been updated yet. NULL slots are generated

[issue34410] Segfault/TimeoutError: itertools.tee of multiprocessing.pool.imap_unordered

Change by Xiang Zhang : -- nosy: +xiang.zhang ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue34410] Segfault/TimeoutError: itertools.tee of multiprocessing.pool.imap_unordered

Carlo Rosati added the comment: `for i in itertools.count()` in the first implementation I posted should be `while True`. I was using that for debugging. -- ___ Python tracker

[issue34410] Segfault/TimeoutError: itertools.tee of multiprocessing.pool.imap_unordered

Carlo Rosati added the comment: I've actually written a few workarounds that should be considered a multiprocessing specific tee function. I need feedback/critique on these. Hopefully we can all agree on one solution that's the best. It is unfortunate that the multiprocessing manager does

[issue34410] Segfault/TimeoutError: itertools.tee of multiprocessing.pool.imap_unordered

Raymond Hettinger added the comment: Davin, is there anything itertools.tee() can do about this or is this a multiprocessing issue? -- nosy: +davin, rhettinger ___ Python tracker

[issue34410] Segfault/TimeoutError: itertools.tee of multiprocessing.pool.imap_unordered

Karthikeyan Singaravelan added the comment: Thanks for the script. I can reproduce this on master and Python 3.6 too. Sometimes the attached script causes timeout error. Running it under gdb gives me below : [New Thread 0x18ab of process 10682] [New Thread 0x1903 of process 10682] [New

[issue34410] Segfault/TimeoutError: itertools.tee of multiprocessing.pool.imap_unordered

Carlo Rosati added the comment: You'll also need to lock when modifying the manager's list. Does anyone know how to do this using the multiprocessing.Queues without deadlocking? -- ___ Python tracker

[issue34410] Segfault/TimeoutError: itertools.tee of multiprocessing.pool.imap_unordered

Carlo Rosati added the comment: Okay I needed to do .pop(0) instead of .pop(-1) which is probably O(N) -- ___ Python tracker ___

[issue34410] Segfault/TimeoutError: itertools.tee of multiprocessing.pool.imap_unordered

Carlo Rosati added the comment: I figured out that the problem is itertools.tee does not use a multiprocessing.Manager proxied object for shared state. I was able to create a workaround tee as follows. def multiprocessing_tee(iterable, n=2): """Write a multiprocessing safe

[issue34410] Segfault/TimeoutError: itertools.tee of multiprocessing.pool.imap_unordered

New submission from Carlo Rosati : Hello, When I run the attached code, I encounter a segmentation fault. Thanks, Carlo -- files: 3.py messages: 323546 nosy: carlorosati priority: normal severity: normal status: open title: Segfault/TimeoutError: itertools.tee of