New issue 3111: Multiprocessing occasionally hangs
https://bitbucket.org/pypy/pypy/issues/3111/multiprocessing-occasionally-hangs
Aaron Wise:
I’ve had an issue with some multiprocessed code occasionally failing, and have
managed to work it to a \(fairly\) minimal example \(below\). Running this
code with pypy3.6-7.2 on linux \(ubuntu 18.04\) causes the loop to hang
occasionally \(about 1 in 10 runs\). When I ctrl\+c, I get the following error
message \(interrupted by the KeyboardInterrupt\):
> Traceback \(most recent call last\):
> File "/path/to/pypy3.6-7.2/lib-python/3/multiprocessing/pool.py", line 720,
> in next
> item = self.\_items.popleft\(\)
> IndexError: pop from an empty deque
As far as I can tell, if I do not define maxtasksperchild=1 \(and thus reuse
processes\) the error does not occur. \(But it may occur very rarely, I’ve
only tested it about 1000 times.\)
```python
import multiprocessing
def process_chunk(chunk_index):
return ([],[],[])
def main():
i = 0
with multiprocessing.Pool(5, maxtasksperchild=1) as p:
for sub_headers, sub_results, sub_scores in
p.imap_unordered(process_chunk, range(5000)):
print(i)
i += 1
if __name__ == '__main__':
main()
```
_______________________________________________
pypy-issue mailing list
pypy-issue@python.org
https://mail.python.org/mailman/listinfo/pypy-issue
