New issue 3154: Multiprocessing leaking file descriptors
https://bitbucket.org/pypy/pypy/issues/3154/multiprocessing-leaking-file-descriptors
Andrew Robinson:
Have migrated a multiprocessing project from PyPy 6.0 \(2.7\) to PyPy 7.3
\(3.6\) and have identified an issue with multiprocessing leaking descriptors.
I believe this may also relate to the issue reported as #3060.
I have identified this issue as repeatable on both Ubuntu 18.04.2 and CentOS
7.0.
Simple code to reproduce issue \(Taken from #3111 which is raised a different
issue, but current issue is reproduced with the same code sample\):
```
import multiprocessing
def process_chunk(chunk_index):
return ([],[],[])
def main():
i = 0
with multiprocessing.Pool(5, maxtasksperchild=1) as p:
for sub_headers, sub_results, sub_scores in
p.imap_unordered(process_chunk, range(5000)):
print(i)
i += 1
if __name__ == '__main__':
main()
```
If saving the above as “test.py” the descriptor leak can be verified by running
“/path/to/pypy3 test.py” then taking the PID from ps and doing an “ls -l
/proc/PID/fd” and comparing to the same when running native Python Ie. Running
“/path/to/python3 test.py” taking PID and “ls -l /proc/PID/fd”.
Under native Python3 the descriptor list remains constant at 9 pipe entries.
Under PyPy the file descriptor list keeps growing until ulimit is reached and
it will throw an exception as “OSError: \[Errno 24\] Too many open files“. In
my case ulimit is set to 1024.
Traceback below:
```
Exception in thread Thread-1:
Traceback (most recent call last):
File "/opt/callation/pypy3/lib-python/3/threading.py", line 916, in
_bootstrap_inner
self.run()
File "/opt/callation/pypy3/lib-python/3/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/opt/callation/pypy3/lib-python/3/multiprocessing/pool.py", line 405,
in _handle_workers
pool._maintain_pool()
File "/opt/callation/pypy3/lib-python/3/multiprocessing/pool.py", line 246,
in _maintain_pool
self._repopulate_pool()
File "/opt/callation/pypy3/lib-python/3/multiprocessing/pool.py", line 239,
in _repopulate_pool
w.start()
File "/opt/callation/pypy3/lib-python/3/multiprocessing/process.py", line
105, in start
self._popen = self._Popen(self)
File "/opt/callation/pypy3/lib-python/3/multiprocessing/context.py", line
277, in _Popen
return Popen(process_obj)
File "/opt/callation/pypy3/lib-python/3/multiprocessing/popen_fork.py", line
19, in __init__
self._launch(process_obj)
File "/opt/callation/pypy3/lib-python/3/multiprocessing/popen_fork.py", line
65, in _launch
parent_r, child_w = os.pipe()
OSError: [Errno 24] Too many open files
```
_______________________________________________
pypy-issue mailing list
pypy-issue@python.org
https://mail.python.org/mailman/listinfo/pypy-issue
