And it's on Python stdlib 3 https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.Pool
On Sat, 26 Jun 2021, 17:24 Thomas Grainger, <tagr...@gmail.com> wrote: > billiard a multiprocessing py2 fork/backport has > https://billiard.readthedocs.io/en/latest/library/multiprocessing.html#multiprocessing.pool.multiprocessing.Pool > with maxtasksperchild > > On Sat, 26 Jun 2021, 16:57 Ram Rachum, <r...@rachum.com> wrote: > >> Hi guys, >> >> I want to have a version of `concurrent.futures.ProcessPoolExecutor` in >> which every worker process shuts down after each task. I want this because >> I believe I have a memory leak in my program that I wasn't able to fix. >> (Possibly it's in C extensions that I use.) >> >> Since I'm going to use a process pool anyway, I'm hoping to sidestep this >> memory leak by launching every task on a worker process that shuts down >> after it's done. >> >> A few years back when I was working at Dell, they did something similar. >> They had a Python server that had a memory leak, and they worked around it >> by having a process that can be easily restarted, and then they restarted >> it often. >> >> The plan is that each new work item will run on a fresh process that has >> no memory leaks from previous work items. >> >> I looked at the `futures/process.py` file to see whether I could subclass >> `ProcessPoolExecutor` and add this functionality, and boy oh boy this is >> not easy. >> >> Do you think it's worthwhile to add this feature to `ProcessPoolExecutor`? >> >> >> Thanks, >> Ram. >> _______________________________________________ >> Python-ideas mailing list -- python-ideas@python.org >> To unsubscribe send an email to python-ideas-le...@python.org >> https://mail.python.org/mailman3/lists/python-ideas.python.org/ >> Message archived at >> https://mail.python.org/archives/list/python-ideas@python.org/message/UNDY7YKBJ4DOB6H45N7T6QK7PVJBIKZV/ >> Code of Conduct: http://python.org/psf/codeofconduct/ >> >
_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/4UT4F7KHLGS2YV2UQAZZKCCOONP6VNGQ/ Code of Conduct: http://python.org/psf/codeofconduct/
