billiard a multiprocessing py2 fork/backport has https://billiard.readthedocs.io/en/latest/library/multiprocessing.html#multiprocessing.pool.multiprocessing.Pool with maxtasksperchild
On Sat, 26 Jun 2021, 16:57 Ram Rachum, <r...@rachum.com> wrote: > Hi guys, > > I want to have a version of `concurrent.futures.ProcessPoolExecutor` in > which every worker process shuts down after each task. I want this because > I believe I have a memory leak in my program that I wasn't able to fix. > (Possibly it's in C extensions that I use.) > > Since I'm going to use a process pool anyway, I'm hoping to sidestep this > memory leak by launching every task on a worker process that shuts down > after it's done. > > A few years back when I was working at Dell, they did something similar. > They had a Python server that had a memory leak, and they worked around it > by having a process that can be easily restarted, and then they restarted > it often. > > The plan is that each new work item will run on a fresh process that has > no memory leaks from previous work items. > > I looked at the `futures/process.py` file to see whether I could subclass > `ProcessPoolExecutor` and add this functionality, and boy oh boy this is > not easy. > > Do you think it's worthwhile to add this feature to `ProcessPoolExecutor`? > > > Thanks, > Ram. > _______________________________________________ > Python-ideas mailing list -- python-ideas@python.org > To unsubscribe send an email to python-ideas-le...@python.org > https://mail.python.org/mailman3/lists/python-ideas.python.org/ > Message archived at > https://mail.python.org/archives/list/python-ideas@python.org/message/UNDY7YKBJ4DOB6H45N7T6QK7PVJBIKZV/ > Code of Conduct: http://python.org/psf/codeofconduct/ >
_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/VBMR73IJ52C7HCQ4HJJV7TVB4BEPCLAQ/ Code of Conduct: http://python.org/psf/codeofconduct/
