[issue44733] Feature request: maxtasksperchild for ProcessPoolExecutor

[Logan Jones]

Logan Jones  added the comment:

Ok I now have a PR up with the features requested. Let me know if you need 
anything else!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44733] Feature request: maxtasksperchild for ProcessPoolExecutor

[Logan Jones]

Change by Logan Jones :


--
pull_requests: +30264
stage: needs patch -> patch review
pull_request: https://github.com/python/cpython/pull/32187

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44733] Feature request: maxtasksperchild for ProcessPoolExecutor

[Gregory P. Smith]

Gregory P. Smith  added the comment:

yep, branch off of a recent main.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44733] Feature request: maxtasksperchild for ProcessPoolExecutor

[Logan Jones]

Logan Jones  added the comment:

Okay, I'm actually able to work on this again. What is the best way to make 
this change real. Should I be working off of main?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44733] Feature request: maxtasksperchild for ProcessPoolExecutor

[Gregory P. Smith]

Gregory P. Smith  added the comment:

As this is a new feature, it would also be reasonable to have specifying 
max_tasks_per_child without explicitly specifying a mp_context default to a 
safe mp_context.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44733] Feature request: maxtasksperchild for ProcessPoolExecutor

[Gregory P. Smith]

Gregory P. Smith  added the comment:

from discussion on the other bug it looks like we should have a way to keep 
this; we just need to not allow it when the mp_context to be used is the 'fork' 
one.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44733] Feature request: maxtasksperchild for ProcessPoolExecutor

[Logan Jones]

Logan Jones  added the comment:

Based on my reading I’m hopeful that this change can make it in quickly once I 
find the time. The previous implementation didn’t care how the processes were 
created. 

I will look as soon as I can

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44733] Feature request: maxtasksperchild for ProcessPoolExecutor

[Ram Rachum]

Ram Rachum  added the comment:

Oh that sucks. Logan and Antoine worked on this feature for so long. Thanks for 
reporting Greg.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44733] Feature request: maxtasksperchild for ProcessPoolExecutor

[Gregory P. Smith]

Change by Gregory P. Smith :


--
assignee:  -> gregory.p.smith
resolution: fixed -> 

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44733] Feature request: maxtasksperchild for ProcessPoolExecutor

[Gregory P. Smith]

Gregory P. Smith  added the comment:

If we want this feature to stay in, it needs an implementation that does not 
rely on a multithreaded processing calling fork(). I'm going to have to revert 
the existing implementation for bpo-46464.

--
stage: resolved -> needs patch
status: closed -> open

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44733] Feature request: maxtasksperchild for ProcessPoolExecutor

[Gregory P. Smith]

Gregory P. Smith  added the comment:

[bpo-46464](https://bugs.python.org/issue46464) describes a deadlock that the 
pre-requisite for this feature causes.

Spawning new children directly via fork() is impossible once a thread in the 
parent process has been started and concurrent.futures.process starts a thread.

--
nosy: +gregory.p.smith

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44733] Feature request: maxtasksperchild for ProcessPoolExecutor

[Antoine Pitrou]

Change by Antoine Pitrou :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44733] Feature request: maxtasksperchild for ProcessPoolExecutor

[Antoine Pitrou]

Antoine Pitrou  added the comment:


New changeset fdc0e09c3316098b038996c428e88931f0a4fcdb by Logan Jones in branch 
'main':
bpo-44733: Add max_tasks_per_child to ProcessPoolExecutor (GH-27373)
https://github.com/python/cpython/commit/fdc0e09c3316098b038996c428e88931f0a4fcdb


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44733] Feature request: maxtasksperchild for ProcessPoolExecutor

[Logan Jones]

Change by Logan Jones :


--
keywords: +patch
pull_requests: +25911
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/27373

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44733] Feature request: maxtasksperchild for ProcessPoolExecutor

[Ram Rachum]

Ram Rachum  added the comment:

Awesome. You can link to experimental code here. Even if it were not accepted 
to Python, it could be useful for me and for other people who might see this 
issue.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44733] Feature request: maxtasksperchild for ProcessPoolExecutor

[Logan Jones]

Logan Jones  added the comment:

I think I have a solution for this, but I'm pretty new to contributing. Still 
writing up some tests.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44733] Feature request: maxtasksperchild for ProcessPoolExecutor

[Logan Jones]

Change by Logan Jones :


--
nosy: +loganasherjones

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44733] Feature request: maxtasksperchild for ProcessPoolExecutor

[Ram Rachum]

New submission from Ram Rachum :

I love `concurrent.futures`, and I'd like to use it wherever I can. There's a 
feature in `multiprocessing.Pool` that I wish would also be available in 
`ProcessPoolExecutor`: The `maxtasksperchild` argument.

Documentation: "maxtasksperchild is the number of tasks a worker process can 
complete before it will exit and be replaced with a fresh worker process, to 
enable unused resources to be freed. The default maxtasksperchild is None, 
which means worker processes will live as long as the pool."

I want to be able to set it to 1, so each process will only execute one task 
and then be replaced with a fresh process.

--
components: Library (Lib)
messages: 398143
nosy: cool-RR, pitrou
priority: normal
severity: normal
status: open
title: Feature request: maxtasksperchild for ProcessPoolExecutor
type: enhancement
versions: Python 3.11

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com