[issue35238] Alleviate memory reservation of fork_exec in subprocess.Popen via forkserver

STINNER Victor added the comment: See bpo-34663 for posix_spawn() & vfork. -- ___ Python tracker ___ ___ Python-bugs-list mailing

[issue35238] Alleviate memory reservation of fork_exec in subprocess.Popen via forkserver

Oscar Esteban added the comment: Hi Victor, That would be great. However, we played a bit with an alternative implementation of posix_spawn (one I got from one related bpo), and it didn't seem to make any difference in terms of memory allocation. Then, we found out that posix_spawn uses

[issue35238] Alleviate memory reservation of fork_exec in subprocess.Popen via forkserver

STINNER Victor added the comment: > By the way you could open an issue so that subprocess uses posix_spawn() > where possible. FYI I'm working on an implementation of this ;-) -- nosy: +vstinner ___ Python tracker

[issue35238] Alleviate memory reservation of fork_exec in subprocess.Popen via forkserver

Antoine Pitrou added the comment: By the way you could open an issue so that subprocess uses posix_spawn() where possible. (or you could ask to reopen issue31814, which is basically that request but for a different reason than yours) -- ___

[issue35238] Alleviate memory reservation of fork_exec in subprocess.Popen via forkserver

Oscar Esteban added the comment: Thanks for your response. The idea would be to enable ``subprocess.Popen`` to use an existing fork server in its fork_exec. The rationale: I can start a pool of n workers very early in the execution flow. They will have ~350MB memory fingerprint in the

[issue35238] Alleviate memory reservation of fork_exec in subprocess.Popen via forkserver

Antoine Pitrou added the comment: At any rate, given the constraints you're working with (thousands of child processes, memory conservation issues), I suggest you abandon the idea of using multiprocessing and write your own subprocess-server instead. I would suggest doing so using asyncio,

[issue35238] Alleviate memory reservation of fork_exec in subprocess.Popen via forkserver

Antoine Pitrou added the comment: I'm not sure I understand the proposed solution. Do you mean you would replace this: Parent -> forkserver -> fork child then exec with: Parent -> forkserver -> posix_spawn child? -- nosy: +pitrou ___ Python

[issue35238] Alleviate memory reservation of fork_exec in subprocess.Popen via forkserver

Change by 4-launchpad-kalvdans-no-ip-org : -- nosy: +4-launchpad-kalvdans-no-ip-org ___ Python tracker ___ ___ Python-bugs-list

[issue35238] Alleviate memory reservation of fork_exec in subprocess.Popen via forkserver

New submission from Oscar Esteban : ## Context We are developers of nipype (https://github.com/nipype) which is a workflow engine for neuroimaging software. We are experiencing problems that gave rise to the addition of ``os.posix_spawn`` to Python 3.8, and particularly, this -