[issue8713] multiprocessing needs option to eschew fork() under Linux
Roundup Robot added the comment: New changeset b941a320601a by R David Murray in branch 'default': whatsnew: multiprocessing start methods and context (#8713 and #18999) http://hg.python.org/cpython/rev/b941a320601a -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8713] multiprocessing needs option to eschew fork() under Linux
Changes by Piotr Dobrogost p...@bugs.python.dobrogost.net: -- nosy: +piotr.dobrogost ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8713] multiprocessing needs option to eschew fork() under Linux
Roundup Robot added the comment: New changeset f6c7ad7d029a by Richard Oudkerk in branch 'default': Issue #8713: Test should not print message about start method. http://hg.python.org/cpython/rev/f6c7ad7d029a New changeset e99832a60e63 by Richard Oudkerk in branch 'default': Issue #8713: Cleanup before saving process._dangling. http://hg.python.org/cpython/rev/e99832a60e63 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8713] multiprocessing needs option to eschew fork() under Linux
Roundup Robot added the comment: New changeset 6d998a43102b by Richard Oudkerk in branch 'default': Issue #8713: Print dangling processes/threads, if any. http://hg.python.org/cpython/rev/6d998a43102b -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8713] multiprocessing needs option to eschew fork() under Linux
Richard Oudkerk added the comment: I've seen test_multiprocessing_forkserver giving warnings too, while running the whole test suite, but can't reproduce them while running it alone. The warnings seems quite similar though, so a single fix might resolve the problem with all the tests. The Using start method '...' should also be displayed only when the tests are run in verbose mode. Seems to be fixed now. -- status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8713] multiprocessing needs option to eschew fork() under Linux
Ezio Melotti added the comment: The new tests produce a few warnings: $ ./python -m test -uall test_multiprocessing_spawn [1/1] test_multiprocessing_spawn Using start method 'spawn' Warning -- threading._dangling was modified by test_multiprocessing_spawn Warning -- multiprocessing.process._dangling was modified by test_multiprocessing_spawn $ ./python -m test -uall -v -j2 test_multiprocessing_fork OK (skipped=4) Warning -- threading._dangling was modified by test_multiprocessing_fork Warning -- multiprocessing.process._dangling was modified by test_multiprocessing_fork 1 test altered the execution environment: test_multiprocessing_fork I've seen test_multiprocessing_forkserver giving warnings too, while running the whole test suite, but can't reproduce them while running it alone. The warnings seems quite similar though, so a single fix might resolve the problem with all the tests. The Using start method '...' should also be displayed only when the tests are run in verbose mode. -- nosy: +ezio.melotti ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8713] multiprocessing needs option to eschew fork() under Linux
Antoine Pitrou added the comment: Using the custom builders, it seems to happen randomly in test_rlock: test_rlock (test.test_multiprocessing_spawn.WithManagerTestLock) ... Assertion failed: !collecting, file ..\Modules\gcmodule.c, line 1617 ok http://buildbot.python.org/all/builders/AMD64%20Windows%20Server%202008%20%5BSB%5D%20custom -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8713] multiprocessing needs option to eschew fork() under Linux
Antoine Pitrou added the comment: Ok, I enabled faulthandler in the child process and I got the explanation: http://buildbot.python.org/all/builders/AMD64%20Windows%20Server%202008%20%5BSB%5D%20custom/builds/5/steps/test/logs/stdio multiprocessing's manager Server uses daemon threads... Daemon threads are not joined when the interpreter shuts down, they are simply frozen at some point. Unfortunately, it may happen that a deamon thread is frozen while it was doing a cyclic garbage collection, which later triggers the assert. I'm gonna replace the assert by a plain if, then. -- resolution: - fixed stage: needs patch - committed/rejected ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8713] multiprocessing needs option to eschew fork() under Linux
Antoine Pitrou added the comment: I have added documentation now so I think it is ready to merge (except for a change to Makefile). Good for me. This is a very nice addition! -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8713] multiprocessing needs option to eschew fork() under Linux
Roundup Robot added the comment: New changeset 3b82e0d83bf9 by Richard Oudkerk in branch 'default': Issue #8713: Support alternative start methods in multiprocessing on Unix. http://hg.python.org/cpython/rev/3b82e0d83bf9 -- nosy: +python-dev ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8713] multiprocessing needs option to eschew fork() under Linux
Richard Oudkerk added the comment: Good for me. This is a very nice addition! Thanks. I do see a couple of failed assertions on Windows which presumably happen in a child process because they do not cause a failure: Assertion failed: !collecting, file ..\Modules\gcmodule.c, line 1617 The assertion is in _PyGC_CollectNoFail() and checks that it is not called recursively. See http://buildbot.python.org/all/builders/AMD64%20Windows7%20SP1%203.x/builds/2510/steps/test/logs/stdio -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8713] multiprocessing needs option to eschew fork() under Linux
Antoine Pitrou added the comment: I do see a couple of failed assertions on Windows which presumably happen in a child process because they do not cause a failure: Assertion failed: !collecting, file ..\Modules\gcmodule.c, line 1617 The assertion is in _PyGC_CollectNoFail() and checks that it is not called recursively. See http://buildbot.python.org/all/builders/AMD64%20Windows7%20SP1%203.x/builds/2510/steps/test/logs/stdio That's extremely weird. _PyGC_CollectNoFail() is only called from PyImport_Cleanup, which itself is only called from Py_Finalize() and Py_EndInterpreter(). It should be basically impossible for the GC to be already collecting garbage at that point... Perhaps you could try to find out in which test this happens? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8713] multiprocessing needs option to eschew fork() under Linux
Changes by Richard Oudkerk shibt...@gmail.com: Added file: http://bugs.python.org/file31282/4fc7c72b1c5d.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8713] multiprocessing needs option to eschew fork() under Linux
Richard Oudkerk added the comment: I have added documentation now so I think it is ready to merge (except for a change to Makefile). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8713] multiprocessing needs option to eschew fork() under Linux
Changes by Richard Oudkerk shibt...@gmail.com: Added file: http://bugs.python.org/file31214/c7aa0005f231.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8713] multiprocessing needs option to eschew fork() under Linux
Richard Oudkerk added the comment: The forkserver process is now started using _posixsubprocess.fork_exec(). This should fix the order dependent problem mentioned before. Also the forkserver tests are now reenabled on OSX. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8713] multiprocessing needs option to eschew fork() under Linux
Ned Deily added the comment: Richard, can you say what failed on the OS X 10.4 (Tiger) buildbot? FWIW, I tested b3620777f54c.diff (and commented out the darwin skip of test_multiprocessing_forkserver) on OS X 10.4, 10.5, and 10.8. There were no failures on any of them. The only vaguely suspicious message when running with -v was: ./python -m test -v test_multiprocessing_forkserver [...] test_semaphore_tracker (test.test_multiprocessing_forkserver.TestSemaphoreTracker) ... [semaphore_tracker] '/mp18203-0': [Errno 22] Invalid argument [semaphore_tracker] '/mp18203-1': successfully unlinked ok [...] -- Ran 233 tests in 97.162s OK (skipped=5) # on 32-bit 'largest assignable fd number is too small' OK (skipped=4) # on 64-bit 1 test OK. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8713] multiprocessing needs option to eschew fork() under Linux
Richard Oudkerk added the comment: Richard, can you say what failed on the OS X 10.4 (Tiger) buildbot? There seems to be a problem which depends on the order in which you run the test, and it happens on Linux also. For example if I do ./python -m test -v \ test_multiprocessing_fork \ test_multiprocessing_forkserver Then I get lots of failures when forkserver runs. I have tracked down the changeset which caused the problem, but I have not had time to look in to it. The only vaguely suspicious message when running with -v was: [...] [semaphore_tracker] '/mp18203-0': [Errno 22] Invalid argument [semaphore_tracker] '/mp18203-1': successfully unlinked [...] That is expected and it shows the semaphore tracker is working as expected. Maybe I should print a note to stderr to expect this. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8713] multiprocessing needs option to eschew fork() under Linux
Changes by Richard Oudkerk shibt...@gmail.com: Added file: http://bugs.python.org/file31186/b3620777f54c.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8713] multiprocessing needs option to eschew fork() under Linux
Richard Oudkerk added the comment: I have done quite a bit of refactoring and added some extra tests. When I try using the forkserver start method on the OSX Tiger buildbot (the only OSX one available) I get errors. I have disabled the tests for OSX, but it seemed to be working before. Maybe that was with a different buildbot. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8713] multiprocessing needs option to eschew fork() under Linux
Changes by Antoine Pitrou pit...@free.fr: Added file: http://bugs.python.org/file31181/d9fe9757ba0c.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8713] multiprocessing needs option to eschew fork() under Linux
Richard Oudkerk added the comment: The spawn branch is in decent shape, although the documentation is not up-to-date. I would like to commit before the first alpha. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8713] multiprocessing needs option to eschew fork() under Linux
Changes by Stan Seibert s...@mtrr.org: -- nosy: +Stan.Seibert ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8713] multiprocessing needs option to eschew fork() under Linux
Richard Oudkerk added the comment: Richard, apart from performance, what's the advantage of this approach over the fork+exec version? It is really just performance. For context running the unittests in a 1 cpu linux VM gives me fork: real0m53.868s user0m1.496s sys 0m9.757s fork+exec: real1m30.951s user0m24.598s sys 0m25.614s forkserver: real0m54.087s user0m1.572s # excludes descendant processes sys 0m2.336s # excludes descendant processes So running the unit tests using fork+exec takes about 4 times as much cpu time. Starting then immediately joining a trivial process in a loop gives fork:0.025 seconds/process fork+exec: 0.245 seconds/process forkserver: 0.016 seconds/process So latency is about 10 times higher with fork+exec. Because it seems more complicated, and although I didn't have a look a this last patch, I guess that most of the fork+exec version could be factorized with the Windows version, no? The different fork methods are now implemented in separate files. The line counts are 117 popen_spawn_win32.py 80 popen_fork.py 184 popen_spawn_posix.py 191 popen_forkserver.py I don't think any more sharing between the win32 and posix cases is possible. (Note that popen_spawn_posix.py implements a cleanup helper process which is also used by the forkserver method.) Since it's only intented to be used as a debugging/special-purpose replacement - it would probably be better if it could be made as simple as possible. Actually, avoiding the whole fork+threads mess is a big motivation. multiprocessing uses threads in a few places (like implementing Queue), and tries to do so as safely as possible. But unless you turn off garbage collection you cannot really control what code might be running in a background thread when the main thread forks. Also, as you've noted, FD passing isn't supported by all Unices out there (and we've had some reliability issues on OS-X, too). OSX does not seem to allow passing multiple ancilliary messages at once -- but you can send multiple fds in a single ancilliary message. Also, when you send fds on OSX you have to wait for a response from the other end before doing anything else. Not doing that was the cause of the previous fd passing failures in test_multiprocessing. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8713] multiprocessing needs option to eschew fork() under Linux
Richard Oudkerk added the comment: Numbers when running on Linux on a laptop with 2 cores + hyperthreading. RUNNING UNITTESTS: fork: real0m50.687s user0m9.213s sys 0m4.012s fork+exec: real1m9.062s user0m48.579s sys 0m6.648s forkserver: real0m50.702s user0m4.140s# excluding descendants sys 0m0.708s# excluding descendants LATENCY: fork: 0.0071 secs/proc fork+exec: 0.0622 secs/proc forkserver: 0.0035 secs/proc Still 4 times the cpu time and 10 times the latency. But the latency is far lower than in the VM. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8713] multiprocessing needs option to eschew fork() under Linux
Changes by Ram Rachum r...@rachum.com: -- nosy: -cool-RR ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8713] multiprocessing needs option to eschew fork() under Linux
Gregory P. Smith added the comment: I think the forkserver approach is a good idea. It is what a lot of users will choose. forkserver won't work everywhere though so the fork+exec option is still desirable to have available. Threads can be started by non-python code (extension modules, or the larger C/C++ program that is embedding the Python interpreter within it). In that context, by the time the multiprocessing module is can be too late to start a fork server and there is no easy way for Python code to determine if that is the case. The safest default would be fork+exec though we need to implement the fork+exec code as a C extension module or have it use subprocess (as I noted in the mb_fork_exec.patch review). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8713] multiprocessing needs option to eschew fork() under Linux
Changes by Richard Oudkerk shibt...@gmail.com: Added file: http://bugs.python.org/file28461/8f08d83264a0.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8713] multiprocessing needs option to eschew fork() under Linux
Richard Oudkerk added the comment: The safest default would be fork+exec though we need to implement the fork+exec code as a C extension module or have it use subprocess (as I noted in the mb_fork_exec.patch review). That was an old version of the patch. In the branch http://hg.python.org/sandbox/sbt#spawn _posixsubprocess is used instead of fork+exec, and all unnecessary fds are closed. See http://hg.python.org/sandbox/sbt/file/8f08d83264a0/Lib/multiprocessing/popen_spawn_posix.py -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8713] multiprocessing needs option to eschew fork() under Linux
Gregory P. Smith added the comment: ah, i missed that update. cool! +1 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8713] multiprocessing needs option to eschew fork() under Linux
Charles-François Natali added the comment: Richard, apart from performance, what's the advantage of this approach over the fork+exec version? Because it seems more complicated, and although I didn't have a look a this last patch, I guess that most of the fork+exec version could be factorized with the Windows version, no? Since it's only intented to be used as a debugging/special-purpose replacement - it would probably be better if it could be made as simple as possible. Also, as you've noted, FD passing isn't supported by all Unices out there (and we've had some reliability issues on OS-X, too). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8713] multiprocessing needs option to eschew fork() under Linux
Changes by Gregory P. Smith g...@krypto.org: -- nosy: +gregory.p.smith ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8713] multiprocessing needs option to eschew fork() under Linux
Richard Oudkerk added the comment:
http://hg.python.org/sandbox/sbt#spawn now contains support for starting
processes via a separate server process. This depends on fd passing support.
This also solves the problem of mixing threads and processes, but is much
faster than using fork+exec. It seems to be just as fast as using plain fork.
I have tested it successfully on Linux and a MacOSX buildbot. (OpenSolaris
does not seem to support fd passing.)
At the begining of your program you write
multiprocessing.set_start_method('forkserver')
to use the fork server.
Alternatively you can use
multiprocessing.set_start_method('spawn')
to use _posixsubprocess.fork_exec() with closefds=True on Unix or
multiprocessing.set_start_method('fork')
to use the standard fork method.
This branch also stops child processes on Windows from automatically inheriting
inheritable handles.
The test suite can be run with the different start methods by doing
python -m test.test_multiprocessing_fork
python -m test.test_multiprocessing_spawn
python -m test.test_multiprocessing_forkserver
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8713
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8713] multiprocessing needs option to eschew fork() under Linux
Richard Oudkerk added the comment: For updated code see http://hg.python.org/sandbox/sbt#spawn This uses _posixsubprocess and closefds=True. -- hgrepos: +157 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8713] multiprocessing needs option to eschew fork() under Linux
Richard Oudkerk added the comment: A use case for not using fork() is when your parent process opens some system resources of some sort (for example a listening TCP socket). The child will then inherit those resources, which can have all kinds of unforeseen and troublesome consequences (for example that listening TCP socket will be left open in the child when it is closed in the parent, and so trying to bind() to the same port again will fail). Generally, I think having an option for zero-sharing spawning of processes would help code quality. The patch as it stands still depends on fd inheritance, so you would need to use FD_CLOEXEC on your listening socket. But yes, it should be possible to use the closefds feature of _posixsubprocess. BTW, I also have working code (which passes the unittests) that starts a helper process at the beginning of the program and which will fork processes on behalf of the other processes. This also solves the issue of unintended inheritance of resources (and the mixing of fork with threads) but is as fast as doing normal forks. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8713] multiprocessing needs option to eschew fork() under Linux
Antoine Pitrou added the comment: A use case for not using fork() is when your parent process opens some system resources of some sort (for example a listening TCP socket). The child will then inherit those resources, which can have all kinds of unforeseen and troublesome consequences (for example that listening TCP socket will be left open in the child when it is closed in the parent, and so trying to bind() to the same port again will fail). Generally, I think having an option for zero-sharing spawning of processes would help code quality. -- versions: +Python 3.4 -Python 3.3 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8713] multiprocessing needs option to eschew fork() under Linux
Antoine Pitrou added the comment: By the way, instead of doing fork() + exec() in pure Python, you probably want to use _posixsubprocess.fork_exec(). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8713] multiprocessing needs option to eschew fork() under Linux
Christian Heimes added the comment: +1 I still have to use parallel python (pp) in our application stack because the fork() approach causes a lot of strange issues in our application. It might be the punishment for embedding a Java runtime env into a Python process, too. :) -- nosy: +christian.heimes ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8713] multiprocessing needs option to eschew fork() under Linux
Changes by Catalin Iacob iacobcata...@gmail.com: -- nosy: +catalin.iacob ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8713] multiprocessing needs option to eschew fork() under Linux
sbt shibt...@gmail.com added the comment: Attached is an updated version of the mp_fork_exec.patch. This one is able to reliably clean up any unlinked semaphores if the program exits abnormally. -- Added file: http://bugs.python.org/file24297/mp_fork_exec.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8713] multiprocessing needs option to eschew fork() under Linux
sbt shibt...@gmail.com added the comment: mp_split_tests.patch splits up the test_multiprocessing.py: test_multiprocessing_misc.py miscellaneous tests which need not be run with multiple configurations mp_common.py testcases which should be run with multiple configurations test_multiprocessing_fork.py test_multiprocessing_nofork.py test_multiprocessing_manager_fork.py test_multiprocessing_manager_nofork.py test_multiprocessing_threads.py run the testcases in mp_common.py with various configurations -- Added file: http://bugs.python.org/file24298/mp_split_tests.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8713] multiprocessing needs option to eschew fork() under Linux
Changes by Santoso Wijaya santoso.wij...@gmail.com: -- nosy: -santa4nt ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8713] multiprocessing needs option to eschew fork() under Linux
Charles-François Natali neolo...@free.fr added the comment: I don't know what the others think, but I'm still -1 on this patch. Not that I don't like the principle - it's actually the contrary: in a perfect world, I think this should be made the default -and only - behavior on POSIX. But since it may break existing code, we'd have to keep both implementations for POSIX systems, with - at least to me - little benefit. Having three different implementations, with different codepaths, will increase the cognitive burden, make the code less readable, and debugging harder: - user: I'm getting this error with multiprocessing - dev: On Windows or on Unix? - user: On Unix - dev: Do you use the fork()+exec() version or the bare fork() version? - user: what's fork() and exec()? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8713] multiprocessing needs option to eschew fork() under Linux
Ned Deily n...@acm.org added the comment: See also consolidated Issue13558 for additional justification for processes option on OS X. -- nosy: +mrmekon, ned.deily ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8713] multiprocessing needs option to eschew fork() under Linux
Charles-François Natali neolo...@free.fr added the comment: Thanks for the patch sbt. I think this is indeed useful, but I'm tempted to go further and say we should make this the default - and only - behavior. This will probably break existing code that accidentaly relied the fact that the implementation uses a bare fork(), but i'd say it's worth it: - it's cleaner - it will make it possible to remove all the ad-hoc handlers called after fork() - it will remove the only place in the whole stdlib where fork() isn't followed by exec(): people who get bitten by issue #6721 will thus only be people calling explicitely fork(), in which case they're the sole responsibles for their misery ;-) Another - although less common - advantage over the current implementation is that now one can run out of memory pretty easily if the operating system doesn't do overcommitting if you work with a large dataset. If fork() is followed by an exec, no problem. Thoughts? -- nosy: +neologix, pitrou ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8713] multiprocessing needs option to eschew fork() under Linux
Antoine Pitrou pit...@free.fr added the comment: Thanks for the patch sbt. I think this is indeed useful, but I'm tempted to go further and say we should make this the default - and only - behavior. This will probably break existing code that accidentaly relied the fact that the implementation uses a bare fork(), There is probably lots of such code: - code that passes non-pickleable function object / function args to execute in the child process - code that executes code with side effects at module top level - code that relies (willingly or not) on other stuff such as fds being inherited -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8713] multiprocessing needs option to eschew fork() under Linux
sbt shibt...@gmail.com added the comment: I think this is indeed useful, but I'm tempted to go further and say we should make this the default - and only - behavior. This will probably break existing code that accidentaly relied the fact that the implementation uses a bare fork(), but i'd say it's worth it: I'm not convinced about making it the default behaviour, and certainly not the only one. I have a working patch which ensures that leaked semaphores get cleaned up on exit. However, I think to add proper tests for the patch, test_multiprocessing needs to be refactored. Maybe we could end up with multiprocessing_common.py test_multiprocessing_processes_fork.py test_multiprocessing_processes_nofork.py test_multiprocessing_manager_fork.py test_multiprocessing_manager_nofork.py test_multiprocessing_threads.py test_multiprocessing_others.py The actual unittests would be in multiprocessing_common.py and test_multiprocessing_others.py. The other files would run the unittests in multiprocessing_common.py using different configurations. Thoughts? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8713] multiprocessing needs option to eschew fork() under Linux
Charles-François Natali neolo...@free.fr added the comment: There is probably lots of such code: I'm not convinced about making it the default behaviour, and certainly not the only one. Then I'm not convinced that this patch is useful. Having three different implentations and code paths doesn't sound like a good idea to me. fork() vs fork() + exec() is an implementation detail, and exposing such tweakables to the user will only make confusion worse. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8713] multiprocessing needs option to eschew fork() under Linux
Jesse Noller jnol...@gmail.com added the comment: On Wednesday, December 21, 2011 at 10:04 AM, Charles-François Natali wrote: While I would tend to agree with you in theory - I don't think we should make it the default - at least not without a LOT of lead time. There's a surprising amount of code relying on the current behavior that I think the best course is to enable this option, and change the docs to steer users in this direction. For users jumping from 2.x into 3.x, I think the less surprises they have the better, and changing the default behavior of the stdlib module in this was would qualify as surprising. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8713] multiprocessing needs option to eschew fork() under Linux
sbt shibt...@gmail.com added the comment: Here is a patch which adds the following functions: forking_disable() forking_enable() forking_is_enabled() set_semaphore_prefix() get_semaphore_prefix() To create child processes using fork+exec on Unix, call forking_disable() at the beginning of the program. I have tested the patch on Linux (by adding forking_disable() to test_multiprocessing), and it seems to work. However, the patch does not modify test_multiprocessing, and I am not sure of the best way to do so. (See below.) There are some issues with named semaphores. When forking is disabled, the name of the semaphore must be left unlinked so that child processes can use sem_open() on the name. The patch therefore delays unlinking the name (only when forking is disabled) until the original SemLock object is garbage collected or the process which created it exits. But if a process is killed without exiting cleanly then the name may be left unlinked. This happens, for instance, if I run test_multiprocessing and then keep hitting ^C until all the processes exit. On Linux this leaves files with names like /dev/shm/sem.mp-fa012c80-4019-2 which represent leaked semaphores. These won't be destroyed until the computer reboots or the semaphores are manually removed (by using sem_unlink() or by unlinking the entry from the file system). If some form of this patch is accepted, then the problem of leaked semaphores needs to be addressed, otherwise the buildbots are likely run out of named semaphores. But I am not sure how best to do this in a platform agnostic way. (Maybe a forked process could collect names of all semaphores created, via a pipe. Then it could try to sem_unlink() all those names when all write-ends of the pipe are closed.) -- keywords: +patch nosy: +sbt Added file: http://bugs.python.org/file23141/mp_fork_exec.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8713] multiprocessing needs option to eschew fork() under Linux
Changes by sbt shibt...@gmail.com: Removed file: http://bugs.python.org/file23141/mp_fork_exec.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8713] multiprocessing needs option to eschew fork() under Linux
sbt shibt...@gmail.com added the comment: Small fix to patch. -- Added file: http://bugs.python.org/file23142/mp_fork_exec.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8713] multiprocessing needs option to eschew fork() under Linux
Ask Solem a...@celeryproject.org added the comment: I have suspected that this may be necessary, not just merely useful, for some time, and issue6721 seems to verify that. In addition to adding the keyword arg to Process, it should also be added to Pool and Manager. Is anyone working on a patch? If not I will work on a patch asap. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8713] multiprocessing needs option to eschew fork() under Linux
Jesse Noller jnol...@gmail.com added the comment: No one is currently working on a patch AFAIK -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8713] multiprocessing needs option to eschew fork() under Linux
Changes by Vivek Sekhar vi...@viveksekhar.ca: -- nosy: +vsekhar ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8713] multiprocessing needs option to eschew fork() under Linux
Changes by Matt M numbernine.pyt...@complexnumber.net: -- nosy: +numbernine ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8713] multiprocessing needs option to eschew fork() under Linux
dholth dho...@fastmail.fm added the comment: +1 -- nosy: +dholth ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8713] multiprocessing needs option to eschew fork() under Linux
Changes by Antoine Pitrou pit...@free.fr: -- nosy: +asksol stage: - needs patch versions: +Python 3.3 -Python 3.2 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8713] multiprocessing needs option to eschew fork() under Linux
Changes by Terry J. Reedy tjre...@udel.edu: -- versions: -Python 2.6, Python 2.7, Python 3.1, Python 3.3 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8713] multiprocessing needs option to eschew fork() under Linux
Changes by Ryan Coyner rcoy...@gmail.com: -- nosy: +rcoyner ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8713] multiprocessing needs option to eschew fork() under Linux
Ram Rachum cool...@cool-rr.com added the comment: +1 for this issue; I've also wished for this feature in the past. -- nosy: +cool-RR ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8713] multiprocessing needs option to eschew fork() under Linux
New submission from Brandon Craig Rhodes bran...@rhodesmill.org: The multiprocessing module uses a bare fork() to create child processes under Linux, so the children get a copy of the entire state of the parent process. But under Windows, child processes are freshly spun-up Python interpreters with none of the data structures or open connections of the parent process available. This means that code that tests fine under Linux, because it is depending on residual parent state in a way that the programmer has not noticed, can fail spectacularly under Windows. Therefore, the multiprocessing module should offer an option under Linux that ignores the advantage of being able to do a bare fork() and instead spins up a new interpreter instance just like Windows does. Some developers will just use this for testing under Linux, so their test results are valid for Windows too; and some developers might even use this in production, preferring to give up a bit of efficiency under Linux in return for an application that will show the same behavior on both platforms. Either way, an option that lets the developer subvert the simple sys.platform != 'win32' check in forking.py would go a long way towards helping us write platform-agnostic Python programs. -- components: Library (Lib) messages: 105719 nosy: brandon-rhodes priority: normal severity: normal status: open title: multiprocessing needs option to eschew fork() under Linux type: feature request versions: Python 2.6, Python 2.7, Python 3.1, Python 3.2, Python 3.3 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8713] multiprocessing needs option to eschew fork() under Linux
Changes by R. David Murray rdmur...@bitdance.com: -- nosy: +jnoller ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8713] multiprocessing needs option to eschew fork() under Linux
Jesse Noller jnol...@gmail.com added the comment: This is on my wish list; but I have not had time to do it. Patch welcome. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8713] multiprocessing needs option to eschew fork() under Linux
Brandon Craig Rhodes bran...@rhodesmill.org added the comment: Jesse, it's great to learn it's on your wish list too! Should I design the patch so that (a) there is some global in the module that needs tweaking to choose the child creation technique, or (b) that an argument to the Process() constructor forces a full interpreter exec to make all platforms match, or (c) that a process object once created has an attribute (like .daemon) that you set before starting it off? Or (d) should there be a subclass of Process that, if specifically used, has the fork/exec behavior instead of just doing the fork? My vote would probably be for (b), but you have a much better feel for the library and its style than I do. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8713] multiprocessing needs option to eschew fork() under Linux
Jesse Noller jnol...@gmail.com added the comment: I pretty much agree with (b) an argument - your gut instinct is correct - there's a long standing thread in python-dev which pretty much solidified my thinking about whether or not we need this (we do). Any patch has to be backwards compatible by the way, it can not alter the current default behavior, also, it has to target python3 as 2.7 is nearing final, and this is a behavioral change. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8713] multiprocessing needs option to eschew fork() under Linux
Changes by Santoso Wijaya santa@me.com: -- nosy: +santa4nt ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
