[issue8713] multiprocessing needs option to eschew fork() under Linux

2014-03-10 Thread Roundup Robot
Roundup Robot added the comment: New changeset b941a320601a by R David Murray in branch 'default': whatsnew: multiprocessing start methods and context (#8713 and #18999) http://hg.python.org/cpython/rev/b941a320601a -- ___ Python tracker

[issue8713] multiprocessing needs option to eschew fork() under Linux

2013-09-12 Thread Piotr Dobrogost
Changes by Piotr Dobrogost p...@bugs.python.dobrogost.net: -- nosy: +piotr.dobrogost ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___

[issue8713] multiprocessing needs option to eschew fork() under Linux

2013-08-29 Thread Roundup Robot
Roundup Robot added the comment: New changeset f6c7ad7d029a by Richard Oudkerk in branch 'default': Issue #8713: Test should not print message about start method. http://hg.python.org/cpython/rev/f6c7ad7d029a New changeset e99832a60e63 by Richard Oudkerk in branch 'default': Issue #8713:

[issue8713] multiprocessing needs option to eschew fork() under Linux

2013-08-29 Thread Roundup Robot
Roundup Robot added the comment: New changeset 6d998a43102b by Richard Oudkerk in branch 'default': Issue #8713: Print dangling processes/threads, if any. http://hg.python.org/cpython/rev/6d998a43102b -- ___ Python tracker rep...@bugs.python.org

[issue8713] multiprocessing needs option to eschew fork() under Linux

2013-08-29 Thread Richard Oudkerk
Richard Oudkerk added the comment: I've seen test_multiprocessing_forkserver giving warnings too, while running the whole test suite, but can't reproduce them while running it alone. The warnings seems quite similar though, so a single fix might resolve the problem with all the tests.

[issue8713] multiprocessing needs option to eschew fork() under Linux

2013-08-25 Thread Ezio Melotti
Ezio Melotti added the comment: The new tests produce a few warnings: $ ./python -m test -uall test_multiprocessing_spawn [1/1] test_multiprocessing_spawn Using start method 'spawn' Warning -- threading._dangling was modified by test_multiprocessing_spawn Warning --

[issue8713] multiprocessing needs option to eschew fork() under Linux

2013-08-15 Thread Antoine Pitrou
Antoine Pitrou added the comment: Using the custom builders, it seems to happen randomly in test_rlock: test_rlock (test.test_multiprocessing_spawn.WithManagerTestLock) ... Assertion failed: !collecting, file ..\Modules\gcmodule.c, line 1617 ok

[issue8713] multiprocessing needs option to eschew fork() under Linux

2013-08-15 Thread Antoine Pitrou
Antoine Pitrou added the comment: Ok, I enabled faulthandler in the child process and I got the explanation: http://buildbot.python.org/all/builders/AMD64%20Windows%20Server%202008%20%5BSB%5D%20custom/builds/5/steps/test/logs/stdio multiprocessing's manager Server uses daemon threads... Daemon

[issue8713] multiprocessing needs option to eschew fork() under Linux

2013-08-14 Thread Antoine Pitrou
Antoine Pitrou added the comment: I have added documentation now so I think it is ready to merge (except for a change to Makefile). Good for me. This is a very nice addition! -- ___ Python tracker rep...@bugs.python.org

[issue8713] multiprocessing needs option to eschew fork() under Linux

2013-08-14 Thread Roundup Robot
Roundup Robot added the comment: New changeset 3b82e0d83bf9 by Richard Oudkerk in branch 'default': Issue #8713: Support alternative start methods in multiprocessing on Unix. http://hg.python.org/cpython/rev/3b82e0d83bf9 -- nosy: +python-dev ___

[issue8713] multiprocessing needs option to eschew fork() under Linux

2013-08-14 Thread Richard Oudkerk
Richard Oudkerk added the comment: Good for me. This is a very nice addition! Thanks. I do see a couple of failed assertions on Windows which presumably happen in a child process because they do not cause a failure: Assertion failed: !collecting, file ..\Modules\gcmodule.c, line 1617

[issue8713] multiprocessing needs option to eschew fork() under Linux

2013-08-14 Thread Antoine Pitrou
Antoine Pitrou added the comment: I do see a couple of failed assertions on Windows which presumably happen in a child process because they do not cause a failure: Assertion failed: !collecting, file ..\Modules\gcmodule.c, line 1617 The assertion is in _PyGC_CollectNoFail() and

[issue8713] multiprocessing needs option to eschew fork() under Linux

2013-08-13 Thread Richard Oudkerk
Changes by Richard Oudkerk shibt...@gmail.com: Added file: http://bugs.python.org/file31282/4fc7c72b1c5d.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___

[issue8713] multiprocessing needs option to eschew fork() under Linux

2013-08-13 Thread Richard Oudkerk
Richard Oudkerk added the comment: I have added documentation now so I think it is ready to merge (except for a change to Makefile). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713

[issue8713] multiprocessing needs option to eschew fork() under Linux

2013-08-10 Thread Richard Oudkerk
Changes by Richard Oudkerk shibt...@gmail.com: Added file: http://bugs.python.org/file31214/c7aa0005f231.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___

[issue8713] multiprocessing needs option to eschew fork() under Linux

2013-08-10 Thread Richard Oudkerk
Richard Oudkerk added the comment: The forkserver process is now started using _posixsubprocess.fork_exec(). This should fix the order dependent problem mentioned before. Also the forkserver tests are now reenabled on OSX. -- ___ Python tracker

[issue8713] multiprocessing needs option to eschew fork() under Linux

2013-08-08 Thread Ned Deily
Ned Deily added the comment: Richard, can you say what failed on the OS X 10.4 (Tiger) buildbot? FWIW, I tested b3620777f54c.diff (and commented out the darwin skip of test_multiprocessing_forkserver) on OS X 10.4, 10.5, and 10.8. There were no failures on any of them. The only vaguely

[issue8713] multiprocessing needs option to eschew fork() under Linux

2013-08-08 Thread Richard Oudkerk
Richard Oudkerk added the comment: Richard, can you say what failed on the OS X 10.4 (Tiger) buildbot? There seems to be a problem which depends on the order in which you run the test, and it happens on Linux also. For example if I do ./python -m test -v \

[issue8713] multiprocessing needs option to eschew fork() under Linux

2013-08-07 Thread Richard Oudkerk
Changes by Richard Oudkerk shibt...@gmail.com: Added file: http://bugs.python.org/file31186/b3620777f54c.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___

[issue8713] multiprocessing needs option to eschew fork() under Linux

2013-08-07 Thread Richard Oudkerk
Richard Oudkerk added the comment: I have done quite a bit of refactoring and added some extra tests. When I try using the forkserver start method on the OSX Tiger buildbot (the only OSX one available) I get errors. I have disabled the tests for OSX, but it seemed to be working before.

[issue8713] multiprocessing needs option to eschew fork() under Linux

2013-08-06 Thread Antoine Pitrou
Changes by Antoine Pitrou pit...@free.fr: Added file: http://bugs.python.org/file31181/d9fe9757ba0c.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___

[issue8713] multiprocessing needs option to eschew fork() under Linux

2013-07-30 Thread Richard Oudkerk
Richard Oudkerk added the comment: The spawn branch is in decent shape, although the documentation is not up-to-date. I would like to commit before the first alpha. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713

[issue8713] multiprocessing needs option to eschew fork() under Linux

2013-02-01 Thread Stan Seibert
Changes by Stan Seibert s...@mtrr.org: -- nosy: +Stan.Seibert ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list mailing

[issue8713] multiprocessing needs option to eschew fork() under Linux

2012-12-27 Thread Richard Oudkerk
Richard Oudkerk added the comment: Richard, apart from performance, what's the advantage of this approach over the fork+exec version? It is really just performance. For context running the unittests in a 1 cpu linux VM gives me fork: real0m53.868s user0m1.496s sys 0m9.757s

[issue8713] multiprocessing needs option to eschew fork() under Linux

2012-12-27 Thread Richard Oudkerk
Richard Oudkerk added the comment: Numbers when running on Linux on a laptop with 2 cores + hyperthreading. RUNNING UNITTESTS: fork: real0m50.687s user0m9.213s sys 0m4.012s fork+exec: real1m9.062s user0m48.579s sys 0m6.648s forkserver:

[issue8713] multiprocessing needs option to eschew fork() under Linux

2012-12-27 Thread Ram Rachum
Changes by Ram Rachum r...@rachum.com: -- nosy: -cool-RR ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list mailing list

[issue8713] multiprocessing needs option to eschew fork() under Linux

2012-12-27 Thread Gregory P. Smith
Gregory P. Smith added the comment: I think the forkserver approach is a good idea. It is what a lot of users will choose. forkserver won't work everywhere though so the fork+exec option is still desirable to have available. Threads can be started by non-python code (extension modules, or

[issue8713] multiprocessing needs option to eschew fork() under Linux

2012-12-27 Thread Richard Oudkerk
Changes by Richard Oudkerk shibt...@gmail.com: Added file: http://bugs.python.org/file28461/8f08d83264a0.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___

[issue8713] multiprocessing needs option to eschew fork() under Linux

2012-12-27 Thread Richard Oudkerk
Richard Oudkerk added the comment: The safest default would be fork+exec though we need to implement the fork+exec code as a C extension module or have it use subprocess (as I noted in the mb_fork_exec.patch review). That was an old version of the patch. In the branch

[issue8713] multiprocessing needs option to eschew fork() under Linux

2012-12-27 Thread Gregory P. Smith
Gregory P. Smith added the comment: ah, i missed that update. cool! +1 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___

[issue8713] multiprocessing needs option to eschew fork() under Linux

2012-12-26 Thread Charles-François Natali
Charles-François Natali added the comment: Richard, apart from performance, what's the advantage of this approach over the fork+exec version? Because it seems more complicated, and although I didn't have a look a this last patch, I guess that most of the fork+exec version could be factorized

[issue8713] multiprocessing needs option to eschew fork() under Linux

2012-12-04 Thread Gregory P. Smith
Changes by Gregory P. Smith g...@krypto.org: -- nosy: +gregory.p.smith ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list

[issue8713] multiprocessing needs option to eschew fork() under Linux

2012-11-20 Thread Richard Oudkerk
Richard Oudkerk added the comment: http://hg.python.org/sandbox/sbt#spawn now contains support for starting processes via a separate server process. This depends on fd passing support. This also solves the problem of mixing threads and processes, but is much faster than using fork+exec. It

[issue8713] multiprocessing needs option to eschew fork() under Linux

2012-10-26 Thread Richard Oudkerk
Richard Oudkerk added the comment: For updated code see http://hg.python.org/sandbox/sbt#spawn This uses _posixsubprocess and closefds=True. -- hgrepos: +157 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713

[issue8713] multiprocessing needs option to eschew fork() under Linux

2012-10-24 Thread Richard Oudkerk
Richard Oudkerk added the comment: A use case for not using fork() is when your parent process opens some system resources of some sort (for example a listening TCP socket). The child will then inherit those resources, which can have all kinds of unforeseen and troublesome consequences

[issue8713] multiprocessing needs option to eschew fork() under Linux

2012-10-23 Thread Antoine Pitrou
Antoine Pitrou added the comment: A use case for not using fork() is when your parent process opens some system resources of some sort (for example a listening TCP socket). The child will then inherit those resources, which can have all kinds of unforeseen and troublesome consequences (for

[issue8713] multiprocessing needs option to eschew fork() under Linux

2012-10-23 Thread Antoine Pitrou
Antoine Pitrou added the comment: By the way, instead of doing fork() + exec() in pure Python, you probably want to use _posixsubprocess.fork_exec(). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713

[issue8713] multiprocessing needs option to eschew fork() under Linux

2012-10-23 Thread Christian Heimes
Christian Heimes added the comment: +1 I still have to use parallel python (pp) in our application stack because the fork() approach causes a lot of strange issues in our application. It might be the punishment for embedding a Java runtime env into a Python process, too. :) -- nosy:

[issue8713] multiprocessing needs option to eschew fork() under Linux

2012-07-18 Thread Catalin Iacob
Changes by Catalin Iacob iacobcata...@gmail.com: -- nosy: +catalin.iacob ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___

[issue8713] multiprocessing needs option to eschew fork() under Linux

2012-01-23 Thread sbt
sbt shibt...@gmail.com added the comment: Attached is an updated version of the mp_fork_exec.patch. This one is able to reliably clean up any unlinked semaphores if the program exits abnormally. -- Added file: http://bugs.python.org/file24297/mp_fork_exec.patch

[issue8713] multiprocessing needs option to eschew fork() under Linux

2012-01-23 Thread sbt
sbt shibt...@gmail.com added the comment: mp_split_tests.patch splits up the test_multiprocessing.py: test_multiprocessing_misc.py miscellaneous tests which need not be run with multiple configurations mp_common.py testcases which should be run with multiple configurations

[issue8713] multiprocessing needs option to eschew fork() under Linux

2012-01-23 Thread Santoso Wijaya
Changes by Santoso Wijaya santoso.wij...@gmail.com: -- nosy: -santa4nt ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___

[issue8713] multiprocessing needs option to eschew fork() under Linux

2012-01-23 Thread Charles-François Natali
Charles-François Natali neolo...@free.fr added the comment: I don't know what the others think, but I'm still -1 on this patch. Not that I don't like the principle - it's actually the contrary: in a perfect world, I think this should be made the default -and only - behavior on POSIX. But since

[issue8713] multiprocessing needs option to eschew fork() under Linux

2012-01-03 Thread Ned Deily
Ned Deily n...@acm.org added the comment: See also consolidated Issue13558 for additional justification for processes option on OS X. -- nosy: +mrmekon, ned.deily ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713

[issue8713] multiprocessing needs option to eschew fork() under Linux

2011-12-21 Thread Charles-François Natali
Charles-François Natali neolo...@free.fr added the comment: Thanks for the patch sbt. I think this is indeed useful, but I'm tempted to go further and say we should make this the default - and only - behavior. This will probably break existing code that accidentaly relied the fact that the

[issue8713] multiprocessing needs option to eschew fork() under Linux

2011-12-21 Thread Antoine Pitrou
Antoine Pitrou pit...@free.fr added the comment: Thanks for the patch sbt. I think this is indeed useful, but I'm tempted to go further and say we should make this the default - and only - behavior. This will probably break existing code that accidentaly relied the fact that the

[issue8713] multiprocessing needs option to eschew fork() under Linux

2011-12-21 Thread sbt
sbt shibt...@gmail.com added the comment: I think this is indeed useful, but I'm tempted to go further and say we should make this the default - and only - behavior. This will probably break existing code that accidentaly relied the fact that the implementation uses a bare fork(), but i'd

[issue8713] multiprocessing needs option to eschew fork() under Linux

2011-12-21 Thread Charles-François Natali
Charles-François Natali neolo...@free.fr added the comment: There is probably lots of such code: I'm not convinced about making it the default behaviour, and certainly not the only one. Then I'm not convinced that this patch is useful. Having three different implentations and code paths

[issue8713] multiprocessing needs option to eschew fork() under Linux

2011-12-21 Thread Jesse Noller
Jesse Noller jnol...@gmail.com added the comment: On Wednesday, December 21, 2011 at 10:04 AM, Charles-François Natali wrote: While I would tend to agree with you in theory - I don't think we should make it the default - at least not without a LOT of lead time. There's a surprising amount of

[issue8713] multiprocessing needs option to eschew fork() under Linux

2011-09-13 Thread sbt
sbt shibt...@gmail.com added the comment: Here is a patch which adds the following functions: forking_disable() forking_enable() forking_is_enabled() set_semaphore_prefix() get_semaphore_prefix() To create child processes using fork+exec on Unix, call forking_disable() at the

[issue8713] multiprocessing needs option to eschew fork() under Linux

2011-09-13 Thread sbt
Changes by sbt shibt...@gmail.com: Removed file: http://bugs.python.org/file23141/mp_fork_exec.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___

[issue8713] multiprocessing needs option to eschew fork() under Linux

2011-09-13 Thread sbt
sbt shibt...@gmail.com added the comment: Small fix to patch. -- Added file: http://bugs.python.org/file23142/mp_fork_exec.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___

[issue8713] multiprocessing needs option to eschew fork() under Linux

2011-08-24 Thread Ask Solem
Ask Solem a...@celeryproject.org added the comment: I have suspected that this may be necessary, not just merely useful, for some time, and issue6721 seems to verify that. In addition to adding the keyword arg to Process, it should also be added to Pool and Manager. Is anyone working on a

[issue8713] multiprocessing needs option to eschew fork() under Linux

2011-08-24 Thread Jesse Noller
Jesse Noller jnol...@gmail.com added the comment: No one is currently working on a patch AFAIK -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___

[issue8713] multiprocessing needs option to eschew fork() under Linux

2011-08-20 Thread Vivek Sekhar
Changes by Vivek Sekhar vi...@viveksekhar.ca: -- nosy: +vsekhar ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list

[issue8713] multiprocessing needs option to eschew fork() under Linux

2011-08-10 Thread Matt M
Changes by Matt M numbernine.pyt...@complexnumber.net: -- nosy: +numbernine ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___

[issue8713] multiprocessing needs option to eschew fork() under Linux

2011-01-21 Thread dholth
dholth dho...@fastmail.fm added the comment: +1 -- nosy: +dholth ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list

[issue8713] multiprocessing needs option to eschew fork() under Linux

2011-01-21 Thread Antoine Pitrou
Changes by Antoine Pitrou pit...@free.fr: -- nosy: +asksol stage: - needs patch versions: +Python 3.3 -Python 3.2 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___

[issue8713] multiprocessing needs option to eschew fork() under Linux

2010-08-07 Thread Terry J. Reedy
Changes by Terry J. Reedy tjre...@udel.edu: -- versions: -Python 2.6, Python 2.7, Python 3.1, Python 3.3 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___

[issue8713] multiprocessing needs option to eschew fork() under Linux

2010-05-28 Thread Ryan Coyner
Changes by Ryan Coyner rcoy...@gmail.com: -- nosy: +rcoyner ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list mailing

[issue8713] multiprocessing needs option to eschew fork() under Linux

2010-05-17 Thread Ram Rachum
Ram Rachum cool...@cool-rr.com added the comment: +1 for this issue; I've also wished for this feature in the past. -- nosy: +cool-RR ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___

[issue8713] multiprocessing needs option to eschew fork() under Linux

2010-05-14 Thread Brandon Craig Rhodes
New submission from Brandon Craig Rhodes bran...@rhodesmill.org: The multiprocessing module uses a bare fork() to create child processes under Linux, so the children get a copy of the entire state of the parent process. But under Windows, child processes are freshly spun-up Python

[issue8713] multiprocessing needs option to eschew fork() under Linux

2010-05-14 Thread R. David Murray
Changes by R. David Murray rdmur...@bitdance.com: -- nosy: +jnoller ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list

[issue8713] multiprocessing needs option to eschew fork() under Linux

2010-05-14 Thread Jesse Noller
Jesse Noller jnol...@gmail.com added the comment: This is on my wish list; but I have not had time to do it. Patch welcome. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___

[issue8713] multiprocessing needs option to eschew fork() under Linux

2010-05-14 Thread Brandon Craig Rhodes
Brandon Craig Rhodes bran...@rhodesmill.org added the comment: Jesse, it's great to learn it's on your wish list too! Should I design the patch so that (a) there is some global in the module that needs tweaking to choose the child creation technique, or (b) that an argument to the Process()

[issue8713] multiprocessing needs option to eschew fork() under Linux

2010-05-14 Thread Jesse Noller
Jesse Noller jnol...@gmail.com added the comment: I pretty much agree with (b) an argument - your gut instinct is correct - there's a long standing thread in python-dev which pretty much solidified my thinking about whether or not we need this (we do). Any patch has to be backwards

[issue8713] multiprocessing needs option to eschew fork() under Linux

2010-05-14 Thread Santoso Wijaya
Changes by Santoso Wijaya santa@me.com: -- nosy: +santa4nt ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8713 ___ ___ Python-bugs-list mailing