Roundup Robot added the comment:
New changeset b941a320601a by R David Murray in branch 'default':
whatsnew: multiprocessing start methods and context (#8713 and #18999)
http://hg.python.org/cpython/rev/b941a320601a
--
___
Python tracker
Changes by Piotr Dobrogost p...@bugs.python.dobrogost.net:
--
nosy: +piotr.dobrogost
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8713
___
___
Roundup Robot added the comment:
New changeset f6c7ad7d029a by Richard Oudkerk in branch 'default':
Issue #8713: Test should not print message about start method.
http://hg.python.org/cpython/rev/f6c7ad7d029a
New changeset e99832a60e63 by Richard Oudkerk in branch 'default':
Issue #8713:
Roundup Robot added the comment:
New changeset 6d998a43102b by Richard Oudkerk in branch 'default':
Issue #8713: Print dangling processes/threads, if any.
http://hg.python.org/cpython/rev/6d998a43102b
--
___
Python tracker rep...@bugs.python.org
Richard Oudkerk added the comment:
I've seen test_multiprocessing_forkserver giving warnings too, while
running the whole test suite, but can't reproduce them while running it
alone. The warnings seems quite similar though, so a single fix might
resolve the problem with all the tests.
Ezio Melotti added the comment:
The new tests produce a few warnings:
$ ./python -m test -uall test_multiprocessing_spawn
[1/1] test_multiprocessing_spawn
Using start method 'spawn'
Warning -- threading._dangling was modified by test_multiprocessing_spawn
Warning --
Antoine Pitrou added the comment:
Using the custom builders, it seems to happen randomly in test_rlock:
test_rlock (test.test_multiprocessing_spawn.WithManagerTestLock) ... Assertion
failed: !collecting, file ..\Modules\gcmodule.c, line 1617
ok
Antoine Pitrou added the comment:
Ok, I enabled faulthandler in the child process and I got the explanation:
http://buildbot.python.org/all/builders/AMD64%20Windows%20Server%202008%20%5BSB%5D%20custom/builds/5/steps/test/logs/stdio
multiprocessing's manager Server uses daemon threads... Daemon
Antoine Pitrou added the comment:
I have added documentation now so I think it is ready to merge
(except for a change to Makefile).
Good for me. This is a very nice addition!
--
___
Python tracker rep...@bugs.python.org
Roundup Robot added the comment:
New changeset 3b82e0d83bf9 by Richard Oudkerk in branch 'default':
Issue #8713: Support alternative start methods in multiprocessing on Unix.
http://hg.python.org/cpython/rev/3b82e0d83bf9
--
nosy: +python-dev
___
Richard Oudkerk added the comment:
Good for me. This is a very nice addition!
Thanks.
I do see a couple of failed assertions on Windows which presumably happen in a
child process because they do not cause a failure:
Assertion failed: !collecting, file ..\Modules\gcmodule.c, line 1617
Antoine Pitrou added the comment:
I do see a couple of failed assertions on Windows which presumably
happen in a child process because they do not cause a failure:
Assertion failed: !collecting, file ..\Modules\gcmodule.c, line
1617
The assertion is in _PyGC_CollectNoFail() and
Changes by Richard Oudkerk shibt...@gmail.com:
Added file: http://bugs.python.org/file31282/4fc7c72b1c5d.diff
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8713
___
Richard Oudkerk added the comment:
I have added documentation now so I think it is ready to merge (except for a
change to Makefile).
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8713
Changes by Richard Oudkerk shibt...@gmail.com:
Added file: http://bugs.python.org/file31214/c7aa0005f231.diff
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8713
___
Richard Oudkerk added the comment:
The forkserver process is now started using _posixsubprocess.fork_exec(). This
should fix the order dependent problem mentioned before.
Also the forkserver tests are now reenabled on OSX.
--
___
Python tracker
Ned Deily added the comment:
Richard, can you say what failed on the OS X 10.4 (Tiger) buildbot? FWIW, I
tested b3620777f54c.diff (and commented out the darwin skip of
test_multiprocessing_forkserver) on OS X 10.4, 10.5, and 10.8. There were no
failures on any of them. The only vaguely
Richard Oudkerk added the comment:
Richard, can you say what failed on the OS X 10.4 (Tiger) buildbot?
There seems to be a problem which depends on the order in which you run
the test, and it happens on Linux also. For example if I do
./python -m test -v \
Changes by Richard Oudkerk shibt...@gmail.com:
Added file: http://bugs.python.org/file31186/b3620777f54c.diff
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8713
___
Richard Oudkerk added the comment:
I have done quite a bit of refactoring and added some extra tests.
When I try using the forkserver start method on the OSX Tiger buildbot (the
only OSX one available) I get errors. I have disabled the tests for OSX, but
it seemed to be working before.
Changes by Antoine Pitrou pit...@free.fr:
Added file: http://bugs.python.org/file31181/d9fe9757ba0c.diff
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8713
___
Richard Oudkerk added the comment:
The spawn branch is in decent shape, although the documentation is not
up-to-date.
I would like to commit before the first alpha.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8713
Changes by Stan Seibert s...@mtrr.org:
--
nosy: +Stan.Seibert
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8713
___
___
Python-bugs-list mailing
Richard Oudkerk added the comment:
Richard, apart from performance, what's the advantage of this approach over
the
fork+exec version?
It is really just performance. For context running the unittests in a 1 cpu
linux VM gives me
fork:
real0m53.868s
user0m1.496s
sys 0m9.757s
Richard Oudkerk added the comment:
Numbers when running on Linux on a laptop with 2 cores + hyperthreading.
RUNNING UNITTESTS:
fork:
real0m50.687s
user0m9.213s
sys 0m4.012s
fork+exec:
real1m9.062s
user0m48.579s
sys 0m6.648s
forkserver:
Changes by Ram Rachum r...@rachum.com:
--
nosy: -cool-RR
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8713
___
___
Python-bugs-list mailing list
Gregory P. Smith added the comment:
I think the forkserver approach is a good idea. It is what a lot of users will
choose.
forkserver won't work everywhere though so the fork+exec option is still
desirable to have available. Threads can be started by non-python code
(extension modules, or
Changes by Richard Oudkerk shibt...@gmail.com:
Added file: http://bugs.python.org/file28461/8f08d83264a0.diff
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8713
___
Richard Oudkerk added the comment:
The safest default would be fork+exec though we need to implement the
fork+exec code as a C extension module or have it use subprocess (as I
noted in the mb_fork_exec.patch review).
That was an old version of the patch.
In the branch
Gregory P. Smith added the comment:
ah, i missed that update. cool! +1
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8713
___
___
Charles-François Natali added the comment:
Richard, apart from performance, what's the advantage of this approach over the
fork+exec version?
Because it seems more complicated, and although I didn't have a look a this
last patch, I guess that most of the fork+exec version could be factorized
Changes by Gregory P. Smith g...@krypto.org:
--
nosy: +gregory.p.smith
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8713
___
___
Python-bugs-list
Richard Oudkerk added the comment:
http://hg.python.org/sandbox/sbt#spawn now contains support for starting
processes via a separate server process. This depends on fd passing support.
This also solves the problem of mixing threads and processes, but is much
faster than using fork+exec. It
Richard Oudkerk added the comment:
For updated code see http://hg.python.org/sandbox/sbt#spawn
This uses _posixsubprocess and closefds=True.
--
hgrepos: +157
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8713
Richard Oudkerk added the comment:
A use case for not using fork() is when your parent process opens some
system resources of some sort (for example a listening TCP socket). The
child will then inherit those resources, which can have all kinds of
unforeseen and troublesome consequences
Antoine Pitrou added the comment:
A use case for not using fork() is when your parent process opens some system
resources of some sort (for example a listening TCP socket). The child will
then inherit those resources, which can have all kinds of unforeseen and
troublesome consequences (for
Antoine Pitrou added the comment:
By the way, instead of doing fork() + exec() in pure Python, you probably want
to use _posixsubprocess.fork_exec().
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8713
Christian Heimes added the comment:
+1
I still have to use parallel python (pp) in our application stack because the
fork() approach causes a lot of strange issues in our application. It might be
the punishment for embedding a Java runtime env into a Python process, too. :)
--
nosy:
Changes by Catalin Iacob iacobcata...@gmail.com:
--
nosy: +catalin.iacob
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8713
___
___
sbt shibt...@gmail.com added the comment:
Attached is an updated version of the mp_fork_exec.patch. This one is able to
reliably clean up any unlinked semaphores if the program exits abnormally.
--
Added file: http://bugs.python.org/file24297/mp_fork_exec.patch
sbt shibt...@gmail.com added the comment:
mp_split_tests.patch splits up the test_multiprocessing.py:
test_multiprocessing_misc.py
miscellaneous tests which need not be run with multiple configurations
mp_common.py
testcases which should be run with multiple configurations
Changes by Santoso Wijaya santoso.wij...@gmail.com:
--
nosy: -santa4nt
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8713
___
___
Charles-François Natali neolo...@free.fr added the comment:
I don't know what the others think, but I'm still -1 on this patch.
Not that I don't like the principle - it's actually the contrary: in a
perfect world, I think this should be made the default -and only -
behavior on POSIX. But since
Ned Deily n...@acm.org added the comment:
See also consolidated Issue13558 for additional justification for processes
option on OS X.
--
nosy: +mrmekon, ned.deily
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8713
Charles-François Natali neolo...@free.fr added the comment:
Thanks for the patch sbt.
I think this is indeed useful, but I'm tempted to go further and say we should
make this the default - and only - behavior. This will probably break existing
code that accidentaly relied the fact that the
Antoine Pitrou pit...@free.fr added the comment:
Thanks for the patch sbt.
I think this is indeed useful, but I'm tempted to go further and say
we should make this the default - and only - behavior. This will
probably break existing code that accidentaly relied the fact that the
sbt shibt...@gmail.com added the comment:
I think this is indeed useful, but I'm tempted to go further and say we
should make this the default - and only - behavior. This will probably
break existing code that accidentaly relied the fact that the
implementation uses a bare fork(), but i'd
Charles-François Natali neolo...@free.fr added the comment:
There is probably lots of such code:
I'm not convinced about making it the default behaviour, and
certainly not the only one.
Then I'm not convinced that this patch is useful.
Having three different implentations and code paths
Jesse Noller jnol...@gmail.com added the comment:
On Wednesday, December 21, 2011 at 10:04 AM, Charles-François Natali wrote:
While I would tend to agree with you in theory - I don't think we should make
it the default - at least not without a LOT of lead time. There's a surprising
amount of
sbt shibt...@gmail.com added the comment:
Here is a patch which adds the following functions:
forking_disable()
forking_enable()
forking_is_enabled()
set_semaphore_prefix()
get_semaphore_prefix()
To create child processes using fork+exec on Unix, call
forking_disable() at the
Changes by sbt shibt...@gmail.com:
Removed file: http://bugs.python.org/file23141/mp_fork_exec.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8713
___
sbt shibt...@gmail.com added the comment:
Small fix to patch.
--
Added file: http://bugs.python.org/file23142/mp_fork_exec.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8713
___
Ask Solem a...@celeryproject.org added the comment:
I have suspected that this may be necessary, not just merely useful, for some
time, and issue6721 seems to verify that. In addition to adding the keyword
arg to Process, it should also be added to Pool and Manager.
Is anyone working on a
Jesse Noller jnol...@gmail.com added the comment:
No one is currently working on a patch AFAIK
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8713
___
Changes by Vivek Sekhar vi...@viveksekhar.ca:
--
nosy: +vsekhar
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8713
___
___
Python-bugs-list
Changes by Matt M numbernine.pyt...@complexnumber.net:
--
nosy: +numbernine
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8713
___
___
dholth dho...@fastmail.fm added the comment:
+1
--
nosy: +dholth
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8713
___
___
Python-bugs-list
Changes by Antoine Pitrou pit...@free.fr:
--
nosy: +asksol
stage: - needs patch
versions: +Python 3.3 -Python 3.2
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8713
___
Changes by Terry J. Reedy tjre...@udel.edu:
--
versions: -Python 2.6, Python 2.7, Python 3.1, Python 3.3
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8713
___
Changes by Ryan Coyner rcoy...@gmail.com:
--
nosy: +rcoyner
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8713
___
___
Python-bugs-list mailing
Ram Rachum cool...@cool-rr.com added the comment:
+1 for this issue; I've also wished for this feature in the past.
--
nosy: +cool-RR
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8713
___
New submission from Brandon Craig Rhodes bran...@rhodesmill.org:
The multiprocessing module uses a bare fork() to create child processes under
Linux, so the children get a copy of the entire state of the parent process.
But under Windows, child processes are freshly spun-up Python
Changes by R. David Murray rdmur...@bitdance.com:
--
nosy: +jnoller
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8713
___
___
Python-bugs-list
Jesse Noller jnol...@gmail.com added the comment:
This is on my wish list; but I have not had time to do it. Patch welcome.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8713
___
Brandon Craig Rhodes bran...@rhodesmill.org added the comment:
Jesse, it's great to learn it's on your wish list too!
Should I design the patch so that (a) there is some global in the module that
needs tweaking to choose the child creation technique, or (b) that an argument
to the Process()
Jesse Noller jnol...@gmail.com added the comment:
I pretty much agree with (b) an argument - your gut instinct is correct -
there's a long standing thread in python-dev which pretty much solidified my
thinking about whether or not we need this (we do).
Any patch has to be backwards
Changes by Santoso Wijaya santa@me.com:
--
nosy: +santa4nt
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8713
___
___
Python-bugs-list mailing
67 matches
Mail list logo