Hello
Some update about the spawnl() thingy ;
I've adapted the win32 code to have a new unix Popen object, which works
with a spawn() semantic. It's quite straightforward, and the
mutiprocessing call of a python functions works OK.
But I've run into some trouble : synchronization
Matt Knox a écrit :
Jesse Noller jnoller at gmail.com writes:
We already have an implementation that spawns a
subprocess and then pushes the required state to the child. The
fundamental need for things to be pickleable *all the time* kinda
makes it annoying to work with.
just a
Pascal Chambon writes:
I don't really get it there... it seems to me that multiprocessing only
requires picklability for the objects it needs to transfer, i.e those
given as arguments to the called function, and thsoe put into
multiprocessing queues/pipes. Global program data needn't be
On 04:58 pm, jaeda...@gmail.com wrote:
Jesse Noller jnoller at gmail.com writes:
We already have an implementation that spawns a
subprocess and then pushes the required state to the child. The
fundamental need for things to be pickleable *all the time* kinda
makes it annoying to work with.
Jesse Noller jnoller at gmail.com writes:
We already have an implementation that spawns a
subprocess and then pushes the required state to the child. The
fundamental need for things to be pickleable *all the time* kinda
makes it annoying to work with.
just a lurker here... but this topic
Although I would be in favor of an atfork callback registration system
(similar to atexit), it seems there is no way to solve the fork()
problem automatically with this. Any attempt to acquire/release locks
automatically will lead to deadlocks, as it is necessary to know the
exact program
Pascal Chambon wrote:
I guess spawnl semantic (i.e, like win32's CreateProcess()) can't become
the default multiprocessing behaviour, as too many programs implicitly
rely on the whole sharing of data under unix (and py3k itself is maybe
becoming a little too mature for new compatility breaks)
I guess spawnl semantic (i.e, like win32's CreateProcess()) can't
become the default multiprocessing behaviour...
Nick It would also make it much easier to write cross-platform
Nick multiprocessing code (by always using the non-forking semantics
Nick even on fork-capable
skip at pobox.com writes:
I don't understand. On Unix-y systems isn't spawn* layered on top of
fork/exec?
The point is that exec() relinquishes all the existing resources, so the initial
fork() becomes an implementation detail (IIUC).
If you were going to
combine both threading and
Tres Seaver tseaver at palladion.com writes:
Note that the we in your sentence is not anything like the quod
semper quod ubique quod ab omnibus criterion for accepting dogma:
mutliprocessing is a tool, and needs to be used according to its nature,
just as with threading.
I don't remember
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Antoine Pitrou wrote:
Tres Seaver tseaver at palladion.com writes:
Note that the we in your sentence is not anything like the quod
semper quod ubique quod ab omnibus criterion for accepting dogma:
mutliprocessing is a tool, and needs to be used
Tres Some applications may seem to work when violating this rule, but
Tres their developers are doomed to hair loss over time.
Then for us bald guys it should be okay, right? ;-)
Skip
___
Python-Dev mailing list
Python-Dev@python.org
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
s...@pobox.com wrote:
Tres Some applications may seem to work when violating this rule, but
Tres their developers are doomed to hair loss over time.
Then for us bald guys it should be okay, right? ;-)
Sure: you might even grow hair
The word dogma is a good one in this context however. We ( ;-)) have
accepted and promoted the dogma that multiprocessing is the solution to
parallelism in the face of the GIL. While it needn't be applicable in any and
every situation, we should make it so that it is applicable often enough.
On Tue, Feb 2, 2010 at 10:34 AM, Pascal Chambon
chambon.pas...@gmail.com wrote:
The word dogma is a good one in this context however. We ( ;-)) have
accepted and promoted the dogma that multiprocessing is the solution to
parallelism in the face of the GIL. While it needn't be applicable in
Seems the problem under discussion is already taken care of in Python.
Possibly remains to verify that the logic described below does not possibly
generate deadlocks.
From the Python docs: http://docs.python.org/c-api/init.html
Another important thing to note about threads is their behaviour in
Le lundi 01 février 2010 à 23:58 +0100, Martin v. Löwis a écrit :
Interestingly, the POSIX pthread_atfork documentation defines how you
are supposed to do that: create an atfork handler set, and acquire all
mutexes in the prepare handler. Then fork, and have the parent and child
handlers
Although I would be in favor of an atfork callback registration system
(similar to atexit), it seems there is no way to solve the fork()
problem automatically with this. Any attempt to acquire/release locks
automatically will lead to deadlocks, as it is necessary to know the
exact program
So, if a patch was proposed for the multiprocessing, allowing an unified
spawnl, thread-safe, semantic, do you think something could prevent
its integration ?
We may ignore the subprocess module, since fork+exec shouldn't be
bothered by the (potentially disastrous) state of child process
On Mon, Feb 1, 2010 at 3:09 PM, Pascal Chambon chambon.pas...@gmail.com wrote:
So, if a patch was proposed for the multiprocessing, allowing an unified
spawnl, thread-safe, semantic, do you think something could prevent its
integration ?
We may ignore the subprocess module, since fork+exec
Jesse Noller jnoller at gmail.com writes:
I don't see the need for the change from fork as of yet (for
multiprocessing) and I am leery to change the internal implementation
and semantics right now, or anytime soon. I'd be interested in seeing
the patch, but if the concern is that global
On Mon, Feb 1, 2010 at 4:32 PM, Antoine Pitrou solip...@pitrou.net wrote:
Jesse Noller jnoller at gmail.com writes:
I don't see the need for the change from fork as of yet (for
multiprocessing) and I am leery to change the internal implementation
and semantics right now, or anytime soon. I'd
Jesse Noller jnoller at gmail.com writes:
I don't see spawnl as a viable alternative to fork. I imagine that I,
and others successfully mix threads and multiprocessing on non-win32
platforms just fine, knowing of course that fork() can cause heartburn
if you have global locks code within the
On Mon, Feb 1, 2010 at 5:08 PM, Antoine Pitrou solip...@pitrou.net wrote:
I don't see spawnl as a viable alternative to fork. I imagine that I,
and others successfully mix threads and multiprocessing on non-win32
platforms just fine, knowing of course that fork() can cause heartburn
if you
Antoine Pitrou wrote:
Jesse Noller jnoller at gmail.com writes:
I don't see the need for the change from fork as of yet (for
multiprocessing) and I am leery to change the internal implementation
and semantics right now, or anytime soon. I'd be interested in seeing
the patch, but if the
On Mon, Feb 1, 2010 at 5:20 PM, Martin v. Löwis mar...@v.loewis.de wrote:
Antoine Pitrou wrote:
Jesse Noller jnoller at gmail.com writes:
I don't see the need for the change from fork as of yet (for
multiprocessing) and I am leery to change the internal implementation
and semantics right now,
Le lundi 01 février 2010 à 23:20 +0100, Martin v. Löwis a écrit :
I don't know what spawnl is supposed to do, but it really sounds like
the wrong solution.
As far as I understand, I think the idea is to use the same mechanism as
under Windows: spawn a new Python interpreter (in a separate
On Mon, Feb 1, 2010 at 5:18 PM, Jesse Noller jnol...@gmail.com wrote:
I don't disagree there; but then again, I haven't seen this issue
arise (in my own code)/no bug reports/no test cases that show this to
be a consistent issue. I'm perfectly OK with being wrong, I'm just
leery to tearing out
On Feb 1, 2010, at 5:35 PM, Reid Kleckner r...@mit.edu wrote:
On Mon, Feb 1, 2010 at 5:18 PM, Jesse Noller jnol...@gmail.com
wrote:
I don't disagree there; but then again, I haven't seen this issue
arise (in my own code)/no bug reports/no test cases that show this to
be a consistent issue.
Instead, we should aim to make Python fork-safe. If the primary concern
is that locks get inherited, we should change the Python locks so that
they get auto-released on fork (unless otherwise specified on lock
creation). This may sound like an uphill battle, but if there was a
smart and easy
On Mon, Feb 1, 2010 at 5:20 PM, Martin v. Löwis mar...@v.loewis.de wrote:
Instead, we should aim to make Python fork-safe. If the primary concern
is that locks get inherited, we should change the Python locks so that
they get auto-released on fork (unless otherwise specified on lock
creation).
We must distinguish between locks owned by the thread which survived the
fork(), and locks owned by other threads. I guess it's possible if we
keep track of the thread id which acquired the lock, and if we give
_whatever_after_fork() the thread id of the thread which initiated the
fork() in
On Mon, Feb 1, 2010 at 5:48 PM, Jesse Noller jnol...@gmail.com wrote:
Your reasonable argument is making it difficult for me to be irrational
about this.
No problem. :)
This begs the question - assuming a patch that clones the behavior of win32
for multiprocessing, would the default
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Reid Kleckner wrote:
On Mon, Feb 1, 2010 at 5:18 PM, Jesse Noller jnol...@gmail.com wrote:
I don't disagree there; but then again, I haven't seen this issue
arise (in my own code)/no bug reports/no test cases that show this to
be a consistent
Le lundi 01 février 2010 à 23:58 +0100, Martin v. Löwis a écrit :
Interestingly, the POSIX pthread_atfork documentation defines how you
are supposed to do that: create an atfork handler set, and acquire all
mutexes in the prepare handler. Then fork, and have the parent and child
handlers
On 01/02/2010 23:03, Reid Kleckner wrote:
On Mon, Feb 1, 2010 at 5:48 PM, Jesse Nollerjnol...@gmail.com wrote:
Your reasonable argument is making it difficult for me to be irrational
about this.
No problem. :)
This begs the question - assuming a patch that clones the behavior
Tres Seaver tseaver at palladion.com writes:
Yup, but that's true for *any* POSIXy envirnnment, not just Python. The
only sane non-exec mixture is to have a single-thread parent fork, and
restrict spawning threads to the children.
The problem is that we're advocating multiprocessing as the
On Feb 1, 2010, at 6:25 PM, Michael Foord fuzzy...@voidspace.org.uk
wrote:
On 01/02/2010 23:03, Reid Kleckner wrote:
On Mon, Feb 1, 2010 at 5:48 PM, Jesse Nollerjnol...@gmail.com
wrote:
Your reasonable argument is making it difficult for me to be
irrational
about this.
No
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Antoine Pitrou wrote:
Tres Seaver tseaver at palladion.com writes:
Yup, but that's true for *any* POSIXy envirnnment, not just Python. The
only sane non-exec mixture is to have a single-thread parent fork, and
restrict spawning threads to the
From: Stefan Behnel stefan...@behnel.de
To: python-dev@python.org
Subject: Re: [Python-Dev] Forking and Multithreading - enemy brothers
Message-ID: hk0k0q$7i...@ger.gmane.org
Content-Type: text/plain; charset=ISO-8859-15
Pascal Chambon, 29.01.2010 22:58:
I've just recently realized the huge
/[...]
What dangers do you refer to specifically? Something reproducible?
-L
/
Since it's a race condition issue, it's not easily reproducible with
normal libraries - which only take threading locks for small moments.
But it can appear if your threads make good use of the threading module.
/[...]
What dangers do you refer to specifically? Something reproducible?
-L
/
Since it's a race condition issue, it's not easily reproducible with
normal libraries - which only take threading locks for small moments.
But it can appear if your threads make good use of the threading module.
Hello,
I've just recently realized the huge problems surrounding the mix of
multithreading and fork() - i.e that only the main thread actually
survived the fork(), and that process data (in particular,
synchronization primitives) could be left in a dangerously broken state
because of such
Pascal Chambon, 29.01.2010 22:58:
I've just recently realized the huge problems surrounding the mix of
multithreading and fork() - i.e that only the main thread actually
survived the fork(), and that process data (in particular,
synchronization primitives) could be left in a dangerously broken
Stefan Behnel, 30.01.2010 07:36:
Pascal Chambon, 29.01.2010 22:58:
I've just recently realized the huge problems surrounding the mix of
multithreading and fork() - i.e that only the main thread actually
survived the fork(), and that process data (in particular,
synchronization primitives)
45 matches
Mail list logo
