Roundup Robot devn...@psf.upfronthosting.co.za added the comment:
New changeset 08d4c2fe51ea by Antoine Pitrou in branch 'default':
Issue #4892: multiprocessing Connections can now be transferred over
multiprocessing Connections.
http://hg.python.org/cpython/rev/08d4c2fe51ea
--
nosy:
Antoine Pitrou pit...@free.fr added the comment:
Thanks, Richard. I have now committed the patch. Hopefully the Windows
buildbots will be ok :)
--
resolution: - fixed
stage: patch review - committed/rejected
status: open - closed
___
Python tracker
sbt shibt...@gmail.com added the comment:
Up to date patch.
--
Added file: http://bugs.python.org/file25270/mp_pickle_conn.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4892
___
sbt shibt...@gmail.com added the comment:
A couple of minor changes based on Antoine's earlier review (which I did not
notice till now).
--
Added file: http://bugs.python.org/file25272/mp_pickle_conn.patch
___
Python tracker rep...@bugs.python.org
Antoine Pitrou pit...@free.fr added the comment:
Could you regenerate your patch now that the win32 - _winapi changes have been
applied?
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4892
Changes by Antoine Pitrou pit...@free.fr:
--
stage: needs patch - patch review
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4892
___
___
sbt shibt...@gmail.com added the comment:
The last patch did not work on Unix.
Here is a new version where the reduction functions are automatically
registered, so allow_connection_pickling() is redundant.
--
Added file: http://bugs.python.org/file25181/mp_pickle_conn.patch
sbt shibt...@gmail.com added the comment:
Updated patch which uses ForkingPickler in Connection.send().
Note that connection sharing still has to be enabled using
allow_connection_pickling().
Support could be enabled automatically, but that would introduce more circular
imports which confuse
Antoine Pitrou pit...@free.fr added the comment:
Support could be enabled automatically, but that would introduce more
circular imports which confuse me.
Are you sure? AFAICT:
- connection depends on forking
- reduction depends on forking and connection
But connection doesn't depend on
sbt shibt...@gmail.com added the comment:
But connection doesn't depend on reduction, neither does forking.
If registration of (Pipe)Connection is done in reduction then you can't make
(Pipe)Connection picklable *automatically* unless you make connection depend on
reduction (possibly
sbt shibt...@gmail.com added the comment:
There is an undocumented function multiprocessing.allow_connection_pickling()
whose docstring claims it allows connection and socket objects to be pickled.
The attached patch fixes the multiprocessing.reduction module so that it works
correctly. This
Antoine Pitrou pit...@free.fr added the comment:
Unless there's a technical barrier, I still think it would be better to use
ForkingPickler in multiprocessing.connection, rather than modify global state
(copyreg). The pickling support is multiprocessing-specific and wouldn't make
sense for
Changes by Kristján Valur Jónsson krist...@ccpgames.com:
--
nosy: +kristjan.jonsson
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4892
___
___
Kristján Valur Jónsson krist...@ccpgames.com added the comment:
I just want to point out that each time socket.share() is called, the resulting
data can only be used once by socket.fromshare(). I'm mentioning this because
I know there is some caching mechanism in reduction.py and that this
Antoine Pitrou pit...@free.fr added the comment:
err, is it possible to edit out those file paths?
I don't know how to do that. If you want I can remove the message altogether.
But I don't see anything confidential or exploitable in your message.
--
nosy: +pitrou
Changes by Antoine Pitrou pit...@free.fr:
--
assignee: jnoller -
nosy: +sbt
versions: +Python 3.3 -Python 3.2
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4892
___
sbt shibt...@gmail.com added the comment:
Jimbofbx wrote:
def main():
from multiprocessing import Pipe, reduction
i, o = Pipe()
print(i);
reduced = reduction.reduce_connection(i)
print(reduced);
newi = reduced[0](*reduced[1])
print(newi);
newi.send(hi)
Antoine Pitrou pit...@free.fr added the comment:
Having said all that I agree multiprocessing.reduction should be
fixed. Maybe an enable_pickling_support() function could be added to
register the necessary things with copyreg.
Why not simply use ForkingPickler?
--
sbt shibt...@gmail.com added the comment:
ForkingPickler is only used when creating a child process. The
multiprocessing.reduction module is only really intended for sending stuff to
*pre-existing* processes.
As things stand, after importing multiprocessing.reduction you can do something
Antoine Pitrou pit...@free.fr added the comment:
ForkingPickler is only used when creating a child process. The
multiprocessing.reduction module is only really intended for sending
stuff to *pre-existing* processes.
But ForkingPickler could be used in multiprocessing.connection, couldn't
James Hutchison jamesghutchi...@gmail.com added the comment:
@pitrou
You can just delete my original post. I'll repost an edited version here for
reference
original post with paths removed:
This is an issue for me (Python 3.2). I have a custom pool that sends arguments
for a function call
Changes by Antoine Pitrou pit...@free.fr:
--
Removed message: http://bugs.python.org/msg157702
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4892
___
sbt shibt...@gmail.com added the comment:
But ForkingPickler could be used in multiprocessing.connection,
couldn't it?
I suppose so.
Note that the way a connection handle is transferred between existing processes
is unnecessarily inefficient on Windows. A background server thread (one per
sbt shibt...@gmail.com added the comment:
There is a simpler way to do this on Windows. The sending process
duplicates the handle, and the receiving process duplicates that second
handle using DuplicateHandle() and the DUPLICATE_CLOSE_SOURCE flag. That
way no server thread is necessary
James Hutchison jamesghutchi...@gmail.com added the comment:
Shouldn't reduce_pipe_connection just be an alias for reduce_connection in unix
so that using reduce_pipe_connection would work for both win and unix? My
understanding after looking at the code is that reduce_pipe_connection isn't
James Hutchison jamesghutchi...@gmail.com added the comment:
This is an issue for me (Python 3.2). I have a custom pool that sends arguments
for a function call over a pipe. I cannot send another pipe as an argument.
Tim's workaround also does not work for me (win xp 32bit and 64bit)
From
James Hutchison jamesghutchi...@gmail.com added the comment:
err, is it possible to edit out those file paths? I didn't intend them to be in
the message. I'd appreciate it if someone with the privileges to do so could
remove them.
--
___
Python
Tim Alexander dragonfyr...@gmail.com added the comment:
Wanted to quickly comment here, as I'm dealing with this issue as well, that I
did find a workaround for avoiding it as far back as 2.6 (and it's not don't
pass a Pipe through a Pipe)
multiprocessing.reduction can already do this, though
Changes by Ask Solem a...@opera.com:
--
nosy: +asksol
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4892
___
___
Python-bugs-list mailing list
Changes by Terry J. Reedy tjre...@udel.edu:
--
versions: -Python 2.7, Python 3.1
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4892
___
___
Changes by Terry J. Reedy tjre...@udel.edu:
--
stage: - needs patch
versions: +Python 2.7, Python 3.1, Python 3.2 -Python 2.6
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4892
___
Changes by Stein Magnus Jodal stein.mag...@jodal.no:
--
nosy: +jodal
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4892
___
___
Python-bugs-list
Daniel Svensson dsvens...@gmail.com added the comment:
A typical use case would be for a server to receive a connection, and
then send that connection over to another process that does the actual
work. This used to work with pyprocessing, and the support seems to be
available in
Daniel Svensson dsvens...@gmail.com added the comment:
And to be clear, I have enabled connection pickling by issuing:
multiprocessing.allow_connection_pickling()
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4892
Daniel Svensson dsvens...@gmail.com added the comment:
When reverting this commit stuff works:
http://svn.python.org/view/python/trunk/Lib/multiprocessing/reduction.py?r1=64257r2=65016
--
___
Python tracker rep...@bugs.python.org
Daniel Svensson dsvens...@gmail.com added the comment:
Ehm.. completly broken url in prev message.. Revision 65016, Apply
Amaury's patch to multiprocessing for issue 3125, removes the copy_reg
and replaces it with ForkingPickler.register(), which should resolve the
conflict with the global
Changes by Ronald Oussoren ronaldousso...@mac.com:
--
components: -Macintosh
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4892
___
___
Jesse Noller jnol...@gmail.com added the comment:
Before I can logically support this, I need a clear use case that supports
the idea that this should be supported in the current version of
multiprocessing.
--
priority: normal - low
___
Python
Changes by Jesse Noller jnol...@gmail.com:
--
priority: - normal
type: behavior - feature request
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4892
___
Henrik Gustafsson henrik.gustafs...@purplescout.se added the comment:
$ cat pipetest2.py
try:
from multiprocessing import Pipe
except ImportError:
from processing import Pipe
c1, c2 = Pipe(duplex=False)
c2.send('asdf')
print c1.recv()
c2.send(c1)
print c1.recv()
Jesse Noller jnol...@gmail.com added the comment:
thanks for filing this. I'll need to compare the two code bases and figure
out why it's either regressed, or Richard removed it prior to the
integration.
___
Python tracker rep...@bugs.python.org
41 matches
Mail list logo
