[issue4892] Sending Connection-objects over multiprocessing connections fails

Antoine Pitrou added the comment: Thanks, Richard. I have now committed the patch. Hopefully the Windows buildbots will be ok :) -- resolution: -> fixed stage: patch review -> committed/rejected status: open -> closed ___ Python tracker

[issue4892] Sending Connection-objects over multiprocessing connections fails

Roundup Robot added the comment: New changeset 08d4c2fe51ea by Antoine Pitrou in branch 'default': Issue #4892: multiprocessing Connections can now be transferred over multiprocessing Connections. http://hg.python.org/cpython/rev/08d4c2fe51ea -- nosy: +python-dev _

[issue4892] Sending Connection-objects over multiprocessing connections fails

sbt added the comment: A couple of minor changes based on Antoine's earlier review (which I did not notice till now). -- Added file: http://bugs.python.org/file25272/mp_pickle_conn.patch ___ Python tracker ___

[issue4892] Sending Connection-objects over multiprocessing connections fails

sbt added the comment: Up to date patch. -- Added file: http://bugs.python.org/file25270/mp_pickle_conn.patch ___ Python tracker ___ _

[issue4892] Sending Connection-objects over multiprocessing connections fails

Changes by Antoine Pitrou : -- stage: needs patch -> patch review ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue4892] Sending Connection-objects over multiprocessing connections fails

Antoine Pitrou added the comment: Could you regenerate your patch now that the win32 -> _winapi changes have been applied? -- ___ Python tracker ___

[issue4892] Sending Connection-objects over multiprocessing connections fails

sbt added the comment: The last patch did not work on Unix. Here is a new version where the reduction functions are automatically registered, so allow_connection_pickling() is redundant. -- Added file: http://bugs.python.org/file25181/mp_pickle_conn.patch

[issue4892] Sending Connection-objects over multiprocessing connections fails

sbt added the comment: > But connection doesn't depend on reduction, neither does forking. If registration of (Pipe)Connection is done in reduction then you can't make (Pipe)Connection picklable *automatically* unless you make connection depend on reduction (possibly indirectly). A circular

[issue4892] Sending Connection-objects over multiprocessing connections fails

Antoine Pitrou added the comment: > Support could be enabled automatically, but that would introduce more > circular imports which confuse me. Are you sure? AFAICT: - connection depends on forking - reduction depends on forking and connection But connection doesn't depend on reduction, neither

[issue4892] Sending Connection-objects over multiprocessing connections fails

sbt added the comment: Updated patch which uses ForkingPickler in Connection.send(). Note that connection sharing still has to be enabled using allow_connection_pickling(). Support could be enabled automatically, but that would introduce more circular imports which confuse me. It might be w

[issue4892] Sending Connection-objects over multiprocessing connections fails

Kristján Valur Jónsson added the comment: I just want to point out that each time socket.share() is called, the resulting data can only be used once by socket.fromshare(). I'm mentioning this because I know there is some caching mechanism in reduction.py and that this data is not cacheable/r

[issue4892] Sending Connection-objects over multiprocessing connections fails

Changes by Kristján Valur Jónsson : -- nosy: +kristjan.jonsson ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: h

[issue4892] Sending Connection-objects over multiprocessing connections fails

Antoine Pitrou added the comment: Unless there's a technical barrier, I still think it would be better to use ForkingPickler in multiprocessing.connection, rather than modify global state (copyreg). The pickling support is multiprocessing-specific and wouldn't make sense for other pickles (e.

[issue4892] Sending Connection-objects over multiprocessing connections fails

sbt added the comment: There is an undocumented function multiprocessing.allow_connection_pickling() whose docstring claims it allows connection and socket objects to be pickled. The attached patch fixes the multiprocessing.reduction module so that it works correctly. This means that TestPic

[issue4892] Sending Connection-objects over multiprocessing connections fails

James Hutchison added the comment: Shouldn't reduce_pipe_connection just be an alias for reduce_connection in unix so that using reduce_pipe_connection would work for both win and unix? My understanding after looking at the code is that reduce_pipe_connection isn't defined for non-win32, alth

[issue4892] Sending Connection-objects over multiprocessing connections fails

sbt added the comment: > There is a simpler way to do this on Windows. The sending process > duplicates the handle, and the receiving process duplicates that second > handle using DuplicateHandle() and the DUPLICATE_CLOSE_SOURCE flag. That > way no server thread is necessary on Windows. No

[issue4892] Sending Connection-objects over multiprocessing connections fails

sbt added the comment: > But ForkingPickler could be used in multiprocessing.connection, > couldn't it? I suppose so. Note that the way a connection handle is transferred between existing processes is unnecessarily inefficient on Windows. A background server thread (one per process) has to

[issue4892] Sending Connection-objects over multiprocessing connections fails

Changes by Antoine Pitrou : -- Removed message: http://bugs.python.org/msg157702 ___ Python tracker ___ ___ Python-bugs-list mailing li

[issue4892] Sending Connection-objects over multiprocessing connections fails

James Hutchison added the comment: @pitrou You can just delete my original post. I'll repost an edited version here for reference original post with paths removed: This is an issue for me (Python 3.2). I have a custom pool that sends arguments for a function call over a pipe. I cannot send a

[issue4892] Sending Connection-objects over multiprocessing connections fails

Antoine Pitrou added the comment: > ForkingPickler is only used when creating a child process. The > multiprocessing.reduction module is only really intended for sending > stuff to *pre-existing* processes. But ForkingPickler could be used in multiprocessing.connection, couldn't it? -

[issue4892] Sending Connection-objects over multiprocessing connections fails

sbt added the comment: ForkingPickler is only used when creating a child process. The multiprocessing.reduction module is only really intended for sending stuff to *pre-existing* processes. As things stand, after importing multiprocessing.reduction you can do something like buf = io.Byte

[issue4892] Sending Connection-objects over multiprocessing connections fails

Antoine Pitrou added the comment: > Having said all that I agree multiprocessing.reduction should be > fixed. Maybe an enable_pickling_support() function could be added to > register the necessary things with copyreg. Why not simply use ForkingPickler? --

[issue4892] Sending Connection-objects over multiprocessing connections fails

sbt added the comment: Jimbofbx wrote: > def main(): > from multiprocessing import Pipe, reduction > i, o = Pipe() > print(i); > reduced = reduction.reduce_connection(i) > print(reduced); > newi = reduced[0](*reduced[1]) > print(newi); > newi.send("hi") > o.re

[issue4892] Sending Connection-objects over multiprocessing connections fails

Changes by Antoine Pitrou : -- assignee: jnoller -> nosy: +sbt versions: +Python 3.3 -Python 3.2 ___ Python tracker ___ ___ Python-bug

[issue4892] Sending Connection-objects over multiprocessing connections fails

Antoine Pitrou added the comment: > err, is it possible to edit out those file paths? I don't know how to do that. If you want I can remove the message altogether. But I don't see anything confidential or exploitable in your message. -- nosy: +pitrou _

[issue4892] Sending Connection-objects over multiprocessing connections fails

James Hutchison added the comment: err, is it possible to edit out those file paths? I didn't intend them to be in the message. I'd appreciate it if someone with the privileges to do so could remove them. -- ___ Python tracker

[issue4892] Sending Connection-objects over multiprocessing connections fails

James Hutchison added the comment: This is an issue for me (Python 3.2). I have a custom pool that sends arguments for a function call over a pipe. I cannot send another pipe as an argument. Tim's workaround also does not work for me (win xp 32bit and 64bit) >From what I can tell, you can on

[issue4892] Sending Connection-objects over multiprocessing connections fails

Tim Alexander added the comment: Wanted to quickly comment here, as I'm dealing with this issue as well, that I did find a workaround for avoiding it as far back as 2.6 (and it's not "don't pass a Pipe through a Pipe") multiprocessing.reduction can already do this, though I don't entirely kno

[issue4892] Sending Connection-objects over multiprocessing connections fails

Changes by Ask Solem : -- nosy: +asksol ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/m

[issue4892] Sending Connection-objects over multiprocessing connections fails

Changes by Terry J. Reedy : -- versions: -Python 2.7, Python 3.1 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue4892] Sending Connection-objects over multiprocessing connections fails

Changes by Terry J. Reedy : -- stage: -> needs patch versions: +Python 2.7, Python 3.1, Python 3.2 -Python 2.6 ___ Python tracker ___

[issue4892] Sending Connection-objects over multiprocessing connections fails

Changes by Stein Magnus Jodal : -- nosy: +jodal ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.pyth

[issue4892] Sending Connection-objects over multiprocessing connections fails

Daniel Svensson added the comment: Ehm.. completly broken url in prev message.. Revision 65016, "Apply Amaury's patch to multiprocessing for issue 3125, removes the copy_reg and replaces it with ForkingPickler.register(), which should resolve the conflict with the global registry/ctypes" is what

[issue4892] Sending Connection-objects over multiprocessing connections fails

Daniel Svensson added the comment: When reverting this commit stuff works: http://svn.python.org/view/python/trunk/Lib/multiprocessing/reduction.py?r1=64257&r2=65016 -- ___ Python tracker _

[issue4892] Sending Connection-objects over multiprocessing connections fails

Daniel Svensson added the comment: And to be clear, I have enabled connection pickling by issuing: multiprocessing.allow_connection_pickling() -- ___ Python tracker ___

[issue4892] Sending Connection-objects over multiprocessing connections fails

Daniel Svensson added the comment: A typical use case would be for a server to receive a connection, and then send that connection over to another process that does the actual work. This used to work with pyprocessing, and the support seems to be available in multiprocessing.c -> multiprocessing

[issue4892] Sending Connection-objects over multiprocessing connections fails

Jesse Noller added the comment: Before I can logically support this, I need a clear use case that supports the idea that this should be supported in the current version of multiprocessing. -- priority: normal -> low ___ Python tracker

[issue4892] Sending Connection-objects over multiprocessing connections fails

Changes by Ronald Oussoren : -- components: -Macintosh ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://m

[issue4892] Sending Connection-objects over multiprocessing connections fails

Changes by Jesse Noller : -- priority: -> normal type: behavior -> feature request ___ Python tracker ___ ___ Python-bugs-list mailing

[issue4892] Sending Connection-objects over multiprocessing connections fails

Jesse Noller added the comment: thanks for filing this. I'll need to compare the two code bases and figure out why it's either regressed, or Richard removed it prior to the integration. ___ Python tracker

[issue4892] Sending Connection-objects over multiprocessing connections fails

Changes by Jesse Noller : -- assignee: -> jnoller nosy: +jnoller ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue4892] Sending Connection-objects over multiprocessing connections fails

Henrik Gustafsson added the comment: $ cat pipetest2.py try: from multiprocessing import Pipe except ImportError: from processing import Pipe c1, c2 = Pipe(duplex=False) c2.send('asdf') print c1.recv() c2.send(c1) print c1.recv() ___ Python

[issue4892] Sending Connection-objects over multiprocessing connections fails

New submission from Henrik Gustafsson : It seems the old pyprocessing (http://pyprocessing.berlios.de/) can do some things that the new multiprocessing package can not; sending/receiving connection objects for one. This is a quite handy functionality, so it would be nice if it were reintroduc