[issue28906] Can't inherit sockets with multiprocessing on Windows

[Eryk Sun]

Changes by Eryk Sun :


--
stage:  -> resolved

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28906] Can't inherit sockets with multiprocessing on Windows

[Preston Landers]

Preston Landers added the comment:

Wow, good call. That does work for me. Wish I had thought to try it. I assume 
you want me to go ahead and close the issue. Sorry for the noise, but this 
really helps!

--
resolution:  -> not a bug
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28906] Can't inherit sockets with multiprocessing on Windows

[Eryk Sun]

Changes by Eryk Sun :


--
components: +Windows
nosy: +paul.moore, steve.dower, tim.golden, zach.ware

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28906] Can't inherit sockets with multiprocessing on Windows

[Eryk Sun]

Eryk Sun added the comment:

You should be able to directly pass the socket to the child process. 
multiprocessing registers a reduction for this. On Windows it uses the 
DupSocket class from multiprocessing.resource_sharer:

class DupSocket(object):
'''Picklable wrapper for a socket.'''
def __init__(self, sock):
new_sock = sock.dup()
def send(conn, pid):
share = new_sock.share(pid)
conn.send_bytes(share)
self._id = _resource_sharer.register(send, new_sock.close)

def detach(self):
'''Get the socket.  This should only be called once.'''
with _resource_sharer.get_connection(self._id) as conn:
share = conn.recv_bytes()
return socket.fromshare(share)

This calls the Windows socket share() method [1], which calls 
WSADuplicateSocket [2]. In the child, socket.fromshare() passes the protocol 
info buffer to the socket constructor, which passes it to WSASocket [3].

[1]: https://docs.python.org/3/library/socket.html#socket.socket.share
[2]: https://msdn.microsoft.com/en-us/library/ms741565
[3]: https://msdn.microsoft.com/en-us/library/ms742212

--
nosy: +eryksun

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28906] Can't inherit sockets with multiprocessing on Windows

[Preston Landers]

New submission from Preston Landers:

I'm porting a Python 2.6 based application to Python 3.6. This app uses a 
customized version of the "flup" package to do FastCGI services on Windows 
using the multiprocessing package. This requires a few sockets to be inherited 
across processes - the main FastCGI protocol socket plus a control socket.

In Python 2.6, the `socket.from_fd` function was not available on Windows. 
However I patched Python's socketmodule.c to provide that function using 
DuplicateHandle. In Python 2.6's version of multiprocessing it spawned a 
process with CreateProcess and bInheritHandles=True. This worked well for me.

Now I'm trying to get this working after moving from Python 2.6 to 3.6 
(currently using 3.6.0b4). Fortunately, the socket module now has a working 
`socket.from_fd` on Windows so I no longer have to patch that. Unfortunately 
for me, though, the multiprocessing module now calls CreateProcess with 
bInheritHandles=False. This causes my scenario to fail.

Here's a short test script which illustrates this problem:
https://gist.github.com/Preston-Landers/712fee10fb557cf0b5592b57561a7c08

If you run with an unpatched multiprocessing, it will fail with an error like:

OSError: [WinError 10038] An operation was attempted on something that is not a 
socket

If you patch multiprocessing to set bInheritHandles=True this now works. 
(Change is in popen_spawn_win32.py where it does _winapi.CreateProcess.)

I'm sure there's a good reason for that change in multiprocessing, whether for 
security or for unrelated/undesired file handles being passed.
https://www.python.org/dev/peps/pep-0446/#inheritance-of-file-descriptors-on-windows

However it does break my scenario and I don't see a way to tell multiprocessing 
to allow certain handles to be inherited. The docs for multiprocessing say "In 
particular, unnecessary file descriptors and handles from the parent process 
will not be inherited." It would be nice to have a way to tell it that my 
sockets are "necessary." You would think that calling 
socket.set_inheritable(True) would do it. In fact you must do that, but you 
must also pass bInheritHandles=True to CreateProcess for it to actually work. 
There doesn't seem to be a way to pass through an argument to multiprocessing 
to tell it to set this flag.

I do realize I could be going about this completely wrong, though. But right 
now it looks like my immediate options are:

a) Go ahead and patch my copy of popen_spawn_win32.py to allow inherited 
handles despite other possible risks.

b) Try to rewrite things to not use multiprocessing at all and directly spawn 
my processes instead. That's not attractive because multiprocessing otherwise 
does what I need to do.

Are there any other options I'm missing? Maybe some way to duplicate the socket 
on the other end without relying on CreateProcess with bInheritHandles=True?

Otherwise, I guess I'm asking for an option to be made available in 
multiprocessing to allow handles to be inherited on Windows.

--
components: Library (Lib)
messages: 282723
nosy: planders
priority: normal
severity: normal
status: open
title: Can't inherit sockets with multiprocessing on Windows
type: behavior
versions: Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com