[issue14310] Socket duplication for windows
sbt shibt...@gmail.com added the comment: Can this issue be reclosed now? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14310 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14310] Socket duplication for windows
Changes by Antoine Pitrou pit...@free.fr: -- status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14310 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14310] Socket duplication for windows
Antoine Pitrou pit...@free.fr added the comment: Any other comments? No, the patch looks ok now. Please watch the buildbots after you commit. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14310 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14310] Socket duplication for windows
Roundup Robot devn...@psf.upfronthosting.co.za added the comment: New changeset 51b4bddd0e92 by Kristján Valur Jónsson in branch 'default': Issue #14310: inter-process socket duplication for windows http://hg.python.org/cpython/rev/51b4bddd0e92 -- nosy: +python-dev ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14310 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14310] Socket duplication for windows
Changes by Kristján Valur Jónsson krist...@ccpgames.com: -- resolution: - fixed status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14310 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14310] Socket duplication for windows
Antoine Pitrou pit...@free.fr added the comment: Re-opening. There is now a buildbot failure: == ERROR: testTypes (test.test_socket.TestSocketSharing) -- Traceback (most recent call last): File D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\test\test_socket.py, line 4738, in testTypes source = socket.socket(f, t) File D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\socket.py, line 94, in __init__ _socket.socket.__init__(self, family, type, proto, fileno) OSError: [Error 10047] An address incompatible with the requested protocol was used http://www.python.org/dev/buildbot/all/builders/x86%20XP-4%203.x/ -- assignee: - krisvale stage: - committed/rejected status: closed - open ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14310 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14310] Socket duplication for windows
Roundup Robot devn...@psf.upfronthosting.co.za added the comment: New changeset 9b858096044e by Kristján Valur Jónsson in branch 'default': Issue #14310: Catch testing errors when trying to create unsupported socket http://hg.python.org/cpython/rev/9b858096044e -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14310 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14310] Socket duplication for windows
Changes by Kristján Valur Jónsson krist...@ccpgames.com: -- Removed message: http://bugs.python.org/msg157189 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14310 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14310] Socket duplication for windows
Kristján Valur Jónsson krist...@ccpgames.com added the comment: Not much happening on python-dev. Perhaps this is not so controversial. http://www.mail-archive.com/python-dev@python.org/msg66206.html -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14310 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14310] Socket duplication for windows
Antoine Pitrou pit...@free.fr added the comment: I have one question: you're simply doing a binary copy of the WSAPROTOCOL_INFO structure. Does it mean sharing wouldn't work between e.g. different MSVC builds of Python (if the structure is different), or between a 32-bit and a 64-bit Python? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14310 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14310] Socket duplication for windows
Antoine Pitrou pit...@free.fr added the comment: Comments about the patch: - versionadded should be 3.3, not 3.4 - MultiprocessingFunc would deserve a meaningful, PEP8 name; it should also probably be a classmethod on TestSocketSharing - please use PEP8 for comments (a space after the #) - it would be nice to have a test for the ValueError when passing a bytes object of the wrong size - also, there should be a test that the family, type and proto attributes are correctly set on the fromshare() result (perhaps timeout too) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14310 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14310] Socket duplication for windows
Kristján Valur Jónsson krist...@ccpgames.com added the comment: The binary copy: This is a winsock2 feature. There is no difference in layout between 32 bits and 64 bits. I also don't think there is a difference between SDK versions. However, strangely, there is a difference between unicode and non-unicode builds. But since we are using this in python3, which always uses the same unicode setting, this shouldn't be an issue. I'll take your other comments into account and produce a new patch. Similar to socket.dup(), the socket shared is assumed to be in blockin mode (this his how its timeout is initialized). We can test for that too. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14310 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14310] Socket duplication for windows
Kristján Valur Jónsson krist...@ccpgames.com added the comment: Here is a new patch, with more tests. Note that the process worker function can't be a member function because of how multiprocessing works on Windows. -- Added file: http://bugs.python.org/file25143/duplicate.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14310 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14310] Socket duplication for windows
Antoine Pitrou pit...@free.fr added the comment: Here is a new patch, with more tests. Note that the process worker function can't be a member function because of how multiprocessing works on Windows. Some comments: - I said classmethod, not member function; this is how test_multiprocessing works, so it should be possible... - why did you change the gethostbyname() tests? - before using AF_INET6, you might have to test that IPv6 is available on the test machine (I think there are variables / functions for that in test_socket) - in compareSockets: +if org.proto != 0: +self.assertEqual(sock.proto, other.proto) `sock` doesn't seem to exist at all... -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14310 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14310] Socket duplication for windows
Kristján Valur Jónsson krist...@ccpgames.com added the comment: by memberfunction I mean a member of the class. I tried staticmethod and it didn't work: I tried a staticmethod and it didn't work: == ERROR: testShare (test.test_socket.TestSocketSharing) -- Traceback (most recent call last): File D:\pydev\cpython\lib\test\test_socket.py, line 4723, in testShare p.start() File D:\pydev\cpython\lib\multiprocessing\process.py, line 136, in start self._popen = Popen(self) File D:\pydev\cpython\lib\multiprocessing\forking.py, line 269, in __init__ dump(process_obj, to_child, HIGHEST_PROTOCOL) File D:\pydev\cpython\lib\multiprocessing\forking.py, line 190, in dump ForkingPickler(file, protocol).dump(obj) _pickle.PicklingError: Can't pickle class 'function': attribute lookup builtins.function failed (Picking this kind of stuff should work of course but that's another story) But changing it to staticmethod strangely fixes it. Interesting. the gethostbynametests were an encoding mistake. Would not have been checked in but it was nice of you to notice. These are windows only tests for python 3.3. There is no need to test for IPV6. Here is yet another patch. -- Added file: http://bugs.python.org/file25145/duplicate.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14310 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14310] Socket duplication for windows
Antoine Pitrou pit...@free.fr added the comment: [...] But changing it to staticmethod strangely fixes it. Interesting. Sorry? Here is yet another patch. Your patch is broken, it has lots of duplicate stuff... -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14310 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14310] Socket duplication for windows
Changes by Kristján Valur Jónsson krist...@ccpgames.com: Removed file: http://bugs.python.org/file25145/duplicate.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14310 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14310] Socket duplication for windows
Changes by Kristján Valur Jónsson krist...@ccpgames.com: Removed file: http://bugs.python.org/file25143/duplicate.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14310 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14310] Socket duplication for windows
Kristján Valur Jónsson krist...@ccpgames.com added the comment: But changing it to staticmethod strangely fixes it. Interesting. Sorry? Don't be, probably not your fault. Today we have learned that we can spawn multipropcessing targets with classmethods, but not staticmethods. Your patch is broken, it has lots of duplicate stuff... Yes, I'm sorry, it was a merging error. I'm still getting to grips with Hg. I removed the last two and added a new one. Hope you like it. -- Added file: http://bugs.python.org/file25146/duplicate.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14310 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14310] Socket duplication for windows
Antoine Pitrou pit...@free.fr added the comment: Yes, I'm sorry, it was a merging error. I'm still getting to grips with Hg. I removed the last two and added a new one. Hope you like it. It's still broken. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14310 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14310] Socket duplication for windows
Kristján Valur Jónsson krist...@ccpgames.com added the comment: Could you possibly be any more specific? It works for me. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14310 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14310] Socket duplication for windows
Antoine Pitrou pit...@free.fr added the comment: Could you possibly be any more specific? It works for me. Just read it by yourself. There are still duplicate portions there: http://bugs.python.org/file25146/duplicate.patch In general, it's nicer to others to review your changes before uploading them. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14310 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14310] Socket duplication for windows
Changes by Kristján Valur Jónsson krist...@ccpgames.com: Removed file: http://bugs.python.org/file25146/duplicate.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14310 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14310] Socket duplication for windows
Kristján Valur Jónsson krist...@ccpgames.com added the comment: Well, I think I have all of the merge errors out now. You´ll forgive me if I didn't notice them all at first, but this is why we review code and you would have helped me by giving me specifics, since after a time, the eyes tend to glaze over. Any other comments? -- Added file: http://bugs.python.org/file25149/duplicate.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14310 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14310] Socket duplication for windows
Changes by Santoso Wijaya santoso.wij...@gmail.com: -- nosy: +santa4nt ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14310 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14310] Socket duplication for windows
Kristján Valur Jónsson krist...@ccpgames.com added the comment: Still no responses? I'll submit this then, unless someone objects. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14310 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14310] Socket duplication for windows
Antoine Pitrou pit...@free.fr added the comment: I would like to see this discussed on python-dev. -- nosy: +pitrou ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14310 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14310] Socket duplication for windows
Kristján Valur Jónsson krist...@ccpgames.com added the comment: You are right, I was thinking more of pyobject attributes. I'll fix this then. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14310 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14310] Socket duplication for windows
Kristján Valur Jónsson krist...@ccpgames.com added the comment: So, what is the consensus, does this go in or out? What about the share() and fromshare() functions, are people ok with that? -- versions: +Python 3.3 -Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14310 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14310] Socket duplication for windows
Kristján Valur Jónsson krist...@ccpgames.com added the comment: A new patch with fixes and documentation. And yes, socket.share() does in fact perform duplication, because socket.close() can be performed immediately afterwards. -- Added file: http://bugs.python.org/file24965/duplicate.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14310 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14310] Socket duplication for windows
Kristján Valur Jónsson krist...@ccpgames.com added the comment: hm, patch doesn't work, removing the [mq] section -- Added file: http://bugs.python.org/file24966/duplicate.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14310 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14310] Socket duplication for windows
Changes by Kristján Valur Jónsson krist...@ccpgames.com: Removed file: http://bugs.python.org/file24966/duplicate.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14310 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14310] Socket duplication for windows
Changes by Kristján Valur Jónsson krist...@ccpgames.com: -- Removed message: http://bugs.python.org/msg156418 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14310 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14310] Socket duplication for windows
Kristján Valur Jónsson krist...@ccpgames.com added the comment: I'm adding an updated version of the patch, taking into account issues raised from reviewers. No documentation yet. -- Added file: http://bugs.python.org/file24940/duplicate.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14310 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14310] Socket duplication for windows
Kristján Valur Jónsson krist...@ccpgames.com added the comment: I like the idea of having a frominfo() function. Though info is maybe too generic? Also, dupinfo() is probably better than duplicate, even though there is no dup involved. But I also wonder if we can make this windows/unix agnostic? on unix, you would achieve the same thing by: 1) getting fd with fileno() 2) sending it over a unix domain socket using sendmsg() (used for duplicating fds betwerrn processes 3) calling fromfd() on the target host. We could have a uniform process (more or less) if dupinfo(pid) would return a string containing the fileno on linux, and frominfo() were to accept such a string too. The differeince between linux/windows would then be that linux needs to transfer the info between processes using the magic sendmsg() control info ona UNIX domain sock, whereas on Windows one can use whatever RPC one wishes. To implement this, we could therefore have special sendinfo and recvinfo functions. The process would then become, on either platform: 1) info = dupinfo(pid) 2) sendinfo(info)/info=recvinfo() 3) frominfo(info) But again: info is very generic here. any suggestions for a better name? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14310 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14310] Socket duplication for windows
Kristján Valur Jónsson krist...@ccpgames.com added the comment: On second thought, sendinfo/recvinfo makes no sense. On unix, one has to create and manage the necessary unix domain sockets, and such code belongs higher up somewhere. So dupinfo(pid), and frominfo() I'll add. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14310 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14310] Socket duplication for windows
Kristján Valur Jónsson krist...@ccpgames.com added the comment: Here is a new version of thepatch. We now have socket.socket.share(pid) and socket.fromshare(data) to do these. I think this captures the functionality better than duplicate or duppid() since there is no actual duplication involved until the fromshare() function is called. Still no documentation. What do you think? -- Added file: http://bugs.python.org/file24944/duplicate.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14310 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14310] Socket duplication for windows
sbt shibt...@gmail.com added the comment: I think this captures the functionality better than duplicate or duppid() since there is no actual duplication involved until the fromshare() function is called. Are you saying the WSADuplicateSocket() call in share() doesn't duplicate the handle in to the target process? I am pretty sure it does. (Delaying handle duplication till WSASocket() is called in the target process would be rather problematic since then you cannot close the original socket in the source processes until you know the duplication has occurred.) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14310 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14310] Socket duplication for windows
Kristján Valur Jónsson krist...@ccpgames.com added the comment: Good point. The docs don't actually say, except that the docs show a pretty elaborate process dance, where the source process only closes its side of the socket once the target process has called WSASocket(). If duplication happened early, then there would have to be a way to unduplicate it in the source process if, say, IPC somehow failed. There is currently no api to undo the effects of WSADuplicateSocket(). It's easy enough to test, of course. I´ll do some test using my unittest. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14310 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14310] Socket duplication for windows
sbt shibt...@gmail.com added the comment: If duplication happened early, then there would have to be a way to unduplicate it in the source process if, say, IPC somehow failed. There is currently no api to undo the effects of WSADuplicateSocket(). If this were a normal handle then you could use the DUPLICATE_CLOSE_SOURCE flag with DuplicateHandle() to close it. But using DuplicateHandle() with socket handles is discouraged. I find the ability to duplicate and close handles in unrelated processes of the same user rather surprising. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14310 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14310] Socket duplication for windows
Changes by Giampaolo Rodola' g.rod...@gmail.com: -- nosy: +giampaolo.rodola ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14310 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14310] Socket duplication for windows
sbt shibt...@gmail.com added the comment: It appears that the 4th argument of the socket constructor is undocumented, so presumably one is expected to use fromfd() instead. Maybe you could have a frominfo(info) function (to match fromfd(fd,...)) and a dupinfo(pid) method. (It appears that multiprocessing uses DuplicateHandle() instead of WSADuplicateSocket() for duplicating socket handles on Windows. That should be fixed.) -- nosy: +sbt ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14310 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14310] Socket duplication for windows
New submission from Kristján Valur Jónsson krist...@ccpgames.com: winsock natively supports the duplciation of sockets for use on different processes. This patch proposes to add that functionality: socket.duplicate(target_pid) gets a bytes representation of the duplicate socket, usable for the target process. socket.socket(x,y,z,data) creates the socket object from the bytes representation. The patch contains a test using multiprocessing that works. Note that multiprocessing already contains its own code to achieve this, and that code can possibly be simplified with this patch. Note also the new function duplicate. Perhaps dup(target_pid) would be better? But it would have different semantics. Also notice how we overload the fromfd parameter in socket.socket() to recreate a socket from a bytes representation. Maybe this is not ideal? Looking for thoughs here. -- components: Interpreter Core files: duplicate.patch keywords: patch messages: 155842 nosy: krisvale priority: normal severity: normal status: open title: Socket duplication for windows type: enhancement versions: Python 3.4 Added file: http://bugs.python.org/file24854/duplicate.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14310 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
