[issue10527] multiprocessing.Pipe problem: handle out of range in select()
Changes by Richard Oudkerk shibt...@gmail.com: -- resolution: - fixed stage: commit review - committed/rejected status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10527 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10527] multiprocessing.Pipe problem: handle out of range in select()
Roundup Robot added the comment: New changeset f07435fa6736 by Richard Oudkerk in branch '2.7': Issue #10527: Remove dead code http://hg.python.org/cpython/rev/f07435fa6736 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10527 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10527] multiprocessing.Pipe problem: handle out of range in select()
Roundup Robot added the comment: New changeset 49d45151b9ed by Richard Oudkerk in branch '3.2': Issue #10527: Remove dead code http://hg.python.org/cpython/rev/49d45151b9ed -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10527 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10527] multiprocessing.Pipe problem: handle out of range in select()
Roundup Robot added the comment: New changeset da5e520a7ba5 by Richard Oudkerk in branch '2.7': Issue #10527: Use poll() instead of select() for multiprocessing pipes http://hg.python.org/cpython/rev/da5e520a7ba5 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10527 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10527] multiprocessing.Pipe problem: handle out of range in select()
Roundup Robot added the comment: New changeset abf111b9a464 by Richard Oudkerk in branch '3.2': Issue #10527: Use poll() instead of select() for multiprocessing pipes http://hg.python.org/cpython/rev/abf111b9a464 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10527 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10527] multiprocessing.Pipe problem: handle out of range in select()
Richard Oudkerk added the comment: The commits did not have the intended effect. They just define a _poll() function (and only on Windows) and it is not referenced anywhere else. I will look in to fixing this -- on 2.7 and 3.2 this will need to be done in the C code. -- resolution: fixed - status: closed - open ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10527 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10527] multiprocessing.Pipe problem: handle out of range in select()
Giampaolo Rodola' added the comment: What do you mean? The intent was to use poll() instead of select() anywhere available in order to avoid running out of fds. The change didn't affect Windows because as of right now select() is the only thing available. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10527 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10527] multiprocessing.Pipe problem: handle out of range in select()
Richard Oudkerk added the comment: What do you mean? The intent was to use poll() instead of select() anywhere available in order to avoid running out of fds. The change didn't affect Windows because as of right now select() is the only thing available. The change *only* effects Windows. Currently the code goes if sys.platform != 'win32': ... else: if hasattr(select, 'poll'): def _poll(fds, timeout): ... else: def _poll(fds, timeout): ... So _poll() is only defined when sys.platform == 'win32'. Furthermore, the _poll() function is never used anywhere: ConnectionBase.poll() uses Connection._poll(), which uses wait(), which uses select(). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10527 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10527] multiprocessing.Pipe problem: handle out of range in select()
Richard Oudkerk added the comment:
It looks like the change to multiprocessing/connection.py committed does not
match the one uploaded as issue10527-3.patch
changeset 81174:e971a70984b8
1.1 --- a/Lib/multiprocessing/connection.py
1.2 +++ b/Lib/multiprocessing/connection.py
1.3 @@ -509,6 +509,27 @@ if sys.platform != 'win32':
1.4 return c1, c2
1.5
1.6 else:
1.7 +if hasattr(select, 'poll'):
1.8 +def _poll(fds, timeout):
1.9 +if timeout is not None:
1.10 +timeout = int(timeout) * 1000 # timeout is in
milliseconds
1.11 +fd_map = {}
1.12 +pollster = select.poll()
1.13 +for fd in fds:
1.14 +pollster.register(fd, select.POLLIN)
1.15 +if hasattr(fd, 'fileno'):
1.16 +fd_map[fd.fileno()] = fd
1.17 +else:
1.18 +fd_map[fd] = fd
1.19 +ls = []
1.20 +for fd, event in pollster.poll(timeout):
1.21 +if event select.POLLNVAL:
1.22 +raise ValueError('invalid file descriptor %i' %
fd)
1.23 +ls.append(fd_map[fd])
1.24 +return ls
1.25 +else:
1.26 +def _poll(fds, timeout):
1.27 +return select.select(fds, [], [], timeout)[0]
1.28
1.29 def Pipe(duplex=True):
1.30 '''
issue10527-3.patch:
diff --git a/Lib/multiprocessing/connection.py
b/Lib/multiprocessing/connection.py
--- a/Lib/multiprocessing/connection.py
+++ b/Lib/multiprocessing/connection.py
@@ -861,6 +861,27 @@
return [o for o in object_list if o in ready_objects]
else:
+if hasattr(select, 'poll'):
+def _poll(fds, timeout):
+if timeout is not None:
+timeout = int(timeout) * 1000 # timeout is in milliseconds
+fd_map = {}
+pollster = select.poll()
+for fd in fds:
+pollster.register(fd, select.POLLIN)
+if hasattr(fd, 'fileno'):
+fd_map[fd.fileno()] = fd
+else:
+fd_map[fd] = fd
+ls = []
+for fd, event in pollster.poll(timeout):
+if event select.POLLNVAL:
+raise ValueError('invalid file descriptor %i' % fd)
+ls.append(fd_map[fd])
+return ls
+else:
+def _poll(fds, timeout):
+return select.select(fds, [], [], timeout)[0]
def wait(object_list, timeout=None):
'''
@@ -870,12 +891,12 @@
'''
if timeout is not None:
if timeout = 0:
-return select.select(object_list, [], [], 0)[0]
+return _poll(object_list, 0)
else:
deadline = time.time() + timeout
while True:
try:
-return select.select(object_list, [], [], timeout)[0]
+return _poll(object_list, timeout)
except OSError as e:
if e.errno != errno.EINTR:
raise
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10527
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10527] multiprocessing.Pipe problem: handle out of range in select()
Giampaolo Rodola' added the comment: Damn, you're right. I must have messed up something while porting the patch from 3.2 all the way up to 3.4 as the merge produced some conflicts. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10527 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10527] multiprocessing.Pipe problem: handle out of range in select()
Roundup Robot added the comment: New changeset 831f49cc00fc by Giampaolo Rodola' in branch 'default': fix for previous commit related to issue 10527 which didn't have the intended effect as per http://bugs.python.org/issue10527#msg179895 http://hg.python.org/cpython/rev/831f49cc00fc -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10527 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10527] multiprocessing.Pipe problem: handle out of range in select()
Giampaolo Rodola' added the comment: 3.3 and 3.4 branches should now be fixed. 2.7 and 3.2 still need to be fixed and the code from connections.py removed. Sorry for the mess up. -- assignee: giampaolo.rodola - ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10527 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10527] multiprocessing.Pipe problem: handle out of range in select()
STINNER Victor added the comment: New changeset c5c27b84d7af by Giampaolo Rodola' in branch '2.7': Fix issue 10527: make multiprocessing use poll() instead of select() if available. http://hg.python.org/cpython/rev/c5c27b84d7af This changeset broke many buildbots, at least: http://buildbot.python.org/all/builders/x86%20XP-5%202.7/builds/439/steps/test/logs/stdio File D:\Buildslave\2.7.moore-windows\build\lib\multiprocessing\connection.py, line 203, in module if hasattr(select, 'poll'): NameError: name 'select' is not defined (I reopen the issue) -- resolution: fixed - status: closed - open ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10527 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10527] multiprocessing.Pipe problem: handle out of range in select()
Roundup Robot added the comment: New changeset 7cf4ea64f603 by Giampaolo Rodola' in branch '2.7': issue 10527: fix missing import http://hg.python.org/cpython/rev/7cf4ea64f603 New changeset d565d862545c by Giampaolo Rodola' in branch '3.2': issue 10527: fix missing import http://hg.python.org/cpython/rev/d565d862545c -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10527 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10527] multiprocessing.Pipe problem: handle out of range in select()
Giampaolo Rodola' added the comment: My bad, sorry. It should be fixed now. -- resolution: - fixed status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10527 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10527] multiprocessing.Pipe problem: handle out of range in select()
Charles-François Natali added the comment: The patch looks good, however there's something really bothering me: in issue #14635, the same type of patch was applied to telnetlib, here, it's multiprocessing and AFAICT, any single use of select() in the standard library is subject to this FD_SETSIZE limitation. That why I think it could probably be a good idea to expose a high-level selector object in the select module, which would use the right syscall transparently (e.g. select, poll or /dev/poll), with a unified API. This would make writing portable and efficient I/O multiplexing code much easier, not only in the stdlib, but also for end-users. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10527 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10527] multiprocessing.Pipe problem: handle out of range in select()
Giampaolo Rodola' added the comment: I know. I proposed something like that here: http://mail.python.org/pipermail/python-ideas/2012-May/015223.html. In theory all the necessary pieces are already there. What's missing is an agreement on what the API should look like, and that's the hard part 'cause it should be the most generic as possible. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10527 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10527] multiprocessing.Pipe problem: handle out of range in select()
Antoine Pitrou added the comment: I know. I proposed something like that here: http://mail.python.org/pipermail/python-ideas/2012-May/015223.html. In theory all the necessary pieces are already there. What's missing is an agreement on what the API should look like, and that's the hard part 'cause it should be the most generic as possible. Well, there was a lot of bikeshedding and pie-in-the-sky arguments in that thread, but I think the original idea of a small wrapper is good enough. Let Guido do the grand async shakeup separately. Also, I've changed my mind: I think select would be an ok module for this :) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10527 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10527] multiprocessing.Pipe problem: handle out of range in select()
Giampaolo Rodola' added the comment: Well, for now I'd say let's just check in this patch as-is. I would be keen on considering this a bug and hence address the patch for Python 2.7, 3.2, 3.3 and 3.4. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10527 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10527] multiprocessing.Pipe problem: handle out of range in select()
Roundup Robot added the comment: New changeset 5530251d9cac by Giampaolo Rodola' in branch '3.2': Fix issue 10527: make multiprocessing use poll() instead of select() if available. http://hg.python.org/cpython/rev/5530251d9cac New changeset d89891f3f769 by Giampaolo Rodola' in branch '3.3': Fix issue 10527: make multiprocessing use poll() instead of select() if available. http://hg.python.org/cpython/rev/d89891f3f769 New changeset e971a70984b8 by Giampaolo Rodola' in branch 'default': Fix issue 10527: make multiprocessing use poll() instead of select() if available. http://hg.python.org/cpython/rev/e971a70984b8 -- nosy: +python-dev ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10527 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10527] multiprocessing.Pipe problem: handle out of range in select()
Roundup Robot added the comment: New changeset c5c27b84d7af by Giampaolo Rodola' in branch '2.7': Fix issue 10527: make multiprocessing use poll() instead of select() if available. http://hg.python.org/cpython/rev/c5c27b84d7af -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10527 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10527] multiprocessing.Pipe problem: handle out of range in select()
Changes by Giampaolo Rodola' g.rod...@gmail.com: -- assignee: - giampaolo.rodola resolution: - fixed stage: patch review - commit review status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10527 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10527] multiprocessing.Pipe problem: handle out of range in select()
Giampaolo Rodola' added the comment: Updated patch including test fixes is in attachment. -- Added file: http://bugs.python.org/file28501/issue10527-3.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10527 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10527] multiprocessing.Pipe problem: handle out of range in select()
Giampaolo Rodola' added the comment: New patch in attachment. It always uses poll() and maintains and internal fd/Connection map. I get one failure due to the returned list being sorted differently than when using select() though. == FAIL: test_wait_integer (__main__.TestWait) -- Traceback (most recent call last): File Lib/test/test_multiprocessing.py, line 3277, in test_wait_integer self.assertEqual(res, [p.sentinel, b]) AssertionError: Lists differ: [multiprocessing.connection.C... != [7, multiprocessing.connectio... First differing element 0: multiprocessing.connection.Connection object at 0x7f8924fccd30 7 - [multiprocessing.connection.Connection object at 0x7f8924fccd30, 7] ? --- + [7, multiprocessing.connection.Connection object at 0x7f8924fccd30] ? +++ I don't how important this is. If it's not tests can be adapted accordingly. -- Added file: http://bugs.python.org/file28469/issue10527-2.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10527 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10527] multiprocessing.Pipe problem: handle out of range in select()
Antoine Pitrou added the comment: The order of the results is probably unimportant. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10527 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10527] multiprocessing.Pipe problem: handle out of range in select()
Charles-François Natali added the comment: This problem affects any single use of select(): instead of using an ad-hoc wrapper in each module, it would probably make sense to add a higher level selector class to the select module which would fallback on the right syscall (i.e. poll() if available, or /dev/poll on Solaris- like). Doesn't Solaris have poll()? If so then I don't see why one would want to use /dev/poll in the single fd case. Because it offers better performance than poll(): you don't have to keep passing the FD at each syscall (note that I'm not talking about the signal FD case, but about a generic polling API). Also note that microbenchmarks with one FD isn't really meaningful, since in real life the FD won't be ready at least part of the time: like Antoine, I think that worrying about performance impact is really a premature optimization (unless real benchmarks prove otherwise). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10527 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10527] multiprocessing.Pipe problem: handle out of range in select()
Giampaolo Rodola' added the comment: A preliminary patch is in attachment. By default it uses select() but looks for ValueError (raised in case FD_SETSIZE gets hit) and falls back on using poll(). This is the failure I get when running tests on Linux. It is related to issue 3321 and I'm not sure what to do with it (remove the test maybe?). == FAIL: test_invalid_handles (__main__.TestInvalidHandle) -- Traceback (most recent call last): File Lib/test/test_multiprocessing.py, line 2852, in test_invalid_handles self.assertRaises((ValueError, IOError), conn.poll) AssertionError: (class 'ValueError', class 'OSError') not raised by poll -- Added file: http://bugs.python.org/file27664/issue10527.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10527 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10527] multiprocessing.Pipe problem: handle out of range in select()
Antoine Pitrou added the comment: I don't like this patch since it makes the implementation poorly testable. Moreover, behaviour might change subtly when the fd becomes 512. I think that instead the code should always use poll() on platforms where it is available. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10527 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10527] multiprocessing.Pipe problem: handle out of range in select()
Antoine Pitrou added the comment: By the way, I think this is a bug rather than a performance issue. -- type: performance - behavior versions: +Python 3.2, Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10527 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10527] multiprocessing.Pipe problem: handle out of range in select()
Giampaolo Rodola' added the comment: Using poll() by default is controversial for 2 reasons, I think: #1 - a certain slowdown is likely to be introduced (I'll measure it) #2 - current wait() implementation allows to specify a list of file descriptors and/or Connections objects. select() can deal with both while poll() does not (it will return a list of integers rather than a list of Connection instances). I'm not sure how public multiprocessing.connection.wait() is considered and how much backward compatibility should matter in this case. behaviour might change subtly when the fd becomes 512 What do you mean? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10527 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10527] multiprocessing.Pipe problem: handle out of range in select()
Richard Oudkerk added the comment:
A preliminary patch is in attachment.
By default it uses select() but looks for ValueError (raised in case
FD_SETSIZE gets hit) and falls back on using poll().
This is the failure I get when running tests on Linux.
It is related to issue 3321 and I'm not sure what to do with it (remove the
test maybe?).
I guess the patch could do
if any(x[1] POLLNVAL for x in ret):
raise ValueError('invalid file descriptor')
Also, I don't think there is any need to unregister the fds since you are just
removing entries from an internal dict which will be garbage collected.
I don't know if patching 2.7, 3.2, 3.3 to use/fallback on poll() would be
allowed as a bug fix. When, if ever, will the next 2.7 release happen?
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10527
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10527] multiprocessing.Pipe problem: handle out of range in select()
Antoine Pitrou added the comment: Using poll() by default is controversial for 2 reasons, I think: #1 - a certain slowdown is likely to be introduced (I'll measure it) That sounds like premature optimization. If you are concerned about that you could add some caching of the poll object. #2 - current wait() implementation allows to specify a list of file descriptors and/or Connections objects. select() can deal with both while poll() does not (it will return a list of integers rather than a list of Connection instances). Well, can't you just create a mapping of the fds to the objects? Your patch already has that problem by the way, and it's even worse since it will trigger in random conditions (when some fd is 512). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10527 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10527] multiprocessing.Pipe problem: handle out of range in select()
Giampaolo Rodola' added the comment: Still not getting what you refer to when you talk about 512 fds problem. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10527 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10527] multiprocessing.Pipe problem: handle out of range in select()
Richard Oudkerk added the comment: Using poll() by default is controversial for 2 reasons, I think: #1 - a certain slowdown is likely to be introduced (I'll measure it) With a single fd poll is a bit faster than select: $ python -m timeit -s 'from select import select' 'select([0],[],[],0)' 10 loops, best of 3: 2.99 usec per loop $ python -m timeit -s 'from select import poll, POLLIN' 'p=poll();p.register(0,POLLIN);p.poll(0)' 10 loops, best of 3: 2.8 usec per loop The single fd case is the most important one -- see below. #2 - current wait() implementation allows to specify a list of file descriptors and/or Connections objects. select() can deal with both while poll() does not (it will return a list of integers rather than a list of Connection instances). I'm not sure how public multiprocessing.connection.wait() is considered and how much backward compatibility should matter in this case. It was introduced in Python 3.3 and is only really there to allow cross platform Windows/Unix multiplexing. It is (now) also used internally by Connection.poll() and Queue.get() with a single fd. In retrospect it would probably have been better to have implemented poll style multiplexing on Windows. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10527 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10527] multiprocessing.Pipe problem: handle out of range in select()
Antoine Pitrou added the comment: Still not getting what you refer to when you talk about 512 fds problem. By 512 I mean FD_SETSIZE :) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10527 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10527] multiprocessing.Pipe problem: handle out of range in select()
Richard Oudkerk added the comment: Still not getting what you refer to when you talk about 512 fds problem. Whether you get back the original objects or only their fds will depend on whether some fd was larger than FD_SETSIZE. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10527 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10527] multiprocessing.Pipe problem: handle out of range in select()
Charles-François Natali added the comment: See also http://bugs.python.org/issue14635. This problem affects any single use of select(): instead of using an ad-hoc wrapper in each module, it would probably make sense to add a higher level selector class to the select module which would fallback on the right syscall (i.e. poll() if available, or /dev/poll on Solaris-like). -- nosy: +neologix ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10527 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10527] multiprocessing.Pipe problem: handle out of range in select()
Richard Oudkerk added the comment: This problem affects any single use of select(): instead of using an ad-hoc wrapper in each module, it would probably make sense to add a higher level selector class to the select module which would fallback on the right syscall (i.e. poll() if available, or /dev/poll on Solaris- like). Doesn't Solaris have poll()? If so then I don't see why one would want to use /dev/poll in the single fd case. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10527 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10527] multiprocessing.Pipe problem: handle out of range in select()
William Edwards added the comment: issue 16259 has just been closed as a dup of this one. Does this mean that this one will be fixed in Python 2.x too? -- nosy: +William.Edwards ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10527 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10527] multiprocessing.Pipe problem: handle out of range in select()
William Edwards added the comment: Apologies, I meant: issue 16269 has just been closed as a dup of this one. Does this mean that this one will be fixed in Python 2.x too? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10527 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10527] multiprocessing.Pipe problem: handle out of range in select()
Giampaolo Rodola' added the comment: Not necessarily. It just means the other one was a duplicate. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10527 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10527] multiprocessing.Pipe problem: handle out of range in select()
William Edwards added the comment: That was my fear; I raise an issue hurting my 2.x servers in production, and its closed as duplicate instead of not-going-to-fix? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10527 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10527] multiprocessing.Pipe problem: handle out of range in select()
Giampaolo Rodola' added the comment: If an issue is a duplicate of another one it gets closed as a duplicate, and that's it basically. This issue is still open and this is where the matter should be discussed. -- nosy: +sbt ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10527 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10527] multiprocessing.Pipe problem: handle out of range in select()
Changes by Giampaolo Rodola' g.rod...@gmail.com: -- nosy: +giampaolo.rodola ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10527 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10527] multiprocessing.Pipe problem: handle out of range in select()
Dave Malcolm dmalc...@redhat.com added the comment: [for reference: issue 11743 covered Antoine's rewrite of the connection class to be pure python, for 3.3 (re msg138310)] -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10527 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10527] multiprocessing.Pipe problem: handle out of range in select()
Dan Kenigsberg dan...@redhat.com added the comment: I would rate this issue as a performance bug, not a mere feature request. If the python process has more than 1023 open file descriptors, multiprocessing.Pipe.poll() becomes unusable. This is a serious barrier to using multiprocessing in a complex server. -- nosy: +danken type: feature request - performance ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10527 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10527] multiprocessing.Pipe problem: handle out of range in select()
Erez Sh ere...@gmail.com added the comment: I support this change. Putting an arbitrary limitation on the amount of supported subprocesses is disastrous for complex software. Gergely's patch seems good. I would only like to suggest a small cosmetic refinement to it, which removes some dead code. -- nosy: +Erez.Sh Added file: http://bugs.python.org/file22364/multiproc2.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10527 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10527] multiprocessing.Pipe problem: handle out of range in select()
Changes by STINNER Victor victor.stin...@haypocalc.com: -- nosy: +haypo, pitrou ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10527 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10527] multiprocessing.Pipe problem: handle out of range in select()
Antoine Pitrou pit...@free.fr added the comment: This code has changed a lot in Python 3.3 (it is now located in Lib/multiprocessing/connection.py). Can you post a patch against the development tip (default branch)? See http://docs.python.org/devguide/setup.html if you need more information. -- versions: +Python 3.3 -Python 3.2 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10527 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10527] multiprocessing.Pipe problem: handle out of range in select()
Changes by Dave Malcolm dmalc...@redhat.com: -- nosy: +dmalcolm ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10527 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10527] multiprocessing.Pipe problem: handle out of range in select()
Dave Malcolm dmalc...@redhat.com added the comment: The analogous code within Modules/selectmodule.c uses #ifdef HAVE_POLL to guard the poll-using code, to support non-Windows platforms that don't have poll. Presumably a patch for this should do the same. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10527 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10527] multiprocessing.Pipe problem: handle out of range in select()
Dave Malcolm dmalc...@redhat.com added the comment: Also, I see that Modules/selectmodule.c has some painful-looking workarounds involving HAVE_BROKEN_POLL, which presumably would also be applicable here. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10527 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10527] multiprocessing.Pipe problem: handle out of range in select()
New submission from Gergely Kálmán kalman.gerg...@duodecad.hu: Hello, I have a code that uses multiprocessing.Pipe to communicate with subprocesses. Spawning 500 subprocesses this way works like a charm, but when spawning about 600 of them the pipe ends raise the exception: handle out of range in select(). I realized that this is because of the FD_SETSIZE limit. To address the situation I quickly hacked together a patch that uses poll() instead of select(), which solves the problem for me. I don't know the reason why select() was chosen for this task (maybe because of windows) but wouldn't it be better to use polling where possible? I've attached the tester. Beware, running it may use up all memory in your system, so be careful! Gergely Kalman -- components: Library (Lib) files: tester.py messages: 122349 nosy: synapse priority: normal severity: normal status: open title: multiprocessing.Pipe problem: handle out of range in select() type: feature request versions: Python 3.2 Added file: http://bugs.python.org/file19813/tester.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10527 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10527] multiprocessing.Pipe problem: handle out of range in select()
Gergely Kálmán kalman.gerg...@duodecad.hu added the comment: And this is the patch that I wrote. It applies to python 3.2. Hope this helps Gergely Kalman -- keywords: +patch Added file: http://bugs.python.org/file19814/multiproc.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10527 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10527] multiprocessing.Pipe problem: handle out of range in select()
Changes by Ned Deily n...@acm.org: -- nosy: +jnoller stage: - patch review ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10527 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10527] multiprocessing.Pipe problem: handle out of range in select()
Changes by Jesse Noller jnol...@gmail.com: -- nosy: +asksol ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10527 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
