[issue14059] Implement multiprocessing.Barrier
Roundup Robot devn...@psf.upfronthosting.co.za added the comment: New changeset 2d2f206d040e by Richard Oudkerk in branch 'default': Issue #14059: Implement multiprocessing.Barrier http://hg.python.org/cpython/rev/2d2f206d040e -- nosy: +python-dev ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14059 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14059] Implement multiprocessing.Barrier
Changes by Richard Oudkerk shibt...@gmail.com: -- resolution: - fixed stage: - committed/rejected status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14059 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14059] Implement multiprocessing.Barrier
Richard Oudkerk shibt...@gmail.com added the comment: The attached patch uses memoryview instead of ctypes. If the patch for Issue #14953 (reimplementing RawArray/RawValue in terms of memoryview) is applied, then it could be simplified a bit. -- Added file: http://bugs.python.org/file25835/mp_barrier.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14059 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14059] Implement multiprocessing.Barrier
Richard Oudkerk shibt...@gmail.com added the comment: RawValue uses ctypes, right? That's problematic for platforms which don't support ctypes. Are there many posix systems (we care about) where ctypes doesn't work? It would be fairly easy to use memoryview instead of ctypes. (In fact Value/RawValue could be implemented that way when using a typecode instead of a ctypes type.) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14059 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14059] Implement multiprocessing.Barrier
Antoine Pitrou pit...@free.fr added the comment: RawValue uses ctypes, right? That's problematic for platforms which don't support ctypes. Are there many posix systems (we care about) where ctypes doesn't work? It depends what you call caring about :-) But proprietary Unix C compilers generally don't mix very well with ctypes. It would be fairly easy to use memoryview instead of ctypes. (In fact Value/RawValue could be implemented that way when using a typecode instead of a ctypes type.) That would be a good idea then. Also, not using ctypes could make code theoretically more robust. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14059 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14059] Implement multiprocessing.Barrier
Antoine Pitrou pit...@free.fr added the comment: RawValue uses ctypes, right? That's problematic for platforms which don't support ctypes. -- nosy: +pitrou ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14059 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14059] Implement multiprocessing.Barrier
Charles-François Natali neolo...@free.fr added the comment: The patch looks good. However, I've had two failures while testing it: - a BrokenBarrierError on test_default_timeout: I see you've already increased the timeout from the original threading code, but you can probably double it (we have some slow buildbots, and creating processes is expensive on Windows) - TestNumberOfObjects is failing too: test_number_of_objects (test.test_multiprocessing.WithManagerTestZZZNumberOfObjects) ... b6cb776c: refcount=1 threading.Barrier object at 0xb6cb776c b6d72dfc: refcount=2 multiprocessing.pool.Pool object at 0xb6d72dfc b6cb776c: refcount=1 threading.Barrier object at 0xb6cb776c b6d72dfc: refcount=2 multiprocessing.pool.Pool object at 0xb6d72dfc FAIL -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14059 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14059] Implement multiprocessing.Barrier
Changes by Antoine Pitrou pit...@free.fr: -- dependencies: +multiprocessing.Condition.wait_for missing ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14059 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14059] Implement multiprocessing.Barrier
sbt shibt...@gmail.com added the comment: Patch which subclasses threading.Barrier. -- keywords: +patch Added file: http://bugs.python.org/file24614/mp_barrier.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14059 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14059] Implement multiprocessing.Barrier
sbt shibt...@gmail.com added the comment: Forgot to mention, mp_barrier.patch needs to be applied on top of cond_wait_for.patch for Issue14087. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14059 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14059] Implement multiprocessing.Barrier
sbt shibt...@gmail.com added the comment: Wouldn't it be simpler with a mp.Condition? Well, it is a fair bit shorter than the implementation in threading.py. But that is not a fair comparison because it does implement reset(). I was trying to avoid using shared memory/ctypes since multiprocessing.synchronize does not currently use them. However, I think it would be better (and much simpler) to just subclass threading.Barrier, making self._state and self._counter properties which delegate to RawValue objects. That gets rid of the differences in behaviour. I have this working, although I had to monkey patch multiprocessing.Condition to add a wait_for() method. See Issue 14087. Otherwise, this should be added to Lib/multiprocesing.synchronize.py, and the tests to test_multiprocessing. I will provide a patch later. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14059 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14059] Implement multiprocessing.Barrier
sbt shibt...@gmail.com added the comment: Here is an initial implementation. Differences from threading.Barrier: - I have not implemented reset(). - wait() returns 0 or -1. One thread returns 0, the remainder return -1. This is different to threading.Barrier where each of the N threads returns a unique index in range(N). - I added an action_args parameter to the constructor. This is a tuple which provides argument to the action callback. (This is because closures are not picklable, making no-argument callbacks rather limiting for multiprocessing.) -- nosy: +sbt Added file: http://bugs.python.org/file24579/barrier.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14059 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14059] Implement multiprocessing.Barrier
sbt shibt...@gmail.com added the comment: barrier_tests.py contains minor modifications of the unit tests for threading.Barrier. (The two tests using reset() are commented out.) The implementation passes for me on Linux and Windows. -- Added file: http://bugs.python.org/file24580/barrier_tests.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14059 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14059] Implement multiprocessing.Barrier
Charles-François Natali neolo...@free.fr added the comment: Here is an initial implementation. Wouldn't it be simpler with a mp.Condition? Otherwise, this should be added to Lib/multiprocesing.synchronize.py, and the tests to test_multiprocessing. -- nosy: +neologix ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14059 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14059] Implement multiprocessing.Barrier
New submission from Matt Joiner anacro...@gmail.com: There is no Barrier in multiprocessing corresponding to threading.Barrier. -- components: Library (Lib) messages: 153744 nosy: anacrolix priority: normal severity: normal status: open title: Implement multiprocessing.Barrier type: enhancement versions: Python 3.3 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14059 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
