[issue31782] Add a timeout to multiprocessing's Pool.join
Will Starms <vilhel...@gmail.com> added the comment: I've realized that my patch may not be ideal for general-purpose use, but it's a good start for a discussion on the proper way to implement a timeout. My patch (which is based on a more involved modification to Pool) assumes that the joins after the first will complete within a timely fashion, which is not necessarily true. While this prevents leaving the pool in a half-joined state, it can still get stuck joining other components or at least take significantly longer than the requested timeout. Assuming that joining an already-joined object is safe, or it can be wrapped in an if statement to check before rejoining, I feel the best solution would be to reduce the timeout as joins complete, either raising (much easier) or returning (like threading.thread, but makes an is_alive function more difficult) when the remaining timeout time hits zero. -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue31782> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31782] Add a timeout to multiprocessing's Pool.join
Will Starms <vilhel...@gmail.com> added the comment: A timeout alternative that raises TimeoutError -- Added file: https://bugs.python.org/file47220/cpython_raise_timeout.patch ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue31782> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31782] Add a timeout to multiprocessing's Pool.join
New submission from Will Starms <vilhel...@gmail.com>: Pool's join function currently (3.6.3) lacks a timeout, which can cause the managing thread to sleep indefinitely when a pool worker hangs or starts misbehaving. Adding a timeout allows the owning thread to attempt a join and, after the timeout, return to other tasks, such as monitoring worker health. In my specific situation, I have a Pool running a task on a large set of files. If any single task fails, the whole operation is ruined and the pool should be terminated. A task can communicate with the main thread through error_callback, but if the thread has already called join, it can't check until join returns, after the Pool has finished all processing. Attached is an incredibly simple patch to the current (3.6) cpython implementation that emulates threading.thread.join's behavior. -- components: Library (Lib) files: cpython_timeout.patch keywords: patch messages: 304350 nosy: Will Starms priority: normal severity: normal status: open title: Add a timeout to multiprocessing's Pool.join type: enhancement versions: Python 3.6 Added file: https://bugs.python.org/file47219/cpython_timeout.patch ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue31782> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
