[issue35242] multiprocessing.Queue in an inconsistent state and a traceback silently suppressed if put an unpickable object and process's target function is finished
Sergei Zobov added the comment: So, should I just close this issue in case we've decided it's not a big problem? -- ___ Python tracker <https://bugs.python.org/issue35242> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35242] multiprocessing.Queue in an inconsistent state and a traceback silently suppressed if put an unpickable object and process's target function is finished
Sergei Zobov added the comment: Thanks for answer, Antoine! I see what you mean, but I'm not sure that it's so extreme corner case. We are speaking about Python and it's easy to create an put an unpickable object into queue. I apologize for the persistence, but I just want make this case more friendly to user. I spent plenty of time to debug it and I think it would be better to prevent someone to do the same things. If it's a difficult to fix, we can at least write a note somewhere in multiprocessing module doc to describe this behavior. -- ___ Python tracker <https://bugs.python.org/issue35242> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35242] multiprocessing.Queue in an inconsistent state and a traceback silently suppressed if put an unpickable object and process's target function is finished
Sergei Zobov added the comment: Yes, I've missed that the daemonic thread won't work if the parent process finished. The similarities between multiprocessing and threading modules are confusing a bit. Anyway, could I make a patch to fix the problem with the silently ignored exception and the broken semaphore? -- ___ Python tracker <https://bugs.python.org/issue35242> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35242] multiprocessing.Queue in an inconsistent state and a traceback silently suppressed if put an unpickable object and process's target function is finished
Change by Sergei Zobov : -- nosy: +pitrou ___ Python tracker <https://bugs.python.org/issue35242> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35242] multiprocessing.Queue in an inconsistent state and a traceback silently suppressed if put an unpickable object and process's target function is finished
Sergei Zobov added the comment: I've decided to make the test more clean, so I added `time.sleep` and now we can be sure that here is enough time for queue's underlying process to synchronize all variables: https://github.com/szobov/cpython/compare/master...regression-test-on-multiprocessing-queue-feeder-ignore-exception It shows that sometimes `Queue._sem` will never be decreased. https://github.com/szobov/cpython/blob/master/Lib/multiprocessing/queues.py#L266 If it's undesirable behavior, would it be better to ignore this lines: https://github.com/szobov/cpython/blob/master/Lib/multiprocessing/queues.py#L257-L259 ? But in my thoughts the problem is in the functions `util._exit_function` and `util.is_exiting`. It's look like that `Queue._thread` share the same variable `util._exiting` with the process that started this thread. If you'll try to print it you see that when function `util._exit_function` executed for process it changes the value of `util._exiting` flag to `True` and after, from `Queue._thread`, the value for this flag stay `True`. But it's the daemon process and it should still work, isn't it? Or I'm missing something important here. -- ___ Python tracker <https://bugs.python.org/issue35242> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35242] multiprocessing.Queue in an inconsistent state and a traceback silently suppressed if put an unpickable object and process's target function is finished
Change by Sergei Zobov : -- components: +Library (Lib) ___ Python tracker <https://bugs.python.org/issue35242> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35242] multiprocessing.Queue in an inconsistent state and a traceback silently suppressed if put an unpickable object and process's target function is finished
New submission from Sergei Zobov : In case if the process target function put something unpickable in the `multiprocessing.Queue` and finish before `ForkingPickler`'s exception is handled we get two weird situations: * Internal queue semaphore stay acquired and never will be released, so we get `qsize()` always returns 1(or more if more then one process) even if there is nothing in queue's buffer and `empty()` returns `True`. * An exception's traceback will be silently suppressed. To reproduce this behavior I wrote a test in cpython repo (be aware sometime it pass sometime no): https://github.com/szobov/cpython/commit/95524f10a7b6510e9ab4647223cc67af85ebff86 I suppose it happens because of this: * The function `util._exit_function()` calls every time process target function finished. Inside this function the global (for current process) flag `utils._exiting` marks as `True`: https://github.com/python/cpython/blob/master/Lib/multiprocessing/process.py#L300 https://github.com/python/cpython/blob/master/Lib/multiprocessing/util.py#L294-L295 * When exception while pickling object happens in `Queue._feed` function we are ignore it because of the clause `is_exiting()` that checks the flag `utils._exiting` described above. https://github.com/python/cpython/blob/master/Lib/multiprocessing/queues.py#L257-L259 So we have unreleased semaphore and ignored exceptions. I think it's quite disappointing for user because they have broken queue and nothing about why it happened. I'm not quite sure that it's a bag but I found nothing in documentation about it. -- messages: 329889 nosy: szobov priority: normal severity: normal status: open title: multiprocessing.Queue in an inconsistent state and a traceback silently suppressed if put an unpickable object and process's target function is finished type: behavior versions: Python 3.7 ___ Python tracker <https://bugs.python.org/issue35242> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
