[issue6721] Locks in python standard library should be sanitized on fork
Changes by Tomaž Šolc tomaz.s...@tablix.org: -- nosy: -avian ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6721 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6721] Locks in python standard library should be sanitized on fork
Tomaž Šolc tomaz.s...@tablix.org added the comment: Except for multiprocessing, does anyone know of any other module in the standard library that uses fork() and threads at the same time? After some grepping through the source I couldn't find any other cases. I'm still in favor of just deprecating using fork() on a multithreaded process (with appropriate warnings and documentation) and I'm prepared to work on a patch that would remove the need for helper threads in the multiprocessing module. I gather that having atfork would be useful beyond attempting to solve the locking problem, so this doesn't mean I'm opposed to it. However debugging rare problems in multithreaded/multiprocess applications is such a frustrating task that I really don't like a solution that only works in the common case. In Python atfork() handlers will never run from signal handlers, and if I understood correctly, Charles-François described a way to re-initialize a Python lock safely under that assumption. Just to clarify: it's not that POSIX atfork() handlers run from signal handlers. It's that after a fork in a multithreaded process POSIX only guarantees calls to safe functions, which is the same set of functions as those that are safe to call from signal handlers. This fact does not change for Python's os.fork(). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6721 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6721] Locks in python standard library should be sanitized on fork
Tomaž Šolc tomaz.s...@tablix.org added the comment: We can't do that, it would break existing code. I would argue that such code is already broken. What do you mean by helper threads? multiprocessing uses threads behind the scenes to handle queue traffic and such for individual forked processes. It's something I also wasn't aware of until Antoine pointed it out. It also has its own implementation of atfork hooks in an attempt to handle the locking issue. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6721 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6721] Locks in python standard library should be sanitized on fork
Tomaž Šolc tomaz.s...@tablix.org added the comment: The way I see it is that Charles-François' patch trades a possibility of a deadlock for a possibility of a child process running with inconsistent states due to forcibly reinitialized locks. Personally, I would rather have an occasional deadlock than an occasional random crash. I don't like increasing complexity with fine-grained locking either. While the general case is unsolvable what Giovanni proposed at least solves the specific case where only the basic IO code is involved after a fork. In hindsight the only real life use-case I can find that it would solve is doing an exec() right after a fork(). There are quite a few bugs in the tracker that seem to have this same root cause, so it appears the impossibility of cleanly handling threads and forks is not something people are generally aware of. Since I think we agree you can't just disable fork() when multiple threads are running, how about at least issuing a warning in that case? That would be a two-line change in threading.py. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6721 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6721] Locks in python standard library should be sanitized on fork
Tomaž Šolc tomaz.s...@tablix.org added the comment: You mean a runtime warning? That would be ugly and clumsy. A warning is probably a good idea, but it should be added somewhere in os.fork() and threading documentation. I was thinking about a run time warning that is emitted if you call os.fork() while multiple threads are active. It is ugly, but at least it tells you you are doing something that will in most cases not work correctly. I certainly agree that a warning should also be added to os.fork() documentation. I'm attaching an example patch that adds it into _after_fork() in threading.py, but there are a number of other places where it might go instead. I believe that the comp.programming.threads post from David Butenhof linked above explains why adding atfork() handlers isn't going to solve this. -- Added file: http://bugs.python.org/file22525/emit_warning_on_fork.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6721 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7123] Multiprocess Process does not always exit when run from a thread.
Tomaž Šolc tomaz.s...@tablix.org added the comment: I am pretty sure this is another instance of issue 6721. multiprocessing is forking a new process and if that happens while the other thread is holding the lock for stdout, the process will deadlock because the lock state is replicated in the subprocess while the thread that is supposed unlock it isn't (even if you remove all prints, the deadlock will happen on a stdout flush). Attaching a gdb to a hung subprocess confirms that it is waiting for a lock in the bufferedio module. I've reproduced this on Python trunk running on Linux (using pthreads) and can confirm that the patch attached to issue 6721 fixes it. I'm also attaching a shorter test case that omits unused code from the original example. -- nosy: +avian Added file: http://bugs.python.org/file22465/testing.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7123 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6721] Locks in python standard library should be sanitized on fork
Changes by Tomaž Šolc tomaz.s...@tablix.org: -- nosy: +avian ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6721 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
