[issue6643] Throw away more radioactive locks that could be held across a fork in threading.py
Maciej Bliziński added the comment: Python version: 2.7.5 OS: Solaris 9 I'm still observing this issue (or Issue5114) on Solaris 9. The symptom is that test_threading hangs indefinitely (tested: overnight) and running pstack on the process, I'm seeing: - lwp# 1 / thread# 1 ff3dc734 lwp_park (0, 0, 0) ff3d3c74 s9_lwp_park (0, 0, 0, 1, feed4f48, 18f5a4) + 28 ff3dc698 s9_handler (0, 0, 0, 1, feed4f48, 18f5a4) + 90 ff1dea70 _sema_wait (0, feee66a0, fed6b054, feee6000, 2a298478, d1f20) + 1d4 ff1dec30 sema_wait (81aa8, ff1dec24, 722a5b4b, 1101c, feed4f48, 134d60) + c feed4f48 sem_wait (81aa8, 0, fed6b1ac, 0, 0, 1) + 20 ff050890 PyThread_acquire_lock (81aa8, 1, fed6b214, 2, 0, 1ae778) + 5c ff05524c lock_PyThread_acquire_lock (0, 22030, 0, 13ee40, 16a298, 55150) + 50 fefa779c PyCFunction_Call (1ae788, 22030, 0, ff0d8eb8, 55150, ff0551fc) + e4 ff016b14 PyEval_EvalFrameEx (18f5a0, 0, 0, d4f66, 16a298, 22030) + 5ee8 ff0185d0 PyEval_EvalCodeEx (12c968, 0, 18f5a0, 4, 1, 18f5a4) + 924 ff0168f8 PyEval_EvalFrameEx (1902b8, 0, 1, 1765c0, 16a298, 1b12d0) + 5ccc ff0185d0 PyEval_EvalCodeEx (13f608, 0, 1902b8, 4, 1, 1902bc) + 924 ff0168f8 PyEval_EvalFrameEx (154748, 0, 1, 31f7f, 16a298, 1b1250) + 5ccc ff0185d0 PyEval_EvalCodeEx (10d650, 54a50, 154748, 2203c, 0, 2203c) + 924 fef8e11c function_call (22038, 22030, 1386f0, 2203c, 130730, 22030) + 168 fef604e8 PyObject_Call (130730, 22030, 1386f0, ff0e0340, fef8dfb4, 0) + 60 ff0137dc PyEval_EvalFrameEx (169110, 0, 22030, 10e62d, 16a298, 22030) + 2bb0 ff017478 PyEval_EvalFrameEx (168f80, 0, 169114, 1769fa, 16a298, 16a298) + 684c ff017478 PyEval_EvalFrameEx (176cb0, 0, 168f84, 12a2c0, 16a298, 16a298) + 684c ff0185d0 PyEval_EvalCodeEx (13f410, 176cb4, 176cb0, 13433c, 1, 0) + 924 fef8e040 function_call (1b26f0, 134330, 0, ff1bc000, 1b26f0, 0) + 8c fef604e8 PyObject_Call (1b26f0, 134330, 0, ff0e0340, fef8dfb4, 134320) + 60 fef6e530 instancemethod_call (0, 134330, 0, 0, 1b26f0, 134bd0) + a4 fef604e8 PyObject_Call (c3b48, 22030, 0, ff0e0340, fef6e48c, 0) + 60 ff01051c PyEval_CallObjectWithKeywords (c3b48, 22030, 0, 0, 0, 0) + 68 ff05568c t_bootstrap (63bd0, 0, 0, 0, 16a298, ff0e2804) + 4c ff1e53a4 _lwp_start (0, 0, 0, 0, 0, 0) - lwp# 2 / thread# 2 ff3dc734 lwp_park (0, 0, 0) ff3d3c74 s9_lwp_park (0, 0, 0, 1, b64a0d58, 136818) + 28 ff3dc698 s9_handler (0, 0, 0, 1, b64a0d58, 136818) + 90 ff1dea70 _sema_wait (0, feee66a0, fec6b054, feee6000, 2a298478, d1f20) + 1d4 ff1dec30 sema_wait (8ab00, ff1dec24, 722a5b4b, 1101c, feed4f48, 134d60) + c feed4f48 sem_wait (8ab00, 0, fec6b1ac, 0, 0, 1) + 20 ff050890 PyThread_acquire_lock (8ab00, 1, fec6b214, 2, 0, 1ae610) + 5c ff05524c lock_PyThread_acquire_lock (0, 22030, 0, 13ee40, 156168, 55160) + 50 fefa779c PyCFunction_Call (1ae620, 22030, 0, ff0d8eb8, 55160, ff0551fc) + e4 ff016b14 PyEval_EvalFrameEx (18fe60, 0, 0, d4f66, 156168, 22030) + 5ee8 ff0185d0 PyEval_EvalCodeEx (12c968, 0, 18fe60, 4, 1, 18fe64) + 924 ff0168f8 PyEval_EvalFrameEx (18fce8, 0, 1, 1765c0, 156168, 1b11b0) + 5ccc ff0185d0 PyEval_EvalCodeEx (13f608, 0, 18fce8, 4, 1, 18fcec) + 924 ff0168f8 PyEval_EvalFrameEx (18fb88, 0, 1, 136155, 156168, 1a2930) + 5ccc ff0185d0 PyEval_EvalCodeEx (48b60, 18fb8c, 18fb88, 19d41c, 1, 2203c) + 924 fef8e11c function_call (22038, 19d410, 1b3c00, 2203c, 130370, 22030) + 168 fef604e8 PyObject_Call (130370, 19d410, 1b3c00, ff0e0340, fef8dfb4, 19d400) + 60 ff0137dc PyEval_EvalFrameEx (18fa20, 0, 19d410, 10e62d, 156168, 134950) + 2bb0 ff017478 PyEval_EvalFrameEx (18f890, 0, 18fa24, 1769fa, 156168, 156168) + 684c ff017478 PyEval_EvalFrameEx (18f728, 0, 18f894, 12a2c0, 156168, 156168) + 684c ff0185d0 PyEval_EvalCodeEx (13f410, 18f72c, 18f728, 19d3fc, 1, 0) + 924 fef8e040 function_call (1b26f0, 19d3f0, 0, ff1bc000, 1b26f0, 0) + 8c fef604e8 PyObject_Call (1b26f0, 19d3f0, 0, ff0e0340, fef8dfb4, 19d3e0) + 60 fef6e530 instancemethod_call (0, 19d3f0, 0, 0, 1b26f0, 1b1250) + a4 fef604e8 PyObject_Call (1aeaf8, 22030, 0, ff0e0340, fef6e48c, 0) + 60 ff01051c PyEval_CallObjectWithKeywords (1aeaf8, 22030, 0, 0, 0, 0) + 68 ff05568c t_bootstrap (63c30, 0, 0, 0, 156168, ff0e2804) + 4c ff1e53a4 _lwp_start (0, 0, 0, 0, 0, 0) The problem does not occur on Solaris 10. -- nosy: +automatthias ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6643 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6643] Throw away more radioactive locks that could be held across a fork in threading.py
Nadeem Vawda nadeem.va...@gmail.com added the comment: r87710 introduces an AttributeError in test_thread's TestForkInThread test case. If os.fork() is called from a thread created by the _thread module, threading._after_fork() will get a _DummyThread (with no _block attribute) as the current thread. I've attached a patch that checks whether the thread has a _block attribute before trying to reinitialize it. -- nosy: +nvawda Added file: http://bugs.python.org/file20259/test_thread.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6643 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6643] Throw away more radioactive locks that could be held across a fork in threading.py
Changes by Antoine Pitrou pit...@free.fr: -- status: closed - open ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6643 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6643] Throw away more radioactive locks that could be held across a fork in threading.py
Gregory P. Smith g...@krypto.org added the comment: eek, thanks for noticing that! r87740 fixes this in py3k. backporting to 3.1 and 2.7 now. -- nosy: +barry priority: normal - release blocker ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6643 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6643] Throw away more radioactive locks that could be held across a fork in threading.py
Gregory P. Smith g...@krypto.org added the comment: r87741 3.1 r87742 2.7 -- resolution: accepted - fixed status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6643 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6643] Throw away more radioactive locks that could be held across a fork in threading.py
Gregory P. Smith g...@krypto.org added the comment: fwiw a unified fork-and-exec API implemented in C is what I added in Modules/_posixsubprocess.c to at least avoid this issue as much as possible when using subprocess. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6643 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6643] Throw away more radioactive locks that could be held across a fork in threading.py
Gregory P. Smith g...@krypto.org added the comment: patch looks good. committed in r87710 for 3.2. needs back porting to 3.1 and 2.7 and optionally 2.6. -- assignee: rnk - gregory.p.smith resolution: - accepted ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6643 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6643] Throw away more radioactive locks that could be held across a fork in threading.py
Gregory P. Smith g...@krypto.org added the comment: r87726 for release31-maint r87727 for release27-maint - this required a bit more fiddling as _block and _started and _cond were __ private. -- versions: -Python 3.1, Python 3.2 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6643 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6643] Throw away more radioactive locks that could be held across a fork in threading.py
Gregory P. Smith g...@krypto.org added the comment: Attached is a patch for Python 2.6 release26_maint for reference incase someone wants it. That branch is closed - security fixes only. -- keywords: +patch status: open - closed versions: -Python 2.7 Added file: http://bugs.python.org/file20249/issue6643-release26_maint_gps01.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6643 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6643] Throw away more radioactive locks that could be held across a fork in threading.py
Changes by Reid Kleckner r...@mit.edu: -- assignee: - rnk keywords: +needs review -patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6643 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6643] Throw away more radioactive locks that could be held across a fork in threading.py
Adam Olsen rha...@gmail.com added the comment: I don't have any direct opinions on this, as it is just a bandaid. fork, as defined by POSIX, doesn't allow what we do with it, so we're reliant on great deal of OS and library implementation details. The only portable and robust solution would be to replace it with a unified fork-and-exec API that's implemented directly in C. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6643 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6643] Throw away more radioactive locks that could be held across a fork in threading.py
Reid Kleckner r...@mit.edu added the comment: I completely agree, but the cat is out of the bag on this one. I don't see how we could get rid of fork until Py4K, and even then I'm sure there will be people who don't want to see it go, and I'd rather not spend my time arguing this point. The only application of fork that doesn't use exec that I've heard of is pre-forked Python servers. But those don't seem like they would be very useful, since with refcounting the copy-on-write behavior doesn't get you very many wins. The problem that this bandaid solves for me is that test_threading.py already tests thread+fork behaviors, and can fail non-deterministically. This problem was exacerbated while I was working on making the compilation thread. I don't think we can un-support fork and threads in the near future either, because subprocess.py uses fork, and libraries can use fork behind the user's back. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6643 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6643] Throw away more radioactive locks that could be held across a fork in threading.py
Changes by Reid Kleckner r...@mit.edu: -- title: joining a child that forks can deadlock in the forked child process - Throw away more radioactive locks that could be held across a fork in threading.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6643 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6643] Throw away more radioactive locks that could be held across a fork in threading.py
Changes by Antoine Pitrou pit...@free.fr: -- nosy: +Rhamphoryncus, gregory.p.smith ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6643 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
