[issue1683] Thread local storage and PyGILState_* mucked up by os.fork()
Martin v. Löwis mar...@v.loewis.de added the comment: Committed fork-thread-patch-2 as r67736 into 2.5 branch. -- status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1683 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1683] Thread local storage and PyGILState_* mucked up by os.fork()
Changes by Barry A. Warsaw [EMAIL PROTECTED]: -- priority: deferred blocker - release blocker ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue1683 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1683] Thread local storage and PyGILState_* mucked up by os.fork()
Barry A. Warsaw [EMAIL PROTECTED] added the comment: Since this is a Python 2.5.3 issue, I'm lowering to deferred blocker until after 3.0 and 2.6.1 are released. -- nosy: +barry priority: release blocker - deferred blocker ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue1683 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1683] Thread local storage and PyGILState_* mucked up by os.fork()
Christian Heimes [EMAIL PROTECTED] added the comment: I was wrong and the patch is right. Something is wrong in multiprocessings connection_recvbytes_into() function for the old buffer protocol. Somehow PyArg_ParseTuple(args, w#| ...) fucks up the process' memory. Martin, are you fine with the patch? fork-thread-patch-2 still applies with some fuzzying. -- assignee: christian.heimes - loewis nosy: +loewis ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue1683 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1683] Thread local storage and PyGILState_* mucked up by os.fork()
Christian Heimes [EMAIL PROTECTED] added the comment: The fix is required to run multiprocessing on Python 2.4 and 2.5, see #4451. I suggest we fix the issue in 2.5.3. The fork-thread-patch-2 patch doesn't work on Python 2.5. I'm getting a segfault on my system: test_connection (multiprocessing.tests.WithProcessesTestConnection) ... Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7fa2e999f6e0 (LWP 10594)] 0x0052065f in PyCFunction_Call (func=Cannot access memory at address 0x7ffeffd8 ) at Objects/methodobject.c:73 73 return (*meth)(self, arg); Linux on AMD64 with Python 2.5 svn --with-pydebug. -- resolution: fixed - status: closed - open versions: +Python 2.5.3 ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue1683 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1683] Thread local storage and PyGILState_* mucked up by os.fork()
Changes by Benjamin Peterson [EMAIL PROTECTED]: -- versions: -Python 2.5, Python 2.6, Python 3.0 ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue1683 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1683] Thread local storage and PyGILState_* mucked up by os.fork()
Changes by Jesse Noller [EMAIL PROTECTED]: -- nosy: +jnoller ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue1683 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1683] Thread local storage and PyGILState_* mucked up by os.fork()
Changes by Gregory P. Smith [EMAIL PROTECTED]: -- assignee: gregory.p.smith - christian.heimes ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue1683 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1683] Thread local storage and PyGILState_* mucked up by os.fork()
Benjamin Peterson [EMAIL PROTECTED] added the comment: I applied this in r64212. -- resolution: - fixed status: open - closed ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue1683 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1683] Thread local storage and PyGILState_* mucked up by os.fork()
Gregory P. Smith [EMAIL PROTECTED] added the comment: we need this in before 2.6 is released. -- assignee: - gregory.p.smith nosy: +gregory.p.smith priority: high - critical ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue1683 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1683] Thread local storage and PyGILState_* mucked up by os.fork()
Benjamin Peterson [EMAIL PROTECTED] added the comment: Gregory, go ahead and apply and see if can stop the hell in the buildbots. -- nosy: +benjamin.peterson ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue1683 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1683] Thread local storage and PyGILState_* mucked up by os.fork()
Adam Olsen [EMAIL PROTECTED] added the comment: Updated version of roudkerk's patch. Adds the new function to pythread.h and is based off of current trunk. Note that Parser/intrcheck.c isn't used on my box, so it's completely untested. roudkerk's original analysis is correct. The TLS is never informed that the old thread is gone, so when it seems the same id again it assumes it is the old thread, which PyThreadState_Swap doesn't like. Added file: http://bugs.python.org/file10595/fork-thread-patch-2 ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue1683 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1683] Thread local storage and PyGILState_* mucked up by os.fork()
Adam Olsen [EMAIL PROTECTED] added the comment: Incidentally, it doesn't seem necessary to reinitialize the lock. Posix duplicates the lock, so if you hold it when you fork your child will be able to unlock it and use it as normal. Maybe there's some non-Posix behaviour or something even more obscure when #401226 was done? (reinitializing is essentially harmless though, so in no way should this hold up release.) ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue1683 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1683] Thread local storage and PyGILState_* mucked up by os.fork()
Changes by Ralf Schmitt: -- nosy: +schmir __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1683 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1683] Thread local storage and PyGILState_* mucked up by os.fork()
Christian Heimes added the comment: Bug day task -- nosy: +tiran priority: - high versions: -Python 2.4 __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1683 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1683] Thread local storage and PyGILState_* mucked up by os.fork()
Changes by Adam Olsen: -- nosy: +rhamphoryncus __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1683 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1683] Thread local storage and PyGILState_* mucked up by os.fork()
roudkerk added the comment: The included patch against python2.51 fixes the problem for me. -- versions: +Python 2.6, Python 3.0 Added file: http://bugs.python.org/file9076/fork-thread-patch __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1683 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1683] Thread local storage and PyGILState_* mucked up by os.fork()
New submission from roudkerk:
I got a report that one of the tests for processing
(http://cheeseshop.python.org/pypi/processing) was failing with
Fatal Python error: Invalid thread state for this thread
when run with a debug interpreter. This appears to be caused by the
interaction between os.fork() and threads. The following attached
program reliably reproduces the problem for me on Ubuntu edgy for x86.
All that happens is that the parent process starts a subthread then
forks, and then the child process starts a subthread.
With the normal interpreter I get
started thread -1211028576 in process 18683
I am thread -1211028576 in process 18683
started thread -1211028576 in process 18685
I am thread -1211028576 in process 18685
as expected, but with the debug interpreter I get
started thread -1210782816 in process 18687
I am thread -1210782816 in process 18687
started thread -1210782816 in process 18689
Fatal Python error: Invalid thread state for this thread
[5817 refs]
Notice that the child process is reusing a thread id that was
being used by the parent process at the time of the fork.
The code raising the error seems to be in pystate.c:
PyThreadState *
PyThreadState_Swap(PyThreadState *new)
{
PyThreadState *old = _PyThreadState_Current;
_PyThreadState_Current = new;
/* It should not be possible for more than one thread state
to be used for a thread. Check this the best we can in
debug
builds.
*/
#if defined(Py_DEBUG) defined(WITH_THREAD)
if (new) {
PyThreadState *check = PyGILState_GetThisThreadState();
if (check check != new)
Py_FatalError(Invalid thread state for this
thread);
}
#endif
return old;
}
It looks as though PyGILState_GetThisThreadState() is returning the
thread state of the thread from the parent process which has the same
id as the current thread. Therefore the check fails.
I think the thread local storage implementation in thread.c should
provide a function _PyThread_ReInitTLS() which PyOS_AfterFork() can
call. I think _PyThread_ReInitTLS() just needs to remove and free
each struct key in the linked list which does not match the id of
the calling thread.
--
components: Interpreter Core
files: dbg2.py
messages: 58954
nosy: roudkerk
severity: normal
status: open
title: Thread local storage and PyGILState_* mucked up by os.fork()
type: crash
versions: Python 2.4, Python 2.5
Added file: http://bugs.python.org/file9023/dbg2.py
__
Tracker [EMAIL PROTECTED]
http://bugs.python.org/issue1683
__import time, os, thread
def f():
print I am thread %d in process %d % (thread.get_ident(),os.getpid())
time.sleep(1)
ident = thread.start_new_thread(f, ())
print started thread %d in process %d % (ident, os.getpid())
time.sleep(0.5) # give thread a chance to start
pid = os.fork()
if pid == 0:
ident = thread.start_new_thread(f, ())
print started thread %d in process %d % (ident, os.getpid())
time.sleep(0.5) # give thread a chance to start
os._exit(0)
os.waitpid(pid, 0)
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
