[issue37416] If threading is not imported from the main thread it sees the wrong thread as the main thread.

[Antoine Pitrou]

Antoine Pitrou  added the comment:

This is a duplicate of #31517.

--
nosy: +pitrou
resolution:  -> duplicate
stage:  -> resolved
status: open -> closed
superseder:  -> MainThread association logic is fragile

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37416] If threading is not imported from the main thread it sees the wrong thread as the main thread.

[Brett Cannon]

Change by Brett Cannon :


--
nosy:  -brett.cannon

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37416] If threading is not imported from the main thread it sees the wrong thread as the main thread.

[Pavel Minaev]

Pavel Minaev  added the comment:

Debuggers will have to work around this in past Python versions that they 
support (which still includes Python 2 for pretty much all of them), so this is 
solely about resolving the inconsistency for the future. No point rushing it 
for 3.8 specifically.

(The most likely immediate workaround will be that, instead of invoking 
PyEval_InitThreads on the main thread via Py_AddPendingCall, we will simply use 
the same facility to exec "import threading" on the main thread.)

--

___
Python tracker 
<https://bugs.python.org/issue37416>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37416] If threading is not imported from the main thread it sees the wrong thread as the main thread.

[Eric Snow]

Eric Snow  added the comment:

FWIW, this might also have implications for thread shutdown during 
runtime/interpreter finalization.

--
nosy: +vstinner

___
Python tracker 
<https://bugs.python.org/issue37416>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37416] If threading is not imported from the main thread it sees the wrong thread as the main thread.

[Eric Snow]

Eric Snow  added the comment:

Note that in Python 3.7 we consolidated a bunch of the global runtime state.  
The "main_thread" static in ceval.c moved to the internal struct 
_PyRuntimestate and in 3.7 it is accessible as 
_Runtime.ceval.pending.main_thread (but only if you build with Py_BUILD_CORE, 
which extensions shouldn't).  In 3.8 it was moved to the top of the struct as 
_PyRuntimeState.main_thread.  All that said, that thread ID is still not 
exposed directly in Python.

So perhaps it would make sense to add something like sys.get_main_thread_id, 
which Lib/threading.py could then use (rather than assuming the current (during 
import) thread is the main thread).

Unfortunately, this wouldn't help with anything earlier than 3.9.  Currently 
3.8 is already in beta1, where we apply a feature freeze, so it's barely too 
late for 3.8.  I suppose you could ask for a special exception from the release 
manager, given the change would be relatively small, with real benefits, but 
don't expect it. :)

--
nosy: +brett.cannon, eric.snow
versions: +Python 3.8, Python 3.9 -Python 3.7

___
Python tracker 
<https://bugs.python.org/issue37416>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37416] If threading is not imported from the main thread it sees the wrong thread as the main thread.

[Aldwin Pollefeyt]

Aldwin Pollefeyt  added the comment:

Understood. Thank you for the extra info. I'll read up on all the suggestions.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37416] If threading is not imported from the main thread it sees the wrong thread as the main thread.

[Pavel Minaev]

Pavel Minaev  added the comment:

It's also possible to hit if using some native code that starts a background 
thread without going via threading, and runs Python code on that background 
thread. In that case, if that Python code then does "import threading", and 
threading hasn't been imported yet, then you have this same problem.

Here's a pure Python repro using ctypes on Win32:

#--
import sys, time
from ctypes import *

ThreadProc = WINFUNCTYPE(c_uint32, c_void_p)

@ThreadProc
def thread_proc(_):
import threading
print(threading.current_thread() is threading.main_thread())
return 0

assert "threading" not in sys.modules
#import threading  # uncomment to fix

windll.kernel32.CreateThread(None, c_size_t(0), thread_proc, None, c_uint32(0), 
None)
time.sleep(1)

assert "threading" in sys.modules
import threading
print(threading.current_thread() is threading.main_thread())
#--

Here's the output:

>py -3 main.py
True
False
Exception ignored in: 
Traceback (most recent call last):
  File "C:\Python\3.7-64\lib\threading.py", line 1276, in _shutdown
assert tlock.locked()
AssertionError:

--

___
Python tracker 
<https://bugs.python.org/issue37416>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37416] If threading is not imported from the main thread it sees the wrong thread as the main thread.

[Pavel Minaev]

Pavel Minaev  added the comment:

This is a bit tricky to explain... There's no easy way to achieve this effect 
"normally". It manifests due to the way some Python debuggers (specifically, 
pydevd and ptvsd - as used by PyCharm, PyDev, and VSCode) implement 
non-cooperative attaching to a running Python process by PID.

A TL;DR take is that those debuggers have to inject a new thread into a running 
process from the outside, and then run some Python code on that thread. There 
are OS APIs for such thread injection - e.g. CreateRemoteThread on Windows. 
There are various tricks that they then have to use to safely acquire GIL and 
invoke PyEval_InitThreads, but ultimately it comes down to running Python code.

That is the point where this can manifest. Basically, as soon as that injected 
code (i.e. the actual debugger) imports threading, things break. And advanced 
debuggers do need background threads for some functionality...

Here are the technical details - i.e. how thread injection happens exactly, and 
what kind of code it might run - if you're interested.
https://github.com/microsoft/ptvsd/issues/1542

I think that a similar problem can also occur in an embedded Python scenario 
with multithreading. Consider what happens if the hosted interpreter is 
initialized from the main thread of the host app - but some Python code is then 
run from the background thread, and that code happens to be the first in the 
process to import threading. Then that background thread becomes the "main 
thread" for threading, with the same results as described above.

The high-level problem, I think, is that there's an inconsistency between what 
Python itself considers "main thread" (i.e. main_thread in ceval.c, as set by 
PyEval_InitThreads), and what threading module considers "main thread" (i.e. 
_main_thread in threading.py). Logically, these should be in sync.

If PyEval_InitThreads is the source of truth, then the existing thread 
injection technique will "just work" as implemented already in all the 
aforementioned debuggers. It makes sure to invoke PyEval_InitThreads via 
Py_AddPendingCall, rather than directly from the background thread, precisely 
so that the interpreter doesn't get confused. 

Furthermore, on 3.7+, PyEval_InitThreads is already automatically invoked by 
Py_Initialize, and hence when used by python.exe, will mark the actual first 
thread in the process as the main thread. So, using it a the source of truth 
would guarantee that attach by thread injection works correctly in all 
non-embedded Python scenarios.

Apps hosting Python would still need to ensure that they always call 
Py_Initialize on what they want to be the main thread, as they already have to 
do; but they wouldn't need to worry about "import threading" anymore.

--

___
Python tracker 
<https://bugs.python.org/issue37416>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37416] If threading is not imported from the main thread it sees the wrong thread as the main thread.

[Aldwin Pollefeyt]

Aldwin Pollefeyt  added the comment:

The second import actually doesn't happen. You need to reload it. This can be 
tested by putting print('loading threading') in threading.py.

Your first method-thread will still think it's main thread. So no idea if this 
is a bug or wrong use. 'import threading' should be one of the first lines in 
your main code/thread?


import _thread
import time
import importlib
barrier = 0

def method():
import threading  # Will make threading the wrong thread.
global barrier
print(threading.main_thread())
print(threading.current_thread())
barrier = 1

_thread.start_new_thread(method, ())

while barrier != 1:
time.sleep(.1)

import threading
importlib.reload(threading)
print(threading.main_thread())
print(threading.current_thread())

--
nosy: +aldwinaldwin

___
Python tracker 
<https://bugs.python.org/issue37416>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37416] If threading is not imported from the main thread it sees the wrong thread as the main thread.

[Pavel Minaev]

Change by Pavel Minaev :


--
nosy: +int19h

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37416] If threading is not imported from the main thread it sees the wrong thread as the main thread.

[Fabio Zadrozny]

New submission from Fabio Zadrozny :

I'm attaching a snippet which shows the issue (i.e.: threading.main_thread() 
and threading.current_thread() should be the same and they aren't).

What I'd see as a possible solution is that the initial thread ident would be 
stored when the interpreter is initialized and then when threading is imported 
the first time it would get that indent to initialize the main thread instead 
of calling `threading._MainThread._set_ident` in the wrong thread.

I'm not sure if this is possible if CPython is embedded in some other C++ 
program, but it seems to be the correct approach when Python is called from the 
command line.

As a note, I found this when doing an attach to pid for the `pydevd` debugger 
where a thread is created to initialize the debugger (the issue on the debugger 
is reported at: https://github.com/microsoft/ptvsd/issues/1542).

--
components: Interpreter Core
files: snippet.py
messages: 346653
nosy: fabioz
priority: normal
severity: normal
status: open
title: If threading is not imported from the main thread it sees the wrong 
thread as the main thread.
type: behavior
versions: Python 3.7
Added file: https://bugs.python.org/file48438/snippet.py

___
Python tracker 
<https://bugs.python.org/issue37416>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10517] test_concurrent_futures crashes with --with-pydebug on RHEL5 with Fatal Python error: Invalid thread state for this thread

[Jesús Cea Avión]

Changes by Jesús Cea Avión j...@jcea.es:


--
nosy: +jcea

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10517
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10517] test_concurrent_futures crashes with --with-pydebug on RHEL5 with Fatal Python error: Invalid thread state for this thread

[Charles-François Natali]

Charles-François Natali neolo...@free.fr added the comment:

I did a quick test (calling fork() from a subinterpreter), and as
expected, I couldn't reproduce the problem.
So I still favor an OOM condition making pthread_setspecific bail out
with ENOMEM, othe other option being a nasty libc bug.
If the problem persists, please open a new issue.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10517
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10517] test_concurrent_futures crashes with --with-pydebug on RHEL5 with Fatal Python error: Invalid thread state for this thread

[Graham Dumpleton]

Graham Dumpleton graham.dumple...@gmail.com added the comment:

Did anyone test this fix for case of fork() being called from Python sub 
interpreter?

Getting a report of fork() failing in sub interpreters under mod_wsgi that may 
be caused by this change. Still investigating.

Specifically throwing up error:

  Couldn't create autoTLSkey mapping

--
nosy: +grahamd

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10517
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10517] test_concurrent_futures crashes with --with-pydebug on RHEL5 with Fatal Python error: Invalid thread state for this thread

[Charles-François Natali]

Charles-François Natali neolo...@free.fr added the comment:

Hello,

 Did anyone test this fix for case of fork() being called from Python sub 
 interpreter?


Not specifically, unless it's part of the test suite.
Anyway, unless this problem is systematic - which I doubt - it
probably wouldn't have helped.

 Getting a report of fork() failing in sub interpreters under mod_wsgi that 
 may be caused by this change. Still investigating.

 Specifically throwing up error:

  Couldn't create autoTLSkey mapping


Hmmm.
If you can, try strace or instrument the code (perror() should be
enough) to see why it's failing.
pthread_setspecific() can fail with:
- EINVAL, if the TLS key is invalid (which would be strange since we
call pthread_key_delete()/pthread_key_create() just before)
- or ENOMEM, if you run out of memory/address space

The later seems much more likely (e.g. if many child processes and
subinterpreters are created).
BTW, if this is a bug report from someone else, tell him to post here,
it'll be easier.
And we don't byte :-)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10517
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10517] test_concurrent_futures crashes with --with-pydebug on RHEL5 with Fatal Python error: Invalid thread state for this thread

[Kristján Valur Jónsson]

Kristján Valur Jónsson krist...@ccpgames.com added the comment:

Antoine, I wonder if we can restore PyThread_set_key_value to behave like a 
canonical TLS api function (always setting) but fix the cases that want to set 
if it has not already been set like the cases you mention.
It is very unorthodox to have such only set if it hasn't been set before 
built into your only TLS function.  This wart on python's TLS api has bugged me 
for yearsand it would be cool to fix it.

The init functions (that internally call the python TLS apis) could simply do a 
TLS get explicitly themselves, to make it explicit and clear that they _want_ 
to use any pre-existing tls value.

Of course, that won't fix _this_ problem (which is that the main thread's tls 
value gets inherited on fork).  The right way to do that is to explicitly 
clearthe main thread's TLS value after fork...

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10517
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10517] test_concurrent_futures crashes with --with-pydebug on RHEL5 with Fatal Python error: Invalid thread state for this thread

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

 Antoine, I wonder if we can restore PyThread_set_key_value to behave
 like a canonical TLS api function (always setting) but fix the cases
 that want to set if it has not already been set like the cases you
 mention.
 It is very unorthodox to have such only set if it hasn't been set
 before built into your only TLS function.  This wart on python's TLS
 api has bugged me for yearsand it would be cool to fix it.

Well, these functions are supposed to be private so, while I agree their
behaviour is a bit unusual, I'm not sure there's any point to fix them
if it shifts the burden of reproducing the old behaviour on another part
of our code.

 The init functions (that internally call the python TLS apis) could
 simply do a TLS get explicitly themselves, to make it explicit and
 clear that they _want_ to use any pre-existing tls value.

Granted.

 Of course, that won't fix _this_ problem (which is that the main
 thread's tls value gets inherited on fork).  The right way to do that
 is to explicitly clearthe main thread's TLS value after fork...

The main thread is fine, actually, it's the other (disappeared) threads
which cause this problem when the same TID is re-used.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10517
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10517] test_concurrent_futures crashes with --with-pydebug on RHEL5 with Fatal Python error: Invalid thread state for this thread

[Kristján Valur Jónsson]

Kristján Valur Jónsson krist...@ccpgames.com added the comment:

Ah, using the fallback implementation of tls?  Surely this isn't a problem with 
the pthreads tls, I'd be surprised if it retains TLS values after fork.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10517
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10517] test_concurrent_futures crashes with --with-pydebug on RHEL5 with Fatal Python error: Invalid thread state for this thread

[Charles-Francois Natali]

Charles-Francois Natali neolo...@free.fr added the comment:

 Ah, using the fallback implementation of tls?  Surely this isn't a 
 problem with the pthreads tls, I'd be surprised if it retains TLS values 
 after fork.

It surprised me too when I found that out, but it's really with the pthread 
TLS, on RHEL 4 and 5 (fixed in RHEL6).
See the attached test_specific.c test script.

 You could add a new _PyGILState_ReInit() function and call it from
 PyOS_AfterFork() or PyEval_ReInitThreads().

See attached tls_reinit.diff patch.
But I really find this redundant with PyThread_ReInitTLS, because what we're 
really doing is reinit the TLS.
Also, this calls this for every thread implementation, while it's only 
necessary for pthreads (and for other implementation it will redo the work done 
by PyThread_ReInitTLS).
So I've written another patch which does this in pthread's PyThread_ReInitTLS.

You've got much more experience than me, so it's really your call.
Actually, I kind of feel bad for adding such a hack for a pthread's bug 
affecting only RHEL 4 and 5, I'm wondering whether it's really worth fixing it.

--
Added file: http://bugs.python.org/file21801/tls_reinit.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10517
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10517] test_concurrent_futures crashes with --with-pydebug on RHEL5 with Fatal Python error: Invalid thread state for this thread

[Charles-Francois Natali]

Changes by Charles-Francois Natali neolo...@free.fr:


Added file: http://bugs.python.org/file21802/tls_reinit_bis.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10517
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10517] test_concurrent_futures crashes with --with-pydebug on RHEL5 with Fatal Python error: Invalid thread state for this thread

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

  You could add a new _PyGILState_ReInit() function and call it from
  PyOS_AfterFork() or PyEval_ReInitThreads().
 
 See attached tls_reinit.diff patch.

Thank you. I like this patch, except that _PyGILState_ReInit() should be
declared in the appropriate .h file, not in signalmodule.c.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10517
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10517] test_concurrent_futures crashes with --with-pydebug on RHEL5 with Fatal Python error: Invalid thread state for this thread

[Charles-Francois Natali]

Charles-Francois Natali neolo...@free.fr added the comment:

 Thank you. I like this patch, except that _PyGILState_ReInit() should be
 declared in the appropriate .h file, not in signalmodule.c.

I asked myself this question when writing the patch: what's the convention  
regarding functions ? Should they always be declared in a header with 
PyAPI_FUNC, or should this be reserved to functions exported through the API?
I've seen a couple external function declarations in several places, so I was 
wondering (and since this one isn't meant to be exported, I chose the later 
option).

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10517
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10517] test_concurrent_futures crashes with --with-pydebug on RHEL5 with Fatal Python error: Invalid thread state for this thread

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

  Thank you. I like this patch, except that _PyGILState_ReInit() should be
  declared in the appropriate .h file, not in signalmodule.c.
 
 I asked myself this question when writing the patch: what's the
 convention  regarding functions ? Should they always be declared in a
 header with PyAPI_FUNC, or should this be reserved to functions
 exported through the API?

IMO they should always be exposed in header files. It makes them easier
to discover and re-use than with some extern decls sprinkled in .c
files. As for PyAPI_FUNC, I think we always use it out of convention,
although it's probably not useful for private API functions.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10517
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10517] test_concurrent_futures crashes with --with-pydebug on RHEL5 with Fatal Python error: Invalid thread state for this thread

[Charles-Francois Natali]

Changes by Charles-Francois Natali neolo...@free.fr:


Removed file: http://bugs.python.org/file21802/tls_reinit_bis.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10517
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10517] test_concurrent_futures crashes with --with-pydebug on RHEL5 with Fatal Python error: Invalid thread state for this thread

[Charles-Francois Natali]

Changes by Charles-Francois Natali neolo...@free.fr:


Removed file: http://bugs.python.org/file21801/tls_reinit.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10517
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10517] test_concurrent_futures crashes with --with-pydebug on RHEL5 with Fatal Python error: Invalid thread state for this thread

[Charles-Francois Natali]

Changes by Charles-Francois Natali neolo...@free.fr:


Removed file: http://bugs.python.org/file21678/thread_invalid_key.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10517
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10517] test_concurrent_futures crashes with --with-pydebug on RHEL5 with Fatal Python error: Invalid thread state for this thread

[Charles-Francois Natali]

Charles-Francois Natali neolo...@free.fr added the comment:

Here's an updated patch, tested on RHEL4U8.

--
Added file: http://bugs.python.org/file21804/tls_reinit.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10517
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10517] test_concurrent_futures crashes with --with-pydebug on RHEL5 with Fatal Python error: Invalid thread state for this thread

[Antoine Pitrou]

Changes by Antoine Pitrou pit...@free.fr:


--
dependencies:  -multiprocessing generates a fatal error
stage:  - commit review
superseder:  - multiprocessing generates a fatal error
versions: +Python 2.7, Python 3.1, Python 3.3

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10517
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10517] test_concurrent_futures crashes with --with-pydebug on RHEL5 with Fatal Python error: Invalid thread state for this thread

[Antoine Pitrou]

Changes by Antoine Pitrou pit...@free.fr:


--
superseder: multiprocessing generates a fatal error - 

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10517
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10517] test_concurrent_futures crashes with --with-pydebug on RHEL5 with Fatal Python error: Invalid thread state for this thread

[Roundup Robot]

Roundup Robot devnull@devnull added the comment:

New changeset f6feed6ec3f9 by Antoine Pitrou in branch '2.7':
Issue #10517: After fork(), reinitialize the TLS used by the PyGILState_*
http://hg.python.org/cpython/rev/f6feed6ec3f9

--
nosy: +python-dev

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10517
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10517] test_concurrent_futures crashes with --with-pydebug on RHEL5 with Fatal Python error: Invalid thread state for this thread

[Roundup Robot]

Roundup Robot devnull@devnull added the comment:

New changeset 7b7ad9a88451 by Antoine Pitrou in branch '3.2':
Issue #10517: After fork(), reinitialize the TLS used by the PyGILState_*
http://hg.python.org/cpython/rev/7b7ad9a88451

New changeset c8f283cd3e6e by Antoine Pitrou in branch 'default':
Issue #10517: After fork(), reinitialize the TLS used by the PyGILState_*
http://hg.python.org/cpython/rev/c8f283cd3e6e

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10517
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10517] test_concurrent_futures crashes with --with-pydebug on RHEL5 with Fatal Python error: Invalid thread state for this thread

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

It should be fixed now! Thank you.

--
resolution:  - fixed
stage: commit review - committed/rejected
status: open - closed
versions:  -Python 3.1

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10517
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10517] test_concurrent_futures crashes with --with-pydebug on RHEL5 with Fatal Python error: Invalid thread state for this thread

[Charles-Francois Natali]

Charles-Francois Natali neolo...@free.fr added the comment:

 Not necessarily. You can have several interpreters (and therefore several 
 thread states) in a single thread, using Py_NewInterpreter(). It's used by 
 mod_wsgi and probably other software. If you overwrite the old value with the 
 new one, it may break such software.


OK, I didn't know. Better not to change that in that case.

 Would it be possible to cleanup the autoTLS mappings in PyOS_AfterFork() 
 instead?


Well, after fork, all threads have exited, so you'll be running on the
behalf of the child process' main - and only - thread, so by
definition you can't access other threads' thread-specific data, no?
As an alternate solution, I was thinking of calling
PyThread_delete_key_value(autoTLSkey) in the path of thread bootstrap,
i.e. starting in Modules/_threadmodule.c t_bootstrap. Obviously, this
should be done before calling _PyThreadState_Init, since it can also
be called from Py_NewInterpreter.
The problem is that it would require exporting autoTLSkey whose scope
is now limited to pystate.c (we could also create a small wrapper
function in pystate.c to delete the autoTLSkey, since it's already
done in PyThreadState_DeleteCurrent and PyThreadState_Delete).

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10517
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10517] test_concurrent_futures crashes with --with-pydebug on RHEL5 with Fatal Python error: Invalid thread state for this thread

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

 Well, after fork, all threads have exited, so you'll be running on the
 behalf of the child process' main - and only - thread, so by
 definition you can't access other threads' thread-specific data, no?

A rather good point :)
How about deleting the mapping (pthread_key_delete) and recreating it
from scratch, then?

 As an alternate solution, I was thinking of calling
 PyThread_delete_key_value(autoTLSkey) in the path of thread bootstrap,
 i.e. starting in Modules/_threadmodule.c t_bootstrap.

That would somewhat alleviate the problem, but only for Python-created
threads. Threads created through other means (for example mod_wsgi, or
database wrappers having their own thread pools) would still face the
original issue.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10517
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10517] test_concurrent_futures crashes with --with-pydebug on RHEL5 with Fatal Python error: Invalid thread state for this thread

[Charles-Francois Natali]

Charles-Francois Natali neolo...@free.fr added the comment:

 How about deleting the mapping (pthread_key_delete) and recreating it
 from scratch, then?

Sounds good.
So the idea would be to retrieve the current thread's tstate, destroy the 
current autoTLSkey, re-create it, and re-associate the current tstate to this 
new key. I just did a quick test on RHEL4 and it works.
PyThread_ReinitTLS looks like a good candidate for that, but it's the same 
problem, autoTLSkey scope is limited to pystates.c (and I'm not sure that the 
tstate should be exposed to platform thread implementations).
There's also PyEval_ReinitThreads in ceval.c, exposing the autoTLSkey would 
make more sense (and it already knows about tstate, of course).
Where would you put it?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10517
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10517] test_concurrent_futures crashes with --with-pydebug on RHEL5 with Fatal Python error: Invalid thread state for this thread

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

  How about deleting the mapping (pthread_key_delete) and recreating it
  from scratch, then?
 
 Sounds good.
 So the idea would be to retrieve the current thread's tstate, destroy
 the current autoTLSkey, re-create it, and re-associate the current
 tstate to this new key. I just did a quick test on RHEL4 and it works.
 PyThread_ReinitTLS looks like a good candidate for that, but it's the
 same problem, autoTLSkey scope is limited to pystates.c (and I'm not
 sure that the tstate should be exposed to platform thread
 implementations).
 There's also PyEval_ReinitThreads in ceval.c, exposing the autoTLSkey
 would make more sense (and it already knows about tstate, of course).
 Where would you put it?

You could add a new _PyGILState_ReInit() function and call it from
PyOS_AfterFork() or PyEval_ReInitThreads().

(perhaps you also need to add a TLS-destroying function to thread.c, I
haven't looked)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10517
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10517] test_concurrent_futures crashes with --with-pydebug on RHEL5 with Fatal Python error: Invalid thread state for this thread

[Charles-Francois Natali]

Charles-Francois Natali neolo...@free.fr added the comment:

 So, if it is possible to fix this and remove this weird special case and cast 
 it into the abyss, then by all means, you have my 10 thumbs up.  Not that it 
 counts for much :)

Me too.
We still have a couple hundred RHEL4/5 boxes at work, and I guess we're not 
alone in this case. It's a really specific case, but I think it would be nice 
to fix it, especially since it also makes the code more understandable and less 
error-prone. Unless of course this special treatment is really necessary, in 
which case I'll have to think of another solution or just drop it.
I'm adding Antoine to the noisy list, since he's noted as thread expert in the 
Experts Index.

--
nosy: +pitrou

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10517
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10517] test_concurrent_futures crashes with --with-pydebug on RHEL5 with Fatal Python error: Invalid thread state for this thread

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

 I think that if we were to call PyThread_set_key_value twice on the
 same key it's either an error, or we want the last version to be
 stored, not the old one.

Not necessarily. You can have several interpreters (and therefore several 
thread states) in a single thread, using Py_NewInterpreter(). It's used by 
mod_wsgi and probably other software. If you overwrite the old value with the 
new one, it may break such software.

Would it be possible to cleanup the autoTLS mappings in PyOS_AfterFork() 
instead?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10517
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10517] test_concurrent_futures crashes with --with-pydebug on RHEL5 with Fatal Python error: Invalid thread state for this thread

[Charles-Francois Natali]

Charles-Francois Natali neolo...@free.fr added the comment:

This is due to a bug in the TLS key management when mixed with fork.
Here's what happens:
When a thread is created, a tstate is allocated and stored in the thread's TLS:
thread_PyThread_start_new_thread - t_bootstrap - _PyThreadState_Init - 
_PyGILState_NoteThreadState:

if (PyThread_set_key_value(autoTLSkey, (void *)tstate)  0)
Py_FatalError(Couldn't create autoTLSkey mapping);

where 
int
PyThread_set_key_value(int key, void *value)
{
int fail;
void *oldValue = pthread_getspecific(key);
if (oldValue != NULL)
return 0;
fail = pthread_setspecific(key, value);
return fail;
}

A pthread_getspecific(key) is performed to see if there was already a value 
associated to this key.
The problem is that, if a process has a thread with a given thread ID (and a 
tstate stored in its TLS), and then the process forks (from another thread), if 
a new thread is created with the same thread ID as the thread in the child 
process, pthread_getspecific(key) will return the value stored by the other 
thread (with the same thread ID). In short, thread-specific values are 
inherited across fork, and if you're unlucky and create a thread with a thread 
ID already existing in the parent process, you're screwed.
To conclude, PyGILState_GetThisThreadState, which calls 
PyThread_get_key_value(autoTLSkey) will return the other thread's tstate, which 
will triggers this fatal error in PyThreadState_Swap.

The patch attached fixes this issue by removing the call to 
pthread_getspecific(key) from PyThread_set_key_value. This solves the problem 
and doesn't seem to cause any regression in test_threading and 
test_multiprocessing, and I think that if we were to call 
PyThread_set_key_value twice on the same key it's either an error, or we want 
the last version to be stored, not the old one.
test_threading and test_multiprocessing now run fine without any fatal error.

Note that this is probably be a bug in RHEL pthread's implementation, but given 
how widespread RHEL and derived distros are, I think this should be fixed.
I've attached a patch and a small test program to check if thread-specific data 
is inherited across a fork.
Here's a sample run on a RHEL4U8 box:

$ /tmp/test
PID: 17922, TID: 3086187424, init value: (nil)
PID: 17924, TID: 3086187424, init value: 0xdeadbeef

The second thread has been created in the child process and inherited the first 
thread's (created by the parent) key's value (one condition for this to happen 
is of course that the second thread is allocated the same thread ID as the 
first one).

--
nosy: +neologix

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10517
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10517] test_concurrent_futures crashes with --with-pydebug on RHEL5 with Fatal Python error: Invalid thread state for this thread

[Charles-Francois Natali]

Changes by Charles-Francois Natali neolo...@free.fr:


Added file: http://bugs.python.org/file21677/test_specific.c

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10517
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10517] test_concurrent_futures crashes with --with-pydebug on RHEL5 with Fatal Python error: Invalid thread state for this thread

[Charles-Francois Natali]

Changes by Charles-Francois Natali neolo...@free.fr:


--
keywords: +patch
Added file: http://bugs.python.org/file21678/thread_invalid_key.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10517
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10517] test_concurrent_futures crashes with --with-pydebug on RHEL5 with Fatal Python error: Invalid thread state for this thread

[Charles-Francois Natali]

Charles-Francois Natali neolo...@free.fr added the comment:

Note: this seems to be fixed in RHEL6.
(Sorry for the noise).

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10517
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10517] test_concurrent_futures crashes with --with-pydebug on RHEL5 with Fatal Python error: Invalid thread state for this thread

[Kristján Valur Jónsson]

Kristján Valur Jónsson krist...@ccpgames.com added the comment:

Now, I'd be super happy to see this strange semantics of PyThread_set_key_value 
go away.  Its very un-standard and complicates the mapping from an native 
implementation to the python one.
But I think I did once bring up this issue, and was told that it was a bad idea.
But your logic is sound.  Doing two Sets, is an error regardless.  Hiding the 
error by ignoring the second set is arbitrarily as bad as ignoring the first 
thing.
So, if it is possible to fix this and remove this weird special case and cast 
it into the abyss, then by all means, you have my 10 thumbs up.  Not that it 
counts for much :)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10517
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10517] test_concurrent_futures crashes with --with-pydebug on RHEL5 with Fatal Python error: Invalid thread state for this thread

[Kristján Valur Jónsson]

Kristján Valur Jónsson krist...@ccpgames.com added the comment:

What is the line that the parent process is executing?  Line numbers don't seem 
to match any more.
And is it possible to set a breakpoint in the child process where the fatal 
error is triggered?  It would be good to know what is being run at that point.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10517
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10517] test_concurrent_futures crashes with --with-pydebug on RHEL5 with Fatal Python error: Invalid thread state for this thread

[Sandro Tosi]

Sandro Tosi sandro.t...@gmail.com added the comment:

Is someone still able to replicate this crash? I'm not, with a fresh built 3.2 
and default (3.3), --with-pydebug enabled. Brian confirmed on msg132418 that he 
can't any longer replicate it.

--
nosy: +sandro.tosi

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10517
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10517] test_concurrent_futures crashes with --with-pydebug on RHEL5 with Fatal Python error: Invalid thread state for this thread

[Dave Malcolm]

Dave Malcolm dmalc...@redhat.com added the comment:

I tried again, and I'm still able to reproduce this bug on a RHEL5 box with 
cpython --with-pydebug as of a recent checkout (69030:00217100b9e7 as it 
happens):

$ ./python -c import multiprocessing.managers ; mpp = multiprocessing.Pool(4); 
sm = multiprocessing.managers.SyncManager(); sm.start()
Fatal Python error: Invalid thread state for this thread
[66448 refs]

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10517
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10517] test_concurrent_futures crashes with --with-pydebug on RHEL5 with Fatal Python error: Invalid thread state for this thread

[Dave Malcolm]

Dave Malcolm dmalc...@redhat.com added the comment:

I spent some time bisecting the SVN history in the py3k branch, and believe 
that r84914 is the commit that introduced this issue.

Details:
  
  Trying on 4-core i386 RHEL 5 box
  $ svn up -r REV
  $ make clean ; make
  (configured --with-pydebug)

Reproducer:

  ./python -c import multiprocessing.managers ; mpp = multiprocessing.Pool(4); 
sm = multiprocessing.managers.SyncManager(); sm.start()

  (running it 3 times; each time, at each build I see it either successfully 
completing 3 times, or having the Py_FatalError 3 times)

Bisecting:
  r86720: CRASHES with r86729 --with-pydebug (from when I was investigating 
this before)
  r73573: Runs OK with r73573 --with-pydebug (r73573 was origin for 3.1 tag)

= somewhere in 73572..86729

  r76195: Runs OK with r76195 --with-pydebug (r76195 added the new GIL)

= somewhere in 76195..86729

  r80724: Runs OK with r80724 --with-pydebug (changed threading code to Make 
(most of) Python's tests pass under Thread Sanitizer.)

= somewhere in 80724..86729

  r83722: Runs OK with r80724 --with-pydebug (touched multiprocessing)

= somewhere in 83722..86729

  r85222: CRASHES with r85222 --with-pydebug (rough midpoint)

= somewhere in 83722..85222

  r84472: Runs OK with r84472 --with-pydebug (arbitrary midpoint)

= somewhere in 84472..85222

  r84847: Runs OK with r84847 --with-pydebug (arbitrary midpoint)

 = somewhere in 84847..85222

  r85033: CRASHES with r85033 --with-pydebug (arbitrary midpoint)

 = somewhere in 84847..85033

  r84938: CRASHES with r84938 --with-pydebug (arbitrary midpoint)

 = somewhere in 84847..84938

  r84892: Runs OK with r84892 --with-pydebug (arbitrary midpoint)

 = somewhere in 84892..84938

  r84915: CRASHES with r84915 --with-pydebug (arbitrary midpoint)

 = somewhere in 84892..84915

  r84903: Runs OK with r84903 --with-pydebug (arbitrary midpoint)

 = somewhere in 84903..84915

  r84914: CRASHES with r84914 --with-pydebug (affects threads)

 = somewhere in 84903..84914

  r84913: Runs OK with r84913 --with-pydebug (previous commit before 84914)

 = r84914 is the commit that triggered this issue

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10517
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10517] test_concurrent_futures crashes with --with-pydebug on RHEL5 with Fatal Python error: Invalid thread state for this thread

[Dave Malcolm]

Changes by Dave Malcolm dmalc...@redhat.com:


--
nosy: +krisvale

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10517
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10517] test_concurrent_futures crashes with --with-pydebug on RHEL5 with Fatal Python error: Invalid thread state for this thread

[Dave Malcolm]

Dave Malcolm dmalc...@redhat.com added the comment:

r84914 was the implementation of issue 9786 (Native TLS support for pthreads)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10517
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10517] test_concurrent_futures crashes with --with-pydebug on RHEL5 with Fatal Python error: Invalid thread state for this thread

[Dave Malcolm]

Dave Malcolm dmalc...@redhat.com added the comment:

This appears to be happening in a child process when the parent process is 
running:
  Lib/multiprocessing/util.py, line 255, in _exit_function ()

Liberally adding printf() and getpid() calls in various places, seems to always 
happen when parent process is within call_py_exitfuncs() within Py_Finalize; 
error is from the child process that was created last.

Using gdb with a breakpoint on call_py_exitfuncs and single-stepping seems to 
confirm a single exitfunc:
  Lib/multiprocessing/util.py, line 255, in _exit_function ()
and that the child dies as that bytecode function is executed.

$ ./python -c import multiprocessing.managers ; mpp = multiprocessing.Pool(4); 
sm = multiprocessing.managers.SyncManager(); sm.start()
Py_InitializeEx called for PID 27824
posix_fork called by PID 27824
child of posix_fork has PID 27825
posix_fork called by PID 27824
child of posix_fork has PID 27826
posix_fork called by PID 27824
child of posix_fork has PID 27827
posix_fork called by PID 27824
child of posix_fork has PID 27828
posix_fork called by PID 27824
child of posix_fork has PID 27832
Py_Finalize called for PID 27824
wait_for_thread_shutdown() finished for PID 27824
Fatal Python error for PID 27832: Invalid thread state for this thread
call_py_exitfuncs() finished for PID 27824
PyOS_FiniInterrupts() finished for PID 27824
[64240 refs]

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10517
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10517] test_concurrent_futures crashes with --with-pydebug on RHEL5 with Fatal Python error: Invalid thread state for this thread

[Dave Malcolm]

Changes by Dave Malcolm dmalc...@redhat.com:


--
title: test_concurrent_futures crashes with Fatal Python error: Invalid thread 
state for this thread - test_concurrent_futures crashes with --with-pydebug 
on RHEL5 with Fatal Python error: Invalid thread state for this thread

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10517
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10517] test_concurrent_futures crashes with Fatal Python error: Invalid thread state for this thread

[Ray.Allen]

Ray.Allen ysj@gmail.com added the comment:

Couldn't repro this on my debian 5.

--
nosy: +ysj.ray

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10517
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10517] test_concurrent_futures crashes with Fatal Python error: Invalid thread state for this thread

[Brian Quinlan]

Brian Quinlan br...@sweetapp.com added the comment:

I've filed a new bug (http://bugs.python.org/issue10632) against 
multiprocessing and this bug dependent on it.

In the meantime, I can't repro this on ubuntu 10.04 LTS so I'm going to install 
Centos and give that a go.

--
dependencies: +multiprocessing gene

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10517
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: To Thread or not to Thread....?

[John Nagle]

On 12/1/2010 1:24 AM, Antoine Pitrou wrote:

On Wed, 1 Dec 2010 02:45:50 +
Jack Keeganwhatsjacksem...@gmail.com  wrote:


Hi there,

I'm currently writing an application to control and take measurements during
an experiments. This is to be done on an embedded computer running XPe so I
am happy to have python available, although I am pretty new to it.
The application basically runs as a state machine, which transitions through
it's states based on inputs read in from a set of general purpose
input/output (GPIO) lines. So when a certain line is pulled low/high, do
something and move to another state. All good so far and since I get through
main loop pretty quickly, I can just do a read of the GPIO lines on each
pass through the loop and respond accordingly.



However, in one of the states I have to start reading in, and storing frames
from a camera. In another, I have to start reading accelerometer data from
an I2C bus (which operates at 400kHz). I haven't implemented either yet but
I would imagine that, in the case of the camera data, reading a frame would
take a large amount of time as compared to other operations. Therefore, if I
just tried to read one (or one set of) frames on each pass through the loop
then I would hold up the rest of the application. Conversely, as the I2C bus
will need to be read at such a high rate, I may not be able to get the
required data rate I need even without the camera data. This naturally leads
me to think I need to use threads.
As I am no expert in either I2C, cameras, python or threading I thought I
would chance asking for some advice on the subject. Do you think I need
threads here or would I be better off using some other method. I was
previously toying with the idea of using generators to create weightless
threads (as detailed in
http://www.ibm.com/developerworks/library/l-pythrd.html) for reading the
GPIOs. Do you think this would work in this situation?


The main question IMO: the I2C bus operates at 400kHz, but how much
received data can it buffer? That will give you a hint as to how much
latency you can tolerate.


   Right.  Neither Windows XPe nor Python is designed for hard real time
work.  Unless your peripheral devices have considerable buffering, don't
expect this to work without missing a measurement once in a while.
If you have to talk to the I2C interface for each bus transaction before
you get another reading, so that you have to respond to the device
within 2ms every time, this probably won't work.

   If you were writing this in C++ on QNX, which is intended for
hard real-time work, you could easily meet those time requirements.
I've done that, and just reading the devices took maybe 3% of a
Pentium 4.  Python on Windows XPe, maybe not.  You'll need to test.
XPe is just XP with some stuff taken out; it's not a real-time OS.

   A good test is to hook up a square wave generator to some I2C device
you can read, and try to read it in real time, measuring the time in
microseconds between each pulse.  Turn up the frequency until you start
missing events.  That will give you a handle on how good your real time
capabilities are.

   Note that the machine you're using matters.  Some computers have
stuff going on in system management mode that results in long
interrupt lockout latencies.

   One option may be to put something like an Arduno on a USB port,
and let it talk to the I2C device.  See

   http://www.arduino.cc/playground/Learning/I2C

Let it queue up and timestamp the accelerometer data, which can then
be read in blocks from the USB port.  If both the accelerometer data
and video frames are buffered, then you can do your processing in
Python without trying to meet hard real-time constraints.  This is
the usual approach in Windows land.

John Nagle
--
http://mail.python.org/mailman/listinfo/python-list

Re: To Thread or not to Thread....?

[Antoine Pitrou]

On Wed, 1 Dec 2010 02:45:50 +
Jack Keegan whatsjacksem...@gmail.com wrote:

 Hi there,
 
 I'm currently writing an application to control and take measurements during
 an experiments. This is to be done on an embedded computer running XPe so I
 am happy to have python available, although I am pretty new to it.
 The application basically runs as a state machine, which transitions through
 it's states based on inputs read in from a set of general purpose
 input/output (GPIO) lines. So when a certain line is pulled low/high, do
 something and move to another state. All good so far and since I get through
 main loop pretty quickly, I can just do a read of the GPIO lines on each
 pass through the loop and respond accordingly.

 However, in one of the states I have to start reading in, and storing frames
 from a camera. In another, I have to start reading accelerometer data from
 an I2C bus (which operates at 400kHz). I haven't implemented either yet but
 I would imagine that, in the case of the camera data, reading a frame would
 take a large amount of time as compared to other operations. Therefore, if I
 just tried to read one (or one set of) frames on each pass through the loop
 then I would hold up the rest of the application. Conversely, as the I2C bus
 will need to be read at such a high rate, I may not be able to get the
 required data rate I need even without the camera data. This naturally leads
 me to think I need to use threads.
 As I am no expert in either I2C, cameras, python or threading I thought I
 would chance asking for some advice on the subject. Do you think I need
 threads here or would I be better off using some other method. I was
 previously toying with the idea of using generators to create weightless
 threads (as detailed in
 http://www.ibm.com/developerworks/library/l-pythrd.html) for reading the
 GPIOs. Do you think this would work in this situation?

The main question IMO: the I2C bus operates at 400kHz, but how much
received data can it buffer? That will give you a hint as to how much
latency you can tolerate.

I don't think soft threads would work at all, since they wouldn't allow
overlapping between frame reading and other ops. If frame reading
releases the GIL (as any properly implemented IO primitive in Python
should), then at least you can try using OS threads.

Then, depending on the tolerable latency for I2C operation, you can try
to run it as an OS thread, or a separate process (if running as a
separate process, make sure it cannot block while sending IO to the
master process).

Regards

Antoine.


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: To Thread or not to Thread....?

[Jack Keegan]

Hi Antoine,

On Wed, Dec 1, 2010 at 9:24 AM, Antoine Pitrou solip...@pitrou.net wrote:


 The main question IMO: the I2C bus operates at 400kHz, but how much
 received data can it buffer? That will give you a hint as to how much
 latency you can tolerate.

I'm not sure on buffering, but I have to ask the device on the I2C bus for a
sample each time I want one. This is done by sending out the address of the
device on the I2C bus, waiting for a reply, then addressing the registers I
want to read data from, then read the data. Therefore, I would need to make
the request at the rate I require.


 I don't think soft threads would work at all, since they wouldn't allow
 overlapping between frame reading and other ops. If frame reading
 releases the GIL (as any properly implemented IO primitive in Python
 should), then at least you can try using OS threads.

It's actually a separate piece of hardware that does the frame
reading/encoding. I then interface to this via a DLL so I'm not sure how it
handles things in the background. Unless I am mistaken, I have to allocate a
chunk of memory for it to read the frames into, while I read from there.
It's probably some sort of ring buffer implementation so I'd have to read
the frames out and store them to disk before the next run through the buffer
overwrites the old frames.

How does that sound, process/thread wise?

Thanks,

Jack


-- 
The earth is a very small stage in a vast cosmic arena. Think of the rivers
of blood spilled by all those generals and emperors so that in glory and in
triumph they could become the momentary masters of a fraction of a dot.
- Carl Sagan [Pale Blue Dot]
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: To Thread or not to Thread....?

[Antoine Pitrou]

On Wed, 1 Dec 2010 11:50:46 +
Jack Keegan whatsjacksem...@gmail.com wrote:
 Hi Antoine,
 
 On Wed, Dec 1, 2010 at 9:24 AM, Antoine Pitrou solip...@pitrou.net wrote:
 
 
  The main question IMO: the I2C bus operates at 400kHz, but how much
  received data can it buffer? That will give you a hint as to how much
  latency you can tolerate.
 
 I'm not sure on buffering, but I have to ask the device on the I2C bus for a
 sample each time I want one. This is done by sending out the address of the
 device on the I2C bus, waiting for a reply, then addressing the registers I
 want to read data from, then read the data. Therefore, I would need to make
 the request at the rate I require.

Sounds like you want to do this part in raw C and in a separate process,
IMO.

  I don't think soft threads would work at all, since they wouldn't allow
  overlapping between frame reading and other ops. If frame reading
  releases the GIL (as any properly implemented IO primitive in Python
  should), then at least you can try using OS threads.
 
 It's actually a separate piece of hardware that does the frame
 reading/encoding. I then interface to this via a DLL so I'm not sure how it
 handles things in the background. Unless I am mistaken, I have to allocate a
 chunk of memory for it to read the frames into, while I read from there.

Well, if you are the one writing the CPython interface against the DLL,
you are responsible for releasing the GIL properly.

 It's probably some sort of ring buffer implementation so I'd have to read
 the frames out and store them to disk before the next run through the buffer
 overwrites the old frames.

You have to evaluate the latency requirements there too.

I have to point out that I have no experience on embedded software, so
the advice I'm giving is out of pure common sense :)

Regards

Antoine.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: To Thread or not to Thread....?

[James Mills]

Surely I2C is just a serial-like interface
and one should be able to do async I/O on it ?

The use of threads is not necessary here and the GIL
doesn't become a problem in async I/O anyway.

I only use threads for operating that might block (not for I/O).

cheers
James

On Wed, Dec 1, 2010 at 7:24 PM, Antoine Pitrou solip...@pitrou.net wrote:
 On Wed, 1 Dec 2010 02:45:50 +
 Jack Keegan whatsjacksem...@gmail.com wrote:

 Hi there,

 I'm currently writing an application to control and take measurements during
 an experiments. This is to be done on an embedded computer running XPe so I
 am happy to have python available, although I am pretty new to it.
 The application basically runs as a state machine, which transitions through
 it's states based on inputs read in from a set of general purpose
 input/output (GPIO) lines. So when a certain line is pulled low/high, do
 something and move to another state. All good so far and since I get through
 main loop pretty quickly, I can just do a read of the GPIO lines on each
 pass through the loop and respond accordingly.

 However, in one of the states I have to start reading in, and storing frames
 from a camera. In another, I have to start reading accelerometer data from
 an I2C bus (which operates at 400kHz). I haven't implemented either yet but
 I would imagine that, in the case of the camera data, reading a frame would
 take a large amount of time as compared to other operations. Therefore, if I
 just tried to read one (or one set of) frames on each pass through the loop
 then I would hold up the rest of the application. Conversely, as the I2C bus
 will need to be read at such a high rate, I may not be able to get the
 required data rate I need even without the camera data. This naturally leads
 me to think I need to use threads.
 As I am no expert in either I2C, cameras, python or threading I thought I
 would chance asking for some advice on the subject. Do you think I need
 threads here or would I be better off using some other method. I was
 previously toying with the idea of using generators to create weightless
 threads (as detailed in
 http://www.ibm.com/developerworks/library/l-pythrd.html) for reading the
 GPIOs. Do you think this would work in this situation?

 The main question IMO: the I2C bus operates at 400kHz, but how much
 received data can it buffer? That will give you a hint as to how much
 latency you can tolerate.

 I don't think soft threads would work at all, since they wouldn't allow
 overlapping between frame reading and other ops. If frame reading
 releases the GIL (as any properly implemented IO primitive in Python
 should), then at least you can try using OS threads.

 Then, depending on the tolerable latency for I2C operation, you can try
 to run it as an OS thread, or a separate process (if running as a
 separate process, make sure it cannot block while sending IO to the
 master process).

 Regards

 Antoine.


 --
 http://mail.python.org/mailman/listinfo/python-list




-- 
-- James Mills
--
-- Problems are solved by method
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: To Thread or not to Thread....?

[Werner Thie]

Hi

I see quite a few alleys to go down when stuck with such types of 
problems, but instead of listing and discussing them have a look at a 
quite complete discussion and comparison of the various async 
programming options available at


http://syncless.googlecode.com

Also have a look at the presentation at

http://syncless.googlecode.com/svn/trunk/doc/slides_2010-11-29/pts_coro_2010-11-29.html

If I were in your shoes I would solve the problem in stackless, if 
exchanging the python interpreter is not possible then Twisted would be 
my second choice, having done a lot of work with it (see the NMEA 
classes for serial ports).


I don't know if syncless handles other types of fd's like serial ports, 
I've never played with it. The monkey patching of syncless might pose 
other problems in your case.


HTH, Werner


Am 01.12.2010 14:48, schrieb James Mills:

Surely I2C is just a serial-like interface
and one should be able to do async I/O on it ?

The use of threads is not necessary here and the GIL
doesn't become a problem in async I/O anyway.

I only use threads for operating that might block (not for I/O).

cheers
James

On Wed, Dec 1, 2010 at 7:24 PM, Antoine Pitrousolip...@pitrou.net  wrote:

On Wed, 1 Dec 2010 02:45:50 +
Jack Keeganwhatsjacksem...@gmail.com  wrote:


Hi there,

I'm currently writing an application to control and take measurements during
an experiments. This is to be done on an embedded computer running XPe so I
am happy to have python available, although I am pretty new to it.
The application basically runs as a state machine, which transitions through
it's states based on inputs read in from a set of general purpose
input/output (GPIO) lines. So when a certain line is pulled low/high, do
something and move to another state. All good so far and since I get through
main loop pretty quickly, I can just do a read of the GPIO lines on each
pass through the loop and respond accordingly.



However, in one of the states I have to start reading in, and storing frames
from a camera. In another, I have to start reading accelerometer data from
an I2C bus (which operates at 400kHz). I haven't implemented either yet but
I would imagine that, in the case of the camera data, reading a frame would
take a large amount of time as compared to other operations. Therefore, if I
just tried to read one (or one set of) frames on each pass through the loop
then I would hold up the rest of the application. Conversely, as the I2C bus
will need to be read at such a high rate, I may not be able to get the
required data rate I need even without the camera data. This naturally leads
me to think I need to use threads.
As I am no expert in either I2C, cameras, python or threading I thought I
would chance asking for some advice on the subject. Do you think I need
threads here or would I be better off using some other method. I was
previously toying with the idea of using generators to create weightless
threads (as detailed in
http://www.ibm.com/developerworks/library/l-pythrd.html) for reading the
GPIOs. Do you think this would work in this situation?


The main question IMO: the I2C bus operates at 400kHz, but how much
received data can it buffer? That will give you a hint as to how much
latency you can tolerate.

I don't think soft threads would work at all, since they wouldn't allow
overlapping between frame reading and other ops. If frame reading
releases the GIL (as any properly implemented IO primitive in Python
should), then at least you can try using OS threads.

Then, depending on the tolerable latency for I2C operation, you can try
to run it as an OS thread, or a separate process (if running as a
separate process, make sure it cannot block while sending IO to the
master process).

Regards

Antoine.


--
http://mail.python.org/mailman/listinfo/python-list





attachment: werner.vcf-- 
http://mail.python.org/mailman/listinfo/python-list

Re: To Thread or not to Thread....?

[Antoine Pitrou]

On Wed, 1 Dec 2010 23:48:38 +1000
James Mills prolo...@shortcircuit.net.au wrote:
 Surely I2C is just a serial-like interface
 and one should be able to do async I/O on it ?
 
 The use of threads is not necessary here and the GIL
 doesn't become a problem in async I/O anyway.

Well, you are missing the point. The OP wants to do operations that
have certain latency requirements while doing other operations in
parallel. Using several preemptively-switching threads (or processes)
is the simplest way of achieving that; you can do cooperative
multithreading (which is conceptually the same as single-threaded async
programming), but you'll have to insert as many explicit
synchronization points as necessary to achieve the latency objectives.
Not very practical obviously.

(and, yes, the GIL amounts to some bastardized, fine-grained form of
cooperative multithreading, which is why separate processes might be a
better answer if the latency requirements are tight - especially on
Python 2.x-3.1 where the GIL is badly implemented)

Regards

Antoine.
-- 
http://mail.python.org/mailman/listinfo/python-list

To Thread or not to Thread....?

[Jack Keegan]

Hi there,

I'm currently writing an application to control and take measurements during
an experiments. This is to be done on an embedded computer running XPe so I
am happy to have python available, although I am pretty new to it.
The application basically runs as a state machine, which transitions through
it's states based on inputs read in from a set of general purpose
input/output (GPIO) lines. So when a certain line is pulled low/high, do
something and move to another state. All good so far and since I get through
main loop pretty quickly, I can just do a read of the GPIO lines on each
pass through the loop and respond accordingly.
However, in one of the states I have to start reading in, and storing frames
from a camera. In another, I have to start reading accelerometer data from
an I2C bus (which operates at 400kHz). I haven't implemented either yet but
I would imagine that, in the case of the camera data, reading a frame would
take a large amount of time as compared to other operations. Therefore, if I
just tried to read one (or one set of) frames on each pass through the loop
then I would hold up the rest of the application. Conversely, as the I2C bus
will need to be read at such a high rate, I may not be able to get the
required data rate I need even without the camera data. This naturally leads
me to think I need to use threads.
As I am no expert in either I2C, cameras, python or threading I thought I
would chance asking for some advice on the subject. Do you think I need
threads here or would I be better off using some other method. I was
previously toying with the idea of using generators to create weightless
threads (as detailed in
http://www.ibm.com/developerworks/library/l-pythrd.html) for reading the
GPIOs. Do you think this would work in this situation?
Another option would be to write separately programs, perhaps even in C, and
spawn these in the background when needed. I'm a little torn as to which way
to go. If it makes a difference and more in case you are wondering, I will
be interfacing to the GPIOs, cameras and I2C bus through a set of C DLLs
using Ctypes.

Any help or suggestions will be greatly appreciated,

Thanks very much,

Jack
-- 
http://mail.python.org/mailman/listinfo/python-list

[issue10517] test_concurrent_futures crashes with Fatal Python error: Invalid thread state for this thread

[Dave Malcolm]

Changes by Dave Malcolm dmalc...@redhat.com:


--
nosy: +dmalcolm

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10517
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10517] test_concurrent_futures crashes with Fatal Python error: Invalid thread state for this thread

[Dave Malcolm]

Dave Malcolm dmalc...@redhat.com added the comment:

I'm able to reliably reproduce this on a RHEL 5 box (i386 in this case).

All of the ProcessPool* unittest subclasses within 
Lib/test/test_concurrent_futures.py exhibit this hang, each printing out just 
the name of the first test (so presumably either within the first test method, 
or in shared setup/teardown).

None of the other subclasses hang.

You need to build with --with-pydebug to see this: the error message is coming 
from this code in PyThreadState_Swap in Python/pystate.c:

   390  #if defined(Py_DEBUG)  defined(WITH_THREAD)
   391  if (newts) {
   392  /* This can be called from PyEval_RestoreThread(). Similar
   393 to it, we need to ensure errno doesn't change.
   394  */
   395  int err = errno;
   396  PyThreadState *check = PyGILState_GetThisThreadState();
   397  if (check  check-interp == newts-interp  check != newts)
398  Py_FatalError(Invalid thread state for this thread);
   399  errno = err;
   400  }
   401  #endif

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10517
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10517] test_concurrent_futures crashes with Fatal Python error: Invalid thread state for this thread

[Dave Malcolm]

Dave Malcolm dmalc...@redhat.com added the comment:

Minimal reproducer:
$ ./python -c from concurrent.futures import * ; e = ProcessPoolExecutor() ; 
e.submit(pow, 2, 5)
Fatal Python error: Invalid thread state for this thread

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10517
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10517] test_concurrent_futures crashes with Fatal Python error: Invalid thread state for this thread

[Dave Malcolm]

Dave Malcolm dmalc...@redhat.com added the comment:

Seems to be an issue within (or triggered by) multiprocessing (test_threads and 
test_threading pass OK, fwiw):
$ ./python -m test.test_multiprocessing
Fatal Python error: Invalid thread state for this thread
Traceback (most recent call last):
  File /home/dmalcolm/coding/python-svn/py3k-clean/Lib/runpy.py, line 160, in 
_run_module_as_main
__main__, fname, loader, pkg_name)
  File /home/dmalcolm/coding/python-svn/py3k-clean/Lib/runpy.py, line 73, in 
_run_code
exec(code, run_globals)
  File 
/home/dmalcolm/coding/python-svn/py3k-clean/Lib/test/test_multiprocessing.py, 
line 2127, in module
main()
  File 
/home/dmalcolm/coding/python-svn/py3k-clean/Lib/test/test_multiprocessing.py, 
line 2124, in main
test_main(unittest.TextTestRunner(verbosity=2).run)
  File 
/home/dmalcolm/coding/python-svn/py3k-clean/Lib/test/test_multiprocessing.py, 
line 2103, in test_main
ManagerMixin.pool = ManagerMixin.manager.Pool(4)
  File 
/home/dmalcolm/coding/python-svn/py3k-clean/Lib/multiprocessing/managers.py, 
line 644, in temp
token, exp = self._create(typeid, *args, **kwds)
  File 
/home/dmalcolm/coding/python-svn/py3k-clean/Lib/multiprocessing/managers.py, 
line 542, in _create
conn = self._Client(self._address, authkey=self._authkey)
  File 
/home/dmalcolm/coding/python-svn/py3k-clean/Lib/multiprocessing/connection.py,
 line 149, in Client
answer_challenge(c, authkey)
  File 
/home/dmalcolm/coding/python-svn/py3k-clean/Lib/multiprocessing/connection.py,
 line 383, in answer_challenge
message = connection.recv_bytes(256) # reject large message
EOFError

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10517
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10517] test_concurrent_futures crashes with Fatal Python error: Invalid thread state for this thread

[Łukasz Langa]

Changes by Łukasz Langa luk...@langa.pl:


--
nosy: +jnoller

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10517
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10517] test_concurrent_futures crashes with Fatal Python error: Invalid thread state for this thread

[Dave Malcolm]

Dave Malcolm dmalc...@redhat.com added the comment:

By strategically adding print() and input() calls, I was able to isolate the 
error to this line in test_multiprocessing.py's test_main:
   ManagerMixin.pool = ManagerMixin.manager.Pool(4)

specifically, to the construction:
  ManagerMixin.manager.Pool(4)

Minimal reproducer seems to be:
 import multiprocessing.managers
 mpp = multiprocessing.Pool(4)
 sm = multiprocessing.managers.SyncManager()
 sm.start()

i.e.:

$ ./python -c import multiprocessing.managers ; mpp = multiprocessing.Pool(4); 
sm = multiprocessing.managers.SyncManager(); sm.start()
Fatal Python error: Invalid thread state for this thread

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10517
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10517] test_concurrent_futures crashes with Fatal Python error: Invalid thread state for this thread

[Dave Malcolm]

Dave Malcolm dmalc...@redhat.com added the comment:

FWIW, I was able to do an almost full run of regrtest on this box with:
  -x test_multiprocessing test_concurrent_futures

Other than those two, all tests pass.

So _something_ is going wrong w.r.t. threads, though I'm not sure what at this 
stage.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10517
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10517] test_concurrent_futures crashes with Fatal Python error: Invalid thread state for this thread

[Łukasz Langa]

New submission from Łukasz Langa luk...@langa.pl:

py3k built from trunk on Centos 5.5 freezes during regrtest on 
test_concurrent_futures with Fatal Python error: Invalid thread state for this 
thread.

A set of hopefully useful diagnostic logs attached as patch.

--
assignee: bquinlan
components: Extension Modules
files: concurrent-futures-freeze.tar.bz2
messages: 122254
nosy: bquinlan, lukasz.langa
priority: critical
severity: normal
status: open
title: test_concurrent_futures crashes with Fatal Python error: Invalid thread 
state for this thread
type: crash
versions: Python 3.2
Added file: http://bugs.python.org/file19792/concurrent-futures-freeze.tar.bz2

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10517
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10517] test_concurrent_futures crashes with Fatal Python error: Invalid thread state for this thread

[Łukasz Langa]

Łukasz Langa luk...@langa.pl added the comment:

A colorful example: http://bpaste.net/show/11493/

(just in case if downloading and extracting logs is not feasible)

Some clarification: as in a typical concurrent problem, subsequent calls freeze 
in different test cases, but the freeze itself is always reproducible and 
always during this test.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10517
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: starting a thread in a nother thread

[Aahz]

In article 4b60a661$0$1598$742ec...@news.sonic.net,
John Nagle  na...@animats.com wrote:

If a C package called from Python crashes, the package is defective.
Nothing you can do from Python should be able to cause a segmentation
fault.

...unless you use ctypes.
-- 
Aahz (a...@pythoncraft.com)   * http://www.pythoncraft.com/

import antigravity
-- 
http://mail.python.org/mailman/listinfo/python-list

starting a thread in a nother thread

[Richard Lamboj]

hello,

just for _curiosity_. What would be if i start a thread in a nother thread and 
acquire a lock in the child thread. Is there anything that could go wrong 
if someone try to start threads in threads?

Kind Regards,

Richi
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: starting a thread in a nother thread

[Stefan Behnel]

Richard Lamboj, 27.01.2010 14:06:
 just for _curiosity_. What would be if i start a thread in a nother thread 
 and 
 acquire a lock in the child thread. Is there anything that could go wrong 
 if someone try to start threads in threads?

There's usually tons of things that can go wrong w.r.t. threads:

http://www.eecs.berkeley.edu/Pubs/TechRpts/2006/EECS-2006-1.pdf

However, there's nothing special to a thread that was started from another
thread, so the problems don't change.

Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: starting a thread in a nother thread

[Richard Lamboj]

Am Wednesday 27 January 2010 14:10:13 schrieb Stefan Behnel:
 Richard Lamboj, 27.01.2010 14:06:
  just for _curiosity_. What would be if i start a thread in a nother
  thread and acquire a lock in the child thread. Is there anything that
  could go wrong if someone try to start threads in threads?

 There's usually tons of things that can go wrong w.r.t. threads:

 http://www.eecs.berkeley.edu/Pubs/TechRpts/2006/EECS-2006-1.pdf

 However, there's nothing special to a thread that was started from another
 thread, so the problems don't change.

 Stefan

Hello,

i have tried a little bit around with psycopg2 and threads,

I'am sharing one connection for all threads. When i'am starting the 
threads normal everything works without any Problem. When i'am starting the 
threads from another thread than i got a segmentation fault - i'am using 
locks. I'am sending 2048 Reqeuest at once. The normal Methode with one 
thread for every Request needs 10 seconds. The other Methode starting a 
thread and starting two other threads from this thread crashes after 10 to 40 
requests with the segmentation fault error. Any Idea why? Its a 64 bit 
maschine. Maybe i'am making something wrong?

Kind Regards

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: starting a thread in a nother thread

[Stefan Behnel]

Richard Lamboj, 27.01.2010 15:23:
 Am Wednesday 27 January 2010 14:10:13 schrieb Stefan Behnel:
 Richard Lamboj, 27.01.2010 14:06:
 just for _curiosity_. What would be if i start a thread in a nother
 thread and acquire a lock in the child thread. Is there anything that
 could go wrong if someone try to start threads in threads?
 There's usually tons of things that can go wrong w.r.t. threads:

 http://www.eecs.berkeley.edu/Pubs/TechRpts/2006/EECS-2006-1.pdf

 However, there's nothing special to a thread that was started from another
 thread, so the problems don't change.
 
 i have tried a little bit around with psycopg2 and threads,
 
 I'am sharing one connection for all threads. When i'am starting the 
 threads normal everything works without any Problem. When i'am starting the 
 threads from another thread than i got a segmentation fault

Sounds like a bug that you might want to report to the maintainers of psycopg2.

Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: starting a thread in a nother thread

[John Nagle]

Stefan Behnel wrote:

Richard Lamboj, 27.01.2010 15:23:

Am Wednesday 27 January 2010 14:10:13 schrieb Stefan Behnel:

Richard Lamboj, 27.01.2010 14:06:

just for _curiosity_. What would be if i start a thread in a nother
thread and acquire a lock in the child thread. Is there anything that
could go wrong if someone try to start threads in threads?

There's usually tons of things that can go wrong w.r.t. threads:

http://www.eecs.berkeley.edu/Pubs/TechRpts/2006/EECS-2006-1.pdf

However, there's nothing special to a thread that was started from another
thread, so the problems don't change.

i have tried a little bit around with psycopg2 and threads,

I'am sharing one connection for all threads. When i'am starting the 
threads normal everything works without any Problem. When i'am starting the 
threads from another thread than i got a segmentation fault


Sounds like a bug that you might want to report to the maintainers of psycopg2.

Stefan


  If a C package called from Python crashes, the package is defective.
Nothing you can do from Python should be able to cause a segmentation fault.

Google search: Results 1 - 10 of about 29,400 for psycopg2 crash.

John Nagle
--
http://mail.python.org/mailman/listinfo/python-list

Re: starting a thread in a nother thread

[Richard Lamboj]

Am Wednesday 27 January 2010 15:30:17 schrieb Stefan Behnel:
 Richard Lamboj, 27.01.2010 15:23:
  Am Wednesday 27 January 2010 14:10:13 schrieb Stefan Behnel:
  Richard Lamboj, 27.01.2010 14:06:
  just for _curiosity_. What would be if i start a thread in a nother
  thread and acquire a lock in the child thread. Is there anything that
  could go wrong if someone try to start threads in threads?
 
  There's usually tons of things that can go wrong w.r.t. threads:
 
  http://www.eecs.berkeley.edu/Pubs/TechRpts/2006/EECS-2006-1.pdf
 
  However, there's nothing special to a thread that was started from
  another thread, so the problems don't change.
 
  i have tried a little bit around with psycopg2 and threads,
 
  I'am sharing one connection for all threads. When i'am starting the
  threads normal everything works without any Problem. When i'am starting
  the threads from another thread than i got a segmentation fault

 Sounds like a bug that you might want to report to the maintainers of
 psycopg2.

 Stefan

Yes i should make a bug report and thanks for your help.

Just some Debugginginfo:
[New Thread 0x429b5950 (LWP 27486)]
python-dbg: ../Objects/stringobject.c:116: PyString_FromString: Assertion 
`str != ((void *)0)' failed.

Program received signal SIGABRT, Aborted.
[Switching to Thread 0x431b6950 (LWP 27484)]
0x7f6d932feed5 in raise () from /lib/libc.so.6
-- 
http://mail.python.org/mailman/listinfo/python-list

an't start a thread Pool from another thread

[Glazner]

Hi all,
I hope someone can help me with this issue
I see that i can't start a thread Pool from another thread, why?

running python 2.6.4 windowsXP
 import multiprocessing.dummy as threads
 def makePool():
threads.Pool(3)

 makePool()
 import thread
 thread.start_new(makePool,())
Unhandled exception in thread started by function makePool at
0x011EA670
6960
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: how can a child thread notify a parent thread its status?

[davidj411]

could i see an example of this maybe?
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: how can a child thread notify a parent thread its status?

[Piet van Oostrum]

 MRAB pyt...@mrabarnett.plus.com (M) wrote:

M scriptlear...@gmail.com wrote:
 I decided to go with one big log file, which will be shared by all
 threads (child and parent).  A log message Queue is used to store all
 log entries, and a customized logger thread will get log entries from
 the Queue.
 
 #from the logger thread#
 def run(self):
 while self.flag == 1: #if the flag is set to 0, the logger
 thread should exit
 try:
 entry = self.q.get()
 except Empty:
 self.logger.debug('cant find any log entry')
 continue
 except:
 self.logger.error(Unexpected error:, sys.exc_info()
 [0])
 raise
 #do whatever that should be done
 self.logger.info(logger thread done) #should see this
 message in log file as well
 def off(self):
 self.logger.info('turning off flag')
 self.flag = 0
 
 
 #in parent thread#
 logItemQ.put('We are done, lets stop the logger now.')
 time.sleep(1) #it seems that the logger thread cannot exit if
 I put a sleep here
 myLog.off() #off is called successfully
 myLog.join()
 
 
 I put an off method to turn off a flag so the logger thread knows it
 should exit.  However, the last log message (the one 'We are done,
 lets stop the logger now.') won't be logged if I call myLog.off() and
 myLog.join() immediately.  So, I put a time.sleep(1) to make sure the
 logger thread has enough time to finish it's job.  Unfortunately, now
 the logger thread simply won't exit, and I don't see the message
 'logger thread done'.  I can't figure out at which point it hangs,
 since I don't any new log entry but the thread simply won't exit.
 Am I taking a right approach by using a flag?  Should I lock the flag?

M self.q.get() will block if the queue is empty, so the Empty exception is
M never raised.

M What's happening is that the parent thread puts the final message into
M the queue, sleeps, and then clears the flag; meanwhile, the logging
M thread gets the message, writes it out, checks the flag, which is still
M set, and then tries to get the next message. The queue is empty, so the
M .get() blocks.

M The simplest solution is not to use a flag, but the sentinel trick. The
M parent thread can put, say, None into the queue after the last message;
M when the logging thread gets None, it knows it should terminate.

To the OP: a sleep will never do it because you can't be sure how long
the logging thread will lag behind. Most multithreaded `solutions' which
depend on timings are buggy.

If you have more than one thread writing to the log queue one sentinel
won't solve the problem. You would need as many sentinels as there are
threads, and the logger should count the sentinels which can have normal
messages between them. It may be a bit fragile to depend on the number
of threads especially when this number is dynamic. One solution would be
to have special StartLogging and StopLogging objects that are put in the
queue when a thread starts and finishes respectively. The logger can stop
then when the number of StopLogging objects equals the number of
StartLogging objects received and that number is greater than 0. This
presupposes that no new thread will be started after all the others have
finished. The StartLogging objects can also be put in the queue by the
parent when it starts up the threads, which may be more robust because
it has fewer timing dependencies.
-- 
Piet van Oostrum p...@cs.uu.nl
URL: http://pietvanoostrum.com [PGP 8DAE142BE17999C4]
Private email: p...@vanoostrum.org
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: how can a child thread notify a parent thread its status?

[Piet van Oostrum]

 davidj411 davidj...@gmail.com (d) wrote:

d could i see an example of this maybe?

queue is a shared Queue instance.

The parent puts the work in the queue with queue.put(work_object).
After all work has been put in the queue it puts a sentinel

queue.put(None)

Each child thread (consumer) has a loop getting things out of the queue:

while True:
work = queue.get()
if work is None: break
#do real stuff using work

queue.put(None) #put back the sentinel for other worker threads.

#finish

(Thanks to Dennis Lee Bieber)

Instead of None you can also create a special sentinel object and use that.
-- 
Piet van Oostrum p...@cs.uu.nl
URL: http://pietvanoostrum.com [PGP 8DAE142BE17999C4]
Private email: p...@vanoostrum.org
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: how can a child thread notify a parent thread its status?

[scriptlear...@gmail.com]

First of all, let me say thank you to all of you.  I have asked many
questions (some of them are dump questions), and you have kindly
helped me.  I am not going to reply every message to say thank-you
since that would be annoying for such group with such high daily
traffics.  Thank you very much.

Let's get back to topic of this message.
Here's how I have implemented it so far, and I am taking the queue of
work load items approach.
In my child thread, I will keep checking for available work load item
until a duration is reached.
#inside the child#
while endTime  time.time():
try:
 item = self.q.get(True, 3)
except Queue.Empty:  #what's wrong?  AttributeError: class
Queue has no attribute 'Empty'
 print 'cant find any work load item, so lets wait and
try again later'
 time.sleep(1) #wait and then check again
 continue
except:
 print Unexpected error:, sys.exc_info()[0]
 raise
#do the real work with load item

In my parent thread, I will initialize X (depending on a cfg file)
child threads and keep adding load items to a shared q until the
duration is reached.
#inside the parent#
callCounter = 0
workers = [] #a list of child threads
totalWorkers = 250
endTime = time.time() + duration
for i in range(totalWorkers):
w = Worker(q, duration, i)
w.start() #worker, do your job now!
workers.append(w)

while endTime  time.time():
time.sleep(1)
q.put(getWorkloadItem()) #add workload itmes
callCounter += 1 #actually can we guarantee that the
call will be sent??
 #should we ask each child to report
the number of calls they make?

for i in range(totalWorkers):
workers[i].join()# Wait for the child threads to
finish


Overall, it seems to be working now.  Though, I still have a couple of
problems to resolve.
1. I got the following error for the codes that attempt to catch Empty
Queue exception.  What's the right way to use it?
except Queue.Empty:
AttributeError: class Queue has no attribute 'Empty'

2. What's the best way to have each child thread to report the number
of requests they send when they are done?  To add the numbers to
another queue?

3. I will need to do some logging for response time as well as some
response contents.  I have two choices, one big log file for all
threads (both child and parent), and one log file for each thread.
Given the fact that I may have to log tons of data, I think opening
and maintaining a bunch of smaller logs may be better than dealing
with a big one (it may grow very fast).  Is there any best prastice
for logging in Python?  If I change my mind and go with one big log
file (pass it to each thread), is there anything I should be aware of
for multi-thread access (writting) to the same log file?

Again, thank you.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: how can a child thread notify a parent thread its status?

[MRAB]

scriptlear...@gmail.com wrote:

First of all, let me say thank you to all of you.  I have asked many
questions (some of them are dump questions), and you have kindly
helped me.  I am not going to reply every message to say thank-you
since that would be annoying for such group with such high daily
traffics.  Thank you very much.

Let's get back to topic of this message.
Here's how I have implemented it so far, and I am taking the queue of
work load items approach.
In my child thread, I will keep checking for available work load item
until a duration is reached.
#inside the child#
while endTime  time.time():
try:
 item = self.q.get(True, 3)
except Queue.Empty:  #what's wrong?  AttributeError: class
Queue has no attribute 'Empty'
 print 'cant find any work load item, so lets wait and
try again later'
 time.sleep(1) #wait and then check again
 continue
except:
 print Unexpected error:, sys.exc_info()[0]
 raise
#do the real work with load item

In my parent thread, I will initialize X (depending on a cfg file)
child threads and keep adding load items to a shared q until the
duration is reached.
#inside the parent#
callCounter = 0
workers = [] #a list of child threads
totalWorkers = 250
endTime = time.time() + duration
for i in range(totalWorkers):
w = Worker(q, duration, i)
w.start() #worker, do your job now!
workers.append(w)

while endTime  time.time():
time.sleep(1)
q.put(getWorkloadItem()) #add workload itmes
callCounter += 1 #actually can we guarantee that the
call will be sent??
 #should we ask each child to report
the number of calls they make?

for i in range(totalWorkers):
workers[i].join()# Wait for the child threads to
finish


Overall, it seems to be working now.  Though, I still have a couple of
problems to resolve.
1. I got the following error for the codes that attempt to catch Empty
Queue exception.  What's the right way to use it?
except Queue.Empty:
AttributeError: class Queue has no attribute 'Empty'



The exception 'Empty' belongs to the module, not the class. Try
importing as:

from Queue import Queue, Empty


2. What's the best way to have each child thread to report the number
of requests they send when they are done?  To add the numbers to
another queue?


Why not? :-)


3. I will need to do some logging for response time as well as some
response contents.  I have two choices, one big log file for all
threads (both child and parent), and one log file for each thread.
Given the fact that I may have to log tons of data, I think opening
and maintaining a bunch of smaller logs may be better than dealing
with a big one (it may grow very fast).  Is there any best prastice
for logging in Python?  If I change my mind and go with one big log
file (pass it to each thread), is there anything I should be aware of
for multi-thread access (writting) to the same log file?

Again, thank you.


If you like threads then you could put the log items into a queue and
have another thread writing them to the logfile. :-)

BTW, do you really need 250 threads? Seems like a lot.

I notice that you stop putting items into the queue when endTime is
reached and also the threads terminate when endTime is reached. If items
are put into the queue faster than they're taken out (or an item is put
in just before endTime) then there might still be unprocessed items in
the queue at endTime.
--
http://mail.python.org/mailman/listinfo/python-list

Re: how can a child thread notify a parent thread its status?

[scriptlear...@gmail.com]

I decided to go with one big log file, which will be shared by all
threads (child and parent).  A log message Queue is used to store all
log entries, and a customized logger thread will get log entries from
the Queue.

#from the logger thread#
def run(self):
while self.flag == 1: #if the flag is set to 0, the logger
thread should exit
try:
 entry = self.q.get()
except Empty:
 self.logger.debug('cant find any log entry')
 continue
except:
 self.logger.error(Unexpected error:, sys.exc_info()
[0])
 raise
#do whatever that should be done
self.logger.info(logger thread done) #should see this
message in log file as well
def off(self):
self.logger.info('turning off flag')
self.flag = 0


#in parent thread#
logItemQ.put('We are done, lets stop the logger now.')
time.sleep(1) #it seems that the logger thread cannot exit if
I put a sleep here
myLog.off() #off is called successfully
myLog.join()


I put an off method to turn off a flag so the logger thread knows it
should exit.  However, the last log message (the one 'We are done,
lets stop the logger now.') won't be logged if I call myLog.off() and
myLog.join() immediately.  So, I put a time.sleep(1) to make sure the
logger thread has enough time to finish it's job.  Unfortunately, now
the logger thread simply won't exit, and I don't see the message
'logger thread done'.  I can't figure out at which point it hangs,
since I don't any new log entry but the thread simply won't exit.
Am I taking a right approach by using a flag?  Should I lock the flag?
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: how can a child thread notify a parent thread its status?

[MRAB]

scriptlear...@gmail.com wrote:

I decided to go with one big log file, which will be shared by all
threads (child and parent).  A log message Queue is used to store all
log entries, and a customized logger thread will get log entries from
the Queue.

#from the logger thread#
def run(self):
while self.flag == 1: #if the flag is set to 0, the logger
thread should exit
try:
 entry = self.q.get()
except Empty:
 self.logger.debug('cant find any log entry')
 continue
except:
 self.logger.error(Unexpected error:, sys.exc_info()
[0])
 raise
#do whatever that should be done
self.logger.info(logger thread done) #should see this
message in log file as well
def off(self):
self.logger.info('turning off flag')
self.flag = 0


#in parent thread#
logItemQ.put('We are done, lets stop the logger now.')
time.sleep(1) #it seems that the logger thread cannot exit if
I put a sleep here
myLog.off() #off is called successfully
myLog.join()


I put an off method to turn off a flag so the logger thread knows it
should exit.  However, the last log message (the one 'We are done,
lets stop the logger now.') won't be logged if I call myLog.off() and
myLog.join() immediately.  So, I put a time.sleep(1) to make sure the
logger thread has enough time to finish it's job.  Unfortunately, now
the logger thread simply won't exit, and I don't see the message
'logger thread done'.  I can't figure out at which point it hangs,
since I don't any new log entry but the thread simply won't exit.
Am I taking a right approach by using a flag?  Should I lock the flag?


self.q.get() will block if the queue is empty, so the Empty exception is
never raised.

What's happening is that the parent thread puts the final message into
the queue, sleeps, and then clears the flag; meanwhile, the logging
thread gets the message, writes it out, checks the flag, which is still
set, and then tries to get the next message. The queue is empty, so the
.get() blocks.

The simplest solution is not to use a flag, but the sentinel trick. The
parent thread can put, say, None into the queue after the last message;
when the logging thread gets None, it knows it should terminate.

--
http://mail.python.org/mailman/listinfo/python-list

Re: how can a child thread notify a parent thread its status?

[MRAB]

scriptlear...@gmail.com wrote:

My parent thread keeps a counter for the number of free child workers
(say 100) and initializes some child threads and call child.start().
Once the number of free child workers reach 0, the parent thread will
wait until some at least one child thread finishes and then it will
initialize another child thread.
My question is, how can a child thread notify the parent that it's
done so that the parent can call join() on it?  I am not sure how a
child thread can send a signal to its parent while it may not even
know anything about it's parent.  Can you guys please provide some
suggestions?  Some code samples will be nice.  Thanks.


I would create a queue (Queue module) and pass it to each child; the 
child would put something (eg a reference to itself) in the queue when

it had finished.
--
http://mail.python.org/mailman/listinfo/python-list

Re: how can a child thread notify a parent thread its status?

[Christian Heimes]

scriptlear...@gmail.com wrote:
 My parent thread keeps a counter for the number of free child workers
 (say 100) and initializes some child threads and call child.start().
 Once the number of free child workers reach 0, the parent thread will
 wait until some at least one child thread finishes and then it will
 initialize another child thread.
 My question is, how can a child thread notify the parent that it's
 done so that the parent can call join() on it?  I am not sure how a
 child thread can send a signal to its parent while it may not even
 know anything about it's parent.  Can you guys please provide some
 suggestions?  Some code samples will be nice.  Thanks.

You are using the wrong approach here. There is a much easier solution
to your problem although it's not obvious in the first place. In this
approach the worker threads don't have to notify the parent thread that
they are ready.

The main thread creates a queue and a bunch of worker threads that are
pulling tasks from the queue. As long as the queue is empty all worker
threads block and do nothing. When a task is put into the queue a random
worker thread acquires the task, does it job and sleeps as soon as its
ready. That way you can reuse a thread over and over again without
creating a new worker threads.

When you need to stop the threads you put a kill object into the queue
that tells the thread to shut down instead of blocking on the queue.

Christian

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: how can a child thread notify a parent thread its status?

[MRAB]

Christian Heimes wrote:

scriptlear...@gmail.com wrote:

My parent thread keeps a counter for the number of free child workers
(say 100) and initializes some child threads and call child.start().
Once the number of free child workers reach 0, the parent thread will
wait until some at least one child thread finishes and then it will
initialize another child thread.
My question is, how can a child thread notify the parent that it's
done so that the parent can call join() on it?  I am not sure how a
child thread can send a signal to its parent while it may not even
know anything about it's parent.  Can you guys please provide some
suggestions?  Some code samples will be nice.  Thanks.


You are using the wrong approach here. There is a much easier solution
to your problem although it's not obvious in the first place. In this
approach the worker threads don't have to notify the parent thread that
they are ready.

The main thread creates a queue and a bunch of worker threads that are
pulling tasks from the queue. As long as the queue is empty all worker
threads block and do nothing. When a task is put into the queue a random
worker thread acquires the task, does it job and sleeps as soon as its
ready. That way you can reuse a thread over and over again without
creating a new worker threads.

When you need to stop the threads you put a kill object into the queue
that tells the thread to shut down instead of blocking on the queue.


The extra trick is that when a thread gets the kill object it puts it
back before terminating so that another thread will also get it.
--
http://mail.python.org/mailman/listinfo/python-list

[issue4451] multiprocessing fails with Invalid thread state for this thread on 2.4 and 2.5

[Christian Heimes]

New submission from Christian Heimes [EMAIL PROTECTED]:

The multiprocessing backport to 2.4/2.5 fails with a fatal error when
the test suite is run with a debug build of Python.

PYTHONPATH=Lib/ /home/heimes/dev/python/release25-maint/python -tt -c
from multiprocessing.tests import main; main()
Fatal Python error: Invalid thread state for this thread
Traceback (most recent call last):
  File string, line 1, in module
  File
/home/heimes/dev/py/python-multiprocessing/Lib/multiprocessing/tests.py,
line 1826, in main
test_main(unittest.TextTestRunner(verbosity=2).run)
  File
/home/heimes/dev/py/python-multiprocessing/Lib/multiprocessing/tests.py,
line 1805, in test_main
ManagerMixin.pool = ManagerMixin.manager.Pool(4)
  File
/home/heimes/dev/py/python-multiprocessing/Lib/multiprocessing/managers.py,
line 637, in temp
token, exp = self._create(typeid, *args, **kwds)
  File
/home/heimes/dev/py/python-multiprocessing/Lib/multiprocessing/managers.py,
line 535, in _create
conn = self._Client(self._address, authkey=self._authkey)
  File
/home/heimes/dev/py/python-multiprocessing/Lib/multiprocessing/connection.py,
line 142, in Client
answer_challenge(c, authkey)
  File
/home/heimes/dev/py/python-multiprocessing/Lib/multiprocessing/connection.py,
line 373, in answer_challenge
message = connection.recv_bytes(256) # reject large message
EOFError

--
components: Interpreter Core
messages: 76528
nosy: christian.heimes, jnoller
priority: high
severity: normal
stage: needs patch
status: open
title: multiprocessing fails with Invalid thread state for this thread on 2.4 
and 2.5
type: crash
versions: Python 2.4, Python 2.5.3

___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue4451
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4451] multiprocessing fails with Invalid thread state for this thread on 2.4 and 2.5

[Benjamin Peterson]

Benjamin Peterson [EMAIL PROTECTED] added the comment:

Christian, to fix this, you need to backport the fix for #1683.

--
nosy: +benjamin.peterson

___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue4451
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4451] multiprocessing fails with Invalid thread state for this thread on 2.4 and 2.5

[Jesse Noller]

Jesse Noller [EMAIL PROTECTED] added the comment:

As ben mentioned, this is already fixed in core. See issue 1683 - this is 
only a problem when running in 2.5/2.4

--
resolution:  - duplicate
status: open - closed

___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue4451
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: Invalid thread state for this thread

[Martin Evans]

Just in case anyone else has seen this problem, after upgrading to 2.4.4 the 
problem appears to have resolved itself.

I know this has been seen before but it is not making too much sense (after 
reading many posts). It all appears to work fine but then dies after about 
40 invocations.

 My app has Python embedded, it is embedded as part of a dll which 
 initializes python and finalizes on load and unload (see below). When a 
 script needs to be run, I create a new thread, let the script complete and 
 close the thread.

 The thread contains the following type arrangement:

 PythonThread()
 {
  hScriptFile = OpenScriptFile(m_szActiveScript);
  PyEval_AcquireLock();
  pInterpreter = Py_NewInterpreter();
  Py_SetProgramName(szModuleFileName);
  PyRun_SimpleFile(hScriptFile,m_szActiveScript);
  PyErr_Clear();
  Py_EndInterpreter(pInterpreter);
  PyEval_ReleaseLock();
 }

 This appears to work fine accept that after around 30-40 invocations I 
 always get the Invalid thread state for this thread. ie the app and dll 
 stay loaded and I click my run button manually about 40 times. In this 
 test it is a simple hello world type script. The size of the script does 
 not appear to matter.

 The dll is coded something like this:

 DllLoad()
 {
  Py_Initialize();
  PyEval_InitThreads();
  m_mainThreadState = PyThreadState_Get();
  PyEval_ReleaseLock();
 }

 DllUnload() (not called as part of this test)
 {
  PyEval_AcquireLock();
  PyThreadState_Swap(m_mainThreadState);
  Py_Finalize();
 }

 The app has been designed to allow a second interpreter to run 
 independently (thus the need for multiple thread support) and this also 
 appears to work fine, but for this test only this one thread was used. I 
 have a compiled version of 2.4.2.

 Any ideas would be appreciated.

 Martin 


-- 
http://mail.python.org/mailman/listinfo/python-list

Invalid thread state for this thread

[Martin Evans]

I know this has been seen before but it is not making too much sense (after 
reading many posts). It all appears to work fine but then dies after about 
40 invocations.

My app has Python embedded, it is embedded as part of a dll which 
initializes python and finalizes on load and unload (see below). When a 
script needs to be run, I create a new thread, let the script complete and 
close the thread.

The thread contains the following type arrangement:

PythonThread()
{
  hScriptFile = OpenScriptFile(m_szActiveScript);
  PyEval_AcquireLock();
  pInterpreter = Py_NewInterpreter();
  Py_SetProgramName(szModuleFileName);
  PyRun_SimpleFile(hScriptFile,m_szActiveScript);
  PyErr_Clear();
  Py_EndInterpreter(pInterpreter);
  PyEval_ReleaseLock();
}

This appears to work fine accept that after around 30-40 invocations I 
always get the Invalid thread state for this thread. ie the app and dll 
stay loaded and I click my run button manually about 40 times. In this 
test it is a simple hello world type script. The size of the script does 
not appear to matter.

The dll is coded something like this:

DllLoad()
{
  Py_Initialize();
  PyEval_InitThreads();
  m_mainThreadState = PyThreadState_Get();
  PyEval_ReleaseLock();
}

DllUnload() (not called as part of this test)
{
  PyEval_AcquireLock();
  PyThreadState_Swap(m_mainThreadState);
  Py_Finalize();
}

The app has been designed to allow a second interpreter to run independently 
(thus the need for multiple thread support) and this also appears to work 
fine, but for this test only this one thread was used. I have a compiled 
version of 2.4.2.

Any ideas would be appreciated.

Martin 


-- 
http://mail.python.org/mailman/listinfo/python-list

how can I block a thread until some other thread finished?

[bearsprite]

I start a thread A. In A, I start thread B,C,D,...

How can I block A until B,C,D,...(All the thread started by A) finished?

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: how can I block a thread until some other thread finished?

[bearsprite]

I think I need some implement like win32 API WaitForMultiObject.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: To thread or not to thread

[snacktime]

On 28 Jul 2005 12:10:12 -0700, Sidd [EMAIL PROTECTED] wrote:
 Hello,
   I was recently reading an article on threading in python and I
 came across Global Interpreter Lock,now as a novince in python I was
 cusrious about
 
 1.Is writing a threaded code in python going to perform well than a
 normal python  code.If so on what basis can it performance be measured.
 
 2.Is writing a threaded code in python better than a code written in
 C/C++ using PTHREADS.
 
 If someone can comment on these questions, it would be great.
 

If you want performance with an application that does a lot of
concurrent activity, you might take a look at
http://www.twistedmatrix.com which is an event driven framework for
python.
Much better performance than threads with a lot less memory and cpu
usage.  Although it does have a bit of a learning curve.  In my own
experience it would be faster  then a comparable application written
in C using pthreads.  We have an application written in twisted that
processes financial applications via bank networks, and at a steady
100tps I get about 1% cpu usage.   We tested it up to around 1000tps
before our database server started to get a bit overloaded.  Twisted
never used more than 20% of the cpu though.

Chris
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: To thread or not to thread

[Fuzzyman]

Some people are of the opinion that threads are evil.

Best Regards,

Fuzzy
http://www.voidspace.org.uk/python

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: To thread or not to thread

[James Richards]

On 2005-07-28, Sidd [EMAIL PROTECTED] wrote:
 Hello,
   I was recently reading an article on threading in python and I
 came across Global Interpreter Lock,now as a novince in python I was
 cusrious about

 1.Is writing a threaded code in python going to perform well than a
 normal python  code.If so on what basis can it performance be measured.

My last real experience with python threads was a while back, on a P-2.
That experience suggested that creating lots of threads (a few hundred)
caused some serious performance impacts.  I determined, in that instance,
that it was better to write my own implementation to simulate threads.
I had set of classes that pretended to be threads.  I had another class
which actually did all the threading for them.  It was pretty ugly.

 2.Is writing a threaded code in python better than a code written in
 C/C++ using PTHREADS.

I agree with the earlier.  Define better.  Do you really have a heavily
multi-threaded app?  Are these threads all CPU-intensive, or do you just 
have a bunch of threads which need some arbitrary scheduling?  Is it really
worth re-writing in PTHREADS?  Or could you buy a new server and save a few
months in development time by writing your own scheduling?

It all depends on your situation.


 If someone can comment on these questions, it would be great.


Heh.  You're on Usenet.  *Anyone* can comment on these questions.  :-)  
You should have asked for *useful* comments.  ;-)


-- 
Liberty means responsibility. That is why most men dread it.
- George Bernard Shaw

-- 
http://mail.python.org/mailman/listinfo/python-list

To thread or not to thread

[Sidd]

Hello,
  I was recently reading an article on threading in python and I
came across Global Interpreter Lock,now as a novince in python I was
cusrious about

1.Is writing a threaded code in python going to perform well than a
normal python  code.If so on what basis can it performance be measured.

2.Is writing a threaded code in python better than a code written in
C/C++ using PTHREADS.

If someone can comment on these questions, it would be great.

-- 
http://mail.python.org/mailman/listinfo/python-list