I've got an application which makes fairly heavy use of daemon threads to 
perform 'background' processing and various other long-running tasks that are 
likely to block.

I originally thought I could safely fire off a daemon threading.Thread and 
essentially forget about managing the thread's lifecycle.  My thinking was that 
from the main thread's perspective its safe to fire off a daemon thread to do 
background twiddling and let the threading machinery manage the thread's 
lifespan assuming that:
        1. the 'background twiddling' is threadsafe and
        2. the thread can safely 'die' at any point without requiring a 
shutdown procedure

Some coding later and I learn this isn't exactly the case, as there as in an 
additional requirement to correctly use 'daemon' threads -- they can't 
reference anything in the global namespace else they may raise exceptions at 
interpreter shutdown.  As part of the shutdown procedure, the interpreter sets 
all global variables to None, a daemon thread may run while this is 
occuring/after it occured and raise an exception.  The exception may get 
printed if the interpreter catches/prints it before the process exits.

I garnered this understanding from this problem description -- (although all 
mistakes in description are my own)

http://bugs.python.org/issue1722344

In this bug report they discuss an interpreter problem which affects non-daemon 
threads -- I'm not attempting to claim that I'm being affected by an 
interpreter bug, I reference this link only because it contains good 
descriptions of the interpreter shutdown process as well as the additional 
requirements the interpreter shutdown process places on 'daemon' threads
> When Python begins to shutdown it takes
> each module and sets each variable in the global namespace to None. If a
> thread has not terminated before the interpreter terminates then the
> thread tries to use a global variable which has been set to None.
> 
> This is not about to change since this occurs because of coding
> "errors". You must make sure that either your thread is as safe as a
> __del__ method (which means no global namespace access) or you can't let
> the app exit until you are positive all of your threads have terminated,
> not just asked them to shutdown since this is all asynchronous.
> 
> > > which means no global namespace access
> > Does that mean that you cannot use len and range in a Thread?
> 
> No, it means you have to be careful if you do. Shutting down properly
> will take care of things. Otherwise you need to save a reference
> locally (either on an object or as a local variable) and use that
> reference instead of relying on the one defined in the global
> namespace.
> 

Here's an example of a variable access from a daemon thread after interpreter 
shutdown started setting module globals to None:

> Exception in thread Thread-11 (most likely raised during interpreter 
> shutdown):
> (pydev-2.6) ncohen$ 

In this run, the process exited before even one traceback finished printing but 
this will be timing dependent, -- sometimes I'll see tracebacks from many 
backgrounds threads and sometimes just snippets like this.  They will typically 
be AttributeError's or TypeError's resulting from failed attempts to access 
variables from the global namespace.  Here's a longer example from another run:

> Traceback (most recent call last):
>   File 
> "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/threading.py",
>  line 525, in __bootstrap_inner
>   File "/Users/ncohen/software/viis/viis/apps/permassh.py", line 184, in run
>   File 
> "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/SocketServer.py",
>  line 224, in serve_forever
> <type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 
> 'select'
> 
> Traceback (most recent call last):
>   File 
> "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/threading.py",
>  line 525, in __bootstrap_inner
>   File "/Users/ncohen/software/viis/viis/apps/permassh.py", line 184, in run
>   File 
> "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/SocketServer.py",
>  line 224, in serve_forever
> <type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 
> 'select'
> Exception in thread Thread-7 (most likely raised during interpreter shutdown):
> Traceback (most recent call last):
>   File 
> "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/threading.py",
>  line 525, in __bootstrap_inner
>   File 
> "/Users/ncohen/pydev-2.6/lib/python2.6/site-packages/paramiko/transport.py", 
> line 1571, in run
>   File 
> "/Users/ncohen/pydev-2.6/lib/python2.6/site-packages/paramiko/transport.py", 
> line 1386, in _log
>   File 
> "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/logging/__init__.py",
>  line 1105, in log
> <type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 
> 'IntType'
> (pydev-2.6)breathe-wifi:viis.webapp ncohen$ 
> (pydev-2.6) ncohen$ 

My questions concern whether or not its 'safe' to use daemon threads this way 
-- is the 'no references to globals()' 'requirement' imposed only by the 
interpreter shutdown process or are there additional reasons for the 
requirement?

0. is it possible for me to consider it 'safe' to have code which can trigger 
the above exceptions during interpreter shutdown?
1. if it is possible, and it turns out I'm certain what I'm doing is safe, is 
there a good way to squelch the printing of those stack traces?

I was hunting around for a way to customize the value that all the globals get 
set to during interpreter shutdown -- i think it would work for me if I could 
get the interpreter to replace all globals with an object which:
        1. returns None as usual if accessed from a non-daemon thread
        2. blocks forever if accessed from a daemon thread (since the process 
is about to exit anyway)

In my particular case I don't always have control over all of the daemon 
threads creation or lifespan -- I'm using the paramiko (www.lag.net/paramiko) 
ssh protocol library which uses daemon thread(s) 'behind the scenes' to 
simplify its networking code ...

Looking forward to any comments/advice/alternate explanations or name calling.

Thanks,
Ben

Ben Cohen
Programmer/Analyst (STS)
Scripps Institution of Oceanography
nco...@ucsd.edu



Ben Cohen
Programmer/Analyst (STS)
Scripps Institution of Oceanography
nco...@ucsd.edu

-- 
http://mail.python.org/mailman/listinfo/python-list

Reply via email to