exceptions from daemon threads which access the global namespace at interpreter shutdown (how to squelch output?)
I've got an application which makes fairly heavy use of daemon threads to perform 'background' processing and various other long-running tasks that are likely to block. I originally thought I could safely fire off a daemon threading.Thread and essentially forget about managing the thread's lifecycle. My thinking was that from the main thread's perspective its safe to fire off a daemon thread to do background twiddling and let the threading machinery manage the thread's lifespan assuming that: 1. the 'background twiddling' is threadsafe and 2. the thread can safely 'die' at any point without requiring a shutdown procedure Some coding later and I learn this isn't exactly the case, as there as in an additional requirement to correctly use 'daemon' threads -- they can't reference anything in the global namespace else they may raise exceptions at interpreter shutdown. As part of the shutdown procedure, the interpreter sets all global variables to None, a daemon thread may run while this is occuring/after it occured and raise an exception. The exception may get printed if the interpreter catches/prints it before the process exits. I garnered this understanding from this problem description -- (although all mistakes in description are my own) http://bugs.python.org/issue1722344 In this bug report they discuss an interpreter problem which affects non-daemon threads -- I'm not attempting to claim that I'm being affected by an interpreter bug, I reference this link only because it contains good descriptions of the interpreter shutdown process as well as the additional requirements the interpreter shutdown process places on 'daemon' threads When Python begins to shutdown it takes each module and sets each variable in the global namespace to None. If a thread has not terminated before the interpreter terminates then the thread tries to use a global variable which has been set to None. This is not about to change since this occurs because of coding errors. You must make sure that either your thread is as safe as a __del__ method (which means no global namespace access) or you can't let the app exit until you are positive all of your threads have terminated, not just asked them to shutdown since this is all asynchronous. which means no global namespace access Does that mean that you cannot use len and range in a Thread? No, it means you have to be careful if you do. Shutting down properly will take care of things. Otherwise you need to save a reference locally (either on an object or as a local variable) and use that reference instead of relying on the one defined in the global namespace. Here's an example of a variable access from a daemon thread after interpreter shutdown started setting module globals to None: Exception in thread Thread-11 (most likely raised during interpreter shutdown): (pydev-2.6) ncohen$ In this run, the process exited before even one traceback finished printing but this will be timing dependent, -- sometimes I'll see tracebacks from many backgrounds threads and sometimes just snippets like this. They will typically be AttributeError's or TypeError's resulting from failed attempts to access variables from the global namespace. Here's a longer example from another run: Traceback (most recent call last): File /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/threading.py, line 525, in __bootstrap_inner File /Users/ncohen/software/viis/viis/apps/permassh.py, line 184, in run File /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/SocketServer.py, line 224, in serve_forever type 'exceptions.AttributeError': 'NoneType' object has no attribute 'select' Traceback (most recent call last): File /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/threading.py, line 525, in __bootstrap_inner File /Users/ncohen/software/viis/viis/apps/permassh.py, line 184, in run File /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/SocketServer.py, line 224, in serve_forever type 'exceptions.AttributeError': 'NoneType' object has no attribute 'select' Exception in thread Thread-7 (most likely raised during interpreter shutdown): Traceback (most recent call last): File /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/threading.py, line 525, in __bootstrap_inner File /Users/ncohen/pydev-2.6/lib/python2.6/site-packages/paramiko/transport.py, line 1571, in run File /Users/ncohen/pydev-2.6/lib/python2.6/site-packages/paramiko/transport.py, line 1386, in _log File /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/logging/__init__.py, line 1105, in log type 'exceptions.AttributeError': 'NoneType' object has no attribute 'IntType' (pydev-2.6)breathe-wifi:viis.webapp ncohen$ (pydev-2.6) ncohen$ My questions concern whether or not its 'safe' to use daemon threads this way -- is the 'no
exceptions from daemon threads which access the global namespace at interpreter shutdown (how to squelch output?)
I've got an application which makes fairly heavy use of daemon threads to perform 'background' processing and various other long-running tasks that are likely to block. My original understanding of threading.Thread's daemon threads was that I could safely fire them off and essentially forget about managing them from the main thread's perspective as long as they don't do anything that's not thread safe -- eg I can fire them off, let them do their background twiddling and safely let the threading machinery manage their lifespan assuming that 1. the 'background twiddling' is threadsafe and 2. the thread can safely 'die' at any point without requiring a shutdown procedure Some coding later and I learn this isn't exactly the case, as there as in an additional requirement on the use of daemon threads -- they can't reference anything in the global namespace else they may raise exceptions at interpreter shutdown. This is because as part of the shutdown procedure, the interpreter sets all global variables to None. A daemon thread may run while this is occuring/after it occured but before the process exits, attempt to access a global variable and then throw an exception. The exception may get printed if the interpreter catches/prints it before the process exits. I garnered this understanding from this problem description -- (although all mistakes in description are my own) http://bugs.python.org/issue1722344 In this bug report they are discussing an interpreter problem which affects non-daemon threads -- I'm not attempting to claim that I'm being affected by an interpreter bug, I reference this link only because it contains good descriptions of the interpreter shutdown process as well as the additional requirements the interpreter places on 'daemon' threads (beyond just being 'thread-safe'): When Python begins to shutdown it takes each module and sets each variable in the global namespace to None. If a thread has not terminated before the interpreter terminates then the thread tries to use a global variable which has been set to None. This is not about to change since this occurs because of coding errors. You must make sure that either your thread is as safe as a __del__ method (which means no global namespace access) or you can't let the app exit until you are positive all of your threads have terminated, not just asked them to shutdown since this is all asynchronous. which means no global namespace access Does that mean that you cannot use len and range in a Thread? No, it means you have to be careful if you do. Shutting down properly will take care of things. Otherwise you need to save a reference locally (either on an object or as a local variable) and use that reference instead of relying on the one defined in the global namespace. Here's an example of one such traceback from one of my application runs: Exception in thread Thread-11 (most likely raised during interpreter shutdown): (pydev-2.6) ncohen$ In this run, the process exited before even one traceback finished printing but this will be timing dependent, -- sometimes I'll see tracebacks from many backgrounds threads and sometimes just snippets like this. They will typically be AttributeError's or TypeError's resulting from failed attempts to access variables from the global namespace. Here's a longer example from another run: Traceback (most recent call last): File /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/threading.py, line 525, in __bootstrap_inner File /Users/ncohen/software/viis/viis/apps/permassh.py, line 184, in run File /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/SocketServer.py, line 224, in serve_forever type 'exceptions.AttributeError': 'NoneType' object has no attribute 'select' Traceback (most recent call last): File /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/threading.py, line 525, in __bootstrap_inner File /Users/ncohen/software/viis/viis/apps/permassh.py, line 184, in run File /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/SocketServer.py, line 224, in serve_forever type 'exceptions.AttributeError': 'NoneType' object has no attribute 'select' Exception in thread Thread-7 (most likely raised during interpreter shutdown): Traceback (most recent call last): File /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/threading.py, line 525, in __bootstrap_inner File /Users/ncohen/pydev-2.6/lib/python2.6/site-packages/paramiko/transport.py, line 1571, in run File /Users/ncohen/pydev-2.6/lib/python2.6/site-packages/paramiko/transport.py, line 1386, in _log File /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/logging/__init__.py, line 1105, in log type 'exceptions.AttributeError': 'NoneType' object has no attribute 'IntType' (pydev-2.6)breathe-wifi:viis.webapp ncohen$ (pydev-2.6)
