Hi everyone, I have a problem and I am hoping that it has been solved already by someone or that I will get some input on at least.I apologize for the lengthy E-mail in advance, but I wanted to provide a detailed discussion as a starting point.
Zope is designed to have very short-lived transactions. If transactions are long-living all sorts of problems arise, most notably: 1. We occupy one thread for a long time. 2. The chance of conflict errors increases. Problem 1 can be addressed by increasing the number of allowed threads or to simply add more Zope servers. But his has clearly its limits and is really just a work-around. Another way to solve the problem is to identify long-running operations and calling them asynchronous. Many of us have implemented solutions for this, one of which is lovely.remotetask. Problem number 2 can only be addressed by identifying the long-running tasks beforehand and move them into an async call, again via lovely.remotetask for example. But what happens, if a something unexpected happens and we have an unanticipated long-runnning process? The worst case being something runs forever. Then whenever this problem occurs, one thread will be locked forever, and we can have a total system lockdown in no-time. So how can this be solved? Effectively, from within Zope we cannot do anything, because (a) Zope makes no assumption about running in a thread, and (b) the application is stuck and won't have a hook to get unstuck. So we have to solve the problem from outside. Currently, Zope is commonly run from an application thread. At least both WSGI servers that we commonly use, twisted and zserver, are implemented this way. This means that by some criterion, probably some timeout, the thread should be killed. But hold on! In Python threads cannot be killed. :-( I have done some research and found issue 221115 , which discusses the shortcoming of not being able to kill a thread. The discussion ended in making a feature request in PEP 42  which has not been implemented as far as I can tell. So I googled some more to find possible implementations. Here are two distinctively different solutions (others I have found are either obviously trivial and will not work, or are derivatives of these two): 1. A Python-only solution using sys.settrace . Besides making everything very slow, sys.settrace() is only called when a new byte code instruction is executed. So in case a low-level call hangs up the process, then the trace intercept will never be called. 2. Use an exception to intercept execution on the C-level . This looked very promising, until I read the following comment on the page: The exception will be raised only when executing python bytecode. If your thread calls a native/built-in blocking function, the exception will be raised only when execution returns to the python code. So my conclusion is that Python threads cannot be unconditionally killed. BTW, if a low-level call is blocking, then all Python threads are blocked. From the Python `thread` library documentation: Not all built-in functions that may block waiting for I/O allow other threads to run. (The most popular ones (time.sleep(), file.read(), select.select()) work as expected.) In all fairness, though, those are very rare occurrences. Most libraries are non-blocking and the above solutions would be just fine. But in my case, I really need to find a way to kill a Zope execution environment when a C call hangs. So what other choices do we have? On Unix-like systems, we can use `os.fork()`. The advantage of this approach is that I can use OS system calls to kill the process. However, ZODB database storages cannot be shared between processes. Nikolay Kim has done some preliminary experiments and found that `db.open()` locks the system (for both, `FileStorage` and `ZeoClientStorage`). I have not verified these results or tried to figure out why it is hanging, but I can see the problem for `FileStorage`. Are there any known side-effects on what happens, if I fork after the connection has been made? Since I am using the original process merely as a control, I guess I should be fine. Of course, the interesting question is: what happens to the ZODB connection, not to mention to the DB, if it is in the middle of writing? I guess the safest solution would be to fork within the constraint of the transaction. Any comments will be very much appreciated. Once we decide on the forking approach, we have to solve the issue for Windows of course too. My googling did not turn out immediately successful, but I think if we use Windows' native threads they will provide us with the necessary API, since I can exit it at any time. .. : http://bugs.python.org/issue221115 .. : http://www.python.org/dev/peps/pep-0042/ .. : http://www.velocityreviews.com/forums/t330554-kill-a-thread-in-python.html .. : http://sebulba.wikispaces.com/recipe+thread2 .. : http://docs.python.org/lib/module-thread.html Regards, Stephan -- Stephan Richter CBU Physics & Chemistry (B.S.) / Tufts Physics (Ph.D. student) Web2k - Web Software Design, Development and Training _______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )