On 1/30/07, Ian Bicking <[EMAIL PROTECTED]> wrote: > Hi quiet Paste list. > > If you are anything like me, you hate it when your HTTP server freezes > up because all the threads are wedged for some reason. If you aren't > using pooling I guess this won't happen, but instead you'll eventually > have tons of wedged threads sitting around and that's not great either. > > Anyway, in the trunk I made some additions to the thread pool and a > small app (egg:Paste#watch_threads) that lets you monitor the pool > through the web and even kill threads. (How reliable the thread killing > is, I'm not sure -- it worked for a couple cases I tried, like reading > past the end of a socket and an infinite loop in Python.) > > Of course it is flawed, since if your thread pool is exhausted you can't > access the app. Plus, while seeing wedged threads is nice for > debugging, it's not really something best managed manually. > > So I'm thinking about how the thread pool could be improved. Here's my > idea; I'm interested in opinions: > > When a request comes in and there are no free threads to handle it, a > new thread should be created up to max_threads (configurable). Maybe > the thread should only live for one request, or maybe it should be added > to the pool and the pool periodically reduced in size if possible. > > When a request comes in and there are already a maximum number of > threads created, the thread most likely to be wedged (the one that's > been working the longest) should be killed and another one added. If > none of the threads has been working very long (wedged_thread_threshold) > then we assume we just have a lot of requests coming in, and we simply > queue the request. That means if like 10 threads all get wedged at > once, and another request comes in, it could end up queued until yet > another request comes in. And then that other request will kill a > thread, the old request gets off the queue, and the new request is back > on the queue. I'm not sure how to deal with that problem, except maybe > to try to empty the queue with multiple kills once a wedged situation is > detected. > > Maybe we should add an API to the request environment to tell the server > that a long-running request is expected. This way a conscientious > programmer could still do long-running requests without being afraid of > being killed, but you have to express real intention to do so. > > We can check if threads that we killed are actually dead (they'll still > be listed in threading._active). If we see an excess of these we can > kill the whole process (assuming that a supervisor process is going to > restart the server). Configurable, zombie_thread_threshold or > something, obviously not on by default. > > Anyway, any thoughts anyone has would be appreciated. Clearly I'll have > to write up a document explaining all this, as it's going to be too long > to go in a docstring. > > I guess I'll also have to clean up some of the lingering issues in > Paste's HTTP server too (I think just the wsgi.input blocking problem > and the limited request methods), as once I start relying on this stuff > it'll be harder to move to another server. So I can no longer vacillate > on what server people should use -- ours!
All of this can get really sticky, and I fear there are no good, general answers. If you do decide to start killing long-running threads, I do like the idea of letting the programmer explicitly state that the thread should be long running. Do you really have a problem of threads hanging? It's just not something I've had a problem with in general. Happy Hacking! -jj -- http://jjinux.blogspot.com/ _______________________________________________ Paste-users mailing list [email protected] http://webwareforpython.org/cgi-bin/mailman/listinfo/paste-users
