On 29 June 2010 19:59, Marwan Al-Sabbagh <[email protected]> wrote: > cool. it's not a big issue for me. I've decided to manage the issue in a > different way. I'm gonna log in a database the start and end time of each > request. Then I can separately monitor any out of control reports/pages and > get general stats on the slow requests in the system. > > Out of curiosity though is it alright to be spawning threads in mod_wsgi if > I'm using daemon mode multi-process single-thread. Assuming I take proper > care of starting and ending the threads. I didn't have a specific > requirement to do that, but I was just wonder if doing that sort of stuff > can cause problems.
Background threads are fine. Possibly preferable that you organise code such that an attempt is made to shut them down when process quits. For an example of that, see how the background thread for code monitor is implemented in: http://code.google.com/p/modwsgi/wiki/ReloadingSourceCode#Monitoring_For_Code_Changes You will see how it does a trick with using an atexit callback to try and set a flag to get background thread to exit. It may not be a big deal, I was just trying to avoid possibility that thread may wake and try and do stuff while Python interpreter is being destroyed, possibly causing process to crash rather than shut down cleanly. Graham > On Tue, Jun 29, 2010 at 9:38 AM, Graham Dumpleton > <[email protected]> wrote: >> >> On 29 June 2010 16:33, Marwan Al-Sabbagh <[email protected]> >> wrote: >> > thanks for the detailed response. How was PyCon? I saw your slides on >> > your >> > blog. I'd like to go to PyCon one of these days. So I like what you've >> > suggested as an approach for dealing with these scripts. In production I >> > plan to give a decent timeout of around 30 seconds, I'm running 10 >> > seconds >> > just to test out the settings. I plan to run the server as you suggested >> > in >> > daemon mode multi-processed but single threaded so that i can safely >> > have >> > processes die that take too long to serve their requests. I added the >> > setting for the inactivity-timeout=10 and ran the following script to do >> > the >> > testing. >> > >> > import time, os >> > >> > def application(environ, start_response): >> > start_response('200 OK', [('Content-Type', 'text/plain')]) >> > yield 'pid: ' + str(os.getpid()) + '\n' >> > time.sleep(30) >> > yield 'done' >> > >> > The first time I ran it after restarting apache the process was killed >> > at >> > 10+5 seconds as expected. But then in subsequent requests the script ran >> > for >> > 30 seconds to completion displaying done at the end. Any ideas what >> > might be >> > going wrong? >> >> Bug in mod_wsgi. See: >> >> http://code.google.com/p/modwsgi/issues/detail?id=182 >> >> Fixed for mod_wsgi 4.0 in subversion. >> >> In short, for the first request after a reset the inactivity timeout >> isn't working properly, instead will only actually timeout when the >> dead lock timeout expires, even though not an actual dead lock. >> >> If this is a big issue, I should look at back porting to 3.X and >> releasing new version. >> >> Graham >> >> > cheers, >> > Marwan >> > >> > On Tue, Jun 29, 2010 at 5:49 AM, Graham Dumpleton >> > <[email protected]> wrote: >> >> >> >> On 28 June 2010 02:19, marwan <[email protected]> wrote: >> >> > Hi all, >> >> > I want to put a limit on the execution time of page requests, to >> >> > protect against some pages that might run large or inefficient >> >> > queries. I've looked around the docs and the Configuration Directives >> >> > but couldn't find out how to do it that way. I'm running the >> >> > following >> >> > stack: >> >> > mod_wsgi 2.8 Apache/2.2.14 Ubuntu 10.04 Python 2.6.5 Linux >> >> > >> >> > my httpd.conf is as follows: >> >> > ServerName 127.0.0.1 >> >> > SetEnv no-gzip >> >> > WSGIDaemonProcess myapp processes=1 threads=1 python-path=/var/www/ >> >> > WSGIProcessGroup myapp >> >> > WSGIScriptAlias /myapp /var/www/myapp.wsgi >> >> > >> >> > I have implemented the following solution, and it seems to work but I >> >> > wanted to get peoples feedback on whether or not this is the best way >> >> > to do this: >> >> > >> >> > this is the contents of /var/www/myapp.wsgi: >> >> > >> >> > import time, os, threading, signal, Queue, cgi >> >> > >> >> > class TimeoutMonitor(threading.Thread): >> >> > def __init__(self, timeoutQ, timeout): >> >> > threading.Thread.__init__(self) >> >> > self.timeoutQ = timeoutQ >> >> > self.timeout = timeout >> >> > >> >> > def run(self): >> >> > try: >> >> > self.timeoutQ.get(timeout=self.timeout) >> >> > except Queue.Empty: >> >> > os.kill(os.getpid(), signal.SIGKILL) >> >> > >> >> > def application(environ, start_response): >> >> > parameters = cgi.parse_qs(environ.get('QUERY_STRING', '')) >> >> > start_response('200 OK', [('Content-type', 'text/plain')]) >> >> > timeoutQ = Queue.Queue() >> >> > TimeoutMonitor(timeoutQ, timeout=10).start() >> >> > >> >> > for i in range(int(parameters['sleep'][0])): >> >> > time.sleep(1) >> >> > yield str(i) + '\n' >> >> > yield 'done' >> >> > timeoutQ.put('done') >> >> > >> >> > I have a few questions: >> >> > - Is there another simpler or better way of doing this? maybe through >> >> > some configuration parameter? >> >> > - In general is it alright to spawn threads in modwsgi (in daemon >> >> > mode)? >> >> > - Is it better to send a SIGKILL, SIGTERM, or SIGINT signal? >> >> >> >> If the application or URL is delegated to a daemon process group where >> >> each process has only a single thread, then you can use the >> >> inactivity-timeout option to WSGIDaemonProcess. Ie., >> >> >> >> WSGIDaemonProcess myapp processes=1 threads=1 python-path=/var/www/ >> >> inactivity-timeout=10 >> >> WSGIProcessGroup myapp >> >> WSGIScriptAlias /myapp /var/www/myapp.wsgi >> >> >> >> What this will do, given that only single thread, is that if a request >> >> itself takes longer than 10 seconds to generate a response, a shutdown >> >> and restart of the process will occur. In doing this there is a >> >> default shutdown timeout of 5 seconds. That is, request is given an >> >> additional 5 seconds to finish normally, but if not complete by then, >> >> then process will be forcibly restarted. Thus, in reality leaving 15 >> >> seconds for request to finish. You could adjust inactivity timeout >> >> down to 5 seconds for overall time of 10, but you really want to be >> >> careful about using such short timeouts. >> >> >> >> Also be aware that although the inactivity timeout applies to where no >> >> request content reading and no response body yielding from active >> >> requests for that timeout, if there are no requests which arrive at >> >> all for that time after process has already handled at least one >> >> request, process will also be shutdown and restarted. >> >> >> >> So, can be made to do what you want, but not necessarily a good thing >> >> to do for a whole application. >> >> >> >> As such, what you may be better off doing is only delegating >> >> problematic URLs to a daemon process with restrictive timeouts. Thus >> >> you might have: >> >> >> >> WSGIDaemonProcess myapp processes=1 threads=15 python-path=/var/www/ >> >> WSGIDaemonProcess myapp+timeout processes=2 threads=1 >> >> python-path=/var/www/ inactivity-timeout=10 >> >> WSGIProcessGroup myapp >> >> WSGIScriptAlias /myapp /var/www/myapp.wsgi >> >> <Location /myap/some/some/url> >> >> WSGIProcessGroup myapp+timeout >> >> </Location> >> >> >> >> That is, all requests normally go to multithread daemon process. If >> >> however URL under sub URL of /myapp/some/url arrives, then that >> >> instead will be delegated to and run in the daemon process group that >> >> has more aggressive timeouts. This way you don't affect operation of >> >> application as a whole. >> >> >> >> BTW, sorry for late reply. Still catching up on emails from past few >> >> days when was at Conference. >> >> >> >> >> >> Graham >> >> >> >> -- >> >> You received this message because you are subscribed to the Google >> >> Groups >> >> "modwsgi" group. >> >> To post to this group, send email to [email protected]. >> >> To unsubscribe from this group, send email to >> >> [email protected]. >> >> For more options, visit this group at >> >> http://groups.google.com/group/modwsgi?hl=en. >> >> >> > >> > -- >> > You received this message because you are subscribed to the Google >> > Groups >> > "modwsgi" group. >> > To post to this group, send email to [email protected]. >> > To unsubscribe from this group, send email to >> > [email protected]. >> > For more options, visit this group at >> > http://groups.google.com/group/modwsgi?hl=en. >> > >> >> -- >> You received this message because you are subscribed to the Google Groups >> "modwsgi" group. >> To post to this group, send email to [email protected]. >> To unsubscribe from this group, send email to >> [email protected]. >> For more options, visit this group at >> http://groups.google.com/group/modwsgi?hl=en. >> > > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/modwsgi?hl=en. > -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/modwsgi?hl=en.
