Hey Graham, Thanks for the detailed explanation. That sounds perfect.
If you haven't documented this (I haven't seen it all in one place), this email would make a good start to a wiki page. Will let you know what I come up with. Thanks! JG PS. mod_wsgi behind nginx on linux whips the pants off any hosting solution I've seen for speed and reliability. My web apps, from > 1,000 miles away, behave like windows apps when using Google Chrome. On Thu, Mar 24, 2011 at 6:31 PM, Graham Dumpleton < [email protected]> wrote: > On 25 March 2011 08:36, Jason Garber <[email protected]> wrote: > > Hello, > > Has anyone put thought into using a couple extra threads in a mod_wsgi to > > handle background tasks (similar to how many of us use cron)? Ideally > this > > would cover multi-process, multi-server scenarios, so one would need a > > method of synchronizing the "worker threads" (database, etc...). > > If during the startup of a given mod_wsgi process (daemon mode), a couple > of > > python threads were spawned... > > 1. are there any fundamental issues with that? > > 2. are there any restrictions to be aware of (signals come to mind)? > > 3. with proper exception handling, could that be a highly reliable > solution? > > To me it would be great to cut crontab out of the loop, and run > everything > > in the same environment. When your app is running (because Apache will > make > > *sure* it is running with a high degree of confidence), then your > background > > processes are doing their thing also. > > Just looking for advice and feedback. Thanks! > > Unless it is intended to specifically affect data in memory for the > application serving the requests, then create a distinct mod_wsgi > daemon process group consisting of a single process of a single > thread. The single thread is just to keep memory use to a minimum as > the request handler thread would never actually be used as we aren't > going to delegate any request handling to the process anyway. > > Having that, use WSGImportScript to import a script of startup of that > daemon process and from that spawn the background threads which are > going to do the work. > > For example: > > WSGIDaemonProcess application-process processes=5 threads=5 > WSGIDaemonProcess tasks-process processes=1 threads=1 > > WSGIScriptAlias / /some/path/application.wsgi > process-group=application-process application-group=%{GLOBAL} > > WSGIImportScript /some/path/tasks.py process-group=tasks-process > application-group=%{GLOBAL} > > The /some/path/tasks.py script should create the background threads > which are to do the work for the background tasks. > > If you want to try and be a bit more graceful about shutdown, then > adapt the code from: > > > http://code.google.com/p/modwsgi/wiki/ReloadingSourceCode#Monitoring_For_Code_Changes > > That code creates a background thread and then uses a Queue object as > a sleep mechanism between runs. An atexit callback is used to push an > object on the Queue object to flag that process is being shutdown. > > Doing it this way with dedicated daemon process group just for > background tasks means that restarting the application process by > touching the WSGI script file doesn't interfere with the background > tasks process. If you need to restart the background tasks process > then you can also send it a SIGTERM to restart and that will not > affect main application process. > > Use of a separate daemon process group just for background tasks means > you don't have to worry about multi process issues and > synchronisation. Obviously though it just cannot affect what is in > memory of the application process. You would still have to use a > background thread in application process for just stuff where want to > affect what is in memory. > > In either case, restart Apache will obviously kill off/restart both > application and background tasks processes. > > Does this achieve what you had in mind? > > Graham > > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/modwsgi?hl=en. > > -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/modwsgi?hl=en.
