On 25 March 2011 10:15, Jason Garber <[email protected]> wrote: > Hey Graham, > Thanks for the detailed explanation. That sounds perfect. > If you haven't documented this (I haven't seen it all in one place), this > email would make a good start to a wiki page.
No, not documented. One of the tricky things I keep to myself and those people who are ask. Bit worried that if documented that people will start abusing it. :-) > Will let you know what I come up with. > Thanks! > JG > PS. mod_wsgi behind nginx on linux whips the pants off any hosting solution > I've seen for speed and reliability. My web apps, from > 1,000 miles away, > behave like windows apps when using Google Chrome. The nginx --> Apache/mod_wsgi setup and why it works well is yet another thing I haven't documented. Some of these things will be easier to document when have New Relic stuff working as will then be able to insert pretty graphs that will hopefully make it bleeding obvious what benefits you get. Graham > On Thu, Mar 24, 2011 at 6:31 PM, Graham Dumpleton > <[email protected]> wrote: >> >> On 25 March 2011 08:36, Jason Garber <[email protected]> wrote: >> > Hello, >> > Has anyone put thought into using a couple extra threads in a mod_wsgi >> > to >> > handle background tasks (similar to how many of us use cron)? Ideally >> > this >> > would cover multi-process, multi-server scenarios, so one would need a >> > method of synchronizing the "worker threads" (database, etc...). >> > If during the startup of a given mod_wsgi process (daemon mode), a >> > couple of >> > python threads were spawned... >> > 1. are there any fundamental issues with that? >> > 2. are there any restrictions to be aware of (signals come to mind)? >> > 3. with proper exception handling, could that be a highly reliable >> > solution? >> > To me it would be great to cut crontab out of the loop, and run >> > everything >> > in the same environment. When your app is running (because Apache will >> > make >> > *sure* it is running with a high degree of confidence), then your >> > background >> > processes are doing their thing also. >> > Just looking for advice and feedback. Thanks! >> >> Unless it is intended to specifically affect data in memory for the >> application serving the requests, then create a distinct mod_wsgi >> daemon process group consisting of a single process of a single >> thread. The single thread is just to keep memory use to a minimum as >> the request handler thread would never actually be used as we aren't >> going to delegate any request handling to the process anyway. >> >> Having that, use WSGImportScript to import a script of startup of that >> daemon process and from that spawn the background threads which are >> going to do the work. >> >> For example: >> >> WSGIDaemonProcess application-process processes=5 threads=5 >> WSGIDaemonProcess tasks-process processes=1 threads=1 >> >> WSGIScriptAlias / /some/path/application.wsgi >> process-group=application-process application-group=%{GLOBAL} >> >> WSGIImportScript /some/path/tasks.py process-group=tasks-process >> application-group=%{GLOBAL} >> >> The /some/path/tasks.py script should create the background threads >> which are to do the work for the background tasks. >> >> If you want to try and be a bit more graceful about shutdown, then >> adapt the code from: >> >> >> http://code.google.com/p/modwsgi/wiki/ReloadingSourceCode#Monitoring_For_Code_Changes >> >> That code creates a background thread and then uses a Queue object as >> a sleep mechanism between runs. An atexit callback is used to push an >> object on the Queue object to flag that process is being shutdown. >> >> Doing it this way with dedicated daemon process group just for >> background tasks means that restarting the application process by >> touching the WSGI script file doesn't interfere with the background >> tasks process. If you need to restart the background tasks process >> then you can also send it a SIGTERM to restart and that will not >> affect main application process. >> >> Use of a separate daemon process group just for background tasks means >> you don't have to worry about multi process issues and >> synchronisation. Obviously though it just cannot affect what is in >> memory of the application process. You would still have to use a >> background thread in application process for just stuff where want to >> affect what is in memory. >> >> In either case, restart Apache will obviously kill off/restart both >> application and background tasks processes. >> >> Does this achieve what you had in mind? >> >> Graham >> >> -- >> You received this message because you are subscribed to the Google Groups >> "modwsgi" group. >> To post to this group, send email to [email protected]. >> To unsubscribe from this group, send email to >> [email protected]. >> For more options, visit this group at >> http://groups.google.com/group/modwsgi?hl=en. >> > > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/modwsgi?hl=en. > -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/modwsgi?hl=en.
