On 25 March 2011 08:36, Jason Garber <[email protected]> wrote:
> Hello,
> Has anyone put thought into using a couple extra threads in a mod_wsgi to
> handle background tasks (similar to how many of us use cron)? Ideally this
> would cover multi-process, multi-server scenarios, so one would need a
> method of synchronizing the "worker threads" (database, etc...).
> If during the startup of a given mod_wsgi process (daemon mode), a couple of
> python threads were spawned...
> 1. are there any fundamental issues with that?
> 2. are there any restrictions to be aware of (signals come to mind)?
> 3. with proper exception handling, could that be a highly reliable solution?
> To me it would be great to cut crontab out of the loop, and run everything
> in the same environment. When your app is running (because Apache will make
> *sure* it is running with a high degree of confidence), then your background
> processes are doing their thing also.
> Just looking for advice and feedback. Thanks!
Unless it is intended to specifically affect data in memory for the
application serving the requests, then create a distinct mod_wsgi
daemon process group consisting of a single process of a single
thread. The single thread is just to keep memory use to a minimum as
the request handler thread would never actually be used as we aren't
going to delegate any request handling to the process anyway.
Having that, use WSGImportScript to import a script of startup of that
daemon process and from that spawn the background threads which are
going to do the work.
For example:
WSGIDaemonProcess application-process processes=5 threads=5
WSGIDaemonProcess tasks-process processes=1 threads=1
WSGIScriptAlias / /some/path/application.wsgi
process-group=application-process application-group=%{GLOBAL}
WSGIImportScript /some/path/tasks.py process-group=tasks-process
application-group=%{GLOBAL}
The /some/path/tasks.py script should create the background threads
which are to do the work for the background tasks.
If you want to try and be a bit more graceful about shutdown, then
adapt the code from:
http://code.google.com/p/modwsgi/wiki/ReloadingSourceCode#Monitoring_For_Code_Changes
That code creates a background thread and then uses a Queue object as
a sleep mechanism between runs. An atexit callback is used to push an
object on the Queue object to flag that process is being shutdown.
Doing it this way with dedicated daemon process group just for
background tasks means that restarting the application process by
touching the WSGI script file doesn't interfere with the background
tasks process. If you need to restart the background tasks process
then you can also send it a SIGTERM to restart and that will not
affect main application process.
Use of a separate daemon process group just for background tasks means
you don't have to worry about multi process issues and
synchronisation. Obviously though it just cannot affect what is in
memory of the application process. You would still have to use a
background thread in application process for just stuff where want to
affect what is in memory.
In either case, restart Apache will obviously kill off/restart both
application and background tasks processes.
Does this achieve what you had in mind?
Graham
--
You received this message because you are subscribed to the Google Groups
"modwsgi" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/modwsgi?hl=en.
