Hey Graham,
Sorry for late reply. It's working out really well. Let me give some
details here:
We've been building a tool where people can generate a graphical
presentation site based on input into a web interface. There is a
rendering process that takes several seconds to generate the graphics.
We wanted to do this in the background, but quickly. Historically we have
used polling cron-jobs for this, but they are clunky to manage, require a
separate setup, and are just a pain to use.
We're running a WSGI Daemon Process in the background which keeps an eye on
the database and will render any sites that need it, right away. We can
communicate with the background process using either a queue or redis
pubsub.
To get the error log going to an application-specific log, we wrapped it in
a <VirtualHost> block. As per our other applications, we use a reserved
port for each "app instance". This is rather convenient too because it
allows us to send commands to the background processing application using
HTTP/WSGI if we want (/restart for example).
Your notes on the background thread doing the work make sense. Also we
need to spawn a few background threads to do different kinds of work all
from the same WSGIDaemonProcess. The vast majority of the time is spent
either waiting (postgresql, subprocess, requests), or waiting for a task to
do.
Listen 127.0.0.1:60010
<VirtualHost 127.0.0.1:60010>
ServerName _default_;
WSGIDaemonProcess Async-ShockBox-60010 processes=1 threads=1
python-path=/home/jason/DevLevel.2/AMSTMT/Python
display-name=Async-ShockBox-60010
WSGIImportScript /home/jason/DevLevel.2/AMSTMT/Async/ShockBox.py
process-group=Async-ShockBox-60010 application-group=%{GLOBAL}
ErrorLog /home/jason/DevLevel.2/AMSTMT/apache-error.log
WSGIScriptAlias /restart
/home/jason/DevLevel.2/AMSTMT/Async/restart.wsgi
</VirtualHost>
All in all, it's really great to have this baked into the application and
run under the appserver (apache in this case). It makes everything so much
cleaner.
Next I need to look at RabbitMQ or something like that to handle the actual
async tasks. We are already using pusher.com and websockets to send
realtime updates back to the web UI.
Thanks for the help. I'll post again when we have it a bit further.
Jason
On Sun, Mar 15, 2015 at 6:43 PM, Graham Dumpleton <
[email protected]> wrote:
> Hi Jason
>
> How is what you wanted to do working out.
>
> I did finally work out what problem you might be having when was off on a
> short holiday over the weekend.
>
> Signals will not work if the script you give to WSGIImportScript never
> returns.
>
> Unfortunately right now because am cheating and using what would otherwise
> be a daemon process for handling web requests, signals don't actually work
> for Python code run by WSGIImportScript.
>
> Right now what you would need to do is create a background thread from the
> script to do the actual work and let the script itself return.
>
> What will happen is that your task would then run in the background thread
> and the main thread would go on to wait for signals and so handle them
> properly.
>
> The only other option is for your script to do an exec of a separate
> application (Python script) which does the work.
>
> I am working on a few ideas of how to change things to make this work
> better.
>
> Graham
>
> On 12/03/2015, at 2:02 PM, Graham Dumpleton <[email protected]>
> wrote:
>
>
> On 12/03/2015, at 1:28 PM, Jason Garber <[email protected]> wrote:
>
> Hi Graham,
>
> I've been starting to work with background processes as a replacement for
> clunky cron-jobs in a major application. Here is what I mean:
>
> WSGIDaemonProcess Async-ShockBox-60011 processes=1 threads=1
> python-path=/home/jason/DevLevel.2/AMSTMT/Python
> display-name=Async-ShockBox-60011
>
> WSGIImportScript /home/jason/DevLevel.2/AMSTMT/Async/ShockBox.py
> process-group=Async-ShockBox-60011 application-group=%{GLOBAL}
>
> This works great:
> 1. the script runs as expected
> 2. if I kill it as root with kill -9 then it is restarted automatically
>
> I have some questions...
>
> *1. Is there any way to select what error log the output goes to.
> Currently it is going to the global apache error log.*
>
>
> Only thing I can think of is to create a fake VirtualHost and set ErrorLog
> for it and have the WSGIDaemonProcess directive in it.
>
> <VirtualHost 127.0.0.255:*>
> ErrorLog /some/path/error.log
> WSGIDaemonProcess Async-ShockBox-60011 processes=1 threads=1
> python-path=/home/jason/DevLevel.2/AMSTMT/Python
> display-name=Async-ShockBox-60011
> WSGIImportScript /home/jason/DevLevel.2/AMSTMT/Async/ShockBox.py
> process-group=Async-ShockBox-60011 application-group=%{GLOBAL}
> </VirtualHost>
>
> I have thought about an option to WSGIDaemonProcess to override the error
> log before, but isn't so simple.
>
> BTW, thanks for asking this question. Will have to allow an error log to
> be specified with this trick for service scripts in mod_wsgi-express.
>
> *2. Is there any way, short of `kill -9` or restarting apache, to restart
> the daemon process that is being used by WSGIImportScript?*
>
>
> The TERM and HUP signals should work as well and will attempt to properly
> shutdown the process.
>
> Very recent versions of mod_wsgi should be a lot smarter about handling
> signals received if from this service process you fork a new process to do
> actual work which you wait on. There was a bug before where if sent a
> signal to one of the forked processes that it would actually shutdown the
> parent mod_wsgi daemon process because of the loopback UNIX pipe used to
> tell process to shutdown due to a signal.
>
> For a forked process, you can register your own signal handlers, but again
> for that to work properly, need a very recent mod_wsgi version.
>
> *3. Is there any way, within a daemon process itself, to restart itself?
> I tried sys.exit() but got this:*
>
> mod_wsgi (pid=25983): SystemExit exception raised by WSGI script
> '/home/jason/DevLevel.2/AMSTMT/Async/ShockBox.py' ignored.
>
>
> Send a SIGINT or SIGTERM to itself should work.
>
> Worst case, us os._exit(). That will do an immediate exit of process and
> not try and shutdown Python interpreter, run atexit callbacks etc.
>
> Am curious to hear more about what you are doing and whether this was
> inspired by the recent long discussion about WebFaction where I showed how
> to use a daemon process to run a multiprocessing module queue manager to
> accept requests from WSGI application to do jobs in forked sub processes
> with timeouts on processes.
>
> There are all sorts of tricks for using daemon processes don't think
> anyone would realise. Want to make my PyCon AU talk this year about it.
>
> For example, did you know that you could fork/exec a redis or memcached
> inside of a daemon process and have Apache manage it?
>
> Did you know you could run in a daemon process a mod_wsgi-express against
> a different Python version (ie., Python 3 rather than 2) and then setup
> proxies from Apache to it for a site name of sub URLs and so have Apache
> manage everything, with both Python 2 and Python 3 parts.
>
> :-)
>
> Graham
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "modwsgi" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To post to this group, send email to [email protected].
> Visit this group at http://groups.google.com/group/modwsgi.
> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google Groups
"modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/modwsgi.
For more options, visit https://groups.google.com/d/optout.
