Stephen Adkins wrote:
So what I think you are saying for option 2 is:* Apache children (web server processes with mod_perl) have two personalities: - user request processors - back-end work processors * When a user submits work to the queue, the child is acting in a "user request" role and it returns the response quickly. * After detaching from the user, however, it checks to see if fewer than four children are processing the queue and if so, it logs into the mainframe and starts processing the queue. * When it finishes the request, it continues to work the queue until no more work is available, at which time, it quits its "back-end processor" personality and returns to wait for another HTTP request.
This just seems a bit odd (and unnecessarily complex).
It does when you put it like that, but it doesn't have to be that way. I would separate the input (user or queue) from the processing part. You'd have a module that runs in mod_perl which knows how to process requests. You have a separate module which can provide a UI for placing requests. Synchronous ones go straight to processing, while asynch ones get added to the queue.
You'd also have a controlling process that polls the queue and if it finds anything it uses LWP to send it to mod_perl for handling. I would make this a tiny script triggered from cron if possible, since cron is robust and can handle outages and error reporting nicely.
Why not let there be web server processes and queue worker processes and they each do their own job? Web servers seem to me to be for synchronous activity, where the user is waiting for the results.
When I think of queue processing, I think of a system for handling tasks in parallel that provides a simple API for plugging in logic, a well-defined control interface, logging, easy configuration... sounds like Apache to me. You just need a tiny control process to trigger it via LWP. Apache is already a system for handling a queue of HTTP requests in parallel, so you just have to make your requests look like HTTP.
You certainly could do this other ways, but you'd probably have to write a lot more code or else use something far less reliable than Apache.
P.S. Another limitation of the "use Apache servers for all server processing"
philosophy seems to be scheduled events or system events (those not
initiated by an HTTP request, which are user events).
Cron/at + LWP. - Perrin
