On Sun, 2005-12-18 at 23:48 -0600, JT Smith wrote: > I added in event handling with both Event > and Event::Lib as seperate trials.
I just used a short sleep with Time::HiRes between polling the database for new jobs. > If everything else is running in Apache, why start a > seperate service to run these tasks? Because everything else sounds much harder and less reliable. That was my reason. > And again, I said I want to go crazy. Let's not > figure out how else we could do that (I already know that), but how > could we do it using > Apache? I came up with about a half dozen possible ways of doing our queue system. Only one of them used apache as the only daemon. I looked into custom protocol handlers and the rest of the mod_perl 2 API and there's nothing I can see that would make a time-based system possible. It would require rewriting some C code and probably changing things that are not part of the module API. The idea I had for handling events without all that goes like this: Run a mod_perl server that the job submitters contact via HTTP. When a process gets a request, it checks to see if there are enough listener processes free for accepting jobs (as opposed to processing jobs). If there are not, it adds the job to the queue and goes back to listening for requests. If there are, it processes the job. This ensures that processing jobs does not starve the ability to accept new ones. A process which has started working on a job will loop (keeping the current request alive), pulling jobs off the queue and working on them until the queue is empty again, when it will allow the request to finish and go back to sleep. In other words, new requests will start child processes working, and all processes that get started will stay working until the queue is empty again. Pros * No polling except by processes that have just finished a job and are deciding whether or not to exit. (This may still be more than the polling done by a simple perl daemon.) * Quick pickup of new jobs. Cons * Clustering would require some kind of custom load balancer that would know which machines were actually busiest. This might involve reading the scoreboard, or something more complex. Much harder than other approaches. * No obvious way to tell how many processes are working vs. listening. Would probably need to use something like Cache::FastMmap to track this. * The whole idea is fairly hard to explain, which probably means it's too complex and will be hard to build and debug. Anyway, feel free to expand on this idea or try it out. As complex as it is, it avoids having to delve into the guts of the httpd code. - Perrin