On Dec 11, 2007 2:35 PM, mike <[EMAIL PROTECTED]> wrote: > When I design a queue system (like MySQL queue table and PHP running > via cron to process jobs) I have a status of scheduled/pending, > processing (currently processing), error, completed, start time, end > time, any message if there is an error, or output in general even with > success (it depends, I suppose) > > I would hope it has most of that functionality.
It has a great deal of that functionality, but also bear in mind that this is just the server. Some of those things are better taken care of by the worker processes that run jobs. For example, recording start and stop time, errors, and other output can all be done by having each worker write to a log file. Each job in beanstalk starts out in the "ready" state, and when a worker wants to run the job it gets marked as "reserved" by that worker. When the worker finishes running the job it deletes it from the server. If there is a transitory error, the worker can release the job (optionally with a delay) back into the ready queue so it can run again later. If the job takes too long (say, the worker process gets wedged), it times out (currently the timeout is fixed at 120 seconds) and goes back into the ready queue. If a job has timed out or been released, those facts are marked in the job stats. So it's easy to tell if something has ever gone wrong for a given job. One feature that we're thinking of adding is a flag to mark jobs as idempotent, so the server knows it's safe to rerun them automatically. > It is cool but I would > not feel comfortable sending jobs into a void that might process it so > fast I can't tell but at the same time I can't tell also that it > processed the job and was successful or errored :) That's understandable. The best way to tell that your jobs are actually running is to watch what the workers are doing, rather than sampling the server stats. We've been using the beantsalk system (including a rails plugin that we're about to release) for almost two months now. The server has been extremely reliable. By far the worst aspect of the system has been the workers. We've had a handful of mishaps during the development and early deployment of this system, and all of them were caused by mistakes in the worker code. At this point, I'd call the server semi-mature. There are a few details that still need to be changed, but the overall design is pretty solid. kr
