I believe the current philosophy of Beanstalk and its handling of releasing to be the most flexible / safest in the long term.
> are not idempotent[1]. If you have a minor network outage you may end > up with jobs being done multiple times which may really mess things I think its up to us application owners to know if a given job has executed already and/or is repeatable. To accomplish this I have a method called "applied?" on my model instance that tells me if that job has already been executed. When I pull a job from Beanstalkd and inflate it into a model instance I check that method - maybe the job creats a record in the DB, at which point I can check for the existence of that record in my "applied?" method. If it exists then I know the job is a duplicate and I can safely delete it. For jobs that interact with the DB its easier to check to see if that job has been applied but if your job just sends emails then yes you have to decide whether you'd rather send a duplicate email or none at all. /Cody On Mon, Aug 16, 2010 at 3:46 PM, Kurt Seifried <[email protected]> wrote: >> Personally, I disagree. If I have an outage (whether network outage or >> worker machine catching fire), I absolutely do not want jobs stuck >> waiting for the host or network to be restored. I think in the face of >> an outage, it's entirely appropriate to recover the jobs for >> delegation to a new worker to take over the processing. > > But this may complete break the logic of someones application if jobs > are not idempotent[1]. If you have a minor network outage you may end > up with jobs being done multiple times which may really mess things > up. This is not always an appropriate assumption to make. However the > assumption of "if you make a mess you need to handle the mess" is > safer on the side of beanstalkd (and makes beanstalkd much easier to > program). > >>And it also >> seems entirely appropriate that the connection to the job broker is as >> good as any other form of heartbeat. If you do need disconnected >> processing, it would seem that you could reserve, bury and disconnect, >> process, and then reconnect to delete the job. > > I think the problem here is more philosophical. Should recovery logic > and idempotency[1] be the job of beanstalkd, or of the application? > > beanstalkd is a generic software solution for queues, you can't stick > business rules or logic in it (yet, and hopefully never). Should > beanstalkd try to be more "clever"? To have the kind of recovery you > are proposing we need to add a lot of business level logic (do we > handle failure, how do we handle failure, what about all the > scorner/special cases? do we give beanstalkd the ability to inspect > and understand jobs? etc.), personally I think this is far better done > in your application than in beanstalkd. If you need to architect > intelligence into your message queue it's not a message queue anymore, > it's an application. > > [1]http://en.wikipedia.org/wiki/Idempotence > > >> -- William > > > -- > Kurt Seifried > [email protected] > tel: 1-703-879-3176 > > -- > You received this message because you are subscribed to the Google Groups > "beanstalk-talk" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/beanstalk-talk?hl=en. > > -- You received this message because you are subscribed to the Google Groups "beanstalk-talk" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/beanstalk-talk?hl=en.
