a simpler solution might be to just have a generic "failed_jobs" tube that you move a job into when it fails (cloning it, adding exception data, etc…)
-- Chad Kouse On Tuesday, October 30, 2012 at 3:35 PM, Jurian Sluiman wrote: > Hi Chad > > On Tuesday, October 30, 2012 5:13:57 PM UTC+1, chadkouse wrote: > > An easy workaround is to add a flag to the cloned job to tell your consumer > > to bury it as soon as it picks it up. > > > > > I have thought about that too, but it all just becomes too messy. You cannot > kick a buried job if you want to modify the data. If you set a bury flag, you > have to remove it again too. There are thus two down sides: > The job gets queued again until it can be buried. If you have a long queue > because of some enormous visitor hit or so, a problem which must be buried > can take quite some time before it gets into the bury list for inspection; > If the job gets a flag, you must remove the flag when you kick it again. That > means the job is cloned again, removed from the queue and inserted again with > a put command. > There are too many steps which all can go wrong, just in order to append some > data to the job. Also, with clones of the original job floating around, you > cannot rely on the job id anymore. I am making a bridge where other > developers can use my code. If anyone uses the job id to track the job over > time (just to name an example) it is not transparent the buried job is > completely unrelated to the inserted job. Or the kicked one. Or the released > one. > > > If you inspect the protocol, the put command is this: > put <pri> <delay> <ttr> <bytes>\r\n <data>\r\n > > The bury command is this: > bury <id> <pri>\r\n > > You can make that into this, where you take care of backwards compatibility > too: > bury <id> <pri>\r\n <data>\t\n > > The data will overwrite any previously set data for this job. If no data is > set, the original data remains. > > As said, I am willing to hire a C expert to take care of this and another > feature I really would like to see (a peek-buried-all or so, returning a list > of ids from all buried jobs, which I can get then with peek <id>). But before > I can take that step, I think I need some more insights in how this all > works, if the repository owner is willing to accept such pull requests and if > I don't overlook something in this process. Or, perhaps there is another job > queue manager which is better than beanstalk at this aspect? > > > What we normally do with buried jobs is to just run the same payload > > against a consumer in our dev environment and observe what happens. We also > > log all exceptions (by just echoing out from our consumer) so we normally > > can easily tell exactly why a buried job failed. > > Well, the cause the job is buried is highly influenced by the environment. I > talk about a webapp here where jobs are things like "create pdf", "send > email" and "resize image". If I connect to a 3rd party service and it fails, > I retry (so a release with a delay). I start a counter there and after x > retries I bury the job. > > I do not want the app to have infinite loops connecting to a 3rd party > service and a user might have a look if something has gone wrong there. I > expect many cases where I bury jobs and I do not want to spit through logs to > see what went wrong. > --- > Jurian Sluiman > > -- > You received this message because you are subscribed to the Google Groups > "beanstalk-talk" group. > To view this discussion on the web visit > https://groups.google.com/d/msg/beanstalk-talk/-/Vs1CMP2jdHsJ. > To post to this group, send email to [email protected] > (mailto:[email protected]). > To unsubscribe from this group, send email to > [email protected] > (mailto:[email protected]). > For more options, visit this group at > http://groups.google.com/group/beanstalk-talk?hl=en. -- You received this message because you are subscribed to the Google Groups "beanstalk-talk" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/beanstalk-talk?hl=en.
