Well, I guess the end result is the same. But I think it's a bit easier/cleaner to put the failed job in a separate tube, say, "failed_db" or "failed_mail" depending on what step failed. That way I don't have to "patch" the job (ie no need to add any meta data to the job itself) and I can have dedicated consumer-threads listening on each tube.
I know about `bury`, but that won't help me since I don't know what do to with buried jobs (ie did it fail the "db" step or the "mail" step). What is missing, I think, is some way to add arbitrary "tags" or something to jobs without having to patch the job itself. I suppose moving them to another tube is a way to achieve that. Den torsdagen den 17:e juli 2014 kl. 21:24:36 UTC+2 skrev chadkouse: > > I still don’t see the point of having a tube just for failed (or partially > failed) jobs but I don’t know about your use case. > > Beanstalkd also has a `bury` command to keep failed jobs around but not > continue trying to process them - could be useful if you’re wanting to do > “later processing” > > -- > Chad Kouse > Sent with Airmail > > On July 17, 2014 at 3:18:20 PM, Wickman ([email protected] > <javascript:>) wrote: > > Yes, that could work, but in that case I might as well put failed jobs in > dedicated fail-tubes for later processing. > > I was looking for an atomic approach though, but I suppose I have to do > without. > > Den torsdagen den 17:e juli 2014 kl. 13:49:26 UTC+2 skrev chadkouse: >> >> If one or more steps fail why not instead of releasing the job just go >> ahead and delete the original job and create a copy of it with some >> metadata attached as to what steps are left to do. Place the job back into >> your validated tube. And have your consumer check for this metadata to >> determine what is left to do. No metadata = all steps. >> >> Sent using CloudMagic >> <https://cloudmagic.com/k/d/mailapp?ct=pi&cv=5.0.82&pv=8.0> >> >> On Thu, Jul 17, 2014 at 4:41 AM, Wickman <[email protected]> wrote: >> >> Hi, thanks for reply! >> >> Let me give a concrete example: >> >> - Each job in the tube is first checked for validness, then put it in >> a "validated" tube. >> - Each valid job should be sent to a database(*) >> - Each valid job should be sent to another database(*) >> - Each valid job should be stored in a file system(*) >> - When all above is done, delete the job >> >> (Validation must be done first, other steps can be done in parallell). >> >> So yes, I could do everything in one consumer, but if one step fails >> (database not available etc) I can't just bury the job, because I don't >> know which step failed when I retry the job later. >> >> I have two ideas which I'd like some feedback on: >> >> Idea 1: When a step fails, move the job to a dedicated fail-tube, like >> "database_fail" or "file_fail". Then have a consumer handling them. This is >> the non-atomic step which i don't really like. >> >> Idea 2: When a step fails, release() the job back to the tube but with a >> new priority. Say 110 for database_fail and 120 for file_fail. When the >> consumer reserves a job, it checks the priority to know what to do next >> with it. >> >> Again, thanks for any input! >> >> >> Den tisdagen den 15:e juli 2014 kl. 21:30:39 UTC+2 skrev chadkouse: >>> >>> The short answer to your questions is: no >>> >>> The long answer is still no, but you can probably *force* things to do >>> what you’re saying with some more logic existing outside of the message >>> queue itself. >>> >>> Might be easier / more robust to have 1 tube and 1 job and then your >>> consumer does what’s needed. Many modern languages allow you to run >>> multiple threads in parallel. >>> >> >>> On July 15, 2014 at 3:12:00 PM, Wickman ([email protected]) wrote: >>> >>> Hi >>> >>> In my case, jobs must pass through several steps before they are done. Some >>> steps can be done in parallell. I figured I solve this by moving jobs >>> between tubes, say I use tubes like "step1" which then moves the job to >>> tubes "step2" and "step3" etc. >>> >>> Is this a sane approach? >>> >>> I see a few issues with this approach: >>> >>> 1. Is there any way of moving a job from one tube to another >>> safely/atomically (put + delete)? >>> 2. Copying the same job to several tubes for parallell processing >>> doubles the amount of RAM needed. Is there any way around this? >>> >>> Just asking for some ideas and "best practices" here. >>> >>> Thanks! >>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "beanstalk-talk" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at http://groups.google.com/group/beanstalk-talk. >> For more options, visit https://groups.google.com/d/optout. >> >> -- > You received this message because you are subscribed to the Google Groups > "beanstalk-talk" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] <javascript:>. > To post to this group, send email to [email protected] > <javascript:>. > Visit this group at http://groups.google.com/group/beanstalk-talk. > For more options, visit https://groups.google.com/d/optout. > > -- You received this message because you are subscribed to the Google Groups "beanstalk-talk" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/beanstalk-talk. For more options, visit https://groups.google.com/d/optout.
