I still don’t see the point of having a tube just for failed (or partially failed) jobs but I don’t know about your use case.
Beanstalkd also has a `bury` command to keep failed jobs around but not continue trying to process them - could be useful if you’re wanting to do “later processing” -- Chad Kouse Sent with Airmail On July 17, 2014 at 3:18:20 PM, Wickman ([email protected]) wrote: Yes, that could work, but in that case I might as well put failed jobs in dedicated fail-tubes for later processing. I was looking for an atomic approach though, but I suppose I have to do without. Den torsdagen den 17:e juli 2014 kl. 13:49:26 UTC+2 skrev chadkouse: If one or more steps fail why not instead of releasing the job just go ahead and delete the original job and create a copy of it with some metadata attached as to what steps are left to do. Place the job back into your validated tube. And have your consumer check for this metadata to determine what is left to do. No metadata = all steps. Sent using CloudMagic On Thu, Jul 17, 2014 at 4:41 AM, Wickman <[email protected]> wrote: Hi, thanks for reply! Let me give a concrete example: Each job in the tube is first checked for validness, then put it in a "validated" tube. Each valid job should be sent to a database(*) Each valid job should be sent to another database(*) Each valid job should be stored in a file system(*) When all above is done, delete the job (Validation must be done first, other steps can be done in parallell). So yes, I could do everything in one consumer, but if one step fails (database not available etc) I can't just bury the job, because I don't know which step failed when I retry the job later. I have two ideas which I'd like some feedback on: Idea 1: When a step fails, move the job to a dedicated fail-tube, like "database_fail" or "file_fail". Then have a consumer handling them. This is the non-atomic step which i don't really like. Idea 2: When a step fails, release() the job back to the tube but with a new priority. Say 110 for database_fail and 120 for file_fail. When the consumer reserves a job, it checks the priority to know what to do next with it. Again, thanks for any input! Den tisdagen den 15:e juli 2014 kl. 21:30:39 UTC+2 skrev chadkouse: The short answer to your questions is: no The long answer is still no, but you can probably *force* things to do what you’re saying with some more logic existing outside of the message queue itself. Might be easier / more robust to have 1 tube and 1 job and then your consumer does what’s needed. Many modern languages allow you to run multiple threads in parallel. On July 15, 2014 at 3:12:00 PM, Wickman ([email protected]) wrote: Hi In my case, jobs must pass through several steps before they are done. Some steps can be done in parallell. I figured I solve this by moving jobs between tubes, say I use tubes like "step1" which then moves the job to tubes "step2" and "step3" etc. Is this a sane approach? I see a few issues with this approach: Is there any way of moving a job from one tube to another safely/atomically (put + delete)? Copying the same job to several tubes for parallell processing doubles the amount of RAM needed. Is there any way around this? Just asking for some ideas and "best practices" here. Thanks! -- You received this message because you are subscribed to the Google Groups "beanstalk-talk" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/beanstalk-talk. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "beanstalk-talk" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/beanstalk-talk. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "beanstalk-talk" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/beanstalk-talk. For more options, visit https://groups.google.com/d/optout.
