Hi, thanks for reply! Let me give a concrete example:
- Each job in the tube is first checked for validness, then put it in a "validated" tube. - Each valid job should be sent to a database(*) - Each valid job should be sent to another database(*) - Each valid job should be stored in a file system(*) - When all above is done, delete the job (Validation must be done first, other steps can be done in parallell). So yes, I could do everything in one consumer, but if one step fails (database not available etc) I can't just bury the job, because I don't know which step failed when I retry the job later. I have two ideas which I'd like some feedback on: Idea 1: When a step fails, move the job to a dedicated fail-tube, like "database_fail" or "file_fail". Then have a consumer handling them. This is the non-atomic step which i don't really like. Idea 2: When a step fails, release() the job back to the tube but with a new priority. Say 110 for database_fail and 120 for file_fail. When the consumer reserves a job, it checks the priority to know what to do next with it. Again, thanks for any input! Den tisdagen den 15:e juli 2014 kl. 21:30:39 UTC+2 skrev chadkouse: > > The short answer to your questions is: no > > The long answer is still no, but you can probably *force* things to do > what you’re saying with some more logic existing outside of the message > queue itself. > > Might be easier / more robust to have 1 tube and 1 job and then your > consumer does what’s needed. Many modern languages allow you to run > multiple threads in parallel. > > On July 15, 2014 at 3:12:00 PM, Wickman ([email protected] > <javascript:>) wrote: > > Hi > > In my case, jobs must pass through several steps before they are done. Some > steps can be done in parallell. I figured I solve this by moving jobs > between tubes, say I use tubes like "step1" which then moves the job to > tubes "step2" and "step3" etc. > > Is this a sane approach? > > I see a few issues with this approach: > > 1. Is there any way of moving a job from one tube to another > safely/atomically (put + delete)? > 2. Copying the same job to several tubes for parallell processing > doubles the amount of RAM needed. Is there any way around this? > > Just asking for some ideas and "best practices" here. > > Thanks! > > -- You received this message because you are subscribed to the Google Groups "beanstalk-talk" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/beanstalk-talk. For more options, visit https://groups.google.com/d/optout.
