What is so bad about patching the job itself? And how is it cleaner to have multiple tubes and consumers doing a job that one consumer can do on one tube?
This type of thing has come up on this group several times over the years but ultimately if you want bells and whistles, use a different service — beanstalk intentionally doesn’t do anything that you can do yourself (ie adding tags to a job) but what it does do it does really well. The only part that is missing is the atomicity — I agree that’s a problem. In the past when this was *critical* I’ve introduced a memcached (or even persisted memcached like couchbase) into the mix where I can stick a flag “this job succeeded” or “this job failed X steps” for example - then when you pull the job out of the queue you check for the flags before going on.. In this way you can just release the job and try again… I don’t love doing this just because it’s additional complexity, points of failure, and cycles spent going back and forth with the cache, but it’s an example of how this could be done. -- Chad Kouse Sent with Airmail On July 17, 2014 at 5:11:51 PM, Wickman ([email protected]) wrote: Well, I guess the end result is the same. But I think it's a bit easier/cleaner to put the failed job in a separate tube, say, "failed_db" or "failed_mail" depending on what step failed. That way I don't have to "patch" the job (ie no need to add any meta data to the job itself) and I can have dedicated consumer-threads listening on each tube. I know about `bury`, but that won't help me since I don't know what do to with buried jobs (ie did it fail the "db" step or the "mail" step). What is missing, I think, is some way to add arbitrary "tags" or something to jobs without having to patch the job itself. I suppose moving them to another tube is a way to achieve that. Den torsdagen den 17:e juli 2014 kl. 21:24:36 UTC+2 skrev chadkouse: I still don’t see the point of having a tube just for failed (or partially failed) jobs but I don’t know about your use case. Beanstalkd also has a `bury` command to keep failed jobs around but not continue trying to process them - could be useful if you’re wanting to do “later processing” -- Chad Kouse Sent with Airmail On July 17, 2014 at 3:18:20 PM, Wickman ([email protected]) wrote: Yes, that could work, but in that case I might as well put failed jobs in dedicated fail-tubes for later processing. I was looking for an atomic approach though, but I suppose I have to do without. Den torsdagen den 17:e juli 2014 kl. 13:49:26 UTC+2 skrev chadkouse: If one or more steps fail why not instead of releasing the job just go ahead and delete the original job and create a copy of it with some metadata attached as to what steps are left to do. Place the job back into your validated tube. And have your consumer check for this metadata to determine what is left to do. No metadata = all steps. Sent using CloudMagic On Thu, Jul 17, 2014 at 4:41 AM, Wickman <[email protected]> wrote: Hi, thanks for reply! Let me give a concrete example: Each job in the tube is first checked for validness, then put it in a "validated" tube. Each valid job should be sent to a database(*) Each valid job should be sent to another database(*) Each valid job should be stored in a file system(*) When all above is done, delete the job (Validation must be done first, other steps can be done in parallell). So yes, I could do everything in one consumer, but if one step fails (database not available etc) I can't just bury the job, because I don't know which step failed when I retry the job later. I have two ideas which I'd like some feedback on: Idea 1: When a step fails, move the job to a dedicated fail-tube, like "database_fail" or "file_fail". Then have a consumer handling them. This is the non-atomic step which i don't really like. Idea 2: When a step fails, release() the job back to the tube but with a new priority. Say 110 for database_fail and 120 for file_fail. When the consumer reserves a job, it checks the priority to know what to do next with it. Again, thanks for any input! Den tisdagen den 15:e juli 2014 kl. 21:30:39 UTC+2 skrev chadkouse: The short answer to your questions is: no The long answer is still no, but you can probably *force* things to do what you’re saying with some more logic existing outside of the message queue itself. Might be easier / more robust to have 1 tube and 1 job and then your consumer does what’s needed. Many modern languages allow you to run multiple threads in parallel. On July 15, 2014 at 3:12:00 PM, Wickman ([email protected]) wrote: Hi In my case, jobs must pass through several steps before they are done. Some steps can be done in parallell. I figured I solve this by moving jobs between tubes, say I use tubes like "step1" which then moves the job to tubes "step2" and "step3" etc. Is this a sane approach? I see a few issues with this approach: Is there any way of moving a job from one tube to another safely/atomically (put + delete)? Copying the same job to several tubes for parallell processing doubles the amount of RAM needed. Is there any way around this? Just asking for some ideas and "best practices" here. Thanks! -- You received this message because you are subscribed to the Google Groups "beanstalk-talk" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/beanstalk-talk. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "beanstalk-talk" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/beanstalk-talk. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "beanstalk-talk" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/beanstalk-talk. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "beanstalk-talk" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/beanstalk-talk. For more options, visit https://groups.google.com/d/optout.
