Thank you for your input, I'll take that into consideration. I'll probably try both approaches and see which works best in this case.
Den fredagen den 18:e juli 2014 kl. 00:15:21 UTC+2 skrev chadkouse: > > What is so bad about patching the job itself? And how is it cleaner to > have multiple tubes and consumers doing a job that one consumer can do on > one tube? > > This type of thing has come up on this group several times over the years > but ultimately if you want bells and whistles, use a different service — > beanstalk intentionally doesn’t do anything that you can do yourself (ie > adding tags to a job) but what it does do it does really well. > > The only part that is missing is the atomicity — I agree that’s a problem. > In the past when this was *critical* I’ve introduced a memcached (or even > persisted memcached like couchbase) into the mix where I can stick a flag > “this job succeeded” or “this job failed X steps” for example - then when > you pull the job out of the queue you check for the flags before going on.. > In this way you can just release the job and try again… I don’t love doing > this just because it’s additional complexity, points of failure, and cycles > spent going back and forth with the cache, but it’s an example of how this > could be done. > > > > -- > Chad Kouse > Sent with Airmail > > On July 17, 2014 at 5:11:51 PM, Wickman ([email protected] > <javascript:>) wrote: > > Well, I guess the end result is the same. But I think it's a bit > easier/cleaner to put the failed job in a separate tube, say, "failed_db" > or "failed_mail" depending on what step failed. That way I don't have to > "patch" the job (ie no need to add any meta data to the job itself) and I > can have dedicated consumer-threads listening on each tube. > > I know about `bury`, but that won't help me since I don't know what do to > with buried jobs (ie did it fail the "db" step or the "mail" step). > > What is missing, I think, is some way to add arbitrary "tags" or something > to jobs without having to patch the job itself. I suppose moving them to > another tube is a way to achieve that. > > Den torsdagen den 17:e juli 2014 kl. 21:24:36 UTC+2 skrev chadkouse: >> >> I still don’t see the point of having a tube just for failed (or >> partially failed) jobs but I don’t know about your use case. >> >> Beanstalkd also has a `bury` command to keep failed jobs around but not >> continue trying to process them - could be useful if you’re wanting to do >> “later processing” >> >> -- >> Chad Kouse >> Sent with Airmail >> >> On July 17, 2014 at 3:18:20 PM, Wickman ([email protected]) wrote: >> >> Yes, that could work, but in that case I might as well put failed jobs >> in dedicated fail-tubes for later processing. >> >> I was looking for an atomic approach though, but I suppose I have to do >> without. >> >> Den torsdagen den 17:e juli 2014 kl. 13:49:26 UTC+2 skrev chadkouse: >>> >>> If one or more steps fail why not instead of releasing the job just go >>> ahead and delete the original job and create a copy of it with some >>> metadata attached as to what steps are left to do. Place the job back into >>> your validated tube. And have your consumer check for this metadata to >>> determine what is left to do. No metadata = all steps. >>> >>> Sent using CloudMagic >>> <https://cloudmagic.com/k/d/mailapp?ct=pi&cv=5.0.82&pv=8.0> >>> >>> On Thu, Jul 17, 2014 at 4:41 AM, Wickman <[email protected]> wrote: >>> >>> Hi, thanks for reply! >>> >>> Let me give a concrete example: >>> >>> - Each job in the tube is first checked for validness, then put it >>> in a "validated" tube. >>> - Each valid job should be sent to a database(*) >>> - Each valid job should be sent to another database(*) >>> - Each valid job should be stored in a file system(*) >>> - When all above is done, delete the job >>> >>> (Validation must be done first, other steps can be done in parallell). >>> >>> So yes, I could do everything in one consumer, but if one step fails >>> (database not available etc) I can't just bury the job, because I don't >>> know which step failed when I retry the job later. >>> >>> I have two ideas which I'd like some feedback on: >>> >>> Idea 1: When a step fails, move the job to a dedicated fail-tube, like >>> "database_fail" or "file_fail". Then have a consumer handling them. This is >>> the non-atomic step which i don't really like. >>> >>> Idea 2: When a step fails, release() the job back to the tube but with a >>> new priority. Say 110 for database_fail and 120 for file_fail. When the >>> consumer reserves a job, it checks the priority to know what to do next >>> with it. >>> >>> Again, thanks for any input! >>> >>> >>> Den tisdagen den 15:e juli 2014 kl. 21:30:39 UTC+2 skrev chadkouse: >>>> >>>> The short answer to your questions is: no >>>> >>>> The long answer is still no, but you can probably *force* things to do >>>> what you’re saying with some more logic existing outside of the message >>>> queue itself. >>>> >>>> Might be easier / more robust to have 1 tube and 1 job and then your >>>> consumer does what’s needed. Many modern languages allow you to run >>>> multiple threads in parallel. >>>> >>> >>>> On July 15, 2014 at 3:12:00 PM, Wickman ([email protected]) wrote: >>>> >>>> Hi >>>> >>>> In my case, jobs must pass through several steps before they are done. >>>> Some >>>> steps can be done in parallell. I figured I solve this by moving jobs >>>> between tubes, say I use tubes like "step1" which then moves the job to >>>> tubes "step2" and "step3" etc. >>>> >>>> Is this a sane approach? >>>> >>>> I see a few issues with this approach: >>>> >>>> 1. Is there any way of moving a job from one tube to another >>>> safely/atomically (put + delete)? >>>> 2. Copying the same job to several tubes for parallell processing >>>> doubles the amount of RAM needed. Is there any way around this? >>>> >>>> Just asking for some ideas and "best practices" here. >>>> >>>> Thanks! >>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "beanstalk-talk" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To post to this group, send email to [email protected]. >>> Visit this group at http://groups.google.com/group/beanstalk-talk. >>> For more options, visit https://groups.google.com/d/optout. >>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "beanstalk-talk" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at http://groups.google.com/group/beanstalk-talk. >> For more options, visit https://groups.google.com/d/optout. >> >> -- > You received this message because you are subscribed to the Google Groups > "beanstalk-talk" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] <javascript:>. > To post to this group, send email to [email protected] > <javascript:>. > Visit this group at http://groups.google.com/group/beanstalk-talk. > For more options, visit https://groups.google.com/d/optout. > > -- You received this message because you are subscribed to the Google Groups "beanstalk-talk" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/beanstalk-talk. For more options, visit https://groups.google.com/d/optout.
