On Tue, Jun 11, 2013 at 10:23 AM, BlackKitten <[email protected]> wrote: > Hi all, > I am new to beanstalk and just moving first steps: everything look very > nice up to now :) > I would like to ask you an advice to manage job dependencies: say for > example that my application "tasks" can be split into several workers that > must be run sequentially, for example: > > Task id=1: WorkerA --> WorkerB --> WorkerC --> Done. > Taks id=2: WorkerC --> Done. > Taks id=3: WorkerC --> WorkerA --> Done. > > etc. The idea is that the output of a Worker is the input of the next one, > but the real game is that the order is not fixed. > I guess I can build a big "do-all" worker, but this will not scale as > nicely... > > My first idea is to have the N worker create the N+1 job in a common tube, > passing the application-specific info as payload, so only one tube.
In this case I don't see how this worker is not "a big do-all worker", unless the worker would first check if it should be handling this task at this point and reinserting the job if not, which is highly wasteful in my opinion, cause tasks could be enqueued several times until the appropriate worker happens to take it. > What do you think? Any ideas? > Thanks! If you want to have independent workers, each doing its own job, I would have one tube per worker, and having, as you even said, the output of a worker to be the input of the next one. You could have in the task payload the info of the sequence, to that each worker can know which tube to reinsert the task. So, in your example, Task1 will be queued in tube "A" and task 2 and 3 in tube "C". Worker A will process task 1, check which is the next one in sequence for this task, and enqueue the task in tube "B". Worker B would then process the task and insert it in tube "C", and so on. In the meantime, worker C would have processed task 2, and then task 3, inserting it in tube "A". For the sequence you could have the list of tubes in an array, and the worker will just remove the first entry and reinsert the task in the now first one. Also here you could have very nice things, such as detecting errors in the processing order, or checking an external source for changes in the sequence dynamically, for example. Hope this helps, Jesus. -- You received this message because you are subscribed to the Google Groups "beanstalk-talk" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/beanstalk-talk?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
