Re: queue with limit to number of simultaneous tasks

2009-07-27 Thread Alexander Sibiryakov
Thank you for this thoughts. It's help to look at my problem at different view. It seems, that I can use slightly modified lock recipe. Ted Dunning wrote: As you look at this, I would be grateful if you can evaluate alternative implementations in which a) each task is a separate file or b) a

Re: queue with limit to number of simultaneous tasks

2009-07-14 Thread Ted Dunning
As you look at this, I would be grateful if you can evaluate alternative implementations in which a) each task is a separate file or b) all tasks are listed and described in a single file that is updated atomically using standard ZK read-modify-write-repeat-on-failure style or c) all tasks are

Re: queue with limit to number of simultaneous tasks

2009-07-14 Thread Patrick Hunt
It's hard to say, there are a number of variables. Some things to think about: Are the tasks idempotent? do they have leases (like SQS)? Is one process responsible for processing the tasks or will you have many vying for the jobs? Are the tasks ordered by creation date, or weighted by some fact

queue with limit to number of simultaneous tasks

2009-07-14 Thread Alexander Sibiryakov
Hello everybody! Please give me advice about designing application on zookeeper. I'd like to implement queue with limit on number of simultaneous tasks. For example I have 10 tasks, and I can process only 2 tasks simultaneously. When one task is finished processing, system should start another,