Let's say I have 100 long-running tasks and 20 nodes.
I want each of them to take up to 10 tasks. Each of the task should be
taken by one and only one node.
Will the following solution solve the problem?
Create a directory /mytasks in zookeeper.
Normally there will be 100 EPHEMERAL children in
This should roughly work. The one thing that I have seen that would not
work well with this would be processes that run anomalously long.
As such, I would include an expected time of completion as well as process
id in the task ephemeral file. Then you can run a period cleanup process to
look
