First post, maybe dumb, apologies in advance.
I am trying to design a pipelined load distribution system, where I have one process (let's call it the distributor) receiving many requests to execute the same task for different parameter combinations; this one process will pass on those tasks to one of N processes (called the workers), which will take care of executing the particular task. Optionally, the worker processes will notify a final process (could be the same as the initial distributor) that this particular task is finished, and wait for more work to do. I would like to hear opinions on several design issues: 1. What would be the practical differences between using a PubSub approach and using Multicast to pass the requests from distributor to workers? 2. By going with PubSub or Multicast, all the workers will receive all task requests and will have to decide whether they are the worker which should process it. What are practical ways of making this decision? It looks like this approach requires the workers to know in advance the total number of workers in the pool, right? 3. How to handle crashed workers? How about workers that are not responding? What if I want to add workers? 4. Maybe I should have the distributor handle the load distribution, not using PubSub or Multicast, but choosing a specific worker and sending the task request directly to it. Same questions apply, right? Any 0mq-specific documentation or examples that might help me answer these questions? Thanks in advance.
_______________________________________________ zeromq-dev mailing list [email protected] http://lists.zeromq.org/mailman/listinfo/zeromq-dev
