Any suggestions for implementing the following in ZMQ?
Imagine a single Q containing millions of entries, which is constantly being
added to. This Q would be fully persistent, probably not managed by ZMQ, and
run in it's own process.
We would like N workers. Those workers need to start/stop ad-hoc, and
reconnect to the Q host process. Each worker would take a single item from the
Q, process, acknowledge completion, then repeat (to request another item).
Processing time for each task is 3ms+ (occasionally minutes).
Because of the variance in compute time it is important that the workers don't
pre-fetch/cache tasks. As an optimization, we'll add a heuristic so we can
batch short-running tasks together (but, we'd like the control -- a
load-balancing algorithm wouldn't know how to route efficiently, unless it
could take hints).
Need a pattern that would allow us to scale to 100s of workers.
MANY THANKS!
Sean Donovan
_______________________________________________
zeromq-dev mailing list
[email protected]
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
