Please read the Guide and specifically the Load-balancing and Majordomo patterns which do precisely this:
* http://zguide.zeromq.org/page:all#The-Load-balancing-Pattern * http://zguide.zeromq.org/page:all#Service-Oriented-Reliable-Queuing-Majordomo-Pattern Majordomo takes load-balancing a step further by doing service-based lookup of the worker to route to. You can find an implementation at https://github.com/zeromq/majordomo although the Guide examples are also a good start if you want to create your own broker. -Pieter On Tue, Nov 13, 2012 at 7:58 AM, Sean Donovan <[email protected]> wrote: > Any suggestions for implementing the following in ZMQ? > > Imagine a single Q containing millions of entries, which is constantly being > added to. This Q would be fully persistent, probably not managed by ZMQ, > and run in it's own process. > > We would like N workers. Those workers need to start/stop ad-hoc, and > reconnect to the Q host process. Each worker would take a single item from > the Q, process, acknowledge completion, then repeat (to request another > item). Processing time for each task is 3ms+ (occasionally minutes). > > Because of the variance in compute time it is important that the workers > don't pre-fetch/cache tasks. As an optimization, we'll add a heuristic so > we can batch short-running tasks together (but, we'd like the control -- a > load-balancing algorithm wouldn't know how to route efficiently, unless it > could take hints). > > Need a pattern that would allow us to scale to 100s of workers. > > MANY THANKS! > > Sean Donovan > > _______________________________________________ > zeromq-dev mailing list > [email protected] > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > _______________________________________________ zeromq-dev mailing list [email protected] http://lists.zeromq.org/mailman/listinfo/zeromq-dev
