And.. of course.. because I love to watch fire shoot from peoples eyes.. there's always IMAP :)
On Wed, Nov 14, 2012 at 9:18 AM, Shane Spencer <[email protected]> wrote: > Yup... it's a ZeroMQ list but I'm gonna deviate a bit: > > Not sure how helpful fhis will be.. but I do a lot of this sort of > work using MongoDB and Redis. Have workers nibble away at a queue (in > variable batch sizes). > > Here's an article I wrote about it a while back: > > > https://docs.google.com/document/pub?id=1CTSka9dlZLi8JYu3IY8-iuKpddLwQe9FI55JwJQUjO4 > > And of course check out Blocking POP/PUSH operations with Redis. You > can put expirable data in a giant queue and let it get nibbled away > and modified. > > - Shane > > > > On Mon, Nov 12, 2012 at 1:58 PM, Sean Donovan <[email protected]> wrote: >> Any suggestions for implementing the following in ZMQ? >> >> Imagine a single Q containing millions of entries, which is constantly being >> added to. This Q would be fully persistent, probably not managed by ZMQ, >> and run in it's own process. >> >> We would like N workers. Those workers need to start/stop ad-hoc, and >> reconnect to the Q host process. Each worker would take a single item from >> the Q, process, acknowledge completion, then repeat (to request another >> item). Processing time for each task is 3ms+ (occasionally minutes). >> >> Because of the variance in compute time it is important that the workers >> don't pre-fetch/cache tasks. As an optimization, we'll add a heuristic so >> we can batch short-running tasks together (but, we'd like the control -- a >> load-balancing algorithm wouldn't know how to route efficiently, unless it >> could take hints). >> >> Need a pattern that would allow us to scale to 100s of workers. >> >> MANY THANKS! >> >> Sean Donovan >> >> _______________________________________________ >> zeromq-dev mailing list >> [email protected] >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >> _______________________________________________ zeromq-dev mailing list [email protected] http://lists.zeromq.org/mailman/listinfo/zeromq-dev
