Re: [zeromq-dev] Distributed Q with **lots** of consumers

: Re: [zeromq-dev] Distributed Q with **lots** of consumers is it possible to estimate the runtime for an item? and what is the metric you are trying to optimise? is it average latency? or total throughput? or minimal idle time? On Nov 12, 2012, at 3:58 PM, Sean Donovan wrote: Any

Re: [zeromq-dev] Distributed Q with **lots** of consumers

description was messy -- and many thanks for the interest, Sean From: Bennie Kloosteman bkloo...@gmail.com To: Sean Donovan sdonovan...@yahoo.com; ZeroMQ development list zeromq-dev@lists.zeromq.org Sent: Monday, November 12, 2012 11:48 PM Subject: Re: [zeromq-dev] Distributed Q with **lots

Re: [zeromq-dev] Distributed Q with **lots** of consumers

Yup... it's a ZeroMQ list but I'm gonna deviate a bit: Not sure how helpful fhis will be.. but I do a lot of this sort of work using MongoDB and Redis. Have workers nibble away at a queue (in variable batch sizes). Here's an article I wrote about it a while back:

Re: [zeromq-dev] Distributed Q with **lots** of consumers

From: Pieter Hintjens p...@imatix.com To: Sean Donovan sdonovan...@yahoo.com; ZeroMQ development list zeromq-dev@lists.zeromq.org Sent: Monday, November 12, 2012 9:43 PM Subject: Re: [zeromq-dev] Distributed Q with **lots** of consumers Please read the Guide and specifically the Load

Re: [zeromq-dev] Distributed Q with **lots** of consumers

sdonovan...@yahoo.com; ZeroMQ development list zeromq-dev@lists.zeromq.org Sent: Monday, November 12, 2012 10:08 PM Subject: Re: [zeromq-dev] Distributed Q with **lots** of consumers is it possible to estimate the runtime for an item? and what is the metric you are trying to optimise

Re: [zeromq-dev] Distributed Q with **lots** of consumers

] Distributed Q with **lots** of consumers This is a fairly standard bus pattern , though personally i HATE persistent queues with  a passion  , they kill all performance , you have to manage the infrastructure and the possibility of poison messages - for most business cases there are better ways

[zeromq-dev] Distributed Q with **lots** of consumers

Any suggestions for implementing the following in ZMQ? Imagine a single Q containing millions of entries, which is constantly being added to.  This Q would be fully persistent, probably not managed by ZMQ, and run in it's own process. We would like N workers.  Those workers need to start/stop

Re: [zeromq-dev] Distributed Q with **lots** of consumers

Please read the Guide and specifically the Load-balancing and Majordomo patterns which do precisely this: * http://zguide.zeromq.org/page:all#The-Load-balancing-Pattern * http://zguide.zeromq.org/page:all#Service-Oriented-Reliable-Queuing-Majordomo-Pattern Majordomo takes load-balancing a step

Re: [zeromq-dev] Distributed Q with **lots** of consumers

is it possible to estimate the runtime for an item? and what is the metric you are trying to optimise? is it average latency? or total throughput? or minimal idle time? On Nov 12, 2012, at 3:58 PM, Sean Donovan wrote: Any suggestions for implementing the following in ZMQ? Imagine a single Q

Re: [zeromq-dev] Distributed Q with **lots** of consumers

This is a fairly standard bus pattern , though personally i HATE persistent queues with a passion , they kill all performance , you have to manage the infrastructure and the possibility of poison messages - for most business cases there are better ways to guarantee delivery eg send update events