w.r.t. persistence, just store the damn queue in memory. i have a 32core machine with 256GB of memory in my home office that cost under $10K. for what your doing, this would resolve the persistence thing.
On Nov 13, 2012, at 10:29 PM, Sean Donovan wrote: > Oh boy, where to start . . . I’ve learnt a lot in the last few days. > > I don’t think it is a standard bus pattern. Don’t laugh, but I had been > experimenting with ActiveMQ. Eventually, we’ll be scaling to millions of > tasks daily. We also need to prioritize messages (“real-time” traffic gets > processed over regular analytics). We learnt that ActiveMQ can’t efficiently > handle that many consumers on one Q, also, the prioritization algorithm > seriously degrades past 60,000 messages in the Q, and the C++ client (CMS) > has bugs. > > Given the # of consumers on one Q, prioritization and millions of messages – > I’m not even sure that other vendors (IBM low-latency MQ, Tibco, etc.) would > work. We also have a requirement to be cross platform (Win32, Win64, > Linux64, SPARC32, SPARC64). Our use-case is very specific, which is why I > started looking at ZeroMQ. It might be worth the effort to build something > ourselves. We write software to handle derivatives software and already have > to support a plethora of 3rd party vendor products -- we'd like to avoid > unnecessary licensing. > > Persistence is interesting. We might not need it. If the system encounters > an error, we simply restart our analytics, and the tasks could get > re-spooled. Persistence of the results is already handled in the distributed > cache (GemFire). My problem is how we handle the spooling of millions of > tasks to ZeroMQ. We could put blocking semantics in if memory is filled, or > offload to disk. So, although it’s still persistence, we might not need to > persist each acknowledgement within a ZeroMQ type broker. > > To handle task prioritization, it should be relatively easy to build a fairly > decent (statistics based) algorithm into the ZeroMQ client/feeder. > > Poison messages are also interesting. In our case, the biggest problem is > receiving an unhandled exception during processing. In that case, we log an > error to the distributed cache -- and should restart the worker. It's > messier than it should be, am relying on millions of lines of code, some that > goes back 15+ years. > > Sorry if my description was messy -- and many thanks for the interest, > > Sean > > From: Bennie Kloosteman <[email protected]> > To: Sean Donovan <[email protected]>; ZeroMQ development list > <[email protected]> > Sent: Monday, November 12, 2012 11:48 PM > Subject: Re: [zeromq-dev] Distributed Q with **lots** of consumers > > This is a fairly standard bus pattern , though personally i HATE persistent > queues with a passion , they kill all performance , you have to manage the > infrastructure and the possibility of poison messages - for most business > cases there are better ways to guarantee delivery eg send update events to > the client which can resend or terminate as desired . > > If you run into a limit you can chain them , eg 20 servers with 20 workes > each serve 400 clients , those 20 servers put everything into 1 queue. > > Why bother using a fast system like ZeroMQ with a persistent queue anyway , > the queue will just bottle neck everything writing the message to disk and > create contention on reading it back. Surely MSMQ or the other Service Bus > queues can do this better - since they have optimized persistence . > > > On Tue, Nov 13, 2012 at 6:58 AM, Sean Donovan <[email protected]> wrote: > Any suggestions for implementing the following in ZMQ? > > Imagine a single Q containing millions of entries, which is constantly being > added to. This Q would be fully persistent, probably not managed by ZMQ, and > run in it's own process. > > We would like N workers. Those workers need to start/stop ad-hoc, and > reconnect to the Q host process. Each worker would take a single item from > the Q, process, acknowledge completion, then repeat (to request another > item). Processing time for each task is 3ms+ (occasionally minutes). > > Because of the variance in compute time it is important that the workers > don't pre-fetch/cache tasks. As an optimization, we'll add a heuristic so we > can batch short-running tasks together (but, we'd like the control -- a > load-balancing algorithm wouldn't know how to route efficiently, unless it > could take hints). > > Need a pattern that would allow us to scale to 100s of workers. > > MANY THANKS! > > Sean Donovan > > _______________________________________________ > zeromq-dev mailing list > [email protected] > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > > > > _______________________________________________ > zeromq-dev mailing list > [email protected] > http://lists.zeromq.org/mailman/listinfo/zeromq-dev ----------------------- Andrew Hume 623-551-2845 (VO and best) 973-236-2014 (NJ) [email protected]
_______________________________________________ zeromq-dev mailing list [email protected] http://lists.zeromq.org/mailman/listinfo/zeromq-dev
