Oh boy, where to start . . . I’ve learnt
a lot in the last few days.
I don’t think it is a standard bus
pattern. Don’t laugh, but I had been
experimenting with ActiveMQ. Eventually,
we’ll be scaling to millions of tasks daily. We also need to prioritize
messages (“real-time” traffic gets processed
over regular analytics). We learnt that
ActiveMQ can’t efficiently handle that many consumers on one Q, also, the
prioritization
algorithm seriously degrades past 60,000 messages in the Q, and the C++ client
(CMS) has bugs.
Given the # of consumers on one Q,
prioritization and millions of messages – I’m not even sure that other vendors
(IBM low-latency MQ, Tibco, etc.) would work. We also have a requirement to be
cross platform (Win32, Win64, Linux64,
SPARC32, SPARC64). Our use-case is very specific, which is
why I started looking at ZeroMQ. It
might be worth the effort to build something ourselves. We write software to
handle derivatives software and already have to support a plethora of 3rd party
vendor products -- we'd like to avoid unnecessary licensing.
Persistence is interesting. We might not need it. If the system encounters an
error, we simply
restart our analytics, and the tasks could get re-spooled. Persistence of the
results is already handled
in the distributed cache (GemFire). My
problem is how we handle the spooling of millions of tasks to ZeroMQ. We could
put blocking semantics in if memory
is filled, or offload to disk. So,
although it’s still persistence, we might not need to persist each
acknowledgement within a ZeroMQ type broker.
To handle task prioritization, it should
be relatively easy to build a fairly decent (statistics based) algorithm into
the ZeroMQ client/feeder.
Poison messages are also interesting. In our case, the biggest problem is
receiving an unhandled exception during processing. In that case, we log an
error to the distributed cache -- and should restart the worker. It's messier
than it should be, am relying on millions of lines of code, some that goes back
15+ years.
Sorry if my description was messy -- and many thanks for the interest,
Sean
________________________________
From: Bennie Kloosteman <[email protected]>
To: Sean Donovan <[email protected]>; ZeroMQ development list
<[email protected]>
Sent: Monday, November 12, 2012 11:48 PM
Subject: Re: [zeromq-dev] Distributed Q with **lots** of consumers
This is a fairly standard bus pattern , though personally i
HATE persistent queues with a passion , they kill all performance , you have
to manage the infrastructure and the possibility of poison messages - for most
business cases there are better ways to guarantee delivery eg send update
events to the client which can resend or terminate as desired .
If you run into a limit you can chain them , eg 20 servers with 20 workes each
serve 400 clients , those 20 servers put everything into 1 queue.
Why bother using a fast system like ZeroMQ with a persistent queue anyway , the
queue will just bottle neck everything writing the message to disk and create
contention on reading it back. Surely MSMQ or the other Service Bus queues can
do this better - since they have optimized persistence .
On Tue, Nov 13, 2012 at 6:58 AM, Sean Donovan <[email protected]> wrote:
Any suggestions for implementing the following in ZMQ?
>
>
>Imagine a single Q containing millions of entries, which is constantly being
>added to. This Q would be fully persistent, probably not managed by ZMQ, and
>run in it's own process.
>
>
>We would like N workers. Those workers need to start/stop ad-hoc, and
>reconnect to the Q host process. Each worker would take a single item from
>the Q, process, acknowledge completion, then repeat (to request another item).
> Processing time for each task is 3ms+ (occasionally minutes).
>
>
>Because of the variance in compute time it is important that the workers don't
>pre-fetch/cache tasks. As an optimization, we'll add a heuristic so we can
>batch short-running tasks together (but, we'd like the control -- a
>load-balancing algorithm wouldn't know how to route efficiently, unless it
>could take hints).
>
>
>Need a pattern that would allow us to scale to 100s of workers.
>
>
>MANY THANKS!
>Sean Donovan
>_______________________________________________
>zeromq-dev mailing list
>[email protected]
>http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
_______________________________________________
zeromq-dev mailing list
[email protected]
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
