then you already have the answer. measuring the roundtrip time will give you the batching constant for quick jobs.
minimising idle time WILL add to latency, but on average, not much. in fact, the best way to minimise idle time (which is the same as maximising throughput), is for each server to have its own queue, use zmq to req/rep jobs from that queue. then you have a second layer which manages topping off those queues from teh central queue. this layer can actually do scheduling stuff (which zmq can't do) and come fairly close to optimal solutions to minimising per-server queue size while (nearly) always maintaing a nonzero per-server queue. we've done this sort of thing before and gotten good results; the one i remember was something like a job set that took 16 hours elapsed time running on 12 servers. jobs varied between 3s and 10mins cpu time. we were able to keep all 12 servers maxed out until 2-3 mins before the run end. your situation is different, but i bet the same techniques would work as well. On Nov 13, 2012, at 10:09 PM, Sean Donovan wrote: > It is possible to estimate, accurately in fact. We can record execution > times from previous runs. However, for many million of tasks that is a lot > of overhead. However, I think there is a simpler approach. Instead, I think > I can batch messages as "very fast execution" (where tasks take 3ms, the > minimum) and "everything else". So, the fast tasks I'd batch (e.g. of 100) > -- then send everything else individually. Determining which messages/tasks > are very fast is relatively trivial -- determining everything else is > laborious. > > Trying (a) to optimize minimum idle time by (b) ensuring that the messaging > adds as little overhead as possible and (c) taking into account there could > be hundreds of cores (either Windows|Linux boxes with 64+cores) or the newer > generation SPARC T-series CPUs. The other factor is the task execution time. > Each task is actually a pricing of a derivative trade. Depending on the > trade type/configuration, there are many complex pricing-models involved. > Optimizing a given pricing-model is a project (well, challenge) in it's own > right. > > Many thanks, > > Sean > > From: Andrew Hume <[email protected]> > To: Sean Donovan <[email protected]>; ZeroMQ development list > <[email protected]> > Sent: Monday, November 12, 2012 10:08 PM > Subject: Re: [zeromq-dev] Distributed Q with **lots** of consumers > > is it possible to estimate the runtime for an item? > > and what is the metric you are trying to optimise? > is it average latency? or total throughput? or minimal idle time? > > On Nov 12, 2012, at 3:58 PM, Sean Donovan wrote: > >> Any suggestions for implementing the following in ZMQ? >> >> Imagine a single Q containing millions of entries, which is constantly being >> added to. This Q would be fully persistent, probably not managed by ZMQ, >> and run in it's own process. >> >> We would like N workers. Those workers need to start/stop ad-hoc, and >> reconnect to the Q host process. Each worker would take a single item from >> the Q, process, acknowledge completion, then repeat (to request another >> item). Processing time for each task is 3ms+ (occasionally minutes). >> >> Because of the variance in compute time it is important that the workers >> don't pre-fetch/cache tasks. As an optimization, we'll add a heuristic so >> we can batch short-running tasks together (but, we'd like the control -- a >> load-balancing algorithm wouldn't know how to route efficiently, unless it >> could take hints). >> >> Need a pattern that would allow us to scale to 100s of workers. >> >> MANY THANKS! >> >> Sean Donovan >> _______________________________________________ >> zeromq-dev mailing list >> [email protected] >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > > ----------------------- > Andrew Hume > 623-551-2845 (VO and best) > 973-236-2014 (NJ) > [email protected] > > > > > ----------------------- Andrew Hume 623-551-2845 (VO and best) 973-236-2014 (NJ) [email protected]
_______________________________________________ zeromq-dev mailing list [email protected] http://lists.zeromq.org/mailman/listinfo/zeromq-dev
