> On Mon, Feb 17, 2014 at 8:08 AM, aman mangal <[email protected]> wrote: > > > I am trying to solve the following problem and I can't think of any model > > which I can use: I have n workers and 1 distributor. Distributor has jobs > > for n workers and every worker can solve only a specific type of job. It has > > to distribute jobs to the workers such that no job is dropped.
It seems to me that if each worker handles a different type of job, then you do not have one job queue, you have N job queues. The question then becomes, who decides which worker handles which type of job? If only the worker knows, then you need one of the "reliable pub/sub" patterns from the guide. None of these are as simple as just a single PUB connected to multiple SUBs, which is inherently unreliable. (Because fan-out can't be implemented in a way that's reliable, fast/scalable, and suitable for a wide range of fan-out problems, so the default pub/sub is fast/scalable and adaptable, but not reliable.) If the distributor knows how to map job types to workers (probably a better idea), then have each worker create its own queue, using push/pull if you can accept occasional undetected failures in the event of disconnects, or one of the reliable req/rep patterns if required, or if results need to be returned from workers. The distributor then connects to those queues (each queue using a *different socket* on the distributor side, not a shared PUSH, which has the round-robin problem you discovered.) Alternately, the distributor can bind a separate socket for each type of queue and workers can connect to that. (The latter allows you to easily grow to support multiple workers per queue without modification to the distributor.) If a static configuration of what workers are where, and what types of jobs are handled, is not acceptable, you can have a separate socket (out of band with respect to the main job queues) on the distributor that workers connect to in order to advertise their services and setup the queues dynamically. Randall _______________________________________________ zeromq-dev mailing list [email protected] http://lists.zeromq.org/mailman/listinfo/zeromq-dev
