Hi list I'm trying to set up a system where certain jobs can be executed through zeromq, but there are currently a few unknowns in how to tackle certain issues. Basically, I have a Redis queue with jobs. I pick one job from the queue and push it to a broker that distributes it to workers that handle the job.
So far so good, but there's a few extra requirements: - one job can have multiple sub-jobs which might or might not need to be executed in a specific order. "item_update 5" could have "cache_update 5" and "clear_proxies 5" as sub-jobs). I'm currently thinking of using the routing slip pattern (http://www.eaipatterns.com/RoutingTable.html) to do this. - some sub-jobs need to wait for other sub-jobs to finish first. - some jobs need to be published across multiple subscribers, other jobs only need to be handled by one worker. - workers should be divided into groups that will only handle specific tasks (majordomo pattern?) - some workers could forward-publish something themselves to a set of subscribers Right now, I have the following setup: (Redis queue) <---- (one or more routers | push) -----> (pull | one or more brokers | push) -----> (pull | multiple workers | push) ----> (pull | sink) The brokers and the sink are the stable part of the architecture. The routers are responsible for getting a job from the queue, deciding the sub-jobs for each job and attaching the routing slip. What I haven't done yet is implementing majordomo to selectively define workers for a certain service, so every worker can handle every task right now. The requirement that some jobs are pub/sub and other are push/pull also isn't fulfilled. I was wondering if this is the right approach and if there are better ways of setting up messaging, keeping into account the requirements? Kind regards, Felix De Vliegher Egeniq.com _______________________________________________ zeromq-dev mailing list [email protected] http://lists.zeromq.org/mailman/listinfo/zeromq-dev
