before i answer, how are you going to implement patterns such as aggregator
from teh EIA book?
i think that means knowing how you identify tasks/jobs and if the tracking and
organising of all
that is going to be centralised or distributed.
andrew
On Jun 29, 2012, at 3:08 AM, Felix De Vliegher wrote:
> Hi list
>
> I'm trying to set up a system where certain jobs can be executed through
> zeromq, but there are currently a few unknowns in how to tackle certain
> issues. Basically, I have a Redis queue with jobs. I pick one job from the
> queue and push it to a broker that distributes it to workers that handle the
> job.
>
> So far so good, but there's a few extra requirements:
> - one job can have multiple sub-jobs which might or might not need to be
> executed in a specific order. "item_update 5" could have "cache_update 5" and
> "clear_proxies 5" as sub-jobs). I'm currently thinking of using the routing
> slip pattern (http://www.eaipatterns.com/RoutingTable.html) to do this.
> - some sub-jobs need to wait for other sub-jobs to finish first.
> - some jobs need to be published across multiple subscribers, other jobs only
> need to be handled by one worker.
> - workers should be divided into groups that will only handle specific tasks
> (majordomo pattern?)
> - some workers could forward-publish something themselves to a set of
> subscribers
>
> Right now, I have the following setup:
> (Redis queue) <---- (one or more routers | push) -----> (pull | one or more
> brokers | push) -----> (pull | multiple workers | push) ----> (pull | sink)
>
>
> The brokers and the sink are the stable part of the architecture. The routers
> are responsible for getting a job from the queue, deciding the sub-jobs for
> each job and attaching the routing slip. What I haven't done yet is
> implementing majordomo to selectively define workers for a certain service,
> so every worker can handle every task right now. The requirement that some
> jobs are pub/sub and other are push/pull also isn't fulfilled.
>
> I was wondering if this is the right approach and if there are better ways of
> setting up messaging, keeping into account the requirements?
>
>
> Kind regards,
>
> Felix De Vliegher
> Egeniq.com
>
>
> _______________________________________________
> zeromq-dev mailing list
> [email protected]
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
------------------
Andrew Hume (best -> Telework) +1 623-551-2845
[email protected] (Work) +1 973-236-2014
AT&T Labs - Research; member of USENIX and LOPSA
_______________________________________________
zeromq-dev mailing list
[email protected]
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
