Actually I need something similar:
Each consumer will have its own job queue. The leader will use round
robin to load balance. These jobs are never-ending and they run as
forked processes. A job will be signaled to stop by the consumer only
when the leader asks to do so.
I am planning implement this with the following znode hierarchy:
/nodes/nodeX/jobs/
start/
j1
j2
j3
stop/
j2
The consumer at nodeX wathches both 'start/' and 'stop/' znodes for
ZOO_CHILD_EVENT. If a node gets created under 'start' the consumer
forks a new worker. If a node gets created under 'stop', the consumer
signals the worker to stop.
Is this approach OK?
I am worried about if the consumer will always see events in 'start/'
and 'stop/' as they are applied by the leader.
For example, consider that leader executes this sequence:
create /start/j2
delete /stop/j2
In this situation, does the consumer have the gurantee that it always
see these events in that sequence? If the consumer sees the change
first at 'stop/' first and then at 'start/', this strategy won't work.
2011/3/8 Scott Fines <[email protected]>
>
> Just to throw it in, https://github.com/openutility/menagerie also has
> blocking and synchronous queue implementations.
>
> If we had a deque, this would be a great opportunity for a work stealing
> scenario as well. You could give each consumer its own deque, then the
> leader could use round robin to load balance. When a consumer has finished a
> task, it removes that item from the deque. Once a consumer runs out of items
> on its deque, it starts taking items off the back of someone else's. That
> way, when a machine dies or becomes slow, another machine can seamlessly
> take over its work.
>
> I'm working on a general purpose implementation for menagerie now, but a
> deque shouldn't be that hard to make, if you follow the queue recipe
> mentioned above
> On Mar 8, 2011 2:50 AM, "Віталій Тимчишин" <[email protected]> wrote:
--
Sabyasachi
