> UC1: Synchronized Jobs
> 1. A job is fired across all nodes
> 2. The nodes wait until the barrier is entered by all participants
> 3. The nodes process the data and leave
> 4. On all nodes leaving the barrier, the Leader node marks the job as
> UC2: Multiple Jobs per Node
> 1. A Job is scheduled for a future time on a specific node (usually the
> same node that's creating the trigger)
> 2. A Trigger can be overwritten and cancelled without the job firing
> 3. In the event of a node failure, the Leader will take all pending jobs
> from the failed node, and partition them across the remaining nodes.
we've implemented UC2 for an internal project with ZK. I'd love to make the
code free, but I've to ask our product owner. It's a small company so this
could go quickly. But I don't know how to convince them. They're so afraid of
giving away stuff.
The basic idea is, that we've two "folders" in ZK, a work queue and a lock
folder. The items (znodes) in the work queue a timestamp prefixed. Every node
consuming the queue tries to create an ephemeral znode in the lock "folder"
before starting on a work item. Work items are actually URLs and we lock on
the domain. Since we also use a lock pool on every worker that only releases
on overflow or timeout, we can reuse locks and also get "weak" locality for
URLs of the same domain. - That's all the magic. Six java classes on top of
our own ZK helper lib.
Thomas Koch, http://www.koch.ro