Todd Nine: > [...] > UC1: Synchronized Jobs > 1. A job is fired across all nodes > 2. The nodes wait until the barrier is entered by all participants > 3. The nodes process the data and leave > 4. On all nodes leaving the barrier, the Leader node marks the job as > complete. > > UC2: Multiple Jobs per Node > 1. A Job is scheduled for a future time on a specific node (usually the > same node that's creating the trigger) > 2. A Trigger can be overwritten and cancelled without the job firing > 3. In the event of a node failure, the Leader will take all pending jobs > from the failed node, and partition them across the remaining nodes.
Hi Todd, we've implemented UC2 for an internal project with ZK. I'd love to make the code free, but I've to ask our product owner. It's a small company so this could go quickly. But I don't know how to convince them. They're so afraid of giving away stuff. The basic idea is, that we've two "folders" in ZK, a work queue and a lock folder. The items (znodes) in the work queue a timestamp prefixed. Every node consuming the queue tries to create an ephemeral znode in the lock "folder" before starting on a work item. Work items are actually URLs and we lock on the domain. Since we also use a lock pool on every worker that only releases on overflow or timeout, we can reuse locks and also get "weak" locality for URLs of the same domain. - That's all the magic. Six java classes on top of our own ZK helper lib. Best regards, Thomas Koch, http://www.koch.ro
