We're using Zookeeper for Leader Election and system monitoring. We're
also using it for synchronizing our cluster wide jobs with barriers. We're
running into an issue where we now have a single job, but each node can fire
the job independently of others with different criteria in the job. In the
event of a system failure, another node in our application cluster will need
to fire this Job. I've used quartz previously (we're running Java 6), but
it simply isn't designed for the use case we have. I found this article on
I've looked at both plugins, but they require hadoop. We're not currently
running hadoop, we only have Cassandra. Here are the 2 basic use cases we
need to support.
UC1: Synchronized Jobs
1. A job is fired across all nodes
2. The nodes wait until the barrier is entered by all participants
3. The nodes process the data and leave
4. On all nodes leaving the barrier, the Leader node marks the job as
UC2: Multiple Jobs per Node
1. A Job is scheduled for a future time on a specific node (usually the same
node that's creating the trigger)
2. A Trigger can be overwritten and cancelled without the job firing
3. In the event of a node failure, the Leader will take all pending jobs
from the failed node, and partition them across the remaining nodes.
Any input would be greatly appreciated.