Hi Vikram, we built s.th. for running/scheduling jobs in replicated (HA) worker nodes, and there we're using Cassandra (lightweight transactions) for job locking: https://github.com/Galeria-Kaufhof/ha-jobs.
I don't know the spark job scheduler, but if it's possible to embed this and control/start/monitor it via some java/scala api, you could build a simple wrapper app that's using ha-jobs to decide on which node the spark job scheduler should be run. Cheers, Martin Am 18.08.2015 20:54 schrieb "Vikram Kone" <[email protected]>: > Hi, > I'm a newbie to Zookeeper, so pardon any naive question I ask here. > I have a cassandra cluster running on linux VMs and have a spark job > scheduler service running on one of the nodes. Since cassandra has a > peer-peer architecture there is no concept of leader. > I want to provide high availability for this job scheduler service using > Zookeeper. I can't make any code changes to the job scheduler service since > it's a 3rd party app. > I'm thinking of copying the application folder on all the servers in the > cluster and use zookeeper to start an instance of the service on the > leader/master node by executing /opt/job-scheduler/bin/start.sh on leader > election. > Is this something easy to do with zookeeper? > > Please point to any documentation or tutorial on how to run a bash script > on the leader node in zoo keeper's ensemble after a node is elected as > leader by the quorum. > > Thanks >
