I'll send out a longer email once I've finished gathering requirements and sketching through a design, but here are my initial thoughts:
- This actually requires two things from Helix: being able to run tasks in the cluster reliably and being able to schedule tasks in the cluster reliably - For the task half of this work, we probably have most of the code available already as the task framework supports things like target resources, DAG-based dependencies, task states, canceling, and correctness in the face of controller failover. - The scheduling half is the part that requires the most new additions. We basically need to be able to (1) store the schedule, (2) know when to wake up to process an item on the schedule, and (3) do this without needing anything in controller memory Kanak ---------------------------------------- > Date: Tue, 18 Mar 2014 16:09:05 -0700 > Subject: Scheduling tasks in the cluster > From: [email protected] > To: [email protected] > > This requirement has come up often and I think its worth while to spend > some time to come up with an elegant solution. We have offered work around > but it still requires users to write write quite a bit of complex code > > Problem statement: > Schedule a Task(s) in the cluster. The task can be Adhoc (one time) or > Recurring (every X minutes or once between 12 to 3 AM etc - basically a > cron expression). Additional criteria as to where the task should be run, > it can be run on any node in the cluster or any node in that cluster that > hosts a particular resource and in a particular state. If the task fails we > might have to retry the task, it can either retry x times before trying on > another node etc. There might be additional constraints that not more than > X tasks should be run on a particular node or across the entire cluster. > > Helix supports all these features in one way or the other but there is no > first class support of API that encapsulates all the above features. > > Any thoughts on how such an API/DSL should look like ? > > thanks, > Kishore G
