Here is my two cents. Currently we are mostly running rebalancer inside controller pipeline, so the rebalancer is triggered by Zookeeper change notifications and it gets a free copy of Zookeeper cluster data snapshot. However, rebalancer may also be triggered in other ways like timers, system load changes, or any external signals. In addition, rebalancer may also need to access data from some monitoring systems, or external services like MySQL.
We could probably separate the logic of rebalancer from controller. Rebalancer is all about setting ideal-state; i.e. set the target mappings of partition-->(host, state). The rebalancer logic will be mostly application specific. On the other hand, controller is all about bringing current-state to ideal-state. The controller logic includes using a semi-greedy algorithm (e.g. shortest path) to calculate the next mappings from current mapping (i.e. current-state) to target mapping (i.e. ideal-state), applying constraints, figuring out optimal parallism, etc. The controller logic will be mostly generic to all applications. The only protocol between rebalancer and controller is ideal-state (i.e. the target partiton-->(host, state) mappings). In this sense, every rebalancer is customized, and we can provide some default implementations like auto or semi-auto ones. Rebalancer can be also running anywhere provided that there is only one instance running. This can be achieved through leader election, running with controller, or use custom-code invoker. On Fri, Oct 10, 2014 at 12:25 PM, kishore g <[email protected]> wrote: > Hi, > > Even though we have couple of ways of writing custom rebalancer (one on > participant side and another on the controller side), I dont think its > trivial for some one to write them without understanding all the internal > details of Helix. > > I am starting this thread to see if others have any thoughts on making it > easier for some to get started and write their own rebalancer as part of > the quick start. > > thanks, > Kishore G >
