Actually I was going to ask whether prevAssignment is preserved if the controller moves to another node. A marker either for the rebalancer or for the returned resource assignment would work. It would be more flexible if there's a flag in the returned resource assignment.
On Wed, Jun 24, 2015 at 11:44 PM, kishore g <[email protected]> wrote: > You are right. I think this is a bug. We need to preserve the previous > assignment and pass it on. Having said that we should not do this for all > types of rebalancers. Doing so will introduce additional latency(additional > writes to Zookeeper) and will impact fail over time. Most rebalancers also > tend to be idempotent, i.e they compute the same output given the same set > of inputs. This property in general is not error prone and makes debugging > easier. > > One idea is to have a marker interface Persistable or something and only > save it when the implementation implements this interface. The fix is easy > but love to hear your feedback. > > Also, the AutoRebalancer that comes with Helix, already provides the > features that you are planning to implement. May be you can clone it and > modify it to suite your needs. > > > https://github.com/apache/helix/blob/master/helix-core/src/main/java/org/apache/helix/controller/rebalancer/FullAutoRebalancer.java > > https://github.com/apache/helix/blob/master/helix-core/src/main/java/org/apache/helix/controller/strategy/AutoRebalanceStrategy.java > > > thanks, > Kishore G > > On Wed, Jun 24, 2015 at 11:13 PM, Changgeng Li <[email protected]> > wrote: > >> yes, it's always null. >> >> actually I checked the code that parameter was never set. >> >> https://github.com/apache/helix/blob/acd902e2433f65e9864ccf49fcd1a04c36b1f206/helix-core/src/main/java/org/apache/helix/controller/stages/BestPossibleStateCalcStage.java#L222 >> >> On Wed, Jun 24, 2015 at 10:41 PM, kishore g <[email protected]> wrote: >> >>> I see that you haven't subscribed to mailing list yet. Please do that my >>> sending an email to [email protected]. >>> >>> I am also available on #apachehelix irc if you have additional >>> questions. http://helix.apache.org/IRC.html >>> >>> >>> ---------- Forwarded message ---------- >>> From: kishore g <[email protected]> >>> Date: Wed, Jun 24, 2015 at 10:37 PM >>> Subject: Re: Customized rebalancer >>> To: "[email protected]" <[email protected]> >>> >>> >>> Hi Changgeng, >>> >>> I think the first invocation will always be null, subsequent invocations >>> should provide you the previous resource assignment. Are you saying its >>> always null?. >>> >>> thanks, >>> Kishore G >>> >>> On Wed, Jun 24, 2015 at 3:42 PM, Changgeng Li <[email protected]> >>> wrote: >>> >>>> Hello, >>>> >>>> We have a use case that when adding or removing a new node, we hope to >>>> minimize the shuffle of partitions between nodes. I'm trying to implement a >>>> customized rebalancer calculating the new resource assignment based on the >>>> previous resource assignment. When a new node is added, just move some >>>> partitions from existing node to the new node, and when a node is down, >>>> move the partitions on this node to the other nodes. Partitions would not >>>> move between two nodes if the both status are not changed. >>>> >>>> The documentation says: >>>> In rebalance(), ... the third is the output of the previous invocation >>>> of this method (if supported) ... >>>> >>>> During my testing I found the previous resource assignment passed in is >>>> always null. my question is what "if supported" means here? Does the >>>> rebalancer need to support something? >>>> >>>> >>>> Any other suggestions are also welcome. >>>> >>>> Thanks, >>>> Changgeng >>>> >>> >>> >>> >> >
