Hi Changgeng, Yes, prevAssignment will be preserved even if controller moves to another node. Agree, we can either have flag in resourceAssignment/marker interface/annotation on the computeAssignment method.
Yeah using current state of each partition should work. One thing to note that is pending state is not guaranteed to reach (the transition might fail). In fact, all the in built rebalancers in Helix work off of current state. Yes, rebalancer is invoked on every change in the cluster. You can check for the event type and return the same assignment for some event types. Just basing on LiveNodes may not work if the participants fails to handle the transition. thanks, Kishore G On Thu, Jun 25, 2015 at 9:55 AM, Changgeng Li <[email protected]> wrote: > I'm also trying to work around this for now by using currentState to > figure out the previous resource assignment, using pending state override > the current state of each partition. Do you think there is any potential > issue with this approach before prevResourceAssignment is persisted? > > I also have another question about when the rebalancer is invoked. My test > shows it's called a lot of times when add just one new node. Looks like > it's also called when the current state is changed. If so, we probably need > implement a shortcut that if livenodes are constraints are not changed, > just return the previous resource assignment. > > > On Thu, Jun 25, 2015 at 9:50 AM, Changgeng Li <[email protected]> > wrote: > >> Actually I was going to ask whether prevAssignment is preserved if the >> controller moves to another node. A marker either for the rebalancer or for >> the returned resource assignment would work. It would be more flexible if >> there's a flag in the returned resource assignment. >> >> >> >> On Wed, Jun 24, 2015 at 11:44 PM, kishore g <[email protected]> wrote: >> >>> You are right. I think this is a bug. We need to preserve the previous >>> assignment and pass it on. Having said that we should not do this for all >>> types of rebalancers. Doing so will introduce additional latency(additional >>> writes to Zookeeper) and will impact fail over time. Most rebalancers also >>> tend to be idempotent, i.e they compute the same output given the same set >>> of inputs. This property in general is not error prone and makes debugging >>> easier. >>> >>> One idea is to have a marker interface Persistable or something and only >>> save it when the implementation implements this interface. The fix is easy >>> but love to hear your feedback. >>> >>> Also, the AutoRebalancer that comes with Helix, already provides the >>> features that you are planning to implement. May be you can clone it and >>> modify it to suite your needs. >>> >>> >>> https://github.com/apache/helix/blob/master/helix-core/src/main/java/org/apache/helix/controller/rebalancer/FullAutoRebalancer.java >>> >>> https://github.com/apache/helix/blob/master/helix-core/src/main/java/org/apache/helix/controller/strategy/AutoRebalanceStrategy.java >>> >>> >>> thanks, >>> Kishore G >>> >>> On Wed, Jun 24, 2015 at 11:13 PM, Changgeng Li <[email protected]> >>> wrote: >>> >>>> yes, it's always null. >>>> >>>> actually I checked the code that parameter was never set. >>>> >>>> https://github.com/apache/helix/blob/acd902e2433f65e9864ccf49fcd1a04c36b1f206/helix-core/src/main/java/org/apache/helix/controller/stages/BestPossibleStateCalcStage.java#L222 >>>> >>>> On Wed, Jun 24, 2015 at 10:41 PM, kishore g <[email protected]> >>>> wrote: >>>> >>>>> I see that you haven't subscribed to mailing list yet. Please do that >>>>> my sending an email to [email protected]. >>>>> >>>>> I am also available on #apachehelix irc if you have additional >>>>> questions. http://helix.apache.org/IRC.html >>>>> >>>>> >>>>> ---------- Forwarded message ---------- >>>>> From: kishore g <[email protected]> >>>>> Date: Wed, Jun 24, 2015 at 10:37 PM >>>>> Subject: Re: Customized rebalancer >>>>> To: "[email protected]" <[email protected]> >>>>> >>>>> >>>>> Hi Changgeng, >>>>> >>>>> I think the first invocation will always be null, subsequent >>>>> invocations should provide you the previous resource assignment. Are you >>>>> saying its always null?. >>>>> >>>>> thanks, >>>>> Kishore G >>>>> >>>>> On Wed, Jun 24, 2015 at 3:42 PM, Changgeng Li <[email protected]> >>>>> wrote: >>>>> >>>>>> Hello, >>>>>> >>>>>> We have a use case that when adding or removing a new node, we hope >>>>>> to minimize the shuffle of partitions between nodes. I'm trying to >>>>>> implement a customized rebalancer calculating the new resource assignment >>>>>> based on the previous resource assignment. When a new node is added, just >>>>>> move some partitions from existing node to the new node, and when a node >>>>>> is >>>>>> down, move the partitions on this node to the other nodes. Partitions >>>>>> would >>>>>> not move between two nodes if the both status are not changed. >>>>>> >>>>>> The documentation says: >>>>>> In rebalance(), ... the third is the output of the previous >>>>>> invocation of this method (if supported) ... >>>>>> >>>>>> During my testing I found the previous resource assignment passed in >>>>>> is always null. my question is what "if supported" means here? Does the >>>>>> rebalancer need to support something? >>>>>> >>>>>> >>>>>> Any other suggestions are also welcome. >>>>>> >>>>>> Thanks, >>>>>> Changgeng >>>>>> >>>>> >>>>> >>>>> >>>> >>> >> >
