Thanks for the details Leo. I think there is a misunderstanding between customized IdealState and user defined rebalancer. Actually, they are different levels of concept.
For customized IdealState, it still uses Helix defined customized rebalancer, but it takes the map field as the final result. Helix only reacts to the map field change for partition/replica assignment. But user defined rebalancer is different. User has to implement the Rebalancer class and include that class in the same JVM when you start the controller. So Helix will try to use the class loader to load your class to do the computation. So the error could be you dont have the class in the same jar of starting the controller. Either you implement Rebalancer class and include it in controller jar. Or you can switch to use default auto rebalancer but with customized mode for the IdealState. Jiajun provided a test class as a good example of implementing the user defined rebalance. You can take a look if you would like to go that route. best, Junkai On Sat, Oct 31, 2020 at 7:19 PM Leo, Li <[email protected]> wrote: > Hi JunKai and Jiajun, > Thank you for your help. When I tried to use a customized Rebalancer, I > saw the rebalancer is triggered and the idea state is set by it. > > 3784 18:38:38.268 [GenericHelixController-event_process] INFO > c.u.k.w.c.CustomizedRebalancer - CustomizedRebalancer.computeNewIdealState > invoked! > > 3785 18:38:38.268 [GenericHelixController-event_process] INFO > c.u.k.w.c.CustomizedRebalancer - set ############### dca1d: > localhost_12001 dca1d_0 online > > 3786 18:38:38.268 [GenericHelixController-event_process] INFO > c.u.k.w.c.CustomizedRebalancer - set ############### dca1d: > localhost_12002 dca1d_1 online > > 3787 18:38:38.269 [GenericHelixController-event_process] WARN > o.a.h.c.s.BestPossibleStateCalcStage - Event b4896f24_DEFAULT : Rebalancer > does not have a mapping calculator, defaulting to SEMI_AUT O, > resource: dca1d > > > However the Helix UI shows "no date to display". The snapshot is attached. > The EXTERNALVIEW and IDEASTATE are: > > > >>> get /Watchdog/EXTERNALVIEW/dca1d > > { > > "id":"dca1d" > > ,"simpleFields":{ > > "BUCKET_SIZE":"0" > > ,"IDEAL_STATE_MODE":"AUTO" > > ,"NUM_PARTITIONS":"2" > > ,"REBALANCER_CLASS_NAME":"mypackage.controller.CustomizedRebalancer" > > ,"REBALANCE_MODE":"USER_DEFINED" > > ,"REBALANCE_STRATEGY":"DEFAULT" > > ,"REPLICAS":"1" > > ,"STATE_MODEL_DEF_REF":"MyStateModel" > > ,"STATE_MODEL_FACTORY_NAME":"DEFAULT" > > } > > ,"listFields":{ > > } > > ,"mapFields":{ > > "dca1d_0":{ > > "localhost_12010":"ONLINE" > > } > > ,"dca1d_1":{ > > "localhost_12000":"ONLINE" > > } > > } > > } > > > >>> get /Watchdog/IDEALSTATES/dca1d > > { > > "id" : "dca1d", > > "simpleFields" : { > > "IDEAL_STATE_MODE" : "AUTO", > > "NUM_PARTITIONS" : "2", > > "REBALANCER_CLASS_NAME" : "mypackage.controller.CustomizedRebalancer", > > "REBALANCE_MODE" : "USER_DEFINED", > > "REBALANCE_STRATEGY" : "DEFAULT", > > "REPLICAS" : "1", > > "STATE_MODEL_DEF_REF" : "MyStateModel", > > "STATE_MODEL_FACTORY_NAME" : "DEFAULT" > > }, > > "mapFields" : { > > "dca1d_0" : { > > }, > > "dca1d_1" : { > > } > > }, > > "listFields" : { > > "dca1d_0" : [ ], > > "dca1d_1" : [ ] > > } > > } > > > Comparing with before, when i use CUSTOMIZED rebalancer and model, The > EXTERNALVIEW and IDEASTATE were working well with Helix UI: > > > >>> get /Watchdog/IDEALSTATES/dca1d > > { > > "id":"dca1d" > > ,"simpleFields":{ > > "IDEAL_STATE_MODE":"CUSTOMIZED" > > ,"MAX_PARTITIONS_PER_INSTANCE":"1" > > ,"NUM_PARTITIONS":"2" > > ,"REBALANCE_MODE":"CUSTOMIZED" > > ,"REPLICAS":"1" > > ,"STATE_MODEL_DEF_REF":"MyStateModel" > > ,"STATE_MODEL_FACTORY_NAME":"DEFAULT" > > } > > ,"listFields":{ > > } > > ,"mapFields":{ > > "0":{ > > "localhost_12004":"ONLINE" > > } > > ,"1":{ > > "localhost_12005":"ONLINE" > > } > > } > > } > > > >>> get /Watchdog/EXTERNALVIEW/dca1d > > { > > "id":"dca1d" > > ,"simpleFields":{ > > "BUCKET_SIZE":"0" > > ,"IDEAL_STATE_MODE":"CUSTOMIZED" > > ,"MAX_PARTITIONS_PER_INSTANCE":"1" > > ,"NUM_PARTITIONS":"2" > > ,"REBALANCE_MODE":"CUSTOMIZED" > > ,"REPLICAS":"1" > > ,"STATE_MODEL_DEF_REF":"MyStateModel" > > ,"STATE_MODEL_FACTORY_NAME":"DEFAULT" > > } > > ,"listFields":{ > > } > > ,"mapFields":{ > > "0":{ > > "localhost_12004":"ONLINE" > > } > > ,"1":{ > > "localhost_12005":"ONLINE" > > } > > } > > } > > > I see that the mapFields key changed after using "USER_DEFINED" > rebalancer. > > Here are the code i used: > > public static void addResource(String resource, int num) { > clusterSetup.addResourceToCluster(CLUSTER_NAME, resource, num, > STATE_MODEL_NAME); > clusterSetup.addResourceProperty(CLUSTER_NAME, resource, > IdealState.IdealStateProperty.REBALANCER_CLASS_NAME.toString(), > CustomizedRebalancer.class.getName()); > clusterSetup.addResourceProperty(CLUSTER_NAME, resource, > IdealState.IdealStateProperty.REBALANCE_MODE.toString(), > IdealState.RebalanceMode.USER_DEFINED.toString()); > clusterSetup.rebalanceStorageCluster(CLUSTER_NAME, resource, 1, ""); > > > What is the right way to set a rebalancer? > > > The rebalancer method has a single resource interface. What is the right > way to rebalance all resources in the cluster for any event change in the > cluster? My concern is the sequence and thread safety. > > @Override > public IdealState computeNewIdealState(String resourceName, IdealState > currentIdealState, > CurrentStateOutput currentStateOutput, ClusterDataCache clusterData) { > > > > Regards, > > Leo > > > > On Fri, Oct 30, 2020 at 10:21 PM Wang Jiajun <[email protected]> > wrote: > >> Hi Leo, >> >> Are you looking for guidance on how to configure the IdealState so the >> resource will be rebalanced by your user_defined rebalancer? Maybe you can >> refer to this test case TestCustomizedIdealStateRebalancer. >> >> Best Regards, >> Jiajun >> >> >> On Fri, Oct 30, 2020 at 10:11 PM Xue Junkai <[email protected]> wrote: >> >>> Hi Leo, >>> >>> What kind of examples are you looking for? Participant side? Or how to >>> build IdealState? >>> >>> Best, >>> >>> Junkai >>> >>> On Fri, Oct 30, 2020 at 9:57 PM Leo, Li <[email protected]> wrote: >>> >>>> Hi, >>>> I am a new user of Helix. I wonder if you could give an example code >>>> of using this user_defined model >>>> https://helix.apache.org/1.0.1-docs/tutorial_user_def_rebalancer.html? >>>> >>>> Thank you, >>>> Leo >>>> >>>
