Hi Junkai, It's my fault to use a misleading class name. I renamed the class name as LeoDefinedRebalancer. It is triggered:
2778 20:15:18.606 [GenericHelixController-event_process] INFO c.u.k.w.c.LeoDefinedRebalancer - LeoDefinedRebalancer.computeNewIdealState invoked! 2779 20:15:18.606 [GenericHelixController-event_process] INFO c.u.k.w.c.LeoDefinedRebalancer - set ############### dca1e: localhost_12001 dca1e_0 online 2780 20:15:18.606 [GenericHelixController-event_process] INFO c.u.k.w.c.LeoDefinedRebalancer - set ############### dca1e: localhost_12002 dca1e_1 online 2781 20:15:18.606 [GenericHelixController-event_process] WARN o.a.h.c.s.BestPossibleStateCalcStage - Event 02e76c7b_DEFAULT : Rebalancer does not have a mapping calculator, defaulting to SEMI_AUT O, resource: dca1d LeoDefinedRebalancer class is the test case code <https://github.com/apache/helix/blob/a5cddd4b8fa51b1ea53b2f7082daf79361cd81de/helix-core/src/test/java/org/apache/helix/integration/rebalancer/TestCustomizedIdealStateRebalancer.java#L54> and runs together with other classes, but the issues remain the same. HelixManager hMan = startStandAloneController(); LeoDefinedRebalancer rebalancer = new LeoDefinedRebalancer(); rebalancer.init(hMan); clusterSetup.addResourceToCluster(CLUSTER_NAME, resource, num, STATE_MODEL_NAME); clusterSetup.addResourceProperty(CLUSTER_NAME, resource, IdealState.IdealStateProperty.REBALANCER_CLASS_NAME.toString(), LeoDefinedRebalancer.class.getName()); clusterSetup.addResourceProperty(CLUSTER_NAME, resource, IdealState.IdealStateProperty.REBALANCE_MODE.toString(), IdealState.RebalanceMode.USER_DEFINED.toString()); clusterSetup.rebalanceStorageCluster(CLUSTER_NAME, resource, 1, ""); On Sat, Oct 31, 2020 at 7:41 PM Xue Junkai <[email protected]> wrote: > Thanks for the details Leo. > > I think there is a misunderstanding between customized IdealState and user > defined rebalancer. Actually, they are different levels of concept. > > For customized IdealState, it still uses Helix defined customized > rebalancer, but it takes the map field as the final result. Helix only > reacts to the map field change for partition/replica assignment. > > But user defined rebalancer is different. User has to implement the > Rebalancer class and include that class in the same JVM when you start the > controller. So Helix will try to use the class loader to load your class to > do the computation. So the error could be you dont have the class in the > same jar of starting the controller. Either you implement Rebalancer class > and include it in controller jar. Or you can switch to use default auto > rebalancer but with customized mode for the IdealState. > > Jiajun provided a test class as a good example of implementing the user > defined rebalance. You can take a look if you would like to go that route. > > best, > > Junkai > > On Sat, Oct 31, 2020 at 7:19 PM Leo, Li <[email protected]> wrote: > >> Hi JunKai and Jiajun, >> Thank you for your help. When I tried to use a customized Rebalancer, I >> saw the rebalancer is triggered and the idea state is set by it. >> >> 3784 18:38:38.268 [GenericHelixController-event_process] INFO >> c.u.k.w.c.CustomizedRebalancer - CustomizedRebalancer.computeNewIdealState >> invoked! >> >> 3785 18:38:38.268 [GenericHelixController-event_process] INFO >> c.u.k.w.c.CustomizedRebalancer - set ############### dca1d: >> localhost_12001 dca1d_0 online >> >> 3786 18:38:38.268 [GenericHelixController-event_process] INFO >> c.u.k.w.c.CustomizedRebalancer - set ############### dca1d: >> localhost_12002 dca1d_1 online >> >> 3787 18:38:38.269 [GenericHelixController-event_process] WARN >> o.a.h.c.s.BestPossibleStateCalcStage - Event b4896f24_DEFAULT : Rebalancer >> does not have a mapping calculator, defaulting to SEMI_AUT O, >> resource: dca1d >> >> >> However the Helix UI shows "no date to display". The snapshot is >> attached. The EXTERNALVIEW and IDEASTATE are: >> >> >> >>> get /Watchdog/EXTERNALVIEW/dca1d >> >> { >> >> "id":"dca1d" >> >> ,"simpleFields":{ >> >> "BUCKET_SIZE":"0" >> >> ,"IDEAL_STATE_MODE":"AUTO" >> >> ,"NUM_PARTITIONS":"2" >> >> ,"REBALANCER_CLASS_NAME":"mypackage.controller.CustomizedRebalancer" >> >> ,"REBALANCE_MODE":"USER_DEFINED" >> >> ,"REBALANCE_STRATEGY":"DEFAULT" >> >> ,"REPLICAS":"1" >> >> ,"STATE_MODEL_DEF_REF":"MyStateModel" >> >> ,"STATE_MODEL_FACTORY_NAME":"DEFAULT" >> >> } >> >> ,"listFields":{ >> >> } >> >> ,"mapFields":{ >> >> "dca1d_0":{ >> >> "localhost_12010":"ONLINE" >> >> } >> >> ,"dca1d_1":{ >> >> "localhost_12000":"ONLINE" >> >> } >> >> } >> >> } >> >> >> >>> get /Watchdog/IDEALSTATES/dca1d >> >> { >> >> "id" : "dca1d", >> >> "simpleFields" : { >> >> "IDEAL_STATE_MODE" : "AUTO", >> >> "NUM_PARTITIONS" : "2", >> >> "REBALANCER_CLASS_NAME" : "mypackage.controller.CustomizedRebalancer", >> >> "REBALANCE_MODE" : "USER_DEFINED", >> >> "REBALANCE_STRATEGY" : "DEFAULT", >> >> "REPLICAS" : "1", >> >> "STATE_MODEL_DEF_REF" : "MyStateModel", >> >> "STATE_MODEL_FACTORY_NAME" : "DEFAULT" >> >> }, >> >> "mapFields" : { >> >> "dca1d_0" : { >> >> }, >> >> "dca1d_1" : { >> >> } >> >> }, >> >> "listFields" : { >> >> "dca1d_0" : [ ], >> >> "dca1d_1" : [ ] >> >> } >> >> } >> >> >> Comparing with before, when i use CUSTOMIZED rebalancer and model, The >> EXTERNALVIEW and IDEASTATE were working well with Helix UI: >> >> >> >>> get /Watchdog/IDEALSTATES/dca1d >> >> { >> >> "id":"dca1d" >> >> ,"simpleFields":{ >> >> "IDEAL_STATE_MODE":"CUSTOMIZED" >> >> ,"MAX_PARTITIONS_PER_INSTANCE":"1" >> >> ,"NUM_PARTITIONS":"2" >> >> ,"REBALANCE_MODE":"CUSTOMIZED" >> >> ,"REPLICAS":"1" >> >> ,"STATE_MODEL_DEF_REF":"MyStateModel" >> >> ,"STATE_MODEL_FACTORY_NAME":"DEFAULT" >> >> } >> >> ,"listFields":{ >> >> } >> >> ,"mapFields":{ >> >> "0":{ >> >> "localhost_12004":"ONLINE" >> >> } >> >> ,"1":{ >> >> "localhost_12005":"ONLINE" >> >> } >> >> } >> >> } >> >> >> >>> get /Watchdog/EXTERNALVIEW/dca1d >> >> { >> >> "id":"dca1d" >> >> ,"simpleFields":{ >> >> "BUCKET_SIZE":"0" >> >> ,"IDEAL_STATE_MODE":"CUSTOMIZED" >> >> ,"MAX_PARTITIONS_PER_INSTANCE":"1" >> >> ,"NUM_PARTITIONS":"2" >> >> ,"REBALANCE_MODE":"CUSTOMIZED" >> >> ,"REPLICAS":"1" >> >> ,"STATE_MODEL_DEF_REF":"MyStateModel" >> >> ,"STATE_MODEL_FACTORY_NAME":"DEFAULT" >> >> } >> >> ,"listFields":{ >> >> } >> >> ,"mapFields":{ >> >> "0":{ >> >> "localhost_12004":"ONLINE" >> >> } >> >> ,"1":{ >> >> "localhost_12005":"ONLINE" >> >> } >> >> } >> >> } >> >> >> I see that the mapFields key changed after using "USER_DEFINED" >> rebalancer. >> >> Here are the code i used: >> >> public static void addResource(String resource, int num) { >> clusterSetup.addResourceToCluster(CLUSTER_NAME, resource, num, >> STATE_MODEL_NAME); >> clusterSetup.addResourceProperty(CLUSTER_NAME, resource, >> IdealState.IdealStateProperty.REBALANCER_CLASS_NAME.toString(), >> CustomizedRebalancer.class.getName()); >> clusterSetup.addResourceProperty(CLUSTER_NAME, resource, >> IdealState.IdealStateProperty.REBALANCE_MODE.toString(), >> IdealState.RebalanceMode.USER_DEFINED.toString()); >> clusterSetup.rebalanceStorageCluster(CLUSTER_NAME, resource, 1, ""); >> >> >> What is the right way to set a rebalancer? >> >> >> The rebalancer method has a single resource interface. What is the right >> way to rebalance all resources in the cluster for any event change in the >> cluster? My concern is the sequence and thread safety. >> >> @Override >> public IdealState computeNewIdealState(String resourceName, IdealState >> currentIdealState, >> CurrentStateOutput currentStateOutput, ClusterDataCache clusterData) { >> >> >> >> Regards, >> >> Leo >> >> >> >> On Fri, Oct 30, 2020 at 10:21 PM Wang Jiajun <[email protected]> >> wrote: >> >>> Hi Leo, >>> >>> Are you looking for guidance on how to configure the IdealState so the >>> resource will be rebalanced by your user_defined rebalancer? Maybe you can >>> refer to this test case TestCustomizedIdealStateRebalancer. >>> >>> Best Regards, >>> Jiajun >>> >>> >>> On Fri, Oct 30, 2020 at 10:11 PM Xue Junkai <[email protected]> wrote: >>> >>>> Hi Leo, >>>> >>>> What kind of examples are you looking for? Participant side? Or how to >>>> build IdealState? >>>> >>>> Best, >>>> >>>> Junkai >>>> >>>> On Fri, Oct 30, 2020 at 9:57 PM Leo, Li <[email protected]> wrote: >>>> >>>>> Hi, >>>>> I am a new user of Helix. I wonder if you could give an example code >>>>> of using this user_defined model >>>>> https://helix.apache.org/1.0.1-docs/tutorial_user_def_rebalancer.html >>>>> ? >>>>> >>>>> Thank you, >>>>> Leo >>>>> >>>>
