What does your class look like? Does it implement the public interface MappingCalculator?This is the interface we casted to get the resource assignment. https://github.com/apache/helix/blob/a5cddd4b8fa51b1ea53b2f7082daf79361cd81de/helix-core/src/main/java/org/apache/helix/controller/rebalancer/internal/MappingCalculator.java
best, Junkai On Sat, Oct 31, 2020 at 9:30 PM Leo, Li <[email protected]> wrote: > Hi Junkai, > > It's my fault to use a misleading class name. I renamed the class name as > LeoDefinedRebalancer. It is triggered: > > 2778 20:15:18.606 [GenericHelixController-event_process] INFO > c.u.k.w.c.LeoDefinedRebalancer - LeoDefinedRebalancer.computeNewIdealState > invoked! > > 2779 20:15:18.606 [GenericHelixController-event_process] INFO > c.u.k.w.c.LeoDefinedRebalancer - set ############### dca1e: > localhost_12001 dca1e_0 online > > 2780 20:15:18.606 [GenericHelixController-event_process] INFO > c.u.k.w.c.LeoDefinedRebalancer - set ############### dca1e: > localhost_12002 dca1e_1 online > > 2781 20:15:18.606 [GenericHelixController-event_process] WARN > o.a.h.c.s.BestPossibleStateCalcStage - Event 02e76c7b_DEFAULT : Rebalancer > does not have a mapping calculator, defaulting to SEMI_AUT O, > resource: dca1d > > > LeoDefinedRebalancer class is the test case code > <https://github.com/apache/helix/blob/a5cddd4b8fa51b1ea53b2f7082daf79361cd81de/helix-core/src/test/java/org/apache/helix/integration/rebalancer/TestCustomizedIdealStateRebalancer.java#L54> > and runs together with other classes, but the issues remain the same. > > HelixManager hMan = startStandAloneController(); > LeoDefinedRebalancer rebalancer = new LeoDefinedRebalancer(); > rebalancer.init(hMan); > > clusterSetup.addResourceToCluster(CLUSTER_NAME, resource, num, > STATE_MODEL_NAME); > clusterSetup.addResourceProperty(CLUSTER_NAME, resource, > IdealState.IdealStateProperty.REBALANCER_CLASS_NAME.toString(), > LeoDefinedRebalancer.class.getName()); > clusterSetup.addResourceProperty(CLUSTER_NAME, resource, > IdealState.IdealStateProperty.REBALANCE_MODE.toString(), > IdealState.RebalanceMode.USER_DEFINED.toString()); > clusterSetup.rebalanceStorageCluster(CLUSTER_NAME, resource, 1, ""); > > > On Sat, Oct 31, 2020 at 7:41 PM Xue Junkai <[email protected]> wrote: > >> Thanks for the details Leo. >> >> I think there is a misunderstanding between customized IdealState and >> user defined rebalancer. Actually, they are different levels of concept. >> >> For customized IdealState, it still uses Helix defined customized >> rebalancer, but it takes the map field as the final result. Helix only >> reacts to the map field change for partition/replica assignment. >> >> But user defined rebalancer is different. User has to implement the >> Rebalancer class and include that class in the same JVM when you start the >> controller. So Helix will try to use the class loader to load your class to >> do the computation. So the error could be you dont have the class in the >> same jar of starting the controller. Either you implement Rebalancer class >> and include it in controller jar. Or you can switch to use default auto >> rebalancer but with customized mode for the IdealState. >> >> Jiajun provided a test class as a good example of implementing the user >> defined rebalance. You can take a look if you would like to go that route. >> >> best, >> >> Junkai >> >> On Sat, Oct 31, 2020 at 7:19 PM Leo, Li <[email protected]> wrote: >> >>> Hi JunKai and Jiajun, >>> Thank you for your help. When I tried to use a customized Rebalancer, I >>> saw the rebalancer is triggered and the idea state is set by it. >>> >>> 3784 18:38:38.268 [GenericHelixController-event_process] INFO >>> c.u.k.w.c.CustomizedRebalancer - CustomizedRebalancer.computeNewIdealState >>> invoked! >>> >>> 3785 18:38:38.268 [GenericHelixController-event_process] INFO >>> c.u.k.w.c.CustomizedRebalancer - set ############### dca1d: >>> localhost_12001 dca1d_0 online >>> >>> 3786 18:38:38.268 [GenericHelixController-event_process] INFO >>> c.u.k.w.c.CustomizedRebalancer - set ############### dca1d: >>> localhost_12002 dca1d_1 online >>> >>> 3787 18:38:38.269 [GenericHelixController-event_process] WARN >>> o.a.h.c.s.BestPossibleStateCalcStage - Event b4896f24_DEFAULT : Rebalancer >>> does not have a mapping calculator, defaulting to SEMI_AUT O, >>> resource: dca1d >>> >>> >>> However the Helix UI shows "no date to display". The snapshot is >>> attached. The EXTERNALVIEW and IDEASTATE are: >>> >>> >>> >>> get /Watchdog/EXTERNALVIEW/dca1d >>> >>> { >>> >>> "id":"dca1d" >>> >>> ,"simpleFields":{ >>> >>> "BUCKET_SIZE":"0" >>> >>> ,"IDEAL_STATE_MODE":"AUTO" >>> >>> ,"NUM_PARTITIONS":"2" >>> >>> ,"REBALANCER_CLASS_NAME":"mypackage.controller.CustomizedRebalancer" >>> >>> ,"REBALANCE_MODE":"USER_DEFINED" >>> >>> ,"REBALANCE_STRATEGY":"DEFAULT" >>> >>> ,"REPLICAS":"1" >>> >>> ,"STATE_MODEL_DEF_REF":"MyStateModel" >>> >>> ,"STATE_MODEL_FACTORY_NAME":"DEFAULT" >>> >>> } >>> >>> ,"listFields":{ >>> >>> } >>> >>> ,"mapFields":{ >>> >>> "dca1d_0":{ >>> >>> "localhost_12010":"ONLINE" >>> >>> } >>> >>> ,"dca1d_1":{ >>> >>> "localhost_12000":"ONLINE" >>> >>> } >>> >>> } >>> >>> } >>> >>> >>> >>> get /Watchdog/IDEALSTATES/dca1d >>> >>> { >>> >>> "id" : "dca1d", >>> >>> "simpleFields" : { >>> >>> "IDEAL_STATE_MODE" : "AUTO", >>> >>> "NUM_PARTITIONS" : "2", >>> >>> "REBALANCER_CLASS_NAME" : >>> "mypackage.controller.CustomizedRebalancer", >>> >>> "REBALANCE_MODE" : "USER_DEFINED", >>> >>> "REBALANCE_STRATEGY" : "DEFAULT", >>> >>> "REPLICAS" : "1", >>> >>> "STATE_MODEL_DEF_REF" : "MyStateModel", >>> >>> "STATE_MODEL_FACTORY_NAME" : "DEFAULT" >>> >>> }, >>> >>> "mapFields" : { >>> >>> "dca1d_0" : { >>> >>> }, >>> >>> "dca1d_1" : { >>> >>> } >>> >>> }, >>> >>> "listFields" : { >>> >>> "dca1d_0" : [ ], >>> >>> "dca1d_1" : [ ] >>> >>> } >>> >>> } >>> >>> >>> Comparing with before, when i use CUSTOMIZED rebalancer and model, The >>> EXTERNALVIEW and IDEASTATE were working well with Helix UI: >>> >>> >>> >>> get /Watchdog/IDEALSTATES/dca1d >>> >>> { >>> >>> "id":"dca1d" >>> >>> ,"simpleFields":{ >>> >>> "IDEAL_STATE_MODE":"CUSTOMIZED" >>> >>> ,"MAX_PARTITIONS_PER_INSTANCE":"1" >>> >>> ,"NUM_PARTITIONS":"2" >>> >>> ,"REBALANCE_MODE":"CUSTOMIZED" >>> >>> ,"REPLICAS":"1" >>> >>> ,"STATE_MODEL_DEF_REF":"MyStateModel" >>> >>> ,"STATE_MODEL_FACTORY_NAME":"DEFAULT" >>> >>> } >>> >>> ,"listFields":{ >>> >>> } >>> >>> ,"mapFields":{ >>> >>> "0":{ >>> >>> "localhost_12004":"ONLINE" >>> >>> } >>> >>> ,"1":{ >>> >>> "localhost_12005":"ONLINE" >>> >>> } >>> >>> } >>> >>> } >>> >>> >>> >>> get /Watchdog/EXTERNALVIEW/dca1d >>> >>> { >>> >>> "id":"dca1d" >>> >>> ,"simpleFields":{ >>> >>> "BUCKET_SIZE":"0" >>> >>> ,"IDEAL_STATE_MODE":"CUSTOMIZED" >>> >>> ,"MAX_PARTITIONS_PER_INSTANCE":"1" >>> >>> ,"NUM_PARTITIONS":"2" >>> >>> ,"REBALANCE_MODE":"CUSTOMIZED" >>> >>> ,"REPLICAS":"1" >>> >>> ,"STATE_MODEL_DEF_REF":"MyStateModel" >>> >>> ,"STATE_MODEL_FACTORY_NAME":"DEFAULT" >>> >>> } >>> >>> ,"listFields":{ >>> >>> } >>> >>> ,"mapFields":{ >>> >>> "0":{ >>> >>> "localhost_12004":"ONLINE" >>> >>> } >>> >>> ,"1":{ >>> >>> "localhost_12005":"ONLINE" >>> >>> } >>> >>> } >>> >>> } >>> >>> >>> I see that the mapFields key changed after using "USER_DEFINED" >>> rebalancer. >>> >>> Here are the code i used: >>> >>> public static void addResource(String resource, int num) { >>> clusterSetup.addResourceToCluster(CLUSTER_NAME, resource, num, >>> STATE_MODEL_NAME); >>> clusterSetup.addResourceProperty(CLUSTER_NAME, resource, >>> IdealState.IdealStateProperty.REBALANCER_CLASS_NAME.toString(), >>> CustomizedRebalancer.class.getName()); >>> clusterSetup.addResourceProperty(CLUSTER_NAME, resource, >>> IdealState.IdealStateProperty.REBALANCE_MODE.toString(), >>> IdealState.RebalanceMode.USER_DEFINED.toString()); >>> clusterSetup.rebalanceStorageCluster(CLUSTER_NAME, resource, 1, ""); >>> >>> >>> What is the right way to set a rebalancer? >>> >>> >>> The rebalancer method has a single resource interface. What is the right >>> way to rebalance all resources in the cluster for any event change in the >>> cluster? My concern is the sequence and thread safety. >>> >>> @Override >>> public IdealState computeNewIdealState(String resourceName, IdealState >>> currentIdealState, >>> CurrentStateOutput currentStateOutput, ClusterDataCache clusterData) { >>> >>> >>> >>> Regards, >>> >>> Leo >>> >>> >>> >>> On Fri, Oct 30, 2020 at 10:21 PM Wang Jiajun <[email protected]> >>> wrote: >>> >>>> Hi Leo, >>>> >>>> Are you looking for guidance on how to configure the IdealState so the >>>> resource will be rebalanced by your user_defined rebalancer? Maybe you can >>>> refer to this test case TestCustomizedIdealStateRebalancer. >>>> >>>> Best Regards, >>>> Jiajun >>>> >>>> >>>> On Fri, Oct 30, 2020 at 10:11 PM Xue Junkai <[email protected]> wrote: >>>> >>>>> Hi Leo, >>>>> >>>>> What kind of examples are you looking for? Participant side? Or how >>>>> to build IdealState? >>>>> >>>>> Best, >>>>> >>>>> Junkai >>>>> >>>>> On Fri, Oct 30, 2020 at 9:57 PM Leo, Li <[email protected]> wrote: >>>>> >>>>>> Hi, >>>>>> I am a new user of Helix. I wonder if you could give an example code >>>>>> of using this user_defined model >>>>>> https://helix.apache.org/1.0.1-docs/tutorial_user_def_rebalancer.html >>>>>> ? >>>>>> >>>>>> Thank you, >>>>>> Leo >>>>>> >>>>>
