Yeah. That's the reason. In the test, it uses CustomRebalancer which extends AbstractRebalancer. And here's the definition of that: https://github.com/apache/helix/blob/a5cddd4b8fa51b1ea53b2f7082daf79361cd81de/helix-core/src/main/java/org/apache/helix/controller/rebalancer/AbstractRebalancer.java .
You can follow the implementation of this class and have a try. best, Junkai On Sat, Oct 31, 2020 at 9:53 PM Leo, Li <[email protected]> wrote: > did not implement MappingCalculator. here is code copied > from TestCustomizedIdealStateRebalancer. The interface you mentioned seems > related with > https://github.com/apache/helix/blob/a5cddd4b8fa51b1ea53b2f7082daf79361cd81de/helix-core/src/test/java/org/apache/helix/integration/rebalancer/TestCustomRebalancer.java#L76 > CustomRebalancer. > > Thanks. > Leo > > public class LeoDefinedRebalancer implements Rebalancer { > private static final Logger LOGGER = > LoggerFactory.getLogger(LeoDefinedRebalancer.class); > > @Override > public void init(HelixManager manager){ > LOGGER.info("LeoDefinedRebalancer.computeNewIdealState setup!"); > } > > > @Override > public IdealState computeNewIdealState(String resourceName, IdealState > currentIdealState, > CurrentStateOutput currentStateOutput, ClusterDataCache clusterData) { > LOGGER.info("LeoDefinedRebalancer.computeNewIdealState invoked!"); > > List<String> liveNodes = > Lists.newArrayList(clusterData.getLiveInstances().keySet()); > int i = 0; > for (String partition : currentIdealState.getPartitionSet()) { > int index = i++ % liveNodes.size(); > String instance = liveNodes.get(index); > currentIdealState.getPreferenceList(partition).clear(); > currentIdealState.getPreferenceList(partition).add(instance); > > currentIdealState.getInstanceStateMap(partition).clear(); > currentIdealState.getInstanceStateMap(partition).put(instance, > "ONLINE"); > LOGGER.info("set ############### {}: {} {} online", resourceName, > instance, partition); > } > currentIdealState.setReplicas("1"); > return currentIdealState; > } > } > > > On Sat, Oct 31, 2020 at 9:39 PM Xue Junkai <[email protected]> wrote: > >> What does your class look like? Does it implement the public interface >> MappingCalculator?This is the interface we casted to get the resource >> assignment. >> >> https://github.com/apache/helix/blob/a5cddd4b8fa51b1ea53b2f7082daf79361cd81de/helix-core/src/main/java/org/apache/helix/controller/rebalancer/internal/MappingCalculator.java >> >> >> best, >> >> Junkai >> >> On Sat, Oct 31, 2020 at 9:30 PM Leo, Li <[email protected]> wrote: >> >>> Hi Junkai, >>> >>> It's my fault to use a misleading class name. I renamed the class name >>> as LeoDefinedRebalancer. It is triggered: >>> >>> 2778 20:15:18.606 [GenericHelixController-event_process] INFO >>> c.u.k.w.c.LeoDefinedRebalancer - LeoDefinedRebalancer.computeNewIdealState >>> invoked! >>> >>> 2779 20:15:18.606 [GenericHelixController-event_process] INFO >>> c.u.k.w.c.LeoDefinedRebalancer - set ############### dca1e: >>> localhost_12001 dca1e_0 online >>> >>> 2780 20:15:18.606 [GenericHelixController-event_process] INFO >>> c.u.k.w.c.LeoDefinedRebalancer - set ############### dca1e: >>> localhost_12002 dca1e_1 online >>> >>> 2781 20:15:18.606 [GenericHelixController-event_process] WARN >>> o.a.h.c.s.BestPossibleStateCalcStage - Event 02e76c7b_DEFAULT : Rebalancer >>> does not have a mapping calculator, defaulting to SEMI_AUT O, >>> resource: dca1d >>> >>> >>> LeoDefinedRebalancer class is the test case code >>> <https://github.com/apache/helix/blob/a5cddd4b8fa51b1ea53b2f7082daf79361cd81de/helix-core/src/test/java/org/apache/helix/integration/rebalancer/TestCustomizedIdealStateRebalancer.java#L54> >>> and runs together with other classes, but the issues remain the same. >>> >>> HelixManager hMan = startStandAloneController(); >>> LeoDefinedRebalancer rebalancer = new LeoDefinedRebalancer(); >>> rebalancer.init(hMan); >>> >>> clusterSetup.addResourceToCluster(CLUSTER_NAME, resource, num, >>> STATE_MODEL_NAME); >>> clusterSetup.addResourceProperty(CLUSTER_NAME, resource, >>> IdealState.IdealStateProperty.REBALANCER_CLASS_NAME.toString(), >>> LeoDefinedRebalancer.class.getName()); >>> clusterSetup.addResourceProperty(CLUSTER_NAME, resource, >>> IdealState.IdealStateProperty.REBALANCE_MODE.toString(), >>> IdealState.RebalanceMode.USER_DEFINED.toString()); >>> clusterSetup.rebalanceStorageCluster(CLUSTER_NAME, resource, 1, ""); >>> >>> >>> On Sat, Oct 31, 2020 at 7:41 PM Xue Junkai <[email protected]> wrote: >>> >>>> Thanks for the details Leo. >>>> >>>> I think there is a misunderstanding between customized IdealState and >>>> user defined rebalancer. Actually, they are different levels of concept. >>>> >>>> For customized IdealState, it still uses Helix defined customized >>>> rebalancer, but it takes the map field as the final result. Helix only >>>> reacts to the map field change for partition/replica assignment. >>>> >>>> But user defined rebalancer is different. User has to implement the >>>> Rebalancer class and include that class in the same JVM when you start the >>>> controller. So Helix will try to use the class loader to load your class to >>>> do the computation. So the error could be you dont have the class in the >>>> same jar of starting the controller. Either you implement Rebalancer class >>>> and include it in controller jar. Or you can switch to use default auto >>>> rebalancer but with customized mode for the IdealState. >>>> >>>> Jiajun provided a test class as a good example of implementing the user >>>> defined rebalance. You can take a look if you would like to go that route. >>>> >>>> best, >>>> >>>> Junkai >>>> >>>> On Sat, Oct 31, 2020 at 7:19 PM Leo, Li <[email protected]> wrote: >>>> >>>>> Hi JunKai and Jiajun, >>>>> Thank you for your help. When I tried to use a customized Rebalancer, >>>>> I saw the rebalancer is triggered and the idea state is set by it. >>>>> >>>>> 3784 18:38:38.268 [GenericHelixController-event_process] INFO >>>>> c.u.k.w.c.CustomizedRebalancer - CustomizedRebalancer.computeNewIdealState >>>>> invoked! >>>>> >>>>> 3785 18:38:38.268 [GenericHelixController-event_process] INFO >>>>> c.u.k.w.c.CustomizedRebalancer - set ############### dca1d: >>>>> localhost_12001 dca1d_0 online >>>>> >>>>> 3786 18:38:38.268 [GenericHelixController-event_process] INFO >>>>> c.u.k.w.c.CustomizedRebalancer - set ############### dca1d: >>>>> localhost_12002 dca1d_1 online >>>>> >>>>> 3787 18:38:38.269 [GenericHelixController-event_process] WARN >>>>> o.a.h.c.s.BestPossibleStateCalcStage - Event b4896f24_DEFAULT : Rebalancer >>>>> does not have a mapping calculator, defaulting to SEMI_AUT O, >>>>> resource: dca1d >>>>> >>>>> >>>>> However the Helix UI shows "no date to display". The snapshot is >>>>> attached. The EXTERNALVIEW and IDEASTATE are: >>>>> >>>>> >>>>> >>> get /Watchdog/EXTERNALVIEW/dca1d >>>>> >>>>> { >>>>> >>>>> "id":"dca1d" >>>>> >>>>> ,"simpleFields":{ >>>>> >>>>> "BUCKET_SIZE":"0" >>>>> >>>>> ,"IDEAL_STATE_MODE":"AUTO" >>>>> >>>>> ,"NUM_PARTITIONS":"2" >>>>> >>>>> >>>>> ,"REBALANCER_CLASS_NAME":"mypackage.controller.CustomizedRebalancer" >>>>> >>>>> ,"REBALANCE_MODE":"USER_DEFINED" >>>>> >>>>> ,"REBALANCE_STRATEGY":"DEFAULT" >>>>> >>>>> ,"REPLICAS":"1" >>>>> >>>>> ,"STATE_MODEL_DEF_REF":"MyStateModel" >>>>> >>>>> ,"STATE_MODEL_FACTORY_NAME":"DEFAULT" >>>>> >>>>> } >>>>> >>>>> ,"listFields":{ >>>>> >>>>> } >>>>> >>>>> ,"mapFields":{ >>>>> >>>>> "dca1d_0":{ >>>>> >>>>> "localhost_12010":"ONLINE" >>>>> >>>>> } >>>>> >>>>> ,"dca1d_1":{ >>>>> >>>>> "localhost_12000":"ONLINE" >>>>> >>>>> } >>>>> >>>>> } >>>>> >>>>> } >>>>> >>>>> >>>>> >>> get /Watchdog/IDEALSTATES/dca1d >>>>> >>>>> { >>>>> >>>>> "id" : "dca1d", >>>>> >>>>> "simpleFields" : { >>>>> >>>>> "IDEAL_STATE_MODE" : "AUTO", >>>>> >>>>> "NUM_PARTITIONS" : "2", >>>>> >>>>> "REBALANCER_CLASS_NAME" : >>>>> "mypackage.controller.CustomizedRebalancer", >>>>> >>>>> "REBALANCE_MODE" : "USER_DEFINED", >>>>> >>>>> "REBALANCE_STRATEGY" : "DEFAULT", >>>>> >>>>> "REPLICAS" : "1", >>>>> >>>>> "STATE_MODEL_DEF_REF" : "MyStateModel", >>>>> >>>>> "STATE_MODEL_FACTORY_NAME" : "DEFAULT" >>>>> >>>>> }, >>>>> >>>>> "mapFields" : { >>>>> >>>>> "dca1d_0" : { >>>>> >>>>> }, >>>>> >>>>> "dca1d_1" : { >>>>> >>>>> } >>>>> >>>>> }, >>>>> >>>>> "listFields" : { >>>>> >>>>> "dca1d_0" : [ ], >>>>> >>>>> "dca1d_1" : [ ] >>>>> >>>>> } >>>>> >>>>> } >>>>> >>>>> >>>>> Comparing with before, when i use CUSTOMIZED rebalancer and model, The >>>>> EXTERNALVIEW and IDEASTATE were working well with Helix UI: >>>>> >>>>> >>>>> >>> get /Watchdog/IDEALSTATES/dca1d >>>>> >>>>> { >>>>> >>>>> "id":"dca1d" >>>>> >>>>> ,"simpleFields":{ >>>>> >>>>> "IDEAL_STATE_MODE":"CUSTOMIZED" >>>>> >>>>> ,"MAX_PARTITIONS_PER_INSTANCE":"1" >>>>> >>>>> ,"NUM_PARTITIONS":"2" >>>>> >>>>> ,"REBALANCE_MODE":"CUSTOMIZED" >>>>> >>>>> ,"REPLICAS":"1" >>>>> >>>>> ,"STATE_MODEL_DEF_REF":"MyStateModel" >>>>> >>>>> ,"STATE_MODEL_FACTORY_NAME":"DEFAULT" >>>>> >>>>> } >>>>> >>>>> ,"listFields":{ >>>>> >>>>> } >>>>> >>>>> ,"mapFields":{ >>>>> >>>>> "0":{ >>>>> >>>>> "localhost_12004":"ONLINE" >>>>> >>>>> } >>>>> >>>>> ,"1":{ >>>>> >>>>> "localhost_12005":"ONLINE" >>>>> >>>>> } >>>>> >>>>> } >>>>> >>>>> } >>>>> >>>>> >>>>> >>> get /Watchdog/EXTERNALVIEW/dca1d >>>>> >>>>> { >>>>> >>>>> "id":"dca1d" >>>>> >>>>> ,"simpleFields":{ >>>>> >>>>> "BUCKET_SIZE":"0" >>>>> >>>>> ,"IDEAL_STATE_MODE":"CUSTOMIZED" >>>>> >>>>> ,"MAX_PARTITIONS_PER_INSTANCE":"1" >>>>> >>>>> ,"NUM_PARTITIONS":"2" >>>>> >>>>> ,"REBALANCE_MODE":"CUSTOMIZED" >>>>> >>>>> ,"REPLICAS":"1" >>>>> >>>>> ,"STATE_MODEL_DEF_REF":"MyStateModel" >>>>> >>>>> ,"STATE_MODEL_FACTORY_NAME":"DEFAULT" >>>>> >>>>> } >>>>> >>>>> ,"listFields":{ >>>>> >>>>> } >>>>> >>>>> ,"mapFields":{ >>>>> >>>>> "0":{ >>>>> >>>>> "localhost_12004":"ONLINE" >>>>> >>>>> } >>>>> >>>>> ,"1":{ >>>>> >>>>> "localhost_12005":"ONLINE" >>>>> >>>>> } >>>>> >>>>> } >>>>> >>>>> } >>>>> >>>>> >>>>> I see that the mapFields key changed after using "USER_DEFINED" >>>>> rebalancer. >>>>> >>>>> Here are the code i used: >>>>> >>>>> public static void addResource(String resource, int num) { >>>>> clusterSetup.addResourceToCluster(CLUSTER_NAME, resource, num, >>>>> STATE_MODEL_NAME); >>>>> clusterSetup.addResourceProperty(CLUSTER_NAME, resource, >>>>> IdealState.IdealStateProperty.REBALANCER_CLASS_NAME.toString(), >>>>> CustomizedRebalancer.class.getName()); >>>>> clusterSetup.addResourceProperty(CLUSTER_NAME, resource, >>>>> IdealState.IdealStateProperty.REBALANCE_MODE.toString(), >>>>> IdealState.RebalanceMode.USER_DEFINED.toString()); >>>>> clusterSetup.rebalanceStorageCluster(CLUSTER_NAME, resource, 1, ""); >>>>> >>>>> >>>>> What is the right way to set a rebalancer? >>>>> >>>>> >>>>> The rebalancer method has a single resource interface. What is the >>>>> right way to rebalance all resources in the cluster for any event change >>>>> in >>>>> the cluster? My concern is the sequence and thread safety. >>>>> >>>>> @Override >>>>> public IdealState computeNewIdealState(String resourceName, IdealState >>>>> currentIdealState, >>>>> CurrentStateOutput currentStateOutput, ClusterDataCache clusterData) { >>>>> >>>>> >>>>> >>>>> Regards, >>>>> >>>>> Leo >>>>> >>>>> >>>>> >>>>> On Fri, Oct 30, 2020 at 10:21 PM Wang Jiajun <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi Leo, >>>>>> >>>>>> Are you looking for guidance on how to configure the IdealState so >>>>>> the resource will be rebalanced by your user_defined rebalancer? Maybe >>>>>> you >>>>>> can refer to this test case TestCustomizedIdealStateRebalancer. >>>>>> >>>>>> Best Regards, >>>>>> Jiajun >>>>>> >>>>>> >>>>>> On Fri, Oct 30, 2020 at 10:11 PM Xue Junkai <[email protected]> wrote: >>>>>> >>>>>>> Hi Leo, >>>>>>> >>>>>>> What kind of examples are you looking for? Participant side? Or how >>>>>>> to build IdealState? >>>>>>> >>>>>>> Best, >>>>>>> >>>>>>> Junkai >>>>>>> >>>>>>> On Fri, Oct 30, 2020 at 9:57 PM Leo, Li <[email protected]> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> I am a new user of Helix. I wonder if you could give an example >>>>>>>> code of using this user_defined model >>>>>>>> https://helix.apache.org/1.0.1-docs/tutorial_user_def_rebalancer.html >>>>>>>> ? >>>>>>>> >>>>>>>> Thank you, >>>>>>>> Leo >>>>>>>> >>>>>>>
