did not implement MappingCalculator. here is code copied
from TestCustomizedIdealStateRebalancer. The interface you mentioned seems
related with
https://github.com/apache/helix/blob/a5cddd4b8fa51b1ea53b2f7082daf79361cd81de/helix-core/src/test/java/org/apache/helix/integration/rebalancer/TestCustomRebalancer.java#L76
CustomRebalancer.
Thanks.
Leo
public class LeoDefinedRebalancer implements Rebalancer {
private static final Logger LOGGER =
LoggerFactory.getLogger(LeoDefinedRebalancer.class);
@Override
public void init(HelixManager manager){
LOGGER.info("LeoDefinedRebalancer.computeNewIdealState setup!");
}
@Override
public IdealState computeNewIdealState(String resourceName,
IdealState currentIdealState,
CurrentStateOutput currentStateOutput, ClusterDataCache clusterData) {
LOGGER.info("LeoDefinedRebalancer.computeNewIdealState invoked!");
List<String> liveNodes =
Lists.newArrayList(clusterData.getLiveInstances().keySet());
int i = 0;
for (String partition : currentIdealState.getPartitionSet()) {
int index = i++ % liveNodes.size();
String instance = liveNodes.get(index);
currentIdealState.getPreferenceList(partition).clear();
currentIdealState.getPreferenceList(partition).add(instance);
currentIdealState.getInstanceStateMap(partition).clear();
currentIdealState.getInstanceStateMap(partition).put(instance, "ONLINE");
LOGGER.info("set ############### {}: {} {} online",
resourceName, instance, partition);
}
currentIdealState.setReplicas("1");
return currentIdealState;
}
}
On Sat, Oct 31, 2020 at 9:39 PM Xue Junkai <[email protected]> wrote:
> What does your class look like? Does it implement the public interface
> MappingCalculator?This is the interface we casted to get the resource
> assignment.
>
> https://github.com/apache/helix/blob/a5cddd4b8fa51b1ea53b2f7082daf79361cd81de/helix-core/src/main/java/org/apache/helix/controller/rebalancer/internal/MappingCalculator.java
>
>
> best,
>
> Junkai
>
> On Sat, Oct 31, 2020 at 9:30 PM Leo, Li <[email protected]> wrote:
>
>> Hi Junkai,
>>
>> It's my fault to use a misleading class name. I renamed the class name
>> as LeoDefinedRebalancer. It is triggered:
>>
>> 2778 20:15:18.606 [GenericHelixController-event_process] INFO
>> c.u.k.w.c.LeoDefinedRebalancer - LeoDefinedRebalancer.computeNewIdealState
>> invoked!
>>
>> 2779 20:15:18.606 [GenericHelixController-event_process] INFO
>> c.u.k.w.c.LeoDefinedRebalancer - set ############### dca1e:
>> localhost_12001 dca1e_0 online
>>
>> 2780 20:15:18.606 [GenericHelixController-event_process] INFO
>> c.u.k.w.c.LeoDefinedRebalancer - set ############### dca1e:
>> localhost_12002 dca1e_1 online
>>
>> 2781 20:15:18.606 [GenericHelixController-event_process] WARN
>> o.a.h.c.s.BestPossibleStateCalcStage - Event 02e76c7b_DEFAULT : Rebalancer
>> does not have a mapping calculator, defaulting to SEMI_AUT O,
>> resource: dca1d
>>
>>
>> LeoDefinedRebalancer class is the test case code
>> <https://github.com/apache/helix/blob/a5cddd4b8fa51b1ea53b2f7082daf79361cd81de/helix-core/src/test/java/org/apache/helix/integration/rebalancer/TestCustomizedIdealStateRebalancer.java#L54>
>> and runs together with other classes, but the issues remain the same.
>>
>> HelixManager hMan = startStandAloneController();
>> LeoDefinedRebalancer rebalancer = new LeoDefinedRebalancer();
>> rebalancer.init(hMan);
>>
>> clusterSetup.addResourceToCluster(CLUSTER_NAME, resource, num,
>> STATE_MODEL_NAME);
>> clusterSetup.addResourceProperty(CLUSTER_NAME, resource,
>> IdealState.IdealStateProperty.REBALANCER_CLASS_NAME.toString(),
>> LeoDefinedRebalancer.class.getName());
>> clusterSetup.addResourceProperty(CLUSTER_NAME, resource,
>> IdealState.IdealStateProperty.REBALANCE_MODE.toString(),
>> IdealState.RebalanceMode.USER_DEFINED.toString());
>> clusterSetup.rebalanceStorageCluster(CLUSTER_NAME, resource, 1, "");
>>
>>
>> On Sat, Oct 31, 2020 at 7:41 PM Xue Junkai <[email protected]> wrote:
>>
>>> Thanks for the details Leo.
>>>
>>> I think there is a misunderstanding between customized IdealState and
>>> user defined rebalancer. Actually, they are different levels of concept.
>>>
>>> For customized IdealState, it still uses Helix defined customized
>>> rebalancer, but it takes the map field as the final result. Helix only
>>> reacts to the map field change for partition/replica assignment.
>>>
>>> But user defined rebalancer is different. User has to implement the
>>> Rebalancer class and include that class in the same JVM when you start the
>>> controller. So Helix will try to use the class loader to load your class to
>>> do the computation. So the error could be you dont have the class in the
>>> same jar of starting the controller. Either you implement Rebalancer class
>>> and include it in controller jar. Or you can switch to use default auto
>>> rebalancer but with customized mode for the IdealState.
>>>
>>> Jiajun provided a test class as a good example of implementing the user
>>> defined rebalance. You can take a look if you would like to go that route.
>>>
>>> best,
>>>
>>> Junkai
>>>
>>> On Sat, Oct 31, 2020 at 7:19 PM Leo, Li <[email protected]> wrote:
>>>
>>>> Hi JunKai and Jiajun,
>>>> Thank you for your help. When I tried to use a customized Rebalancer, I
>>>> saw the rebalancer is triggered and the idea state is set by it.
>>>>
>>>> 3784 18:38:38.268 [GenericHelixController-event_process] INFO
>>>> c.u.k.w.c.CustomizedRebalancer - CustomizedRebalancer.computeNewIdealState
>>>> invoked!
>>>>
>>>> 3785 18:38:38.268 [GenericHelixController-event_process] INFO
>>>> c.u.k.w.c.CustomizedRebalancer - set ############### dca1d:
>>>> localhost_12001 dca1d_0 online
>>>>
>>>> 3786 18:38:38.268 [GenericHelixController-event_process] INFO
>>>> c.u.k.w.c.CustomizedRebalancer - set ############### dca1d:
>>>> localhost_12002 dca1d_1 online
>>>>
>>>> 3787 18:38:38.269 [GenericHelixController-event_process] WARN
>>>> o.a.h.c.s.BestPossibleStateCalcStage - Event b4896f24_DEFAULT : Rebalancer
>>>> does not have a mapping calculator, defaulting to SEMI_AUT O,
>>>> resource: dca1d
>>>>
>>>>
>>>> However the Helix UI shows "no date to display". The snapshot is
>>>> attached. The EXTERNALVIEW and IDEASTATE are:
>>>>
>>>>
>>>> >>> get /Watchdog/EXTERNALVIEW/dca1d
>>>>
>>>> {
>>>>
>>>> "id":"dca1d"
>>>>
>>>> ,"simpleFields":{
>>>>
>>>> "BUCKET_SIZE":"0"
>>>>
>>>> ,"IDEAL_STATE_MODE":"AUTO"
>>>>
>>>> ,"NUM_PARTITIONS":"2"
>>>>
>>>> ,"REBALANCER_CLASS_NAME":"mypackage.controller.CustomizedRebalancer"
>>>>
>>>> ,"REBALANCE_MODE":"USER_DEFINED"
>>>>
>>>> ,"REBALANCE_STRATEGY":"DEFAULT"
>>>>
>>>> ,"REPLICAS":"1"
>>>>
>>>> ,"STATE_MODEL_DEF_REF":"MyStateModel"
>>>>
>>>> ,"STATE_MODEL_FACTORY_NAME":"DEFAULT"
>>>>
>>>> }
>>>>
>>>> ,"listFields":{
>>>>
>>>> }
>>>>
>>>> ,"mapFields":{
>>>>
>>>> "dca1d_0":{
>>>>
>>>> "localhost_12010":"ONLINE"
>>>>
>>>> }
>>>>
>>>> ,"dca1d_1":{
>>>>
>>>> "localhost_12000":"ONLINE"
>>>>
>>>> }
>>>>
>>>> }
>>>>
>>>> }
>>>>
>>>>
>>>> >>> get /Watchdog/IDEALSTATES/dca1d
>>>>
>>>> {
>>>>
>>>> "id" : "dca1d",
>>>>
>>>> "simpleFields" : {
>>>>
>>>> "IDEAL_STATE_MODE" : "AUTO",
>>>>
>>>> "NUM_PARTITIONS" : "2",
>>>>
>>>> "REBALANCER_CLASS_NAME" :
>>>> "mypackage.controller.CustomizedRebalancer",
>>>>
>>>> "REBALANCE_MODE" : "USER_DEFINED",
>>>>
>>>> "REBALANCE_STRATEGY" : "DEFAULT",
>>>>
>>>> "REPLICAS" : "1",
>>>>
>>>> "STATE_MODEL_DEF_REF" : "MyStateModel",
>>>>
>>>> "STATE_MODEL_FACTORY_NAME" : "DEFAULT"
>>>>
>>>> },
>>>>
>>>> "mapFields" : {
>>>>
>>>> "dca1d_0" : {
>>>>
>>>> },
>>>>
>>>> "dca1d_1" : {
>>>>
>>>> }
>>>>
>>>> },
>>>>
>>>> "listFields" : {
>>>>
>>>> "dca1d_0" : [ ],
>>>>
>>>> "dca1d_1" : [ ]
>>>>
>>>> }
>>>>
>>>> }
>>>>
>>>>
>>>> Comparing with before, when i use CUSTOMIZED rebalancer and model, The
>>>> EXTERNALVIEW and IDEASTATE were working well with Helix UI:
>>>>
>>>>
>>>> >>> get /Watchdog/IDEALSTATES/dca1d
>>>>
>>>> {
>>>>
>>>> "id":"dca1d"
>>>>
>>>> ,"simpleFields":{
>>>>
>>>> "IDEAL_STATE_MODE":"CUSTOMIZED"
>>>>
>>>> ,"MAX_PARTITIONS_PER_INSTANCE":"1"
>>>>
>>>> ,"NUM_PARTITIONS":"2"
>>>>
>>>> ,"REBALANCE_MODE":"CUSTOMIZED"
>>>>
>>>> ,"REPLICAS":"1"
>>>>
>>>> ,"STATE_MODEL_DEF_REF":"MyStateModel"
>>>>
>>>> ,"STATE_MODEL_FACTORY_NAME":"DEFAULT"
>>>>
>>>> }
>>>>
>>>> ,"listFields":{
>>>>
>>>> }
>>>>
>>>> ,"mapFields":{
>>>>
>>>> "0":{
>>>>
>>>> "localhost_12004":"ONLINE"
>>>>
>>>> }
>>>>
>>>> ,"1":{
>>>>
>>>> "localhost_12005":"ONLINE"
>>>>
>>>> }
>>>>
>>>> }
>>>>
>>>> }
>>>>
>>>>
>>>> >>> get /Watchdog/EXTERNALVIEW/dca1d
>>>>
>>>> {
>>>>
>>>> "id":"dca1d"
>>>>
>>>> ,"simpleFields":{
>>>>
>>>> "BUCKET_SIZE":"0"
>>>>
>>>> ,"IDEAL_STATE_MODE":"CUSTOMIZED"
>>>>
>>>> ,"MAX_PARTITIONS_PER_INSTANCE":"1"
>>>>
>>>> ,"NUM_PARTITIONS":"2"
>>>>
>>>> ,"REBALANCE_MODE":"CUSTOMIZED"
>>>>
>>>> ,"REPLICAS":"1"
>>>>
>>>> ,"STATE_MODEL_DEF_REF":"MyStateModel"
>>>>
>>>> ,"STATE_MODEL_FACTORY_NAME":"DEFAULT"
>>>>
>>>> }
>>>>
>>>> ,"listFields":{
>>>>
>>>> }
>>>>
>>>> ,"mapFields":{
>>>>
>>>> "0":{
>>>>
>>>> "localhost_12004":"ONLINE"
>>>>
>>>> }
>>>>
>>>> ,"1":{
>>>>
>>>> "localhost_12005":"ONLINE"
>>>>
>>>> }
>>>>
>>>> }
>>>>
>>>> }
>>>>
>>>>
>>>> I see that the mapFields key changed after using "USER_DEFINED"
>>>> rebalancer.
>>>>
>>>> Here are the code i used:
>>>>
>>>> public static void addResource(String resource, int num) {
>>>> clusterSetup.addResourceToCluster(CLUSTER_NAME, resource, num,
>>>> STATE_MODEL_NAME);
>>>> clusterSetup.addResourceProperty(CLUSTER_NAME, resource,
>>>> IdealState.IdealStateProperty.REBALANCER_CLASS_NAME.toString(),
>>>> CustomizedRebalancer.class.getName());
>>>> clusterSetup.addResourceProperty(CLUSTER_NAME, resource,
>>>> IdealState.IdealStateProperty.REBALANCE_MODE.toString(),
>>>> IdealState.RebalanceMode.USER_DEFINED.toString());
>>>> clusterSetup.rebalanceStorageCluster(CLUSTER_NAME, resource, 1, "");
>>>>
>>>>
>>>> What is the right way to set a rebalancer?
>>>>
>>>>
>>>> The rebalancer method has a single resource interface. What is the
>>>> right way to rebalance all resources in the cluster for any event change in
>>>> the cluster? My concern is the sequence and thread safety.
>>>>
>>>> @Override
>>>> public IdealState computeNewIdealState(String resourceName, IdealState
>>>> currentIdealState,
>>>> CurrentStateOutput currentStateOutput, ClusterDataCache clusterData) {
>>>>
>>>>
>>>>
>>>> Regards,
>>>>
>>>> Leo
>>>>
>>>>
>>>>
>>>> On Fri, Oct 30, 2020 at 10:21 PM Wang Jiajun <[email protected]>
>>>> wrote:
>>>>
>>>>> Hi Leo,
>>>>>
>>>>> Are you looking for guidance on how to configure the IdealState so the
>>>>> resource will be rebalanced by your user_defined rebalancer? Maybe you can
>>>>> refer to this test case TestCustomizedIdealStateRebalancer.
>>>>>
>>>>> Best Regards,
>>>>> Jiajun
>>>>>
>>>>>
>>>>> On Fri, Oct 30, 2020 at 10:11 PM Xue Junkai <[email protected]> wrote:
>>>>>
>>>>>> Hi Leo,
>>>>>>
>>>>>> What kind of examples are you looking for? Participant side? Or how
>>>>>> to build IdealState?
>>>>>>
>>>>>> Best,
>>>>>>
>>>>>> Junkai
>>>>>>
>>>>>> On Fri, Oct 30, 2020 at 9:57 PM Leo, Li <[email protected]> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>> I am a new user of Helix. I wonder if you could give an example
>>>>>>> code of using this user_defined model
>>>>>>> https://helix.apache.org/1.0.1-docs/tutorial_user_def_rebalancer.html
>>>>>>> ?
>>>>>>>
>>>>>>> Thank you,
>>>>>>> Leo
>>>>>>>
>>>>>>
