Some resources won't become online

[Grainier Perera]

Hi Devs,

I'm trying to add several resources to the cluster using the following
configurations[1]. However, only some will become `ONLINE`. What could be
the reason? Is there a way to guarantee every resource will become `ONLINE`
if WAGED capacity constraints are met?

You can see with the same IdealState, "_mm:root:_system:cron3" has mapFields
and it is ONLINE, and "_mm:root:_system:cron2" is not. Furthermore, I see
this behavior more often when the replicas count is set to 1.

ResourceInfo:
1. "_mm:root:_system:cron2"

IdealState for _mm:root:_system:cron2:
{
  "id" : "_mm:root:_system:cron2",
  "simpleFields" : {
    "DELAY_REBALANCE_ENABLED" : "true",
    "IDEAL_STATE_MODE" : "AUTO_REBALANCE",
    "MAX_PARTITIONS_PER_INSTANCE" : "1",
    "NUM_PARTITIONS" : "1",
    "REBALANCER_CLASS_NAME" :
"org.apache.helix.controller.rebalancer.waged.WagedRebalancer",
    "REBALANCE_DELAY" : "10000",
    "REBALANCE_MODE" : "FULL_AUTO",
    "REPLICAS" : "1",
    "STATE_MODEL_DEF_REF" : "C8CEPStateModel"
  },
  "mapFields" : {
    "_mm:root:_system:cron2_0" : { }
  },
  "listFields" : {
    "_mm:root:_system:cron2_0" : [ ]
  }
}


ExternalView for _mm:root:_system:cron2:
{
  "id" : "_mm:root:_system:cron2",
  "simpleFields" : {
    "BUCKET_SIZE" : "0",
    "DELAY_REBALANCE_ENABLED" : "true",
    "IDEAL_STATE_MODE" : "AUTO_REBALANCE",
    "MAX_PARTITIONS_PER_INSTANCE" : "1",
    "NUM_PARTITIONS" : "1",
    "REBALANCER_CLASS_NAME" :
"org.apache.helix.controller.rebalancer.waged.WagedRebalancer",
    "REBALANCE_DELAY" : "10000",
    "REBALANCE_MODE" : "FULL_AUTO",
    "REPLICAS" : "1",
    "STATE_MODEL_DEF_REF" : "C8CEPStateModel"
  },
  *"mapFields" : { },*
  "listFields" : { }
}


2. "_mm:root:_system:cron3"

IdealState for _mm:root:_system:cron3:
{
  "id" : "_mm:root:_system:cron3",
  "simpleFields" : {
    "DELAY_REBALANCE_ENABLED" : "true",
    "IDEAL_STATE_MODE" : "AUTO_REBALANCE",
    "MAX_PARTITIONS_PER_INSTANCE" : "1",
    "NUM_PARTITIONS" : "1",
    "REBALANCER_CLASS_NAME" :
"org.apache.helix.controller.rebalancer.waged.WagedRebalancer",
    "REBALANCE_DELAY" : "10000",
    "REBALANCE_MODE" : "FULL_AUTO",
    "REPLICAS" : "1",
    "STATE_MODEL_DEF_REF" : "C8CEPStateModel"
  },
  "mapFields" : {
    "_mm:root:_system:cron3_0" : { }
  },
  "listFields" : {
    "_mm:root:_system:cron3_0" : [ ]
  }
}


ExternalView for _mm:root:_system:cron3:
{
  "id" : "_mm:root:_system:cron3",
  "simpleFields" : {
    "BUCKET_SIZE" : "0",
    "DELAY_REBALANCE_ENABLED" : "true",
    "IDEAL_STATE_MODE" : "AUTO_REBALANCE",
    "MAX_PARTITIONS_PER_INSTANCE" : "1",
    "NUM_PARTITIONS" : "1",
    "REBALANCER_CLASS_NAME" :
"org.apache.helix.controller.rebalancer.waged.WagedRebalancer",
    "REBALANCE_DELAY" : "10000",
    "REBALANCE_MODE" : "FULL_AUTO",
    "REPLICAS" : "1",
    "STATE_MODEL_DEF_REF" : "C8CEPStateModel"
  },
  *"mapFields" : {*
*    "_mm:root:_system:cron3_0" : {*
*      "c8cep-0.c8cep.c8.svc.cluster.local_12000" : "ONLINE"*
*    }*
*  },*
  "listFields" : { }
}


[1]: https://gist.github.com/grainier/aa1c0b279ea99f88d74c1e94d79f5cdb

Thank you.
Grainier Perera.