I see there is a JIRA HELIX-132 <https://issues.apache.org/jira/browse/HELIX-132> which has fixed that issue in 0.7.1; but from my case, after drop the resource and add back, the external view still has the old partitions (got from helix-admin.sh), interesting...
2016-02-05 10:49 GMT+08:00 ShaoFeng Shi <[email protected]>: > I didn't; But after add the transitions between Dropped and Offline, it > got the same behavior. > I also changed my code to drop the resource and create again when its > partition number reaches a limit; But the external view wasn't deleted, > that makes Helix thought all the partitions have already been propery > assigned; Following is a state after the resoruce be recreated, you can see > although its "NUM_PARTITIONS" is 8, the partitions in external view is 20 > (which is the up limit): > > IdealState for Resource_Stream: > { > "id" : "Resource_Stream", > "mapFields" : { > "Resource_Stream_0" : { > }, > "Resource_Stream_1" : { > }, > "Resource_Stream_10" : { > }, > "Resource_Stream_11" : { > }, > "Resource_Stream_12" : { > }, > "Resource_Stream_13" : { > }, > "Resource_Stream_14" : { > }, > "Resource_Stream_15" : { > }, > "Resource_Stream_16" : { > }, > "Resource_Stream_17" : { > }, > "Resource_Stream_18" : { > }, > "Resource_Stream_19" : { > }, > "Resource_Stream_2" : { > }, > "Resource_Stream_3" : { > }, > "Resource_Stream_4" : { > }, > "Resource_Stream_5" : { > }, > "Resource_Stream_6" : { > }, > "Resource_Stream_7" : { > }, > "Resource_Stream_8" : { > }, > "Resource_Stream_9" : { > } > }, > "listFields" : { > "Resource_Stream_0" : [ ], > "Resource_Stream_1" : [ ], > "Resource_Stream_10" : [ ], > "Resource_Stream_11" : [ ], > "Resource_Stream_12" : [ ], > "Resource_Stream_13" : [ ], > "Resource_Stream_14" : [ ], > "Resource_Stream_15" : [ ], > "Resource_Stream_16" : [ ], > "Resource_Stream_17" : [ ], > "Resource_Stream_18" : [ ], > "Resource_Stream_19" : [ ], > "Resource_Stream_2" : [ ], > "Resource_Stream_3" : [ ], > "Resource_Stream_4" : [ ], > "Resource_Stream_5" : [ ], > "Resource_Stream_6" : [ ], > "Resource_Stream_7" : [ ], > "Resource_Stream_8" : [ ], > "Resource_Stream_9" : [ ] > }, > "simpleFields" : { > "IDEAL_STATE_MODE" : "AUTO_REBALANCE", > "INSTANCE_GROUP_TAG" : "Tag_StreamBuilder", > "NUM_PARTITIONS" : "8", > "REBALANCE_MODE" : "FULL_AUTO", > "REPLICAS" : "2", > "STATE_MODEL_DEF_REF" : "LeaderStandby", > "STATE_MODEL_FACTORY_NAME" : "DEFAULT" > } > } > > ExternalView for Resource_Stream: > { > "id" : "Resource_Stream", > "mapFields" : { > "Resource_Stream_0" : { > "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "LEADER", > "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "STANDBY" > }, > "Resource_Stream_1" : { > "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "STANDBY", > "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "LEADER" > }, > "Resource_Stream_10" : { > "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "LEADER", > "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "STANDBY" > }, > "Resource_Stream_11" : { > "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "STANDBY", > "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "LEADER" > }, > "Resource_Stream_12" : { > "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "LEADER", > "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "STANDBY" > }, > "Resource_Stream_13" : { > "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "STANDBY", > "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "LEADER" > }, > "Resource_Stream_14" : { > "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "LEADER", > "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "STANDBY" > }, > "Resource_Stream_15" : { > "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "STANDBY", > "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "LEADER" > }, > "Resource_Stream_16" : { > "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "LEADER", > "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "STANDBY" > }, > "Resource_Stream_17" : { > "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "STANDBY", > "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "LEADER" > }, > "Resource_Stream_18" : { > "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "LEADER", > "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "STANDBY" > }, > "Resource_Stream_19" : { > "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "STANDBY", > "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "LEADER" > }, > "Resource_Stream_2" : { > "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "LEADER", > "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "STANDBY" > }, > "Resource_Stream_3" : { > "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "STANDBY", > "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "LEADER" > }, > "Resource_Stream_4" : { > "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "LEADER", > "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "STANDBY" > }, > "Resource_Stream_5" : { > "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "STANDBY", > "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "LEADER" > }, > "Resource_Stream_6" : { > "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "LEADER", > "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "STANDBY" > }, > "Resource_Stream_7" : { > "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "STANDBY", > "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "LEADER" > }, > "Resource_Stream_8" : { > "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "LEADER", > "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "STANDBY" > }, > "Resource_Stream_9" : { > "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "STANDBY", > "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "LEADER" > } > }, > "listFields" : { > }, > "simpleFields" : { > "BUCKET_SIZE" : "0" > } > } > > 2016-02-04 23:08 GMT+08:00 kishore g <[email protected]>: > >> Ha ave you defined offline to dropped state transition in your state >> transition handler. >> On Feb 3, 2016 11:31 PM, "ShaoFeng Shi" <[email protected]> wrote: >> >>> I found an issue: the numberOfPartitions can only be increased, and >>> couldn't be decreased; Even if I set it to 0, and then do a rebalance, the >>> old partitions still exist in idea view and external view; So, how can >>> descrease or drop a partition? >>> >>> >>> >>> 2016-02-04 15:28 GMT+08:00 ShaoFeng Shi <[email protected]>: >>> >>>> Nice, that you Kishore! >>>> >>>> 2016-02-04 10:35 GMT+08:00 kishore g <[email protected]>: >>>> >>>>> This is needed for another project I am working on. There is no reason >>>>> for Helix to depend on this convention. I will fix this. >>>>> >>>>> >>>>> >>>>> On Wed, Feb 3, 2016 at 5:51 PM, ShaoFeng Shi <[email protected]> >>>>> wrote: >>>>> >>>>>> Hello, >>>>>> >>>>>> I'm trying to use Helix (0.7.1) to manage our resource partitions on a >>>>>> cluster. My scenario is, every 5 minutes a partition will be added; >>>>>> What I >>>>>> do now is, get the ideal state, +1 for the partition number and then >>>>>> update >>>>>> it with ZKHelixAdmin. With a rebalance action, the new partition will >>>>>> be >>>>>> created and assigned to an instance. What the instance can get from >>>>>> Helix >>>>>> is the resource name and partition id. To map the parition to my >>>>>> logical >>>>>> data, I maintained a mapping table in our datastore, which looks like: >>>>>> >>>>>> { >>>>>> "resource_0": 201601010000_ 201601010005, >>>>>> "resource_1": 201601010005_ 201601010010, >>>>>> ... >>>>>> "resource_11": 201601010055_ 201601010100 >>>>>> } >>>>>> >>>>>> Here it has 12 partitions; Now I want to discard some old partitions, >>>>>> say >>>>>> the first 6 partitions; It seems in Helix the partitions must start >>>>>> from >>>>>> 0, so with an update on the IdealState, set # of partitions to 6, the >>>>>> new >>>>>> partitions on the cluster would become to: >>>>>> >>>>>> resource_0,resource_1, ..., resource_5, >>>>>> >>>>>> To make sure the partitions wouldn't be wrongly mapped, I need update >>>>>> my >>>>>> mapping table before the rebalance. While that may not ensure the >>>>>> atomic >>>>>> between the two updates. >>>>>> >>>>>> So my question is, what's the suggested way to do the resource >>>>>> partition >>>>>> mapping? does Helix allow user to specify additional information on a >>>>>> partition (if have this, I don't need maintain the mapping outside)? >>>>>> Can we >>>>>> have some simple APIs like addPartition(String parititionName), >>>>>> dropParitition(String partitionName), just like that for resource? The >>>>>> numeric paritition id can be an internal ID and not exposed to user. >>>>>> >>>>>> I guess many entry users will have such questions. Just raise this >>>>>> for a >>>>>> broad discussion; Any comment and suggestion is welcomed. Thanks for >>>>>> your >>>>>> time! >>>>>> >>>>>> >>>>>> -- >>>>>> Best regards, >>>>>> >>>>>> Shaofeng Shi >>>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> Best regards, >>>> >>>> Shaofeng Shi >>>> >>>> >>> >>> >>> -- >>> Best regards, >>> >>> Shaofeng Shi >>> >>> > > > -- > Best regards, > > Shaofeng Shi > > -- Best regards, Shaofeng Shi
