Hi Kishore, I updated my code, will run it over the night (my side) to see how to runs; Thanks again for your help!
2016-02-05 21:09 GMT+08:00 kishore g <[email protected]>: > oh, looks like you are not removing the entries from Idealstate. > > You can do this. > > Idealstate is = helixadmin.getResourceIdealState(clutername, resourcename). > > //update the idealstate > List<String> partitionsToDrop = .... > for(String partition:partitionsToDrop){ > is.getrecord().getListFields().remove(partition) > is.getrecord().getMapFields().remove(partition) > } > > is.setNumPartitions(is.getRecord().getMapFields().size()); > > can you try this. > > thanks > Kishore G > > > > On Fri, Feb 5, 2016 at 4:52 AM, ShaoFeng Shi <[email protected]> > wrote: > >> I see there is a JIRA HELIX-132 >> <https://issues.apache.org/jira/browse/HELIX-132> which has fixed that >> issue in 0.7.1; but from my case, after drop the resource and add back, the >> external view still has the old partitions (got from helix-admin.sh), >> interesting... >> >> 2016-02-05 10:49 GMT+08:00 ShaoFeng Shi <[email protected]>: >> >>> I didn't; But after add the transitions between Dropped and Offline, it >>> got the same behavior. >>> I also changed my code to drop the resource and create again when its >>> partition number reaches a limit; But the external view wasn't deleted, >>> that makes Helix thought all the partitions have already been propery >>> assigned; Following is a state after the resoruce be recreated, you can see >>> although its "NUM_PARTITIONS" is 8, the partitions in external view is 20 >>> (which is the up limit): >>> >>> IdealState for Resource_Stream: >>> { >>> "id" : "Resource_Stream", >>> "mapFields" : { >>> "Resource_Stream_0" : { >>> }, >>> "Resource_Stream_1" : { >>> }, >>> "Resource_Stream_10" : { >>> }, >>> "Resource_Stream_11" : { >>> }, >>> "Resource_Stream_12" : { >>> }, >>> "Resource_Stream_13" : { >>> }, >>> "Resource_Stream_14" : { >>> }, >>> "Resource_Stream_15" : { >>> }, >>> "Resource_Stream_16" : { >>> }, >>> "Resource_Stream_17" : { >>> }, >>> "Resource_Stream_18" : { >>> }, >>> "Resource_Stream_19" : { >>> }, >>> "Resource_Stream_2" : { >>> }, >>> "Resource_Stream_3" : { >>> }, >>> "Resource_Stream_4" : { >>> }, >>> "Resource_Stream_5" : { >>> }, >>> "Resource_Stream_6" : { >>> }, >>> "Resource_Stream_7" : { >>> }, >>> "Resource_Stream_8" : { >>> }, >>> "Resource_Stream_9" : { >>> } >>> }, >>> "listFields" : { >>> "Resource_Stream_0" : [ ], >>> "Resource_Stream_1" : [ ], >>> "Resource_Stream_10" : [ ], >>> "Resource_Stream_11" : [ ], >>> "Resource_Stream_12" : [ ], >>> "Resource_Stream_13" : [ ], >>> "Resource_Stream_14" : [ ], >>> "Resource_Stream_15" : [ ], >>> "Resource_Stream_16" : [ ], >>> "Resource_Stream_17" : [ ], >>> "Resource_Stream_18" : [ ], >>> "Resource_Stream_19" : [ ], >>> "Resource_Stream_2" : [ ], >>> "Resource_Stream_3" : [ ], >>> "Resource_Stream_4" : [ ], >>> "Resource_Stream_5" : [ ], >>> "Resource_Stream_6" : [ ], >>> "Resource_Stream_7" : [ ], >>> "Resource_Stream_8" : [ ], >>> "Resource_Stream_9" : [ ] >>> }, >>> "simpleFields" : { >>> "IDEAL_STATE_MODE" : "AUTO_REBALANCE", >>> "INSTANCE_GROUP_TAG" : "Tag_StreamBuilder", >>> "NUM_PARTITIONS" : "8", >>> "REBALANCE_MODE" : "FULL_AUTO", >>> "REPLICAS" : "2", >>> "STATE_MODEL_DEF_REF" : "LeaderStandby", >>> "STATE_MODEL_FACTORY_NAME" : "DEFAULT" >>> } >>> } >>> >>> ExternalView for Resource_Stream: >>> { >>> "id" : "Resource_Stream", >>> "mapFields" : { >>> "Resource_Stream_0" : { >>> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "LEADER", >>> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "STANDBY" >>> }, >>> "Resource_Stream_1" : { >>> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "STANDBY", >>> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "LEADER" >>> }, >>> "Resource_Stream_10" : { >>> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "LEADER", >>> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "STANDBY" >>> }, >>> "Resource_Stream_11" : { >>> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "STANDBY", >>> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "LEADER" >>> }, >>> "Resource_Stream_12" : { >>> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "LEADER", >>> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "STANDBY" >>> }, >>> "Resource_Stream_13" : { >>> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "STANDBY", >>> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "LEADER" >>> }, >>> "Resource_Stream_14" : { >>> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "LEADER", >>> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "STANDBY" >>> }, >>> "Resource_Stream_15" : { >>> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "STANDBY", >>> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "LEADER" >>> }, >>> "Resource_Stream_16" : { >>> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "LEADER", >>> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "STANDBY" >>> }, >>> "Resource_Stream_17" : { >>> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "STANDBY", >>> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "LEADER" >>> }, >>> "Resource_Stream_18" : { >>> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "LEADER", >>> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "STANDBY" >>> }, >>> "Resource_Stream_19" : { >>> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "STANDBY", >>> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "LEADER" >>> }, >>> "Resource_Stream_2" : { >>> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "LEADER", >>> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "STANDBY" >>> }, >>> "Resource_Stream_3" : { >>> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "STANDBY", >>> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "LEADER" >>> }, >>> "Resource_Stream_4" : { >>> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "LEADER", >>> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "STANDBY" >>> }, >>> "Resource_Stream_5" : { >>> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "STANDBY", >>> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "LEADER" >>> }, >>> "Resource_Stream_6" : { >>> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "LEADER", >>> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "STANDBY" >>> }, >>> "Resource_Stream_7" : { >>> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "STANDBY", >>> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "LEADER" >>> }, >>> "Resource_Stream_8" : { >>> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "LEADER", >>> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "STANDBY" >>> }, >>> "Resource_Stream_9" : { >>> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "STANDBY", >>> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "LEADER" >>> } >>> }, >>> "listFields" : { >>> }, >>> "simpleFields" : { >>> "BUCKET_SIZE" : "0" >>> } >>> } >>> >>> 2016-02-04 23:08 GMT+08:00 kishore g <[email protected]>: >>> >>>> Ha ave you defined offline to dropped state transition in your state >>>> transition handler. >>>> On Feb 3, 2016 11:31 PM, "ShaoFeng Shi" <[email protected]> wrote: >>>> >>>>> I found an issue: the numberOfPartitions can only be increased, and >>>>> couldn't be decreased; Even if I set it to 0, and then do a rebalance, the >>>>> old partitions still exist in idea view and external view; So, how can >>>>> descrease or drop a partition? >>>>> >>>>> >>>>> >>>>> 2016-02-04 15:28 GMT+08:00 ShaoFeng Shi <[email protected]>: >>>>> >>>>>> Nice, that you Kishore! >>>>>> >>>>>> 2016-02-04 10:35 GMT+08:00 kishore g <[email protected]>: >>>>>> >>>>>>> This is needed for another project I am working on. There is no >>>>>>> reason for Helix to depend on this convention. I will fix this. >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Wed, Feb 3, 2016 at 5:51 PM, ShaoFeng Shi <[email protected] >>>>>>> > wrote: >>>>>>> >>>>>>>> Hello, >>>>>>>> >>>>>>>> I'm trying to use Helix (0.7.1) to manage our resource partitions >>>>>>>> on a >>>>>>>> cluster. My scenario is, every 5 minutes a partition will be added; >>>>>>>> What I >>>>>>>> do now is, get the ideal state, +1 for the partition number and >>>>>>>> then update >>>>>>>> it with ZKHelixAdmin. With a rebalance action, the new partition >>>>>>>> will be >>>>>>>> created and assigned to an instance. What the instance can get from >>>>>>>> Helix >>>>>>>> is the resource name and partition id. To map the parition to my >>>>>>>> logical >>>>>>>> data, I maintained a mapping table in our datastore, which looks >>>>>>>> like: >>>>>>>> >>>>>>>> { >>>>>>>> "resource_0": 201601010000_ 201601010005, >>>>>>>> "resource_1": 201601010005_ 201601010010, >>>>>>>> ... >>>>>>>> "resource_11": 201601010055_ 201601010100 >>>>>>>> } >>>>>>>> >>>>>>>> Here it has 12 partitions; Now I want to discard some old >>>>>>>> partitions, say >>>>>>>> the first 6 partitions; It seems in Helix the partitions must >>>>>>>> start from >>>>>>>> 0, so with an update on the IdealState, set # of partitions to 6, >>>>>>>> the new >>>>>>>> partitions on the cluster would become to: >>>>>>>> >>>>>>>> resource_0,resource_1, ..., resource_5, >>>>>>>> >>>>>>>> To make sure the partitions wouldn't be wrongly mapped, I need >>>>>>>> update my >>>>>>>> mapping table before the rebalance. While that may not ensure the >>>>>>>> atomic >>>>>>>> between the two updates. >>>>>>>> >>>>>>>> So my question is, what's the suggested way to do the resource >>>>>>>> partition >>>>>>>> mapping? does Helix allow user to specify additional information on >>>>>>>> a >>>>>>>> partition (if have this, I don't need maintain the mapping >>>>>>>> outside)? Can we >>>>>>>> have some simple APIs like addPartition(String parititionName), >>>>>>>> dropParitition(String partitionName), just like that for resource? >>>>>>>> The >>>>>>>> numeric paritition id can be an internal ID and not exposed to user. >>>>>>>> >>>>>>>> I guess many entry users will have such questions. Just raise this >>>>>>>> for a >>>>>>>> broad discussion; Any comment and suggestion is welcomed. Thanks >>>>>>>> for your >>>>>>>> time! >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Best regards, >>>>>>>> >>>>>>>> Shaofeng Shi >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Best regards, >>>>>> >>>>>> Shaofeng Shi >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Best regards, >>>>> >>>>> Shaofeng Shi >>>>> >>>>> >>> >>> >>> -- >>> Best regards, >>> >>> Shaofeng Shi >>> >>> >> >> >> -- >> Best regards, >> >> Shaofeng Shi >> >> > -- Best regards, Shaofeng Shi
