I see and is it okay to set the bucket size for a resource which previously did not have buckets - would that also lead to bad behaviour (using setResourceIdealState)
Thanks Varun On Tue, Feb 24, 2015 at 10:27 AM, Zhen Zhang <[email protected]> wrote: > drop ideal state means drop the resource. > > yes. if you set a different bucket size, the old buckets will remains. > since the bucket size is changed, helix controller will not be able to read > the correct current state of each partition and considers them all be in > initial state. This will lead to controller sending incorrect transitions > to existing partitions. > > ------------------------------ > *From:* Varun Sharma [[email protected]] > *Sent:* Tuesday, February 24, 2015 12:37 AM > > *To:* [email protected] > *Subject:* Re: Bucketized resources not working in Helix 0.6.4 > > Do I need to drop the resource in order to drop the ideal state - also > wondering what the behaviour is if i set the bucket size using > setResourceIdealState - would it just keep the previous bucketing or bad > things could happen ? > > On Tue, Feb 24, 2015 at 12:20 AM, Zhen Zhang <[email protected]> wrote: > >> To get bucket size, get the ideal state and then getBucketSize() on it. >> If you want to change the bucket size, you may first drop the ideal state >> and re-create using the new bucket size. >> >> On Mon, Feb 23, 2015 at 3:10 PM, Varun Sharma <[email protected]> >> wrote: >> >>> Also, would changing the bucketSize throw an exception or would it cause >>> undefined behaviour or would the new bucket size simply be ignored ? This >>> is assuming the number of partitions remain the same in the >>> setResourceIdealState call ? >>> >>> Thanks ! >>> Varun >>> >>> On Mon, Feb 23, 2015 at 2:45 PM, Varun Sharma <[email protected]> >>> wrote: >>> >>>> Cool - thanks - for a pre existing resource, how can we retrieve the >>>> bucket size - do we need to pull in the whole ideal state and then run a >>>> getBucketSize() on it ? >>>> >>>> On Mon, Feb 23, 2015 at 2:42 PM, Zhen Zhang <[email protected]> >>>> wrote: >>>> >>>>> bucktSize is the size of each bucket, i.e. the number of partitions >>>>> each bucket will hold. bucket size can't be changed dynamically. >>>>> >>>>> ------------------------------ >>>>> *From:* Varun Sharma [[email protected]] >>>>> *Sent:* Monday, February 23, 2015 2:37 PM >>>>> >>>>> *To:* [email protected] >>>>> *Subject:* Re: Bucketized resources not working in Helix 0.6.4 >>>>> >>>>> OK - thanks. Btw, is bucket size the number of buckets or is it >>>>> the size of one bucket ? Also can the bucket size be changed dynamically >>>>> using setResourceIdealState ? >>>>> >>>>> On Mon, Feb 23, 2015 at 1:53 PM, Zhen Zhang <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi Varun, >>>>>> >>>>>> The partition number is mainly for calculating the buckets. >>>>>> Partition name will remain the same. You can always get and build Helix >>>>>> source code from here: >>>>>> https://github.com/apache/helix >>>>>> >>>>>> Changing the partition naming is not hard, but we need to come up >>>>>> with a general way to achieve this. We will think about this requirement. >>>>>> If you have any good idea for this, you can always submit a pull request >>>>>> on >>>>>> github. >>>>>> >>>>>> Thanks, >>>>>> Jason >>>>>> ------------------------------ >>>>>> *From:* Varun Sharma [[email protected]] >>>>>> *Sent:* Monday, February 23, 2015 12:04 PM >>>>>> >>>>>> *To:* [email protected] >>>>>> *Subject:* Re: Bucketized resources not working in Helix 0.6.4 >>>>>> >>>>>> How hard is this to change to '$' - its difficult for us to now >>>>>> change the partition names now. Is there a place to get the helix 0.6.4 >>>>>> source code, make changes and built it ? >>>>>> >>>>>> Another related question - is the partition number mainly needed >>>>>> for hashing into the buckets - the partition name would still continue to >>>>>> be "resource_name_partition_num", right ? >>>>>> >>>>>> On Mon, Feb 23, 2015 at 11:52 AM, Zhen Zhang <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> The bucketize logic is expecting partition name following the >>>>>>> convention {resource_name}_{partition_number}. In your case, you are >>>>>>> using >>>>>>> $ ? >>>>>>> ------------------------------ >>>>>>> *From:* Varun Sharma [[email protected]] >>>>>>> *Sent:* Monday, February 23, 2015 11:48 AM >>>>>>> >>>>>>> *To:* [email protected] >>>>>>> *Subject:* Re: Bucketized resources not working in Helix 0.6.4 >>>>>>> >>>>>>> Note that we had to prefix each partition with the resource name >>>>>>> to make the partitions unique. >>>>>>> >>>>>>> On Mon, Feb 23, 2015 at 11:48 AM, Varun Sharma <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Here is the latest error I am getting: >>>>>>>> >>>>>>>> 2015-02-23 19:44:48,153 [controller-pool-0] >>>>>>>> (TerrapinControllerServiceImpl.java:129) WARN Resource creation >>>>>>>> failed for >>>>>>>> $terrapin$data$pinpin_board_join$1424377884722, rolling back. >>>>>>>> >>>>>>>> *java.lang.IllegalArgumentException: Could NOT parse partition# >>>>>>>> (join$1424377884722$0) in >>>>>>>> $terrapin$data$pinpin_board_join$1424377884722$0* >>>>>>>> >>>>>>>> *at >>>>>>>> org.apache.helix.ZNRecordBucketizer.getBucketName(ZNRecordBucketizer.java:72)* >>>>>>>> >>>>>>>> *at >>>>>>>> org.apache.helix.ZNRecordBucketizer.bucketize(ZNRecordBucketizer.java:105)* >>>>>>>> >>>>>>>> at >>>>>>>> org.apache.helix.manager.zk.ZKHelixDataAccessor.setProperty(ZKHelixDataAccessor.java:110) >>>>>>>> >>>>>>>> at >>>>>>>> org.apache.helix.manager.zk.ZKHelixAdmin.setResourceIdealState(ZKHelixAdmin.java:686) >>>>>>>> >>>>>>>> On Mon, Feb 23, 2015 at 10:40 AM, Zhen Zhang <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Not really. The bucketize logic just creates child znode under >>>>>>>>> the original ideal state, and each of the child znode holds a subset >>>>>>>>> of the >>>>>>>>> total partitions. >>>>>>>>> ------------------------------ >>>>>>>>> *From:* Varun Sharma [[email protected]] >>>>>>>>> *Sent:* Monday, February 23, 2015 10:37 AM >>>>>>>>> >>>>>>>>> *To:* [email protected] >>>>>>>>> *Subject:* Re: Bucketized resources not working in Helix 0.6.4 >>>>>>>>> >>>>>>>>> One other question - would bucketizing add anything to the >>>>>>>>> partition name - just worried about any special symbols (we use _ and >>>>>>>>> $) >>>>>>>>> causing issues there ? >>>>>>>>> >>>>>>>>> On Mon, Feb 23, 2015 at 10:35 AM, Zhen Zhang <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi Varun, >>>>>>>>>> >>>>>>>>>> HelixAdmin#addResource was initially used for adding an empty >>>>>>>>>> resource only, followed by HelixAdmin#rebalance, which bucketize the >>>>>>>>>> ideal >>>>>>>>>> state, so the bucketize logic is missing from addResource. We should >>>>>>>>>> definitely fix this. We are also considering merging addResource and >>>>>>>>>> rebalance, since separating them are confusing. For now you may use >>>>>>>>>> setResourceIdealState as a workaround. >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Zhen >>>>>>>>>> >>>>>>>>>> ------------------------------ >>>>>>>>>> *From:* Varun Sharma [[email protected]] >>>>>>>>>> *Sent:* Monday, February 23, 2015 10:20 AM >>>>>>>>>> *To:* [email protected] >>>>>>>>>> *Subject:* Re: Bucketized resources not working in Helix 0.6.4 >>>>>>>>>> >>>>>>>>>> Thanks, Zhen. >>>>>>>>>> >>>>>>>>>> What about just calling addResource with an idealState with the >>>>>>>>>> bucket size - is that expected to not work on its own ? >>>>>>>>>> >>>>>>>>>> Varun >>>>>>>>>> >>>>>>>>>> On Mon, Feb 23, 2015 at 10:18 AM, Zhen Zhang <[email protected] >>>>>>>>>> > wrote: >>>>>>>>>> >>>>>>>>>>> Hi Varun, >>>>>>>>>>> >>>>>>>>>>> Did you do the following? >>>>>>>>>>> HelixAdmin#addResource() // set bucket size >>>>>>>>>>> HelixAdmin#setResourceIdealState() // no bucket size >>>>>>>>>>> >>>>>>>>>>> When you set the ideal state, you still need set the bucket size >>>>>>>>>>> in the ideal state. This value will be used to bucketize the ideal >>>>>>>>>>> state. >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> Jason >>>>>>>>>>> >>>>>>>>>>> ________________________________________ >>>>>>>>>>> From: Varun Sharma [[email protected]] >>>>>>>>>>> Sent: Monday, February 23, 2015 1:51 AM >>>>>>>>>>> To: [email protected] >>>>>>>>>>> Subject: Bucketized resources not working in Helix 0.6.4 >>>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> I am trying to bucketize a resource on Helix 0.6.4 - however, am >>>>>>>>>>> unable to do so. I am setting the bucketSize in the idealState and >>>>>>>>>>> using >>>>>>>>>>> addResource() and setResourceIdealState and it ends up complaining >>>>>>>>>>> that ZK >>>>>>>>>>> size of 1M has been exceeded. >>>>>>>>>>> >>>>>>>>>>> I also tried using the addResource() api which creates an empty >>>>>>>>>>> ideal state with a valid bucket size. However, when I tried to set >>>>>>>>>>> the >>>>>>>>>>> ideal state on that bucket (this ideal state did not have the >>>>>>>>>>> bucket size >>>>>>>>>>> set), it failed as well. >>>>>>>>>>> >>>>>>>>>>> Whats the right way to get bucketizing to work ? >>>>>>>>>>> >>>>>>>>>>> Thanks ! >>>>>>>>>>> Varun >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
