Do I need to drop the resource in order to drop the ideal state - also wondering what the behaviour is if i set the bucket size using setResourceIdealState - would it just keep the previous bucketing or bad things could happen ?
On Tue, Feb 24, 2015 at 12:20 AM, Zhen Zhang <[email protected]> wrote: > To get bucket size, get the ideal state and then getBucketSize() on it. If > you want to change the bucket size, you may first drop the ideal state and > re-create using the new bucket size. > > On Mon, Feb 23, 2015 at 3:10 PM, Varun Sharma <[email protected]> wrote: > >> Also, would changing the bucketSize throw an exception or would it cause >> undefined behaviour or would the new bucket size simply be ignored ? This >> is assuming the number of partitions remain the same in the >> setResourceIdealState call ? >> >> Thanks ! >> Varun >> >> On Mon, Feb 23, 2015 at 2:45 PM, Varun Sharma <[email protected]> >> wrote: >> >>> Cool - thanks - for a pre existing resource, how can we retrieve the >>> bucket size - do we need to pull in the whole ideal state and then run a >>> getBucketSize() on it ? >>> >>> On Mon, Feb 23, 2015 at 2:42 PM, Zhen Zhang <[email protected]> wrote: >>> >>>> bucktSize is the size of each bucket, i.e. the number of partitions >>>> each bucket will hold. bucket size can't be changed dynamically. >>>> >>>> ------------------------------ >>>> *From:* Varun Sharma [[email protected]] >>>> *Sent:* Monday, February 23, 2015 2:37 PM >>>> >>>> *To:* [email protected] >>>> *Subject:* Re: Bucketized resources not working in Helix 0.6.4 >>>> >>>> OK - thanks. Btw, is bucket size the number of buckets or is it the >>>> size of one bucket ? Also can the bucket size be changed dynamically using >>>> setResourceIdealState ? >>>> >>>> On Mon, Feb 23, 2015 at 1:53 PM, Zhen Zhang <[email protected]> >>>> wrote: >>>> >>>>> Hi Varun, >>>>> >>>>> The partition number is mainly for calculating the buckets. >>>>> Partition name will remain the same. You can always get and build Helix >>>>> source code from here: >>>>> https://github.com/apache/helix >>>>> >>>>> Changing the partition naming is not hard, but we need to come up >>>>> with a general way to achieve this. We will think about this requirement. >>>>> If you have any good idea for this, you can always submit a pull request >>>>> on >>>>> github. >>>>> >>>>> Thanks, >>>>> Jason >>>>> ------------------------------ >>>>> *From:* Varun Sharma [[email protected]] >>>>> *Sent:* Monday, February 23, 2015 12:04 PM >>>>> >>>>> *To:* [email protected] >>>>> *Subject:* Re: Bucketized resources not working in Helix 0.6.4 >>>>> >>>>> How hard is this to change to '$' - its difficult for us to now >>>>> change the partition names now. Is there a place to get the helix 0.6.4 >>>>> source code, make changes and built it ? >>>>> >>>>> Another related question - is the partition number mainly needed for >>>>> hashing into the buckets - the partition name would still continue to be >>>>> "resource_name_partition_num", right ? >>>>> >>>>> On Mon, Feb 23, 2015 at 11:52 AM, Zhen Zhang <[email protected]> >>>>> wrote: >>>>> >>>>>> The bucketize logic is expecting partition name following the >>>>>> convention {resource_name}_{partition_number}. In your case, you are >>>>>> using >>>>>> $ ? >>>>>> ------------------------------ >>>>>> *From:* Varun Sharma [[email protected]] >>>>>> *Sent:* Monday, February 23, 2015 11:48 AM >>>>>> >>>>>> *To:* [email protected] >>>>>> *Subject:* Re: Bucketized resources not working in Helix 0.6.4 >>>>>> >>>>>> Note that we had to prefix each partition with the resource name >>>>>> to make the partitions unique. >>>>>> >>>>>> On Mon, Feb 23, 2015 at 11:48 AM, Varun Sharma <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Here is the latest error I am getting: >>>>>>> >>>>>>> 2015-02-23 19:44:48,153 [controller-pool-0] >>>>>>> (TerrapinControllerServiceImpl.java:129) WARN Resource creation failed >>>>>>> for >>>>>>> $terrapin$data$pinpin_board_join$1424377884722, rolling back. >>>>>>> >>>>>>> *java.lang.IllegalArgumentException: Could NOT parse partition# >>>>>>> (join$1424377884722$0) in >>>>>>> $terrapin$data$pinpin_board_join$1424377884722$0* >>>>>>> >>>>>>> *at >>>>>>> org.apache.helix.ZNRecordBucketizer.getBucketName(ZNRecordBucketizer.java:72)* >>>>>>> >>>>>>> *at >>>>>>> org.apache.helix.ZNRecordBucketizer.bucketize(ZNRecordBucketizer.java:105)* >>>>>>> >>>>>>> at >>>>>>> org.apache.helix.manager.zk.ZKHelixDataAccessor.setProperty(ZKHelixDataAccessor.java:110) >>>>>>> >>>>>>> at >>>>>>> org.apache.helix.manager.zk.ZKHelixAdmin.setResourceIdealState(ZKHelixAdmin.java:686) >>>>>>> >>>>>>> On Mon, Feb 23, 2015 at 10:40 AM, Zhen Zhang <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Not really. The bucketize logic just creates child znode under >>>>>>>> the original ideal state, and each of the child znode holds a subset >>>>>>>> of the >>>>>>>> total partitions. >>>>>>>> ------------------------------ >>>>>>>> *From:* Varun Sharma [[email protected]] >>>>>>>> *Sent:* Monday, February 23, 2015 10:37 AM >>>>>>>> >>>>>>>> *To:* [email protected] >>>>>>>> *Subject:* Re: Bucketized resources not working in Helix 0.6.4 >>>>>>>> >>>>>>>> One other question - would bucketizing add anything to the >>>>>>>> partition name - just worried about any special symbols (we use _ and >>>>>>>> $) >>>>>>>> causing issues there ? >>>>>>>> >>>>>>>> On Mon, Feb 23, 2015 at 10:35 AM, Zhen Zhang <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi Varun, >>>>>>>>> >>>>>>>>> HelixAdmin#addResource was initially used for adding an empty >>>>>>>>> resource only, followed by HelixAdmin#rebalance, which bucketize the >>>>>>>>> ideal >>>>>>>>> state, so the bucketize logic is missing from addResource. We should >>>>>>>>> definitely fix this. We are also considering merging addResource and >>>>>>>>> rebalance, since separating them are confusing. For now you may use >>>>>>>>> setResourceIdealState as a workaround. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Zhen >>>>>>>>> >>>>>>>>> ------------------------------ >>>>>>>>> *From:* Varun Sharma [[email protected]] >>>>>>>>> *Sent:* Monday, February 23, 2015 10:20 AM >>>>>>>>> *To:* [email protected] >>>>>>>>> *Subject:* Re: Bucketized resources not working in Helix 0.6.4 >>>>>>>>> >>>>>>>>> Thanks, Zhen. >>>>>>>>> >>>>>>>>> What about just calling addResource with an idealState with the >>>>>>>>> bucket size - is that expected to not work on its own ? >>>>>>>>> >>>>>>>>> Varun >>>>>>>>> >>>>>>>>> On Mon, Feb 23, 2015 at 10:18 AM, Zhen Zhang <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi Varun, >>>>>>>>>> >>>>>>>>>> Did you do the following? >>>>>>>>>> HelixAdmin#addResource() // set bucket size >>>>>>>>>> HelixAdmin#setResourceIdealState() // no bucket size >>>>>>>>>> >>>>>>>>>> When you set the ideal state, you still need set the bucket size >>>>>>>>>> in the ideal state. This value will be used to bucketize the ideal >>>>>>>>>> state. >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Jason >>>>>>>>>> >>>>>>>>>> ________________________________________ >>>>>>>>>> From: Varun Sharma [[email protected]] >>>>>>>>>> Sent: Monday, February 23, 2015 1:51 AM >>>>>>>>>> To: [email protected] >>>>>>>>>> Subject: Bucketized resources not working in Helix 0.6.4 >>>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> I am trying to bucketize a resource on Helix 0.6.4 - however, am >>>>>>>>>> unable to do so. I am setting the bucketSize in the idealState and >>>>>>>>>> using >>>>>>>>>> addResource() and setResourceIdealState and it ends up complaining >>>>>>>>>> that ZK >>>>>>>>>> size of 1M has been exceeded. >>>>>>>>>> >>>>>>>>>> I also tried using the addResource() api which creates an empty >>>>>>>>>> ideal state with a valid bucket size. However, when I tried to set >>>>>>>>>> the >>>>>>>>>> ideal state on that bucket (this ideal state did not have the bucket >>>>>>>>>> size >>>>>>>>>> set), it failed as well. >>>>>>>>>> >>>>>>>>>> Whats the right way to get bucketizing to work ? >>>>>>>>>> >>>>>>>>>> Thanks ! >>>>>>>>>> Varun >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
