Also, would changing the bucketSize throw an exception or would it cause undefined behaviour or would the new bucket size simply be ignored ? This is assuming the number of partitions remain the same in the setResourceIdealState call ?
Thanks ! Varun On Mon, Feb 23, 2015 at 2:45 PM, Varun Sharma <[email protected]> wrote: > Cool - thanks - for a pre existing resource, how can we retrieve the > bucket size - do we need to pull in the whole ideal state and then run a > getBucketSize() on it ? > > On Mon, Feb 23, 2015 at 2:42 PM, Zhen Zhang <[email protected]> wrote: > >> bucktSize is the size of each bucket, i.e. the number of partitions >> each bucket will hold. bucket size can't be changed dynamically. >> >> ------------------------------ >> *From:* Varun Sharma [[email protected]] >> *Sent:* Monday, February 23, 2015 2:37 PM >> >> *To:* [email protected] >> *Subject:* Re: Bucketized resources not working in Helix 0.6.4 >> >> OK - thanks. Btw, is bucket size the number of buckets or is it the >> size of one bucket ? Also can the bucket size be changed dynamically using >> setResourceIdealState ? >> >> On Mon, Feb 23, 2015 at 1:53 PM, Zhen Zhang <[email protected]> wrote: >> >>> Hi Varun, >>> >>> The partition number is mainly for calculating the buckets. Partition >>> name will remain the same. You can always get and build Helix source code >>> from here: >>> https://github.com/apache/helix >>> >>> Changing the partition naming is not hard, but we need to come up with >>> a general way to achieve this. We will think about this requirement. If you >>> have any good idea for this, you can always submit a pull request on github. >>> >>> Thanks, >>> Jason >>> ------------------------------ >>> *From:* Varun Sharma [[email protected]] >>> *Sent:* Monday, February 23, 2015 12:04 PM >>> >>> *To:* [email protected] >>> *Subject:* Re: Bucketized resources not working in Helix 0.6.4 >>> >>> How hard is this to change to '$' - its difficult for us to now >>> change the partition names now. Is there a place to get the helix 0.6.4 >>> source code, make changes and built it ? >>> >>> Another related question - is the partition number mainly needed for >>> hashing into the buckets - the partition name would still continue to be >>> "resource_name_partition_num", right ? >>> >>> On Mon, Feb 23, 2015 at 11:52 AM, Zhen Zhang <[email protected]> >>> wrote: >>> >>>> The bucketize logic is expecting partition name following the >>>> convention {resource_name}_{partition_number}. In your case, you are using >>>> $ ? >>>> ------------------------------ >>>> *From:* Varun Sharma [[email protected]] >>>> *Sent:* Monday, February 23, 2015 11:48 AM >>>> >>>> *To:* [email protected] >>>> *Subject:* Re: Bucketized resources not working in Helix 0.6.4 >>>> >>>> Note that we had to prefix each partition with the resource name to >>>> make the partitions unique. >>>> >>>> On Mon, Feb 23, 2015 at 11:48 AM, Varun Sharma <[email protected]> >>>> wrote: >>>> >>>>> Here is the latest error I am getting: >>>>> >>>>> 2015-02-23 19:44:48,153 [controller-pool-0] >>>>> (TerrapinControllerServiceImpl.java:129) WARN Resource creation failed >>>>> for >>>>> $terrapin$data$pinpin_board_join$1424377884722, rolling back. >>>>> >>>>> *java.lang.IllegalArgumentException: Could NOT parse partition# >>>>> (join$1424377884722$0) in >>>>> $terrapin$data$pinpin_board_join$1424377884722$0* >>>>> >>>>> *at >>>>> org.apache.helix.ZNRecordBucketizer.getBucketName(ZNRecordBucketizer.java:72)* >>>>> >>>>> *at >>>>> org.apache.helix.ZNRecordBucketizer.bucketize(ZNRecordBucketizer.java:105)* >>>>> >>>>> at >>>>> org.apache.helix.manager.zk.ZKHelixDataAccessor.setProperty(ZKHelixDataAccessor.java:110) >>>>> >>>>> at >>>>> org.apache.helix.manager.zk.ZKHelixAdmin.setResourceIdealState(ZKHelixAdmin.java:686) >>>>> >>>>> On Mon, Feb 23, 2015 at 10:40 AM, Zhen Zhang <[email protected]> >>>>> wrote: >>>>> >>>>>> Not really. The bucketize logic just creates child znode under the >>>>>> original ideal state, and each of the child znode holds a subset of the >>>>>> total partitions. >>>>>> ------------------------------ >>>>>> *From:* Varun Sharma [[email protected]] >>>>>> *Sent:* Monday, February 23, 2015 10:37 AM >>>>>> >>>>>> *To:* [email protected] >>>>>> *Subject:* Re: Bucketized resources not working in Helix 0.6.4 >>>>>> >>>>>> One other question - would bucketizing add anything to the >>>>>> partition name - just worried about any special symbols (we use _ and $) >>>>>> causing issues there ? >>>>>> >>>>>> On Mon, Feb 23, 2015 at 10:35 AM, Zhen Zhang <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hi Varun, >>>>>>> >>>>>>> HelixAdmin#addResource was initially used for adding an empty >>>>>>> resource only, followed by HelixAdmin#rebalance, which bucketize the >>>>>>> ideal >>>>>>> state, so the bucketize logic is missing from addResource. We should >>>>>>> definitely fix this. We are also considering merging addResource and >>>>>>> rebalance, since separating them are confusing. For now you may use >>>>>>> setResourceIdealState as a workaround. >>>>>>> >>>>>>> Thanks, >>>>>>> Zhen >>>>>>> >>>>>>> ------------------------------ >>>>>>> *From:* Varun Sharma [[email protected]] >>>>>>> *Sent:* Monday, February 23, 2015 10:20 AM >>>>>>> *To:* [email protected] >>>>>>> *Subject:* Re: Bucketized resources not working in Helix 0.6.4 >>>>>>> >>>>>>> Thanks, Zhen. >>>>>>> >>>>>>> What about just calling addResource with an idealState with the >>>>>>> bucket size - is that expected to not work on its own ? >>>>>>> >>>>>>> Varun >>>>>>> >>>>>>> On Mon, Feb 23, 2015 at 10:18 AM, Zhen Zhang <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Varun, >>>>>>>> >>>>>>>> Did you do the following? >>>>>>>> HelixAdmin#addResource() // set bucket size >>>>>>>> HelixAdmin#setResourceIdealState() // no bucket size >>>>>>>> >>>>>>>> When you set the ideal state, you still need set the bucket size in >>>>>>>> the ideal state. This value will be used to bucketize the ideal state. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Jason >>>>>>>> >>>>>>>> ________________________________________ >>>>>>>> From: Varun Sharma [[email protected]] >>>>>>>> Sent: Monday, February 23, 2015 1:51 AM >>>>>>>> To: [email protected] >>>>>>>> Subject: Bucketized resources not working in Helix 0.6.4 >>>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> I am trying to bucketize a resource on Helix 0.6.4 - however, am >>>>>>>> unable to do so. I am setting the bucketSize in the idealState and >>>>>>>> using >>>>>>>> addResource() and setResourceIdealState and it ends up complaining >>>>>>>> that ZK >>>>>>>> size of 1M has been exceeded. >>>>>>>> >>>>>>>> I also tried using the addResource() api which creates an empty >>>>>>>> ideal state with a valid bucket size. However, when I tried to set the >>>>>>>> ideal state on that bucket (this ideal state did not have the bucket >>>>>>>> size >>>>>>>> set), it failed as well. >>>>>>>> >>>>>>>> Whats the right way to get bucketizing to work ? >>>>>>>> >>>>>>>> Thanks ! >>>>>>>> Varun >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
