OK - thanks. Btw, is bucket size the number of buckets or is it the size of one bucket ? Also can the bucket size be changed dynamically using setResourceIdealState ?
On Mon, Feb 23, 2015 at 1:53 PM, Zhen Zhang <[email protected]> wrote: > Hi Varun, > > The partition number is mainly for calculating the buckets. Partition > name will remain the same. You can always get and build Helix source code > from here: > https://github.com/apache/helix > > Changing the partition naming is not hard, but we need to come up with a > general way to achieve this. We will think about this requirement. If you > have any good idea for this, you can always submit a pull request on github. > > Thanks, > Jason > ------------------------------ > *From:* Varun Sharma [[email protected]] > *Sent:* Monday, February 23, 2015 12:04 PM > > *To:* [email protected] > *Subject:* Re: Bucketized resources not working in Helix 0.6.4 > > How hard is this to change to '$' - its difficult for us to now change > the partition names now. Is there a place to get the helix 0.6.4 source > code, make changes and built it ? > > Another related question - is the partition number mainly needed for > hashing into the buckets - the partition name would still continue to be > "resource_name_partition_num", right ? > > On Mon, Feb 23, 2015 at 11:52 AM, Zhen Zhang <[email protected]> wrote: > >> The bucketize logic is expecting partition name following the >> convention {resource_name}_{partition_number}. In your case, you are using >> $ ? >> ------------------------------ >> *From:* Varun Sharma [[email protected]] >> *Sent:* Monday, February 23, 2015 11:48 AM >> >> *To:* [email protected] >> *Subject:* Re: Bucketized resources not working in Helix 0.6.4 >> >> Note that we had to prefix each partition with the resource name to >> make the partitions unique. >> >> On Mon, Feb 23, 2015 at 11:48 AM, Varun Sharma <[email protected]> >> wrote: >> >>> Here is the latest error I am getting: >>> >>> 2015-02-23 19:44:48,153 [controller-pool-0] >>> (TerrapinControllerServiceImpl.java:129) WARN Resource creation failed for >>> $terrapin$data$pinpin_board_join$1424377884722, rolling back. >>> >>> *java.lang.IllegalArgumentException: Could NOT parse partition# >>> (join$1424377884722$0) in $terrapin$data$pinpin_board_join$1424377884722$0* >>> >>> *at >>> org.apache.helix.ZNRecordBucketizer.getBucketName(ZNRecordBucketizer.java:72)* >>> >>> *at >>> org.apache.helix.ZNRecordBucketizer.bucketize(ZNRecordBucketizer.java:105)* >>> >>> at >>> org.apache.helix.manager.zk.ZKHelixDataAccessor.setProperty(ZKHelixDataAccessor.java:110) >>> >>> at >>> org.apache.helix.manager.zk.ZKHelixAdmin.setResourceIdealState(ZKHelixAdmin.java:686) >>> >>> On Mon, Feb 23, 2015 at 10:40 AM, Zhen Zhang <[email protected]> >>> wrote: >>> >>>> Not really. The bucketize logic just creates child znode under the >>>> original ideal state, and each of the child znode holds a subset of the >>>> total partitions. >>>> ------------------------------ >>>> *From:* Varun Sharma [[email protected]] >>>> *Sent:* Monday, February 23, 2015 10:37 AM >>>> >>>> *To:* [email protected] >>>> *Subject:* Re: Bucketized resources not working in Helix 0.6.4 >>>> >>>> One other question - would bucketizing add anything to the >>>> partition name - just worried about any special symbols (we use _ and $) >>>> causing issues there ? >>>> >>>> On Mon, Feb 23, 2015 at 10:35 AM, Zhen Zhang <[email protected]> >>>> wrote: >>>> >>>>> Hi Varun, >>>>> >>>>> HelixAdmin#addResource was initially used for adding an empty >>>>> resource only, followed by HelixAdmin#rebalance, which bucketize the ideal >>>>> state, so the bucketize logic is missing from addResource. We should >>>>> definitely fix this. We are also considering merging addResource and >>>>> rebalance, since separating them are confusing. For now you may use >>>>> setResourceIdealState as a workaround. >>>>> >>>>> Thanks, >>>>> Zhen >>>>> >>>>> ------------------------------ >>>>> *From:* Varun Sharma [[email protected]] >>>>> *Sent:* Monday, February 23, 2015 10:20 AM >>>>> *To:* [email protected] >>>>> *Subject:* Re: Bucketized resources not working in Helix 0.6.4 >>>>> >>>>> Thanks, Zhen. >>>>> >>>>> What about just calling addResource with an idealState with the >>>>> bucket size - is that expected to not work on its own ? >>>>> >>>>> Varun >>>>> >>>>> On Mon, Feb 23, 2015 at 10:18 AM, Zhen Zhang <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi Varun, >>>>>> >>>>>> Did you do the following? >>>>>> HelixAdmin#addResource() // set bucket size >>>>>> HelixAdmin#setResourceIdealState() // no bucket size >>>>>> >>>>>> When you set the ideal state, you still need set the bucket size in >>>>>> the ideal state. This value will be used to bucketize the ideal state. >>>>>> >>>>>> Thanks, >>>>>> Jason >>>>>> >>>>>> ________________________________________ >>>>>> From: Varun Sharma [[email protected]] >>>>>> Sent: Monday, February 23, 2015 1:51 AM >>>>>> To: [email protected] >>>>>> Subject: Bucketized resources not working in Helix 0.6.4 >>>>>> >>>>>> Hi, >>>>>> >>>>>> I am trying to bucketize a resource on Helix 0.6.4 - however, am >>>>>> unable to do so. I am setting the bucketSize in the idealState and using >>>>>> addResource() and setResourceIdealState and it ends up complaining that >>>>>> ZK >>>>>> size of 1M has been exceeded. >>>>>> >>>>>> I also tried using the addResource() api which creates an empty ideal >>>>>> state with a valid bucket size. However, when I tried to set the ideal >>>>>> state on that bucket (this ideal state did not have the bucket size set), >>>>>> it failed as well. >>>>>> >>>>>> Whats the right way to get bucketizing to work ? >>>>>> >>>>>> Thanks ! >>>>>> Varun >>>>>> >>>>> >>>>> >>>> >>> >> >
