Hi Qilong, Does your cube have count-distinct or Top-N measure?
If you observed that there are too many or too small hbase regions, you can adjust some parameters: kylin.cube.size-estimate-ratio=0.25 kylin.cube.size-estimate-countdistinct-ratio=0.05 The default ratio for common case is 0.25, you can set it to smaller if the estimated size is bigger than actual size. These two parameters can be set at Cube level. A better way is when doing merge, using the actual size of existing segments to adjust the estimated size, then get a closer result. 2018-01-23 14:47 GMT+08:00 苏启龙 <[email protected]>: > Hi shaofeng, > > Yes, it’s usually smaller then the sum of each segment, but usually a > small amount compared with the total size. > > But for the statistics estimate, usually result in a N times larger than > it actually be, and results in a huge waste of HBase region numbers。 > > > 1. Do you have any data about deviation of the two ways in statistics? > I mean generally which way will be closer? > 2. Is there any improve plan for this in the roadmap? Or some > consideration to give more options to user to select their own estimate > algo? > > > Thanks > > Qilong > > 发件人: ShaoFeng Shi <[email protected]> > 答复: "[email protected]" <[email protected]> > 日期: 2018年1月23日 星期二 09:43 > 至: user <[email protected]> > 抄送: 林豪（linhao）-技术产品中心 <[email protected]> > 主题: Re: segment size estimate when merging > > Hi Qilong, > > When merging segments, the dimension-measure values (k-v) will be re-orged > and the same key will be merged, so the merged size is not simply a sum of > each segment; usually, it is smaller than before. > > Always using the statistics to estimate the size is for consistency. Of > course, there is room to improve the estimation accuracy. > > > > 2018-01-22 16:54 GMT+08:00 苏启龙 <[email protected]>: > >> >> Hi， >> >> We have some unclear points about the segment size estimate when merging >> multi-segments. >> >> We find that the segment merge job still uses >> CubeStatsReader::getCuboidSizeMap to estimate the total size of the >> merged segment. From our understanding, when building a new segment, Kylin >> uses this way to estimate the total size is OK since no other info we can >> turn to. But in merging we may sum the table size of the segments to be >> merged, which should be more accurate. >> >> So why for this consideration? >> >> >> >> Su Qilong >> > > > > -- > Best regards, > > Shaofeng Shi 史少锋 > > -- Best regards, Shaofeng Shi 史少锋
