as Singh told you it's in RangeKeyDistributionReducer On Thu, Aug 6, 2015 at 11:07 AM, liangmeng <[email protected]> wrote:
> what's the region size refered to "SMALL, MEDIUM, LARGE"? > > > > 梁猛 > 中国移动广东公司 网管维护中心 网管支撑室 > 电话：13802880779 > 邮箱: [email protected] ，[email protected] > 地址：广东省广州市珠江新城珠江西路11号 广东全球通大厦北3楼 > 邮编：510623 > > From: hongbin ma > Date: 2015-08-06 10:32 > To: dev > Subject: Re: step 11": Convert Cuboid Data to HFile" takes too much time > @liangmeng please refer to Singh's hint. > > @Singh, it is a good idea to allow for more flexible cube size > configuration. However there're some issues to address: > > 1. Currently CubeCapacity(SMALL, MEDIUM, LARGE) is at cube level, not at > segment level. We cannot automatically apply tiny cut size to segments, > while apply medium/large cut size to merged large segments. However, it is > possible that we discard the SMALL, MEDIUM, LARGE concept and dynamically > determine each segment's region size depending on its size. > > 2. We do not have enough capacity to do this now, and to be honest it's not > a critical issue for us because most of our cubes are larger than 100G, big > region size is perferrable to us. So if you really need this capability can > you please think about contributing a patch? we'll review it and pack it > into the next 0.7 release > > > On Wed, Aug 5, 2015 at 11:11 PM, Bijeet Singh <[email protected]> > wrote: > > > While we are here, controlling the number of regions while creating a new > > segment, with the cube size options - SMALL, MEDIUM, LARGE - sometimes > > seems too restrictive. Even with SMALL size cubes, number of reducers are > > sometimes too low, which adversely impacts the cube build time. > > > > By modifying the cut size to an even smaller value, I was able to bring > > down the average reduce time of this step, from ~25mins to ~2mins. But I > > understand that setting very low cut sizes will lead to creation of very > > small regions, which isn't very desirable either. But too many smaller > > regions problem for new segments can be handled while merging the > segments. > > > > So, does it make sense to make the setting of cube size option a bit more > > flexible ? Through either making it configurable or by providing more > > options(other than small, medium, large) while creating the cube. > > > > Thanks, > > Bijeet > > > > > > > > On Wed, Aug 5, 2015 at 8:21 PM, Bijeet Singh <[email protected]> > > wrote: > > > > > The number of reducers in this step, depends on the cube size selected > > > while creating the cube. You can try with a small size cube. > > > > > > But if you are getting only one reducer, even with a small cube, you'd > > > have to probably tweak the cut size in RangeKeyDistributionReducer to > > even > > > smaller values. That will help you increase the number of reducers. > > > > > > 2015-08-05 18:13 GMT+05:30 liangmeng <[email protected]>: > > > > > >> there is only one reducer, and reduce take too much time, is it > possible > > >> to increase reducenum? > > >> > > >> > > >> > > >> 梁猛 > > >> 中国移动广东公司 网管维护中心 网管支撑室 > > >> 电话：13802880779 > > >> 邮箱: [email protected] ，[email protected] > > >> 地址：广东省广州市珠江新城珠江西路11号 广东全球通大厦北3楼 > > >> 邮编：510623 > > >> > > > > > > > > > > > > -- > Regards, > > *Bin Mahone | 马洪宾* > Apache Kylin: http://kylin.io > Github: https://github.com/binmahone > -- Regards, *Bin Mahone | 马洪宾* Apache Kylin: http://kylin.io Github: https://github.com/binmahone
