Re: [Discuss]Set block_size for table on table level

2016-09-28 Thread Raghunandan subramanya
Would blocklet division and distribution not help to solve this scenario? On 2016-09-28 12:36 (+0530), Zhangshunyu wrote: > For each table, we can set block size consider the data.size, this is because> > that when execute query, each task will get one block to process one

Re: [Discuss]Set block_size for table on table level

2016-09-28 Thread Zhangshunyu
For each table, we can set block size consider the data.size, this is because that when execute query, each task will get one block to process one time, when the blocks num < parallelism, set a reasonable block size would get most suitable block num, to make the best of parallelism. -- View

Re: [Discuss]Set block_size for table on table level

2016-09-28 Thread Raghunandan subramanya
What problems we want to solve by making it configurable? On 2016-09-26 07:36 (+0530), Zhangshunyu wrote: > Purpose:> > To configure block file size for each table on column level, so that each> > table could has its own blocksize.> > My solution:> > Add a new parameter in

Re: [Discuss]Set block_size for table on table level

2016-09-26 Thread 金铸
this param as an optional field, as long as it is not a required field, we are fine with a defined default block size. Regards. Jihong -Original Message- From: Jacky Li [mailto: jacky.likun@ ] Sent: Monday, September 26, 2016 7:29 AM To: dev@.apache Subject: Re: [Discuss]Set block_size

Re: [Discuss]Set block_size for table on table level

2016-09-26 Thread Jean-Baptiste Onofré
+1 Regards JB On 09/26/2016 04:29 PM, Jacky Li wrote: I am OK with this feature, the only thing I am worrying about is the compatibility of CarbonData file reader. Can you make it compatible when you reading old CarbonData file without this property. We have encountered many times that user

Re: [Discuss]Set block_size for table on table level

2016-09-26 Thread Venkata Gollamudi
ault block size. > > Regards. > > Jihong > > -Original Message- > From: Jacky Li [mailto:jacky.li...@qq.com] > Sent: Monday, September 26, 2016 7:29 AM > To: dev@carbondata.incubator.apache.org > Subject: Re: [Discuss]Set block_size for table on table level >

RE: [Discuss]Set block_size for table on table level

2016-09-26 Thread Jihong Ma
26, 2016 7:29 AM To: dev@carbondata.incubator.apache.org Subject: Re: [Discuss]Set block_size for table on table level I am OK with this feature, the only thing I am worrying about is the compatibility of CarbonData file reader. Can you make it compatible when you reading old CarbonData file

Re: [Discuss]Set block_size for table on table level

2016-09-26 Thread Ravindra Pesala
+1 At same time max and min block size should be restricted and validated while creating table. On 26 September 2016 at 07:36, Zhangshunyu wrote: > Purpose: > To configure block file size for each table on column level, so that each > table could has its own blocksize.