Let me clarify, 1) So initially I had a cube say cube1 with some dimension and measures which was of size 420Mb 2) later I created another cube say cube2 with same dimension as cube1 but with added measures(so in cube1 it was 1 measure while in cube2 it was 8) which was of size 130Mb(expected greater size due to addition of multiple measure but it was lesser than the cube1)
3) thereafter I created another cube say cube3 with same properties as cube2 and made the build and it was 120Mb(slightly lesser than the last build with the same data source) So I was looking out as what's the reason as the cube size is varying and moreover to my surprise the cube2 which is holding extra measures as in comparison to cube1 is even lesser in size. I guess that makes things clear to you. Thanks, On Tue, Jun 16, 2015 at 1:25 PM, ShaoFeng Shi <[email protected]> wrote: > I'm confused; You said these two cubes are exactly same in an early reply, > but now you mentioned the seoncd has more measures... they are NOT the > same; Different measures takes different storage spaces; What kind of the > measures defined in your former cube? Did you use DISTINCT COUNT measure? > > 2015-06-16 3:28 GMT+08:00 Vineet Mishra <[email protected]>: > > > Hi Shi, > > > > Well yes these cubes are build for the same date range without having the > > data changed on the table and moreover hbase compression was not even > > touched so far. > > > > One interesting thing which I got was the later cube which was small in > > size and completing in lesser time was having multiple measures(around 7 > or > > 8) in comparison to the former cube having just 1 or perhaps 2 measure > with > > more time to complete. > > > > Running the same job again, it came up with some different size for the > > same data, so last time it was reduced 130Mb now its even less, around > > 123Mb. This change is observed within 4 hour of the running the last job > > with the same data set. > > > > I am not sure but how the kylin is internally building these cube > dimension > > so as to inflate the data size by 3 and more over when the measures are > > very few as in comparison. > > > > Let me know if you could get anything out of it. > > > > Thanks! > > > > On Mon, Jun 15, 2015 at 8:51 PM, Shi, Shaofeng <[email protected]> wrote: > > > > > Just guess: Are these two cube builds cross the same date range? Was > the > > > data on fact table changed between these two builds? Did HBase enable > > > compression recently? > > > > > > You can build once more with the same cube and date range to see > whether > > > it can be re-produced; The cube size should be consistent in my > > knowledge; > > > > > > On 6/15/15, 10:52 PM, "Vineet Mishra" <[email protected]> wrote: > > > > > > >Thanks Shi, > > > > > > > >By the way I was curious as for the same table, dimension and metric, > I > > > >could see the final cube size varying from 130 Mb to 420Mb. > > > > > > > >The cube is exactly the same as another one but when the cube build > job > > is > > > >run I could see the cube size difference as mentioned below, > > > > > > > >*Earlier Cube Build* > > > >Size - 420 Mb > > > >Time Taken for building cube - 140 mins > > > >Source Records - ~11Million > > > > > > > >*Today's Cube Build* > > > >Size - 130 Mb > > > >Time Taken for building cube - 22 mins > > > >Source Records - ~11Million > > > > > > > >I don't find any reason to have three time cube size difference. Any > > ideas > > > >about this? > > > > > > > >Thanks! > > > > > > > >On Mon, Jun 15, 2015 at 8:08 PM, Shi, Shaofeng <[email protected]> > > wrote: > > > > > > > >> Good catch, and thanks for the sharing; > > > >> > > > >> On 6/15/15, 10:20 PM, "Vineet Mishra" <[email protected]> > wrote: > > > >> > > > >> >Hi All, > > > >> > > > > >> >Well I got it through. > > > >> > > > > >> >Actually it was basically the pre-process which was converting my > 3Gb > > > >>hive > > > >> >table to 60Mb sequence file which due to block size of 128Mb was > > > >>running > > > >> >in > > > >> >a single Mapper job. > > > >> > > > > >> >I changed the split size to 20Mb and it was able to spawn 3 Mapper > > > >>after > > > >> >that. > > > >> > > > > >> >Anyways, thanks all for your quick response. > > > >> > > > > >> >Thanks! > > > >> > > > > >> >On Mon, Jun 15, 2015 at 3:27 PM, Vineet Mishra < > > [email protected] > > > > > > > >> >wrote: > > > >> > > > > >> >> Hi, > > > >> >> > > > >> >> Sorry I misunderstood this value, actually 30Mb was the output > for > > > >>that > > > >> >> job. But I guess I have the got reason, (not exactly confirmed) > as > > > >>why > > > >> >>its > > > >> >> behaving like this, the input to this job is sequence file which > is > > > >> >>60Mb in > > > >> >> size and compressed. > > > >> >> > > > >> >> No doubt when it will be passed to the mapper, it is very much > > > >>certain > > > >> >>to > > > >> >> have the single mapper invoked due to the pre-configured block > > size. > > > >> >>Now I > > > >> >> need to check If I can override the splitSize property for the > job. > > > >> >> > > > >> >> Moreover I can see that the job is running seamlessly slow, > kindly > > > >>find > > > >> >> the stack trace for the job mentioned below. > > > >> >> > > > >> >> 2015-06-13 10:08:00,940 INFO [main] > > org.apache.hadoop.mapred.MapTask: > > > >> >> Processing split: hdfs:// > > > >> >> > > > >> >> > > > >> > > > >> > > > > > > dev-hadoop-namenode.com:8020/tmp/kylin-9b675c66-4ce9-4a33-a356-2ccf9dbaca > > > >> > > > > > > >>>>6a/kylin_intermediate_xyz_20150401000000_20150611000000_9b675c66_4ce9_4 > > > >>>>a3 > > > >> >>3_a356_2ccf9dbaca6a/000000_0:0+60440271 > > > >> >> 2015-06-13 10:08:01,014 INFO [main] > > org.apache.hadoop.mapred.MapTask: > > > >> >> (EQUATOR) 0 kvi 67108860(268435440) > > > >> >> 2015-06-13 10:08:01,015 INFO [main] > > org.apache.hadoop.mapred.MapTask: > > > >> >> mapreduce.task.io.sort.mb: 256 > > > >> >> 2015-06-13 10:08:01,015 INFO [main] > > org.apache.hadoop.mapred.MapTask: > > > >> >>soft > > > >> >> limit at 214748368 > > > >> >> 2015-06-13 10:08:01,015 INFO [main] > > org.apache.hadoop.mapred.MapTask: > > > >> >> bufstart = 0; bufvoid = 268435456 > > > >> >> 2015-06-13 10:08:01,015 INFO [main] > > org.apache.hadoop.mapred.MapTask: > > > >> >> kvstart = 67108860; length = 16777216 > > > >> >> 2015-06-13 10:08:01,023 INFO [main] > > org.apache.hadoop.mapred.MapTask: > > > >> >>Map > > > >> >> output collector class = > > > >> >>org.apache.hadoop.mapred.MapTask$MapOutputBuffer > > > >> >> 2015-06-13 10:08:01,048 INFO [main] > > > >> >> org.apache.hadoop.io.compress.zlib.ZlibFactory: Successfully > > loaded & > > > >> >> initialized native-zlib library > > > >> >> 2015-06-13 10:08:01,048 INFO [main] > > > >> >> org.apache.hadoop.io.compress.CodecPool: Got brand-new > decompressor > > > >> >> [.deflate] > > > >> >> 2015-06-13 10:08:01,052 INFO [main] > > > >> >> org.apache.hadoop.io.compress.CodecPool: Got brand-new > decompressor > > > >> >> [.deflate] > > > >> >> 2015-06-13 10:08:01,052 INFO [main] > > > >> >> org.apache.hadoop.io.compress.CodecPool: Got brand-new > decompressor > > > >> >> [.deflate] > > > >> >> 2015-06-13 10:08:01,053 INFO [main] > > > >> >> org.apache.hadoop.io.compress.CodecPool: Got brand-new > decompressor > > > >> >> [.deflate] > > > >> >> 2015-06-13 10:08:01,058 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.AbstractHadoopJob: The absolute path > > for > > > >> >>meta > > > >> >> dir is > > > >> >> > > > >> > > > > > > >>>>/yarn/nm/usercache/biops/appcache/application_1433833901375_0139/contai > > > >>>>ne > > > >> >>r_1433833901375_0139_01_000002/meta > > > >> >> 2015-06-13 10:08:01,059 INFO [main] > > > >>org.apache.kylin.common.KylinConfig: > > > >> >> Use > > > >> >> > > > >> > > > > > > >>>>KYLIN_CONF=/yarn/nm/usercache/biops/appcache/application_1433833901375_ > > > >>>>01 > > > >> >>39/container_1433833901375_0139_01_000002/meta > > > >> >> 2015-06-13 10:08:01,084 INFO [main] > > > >>org.apache.kylin.cube.CubeManager: > > > >> >> Initializing CubeManager with config > > > >> >> /yarn/nm/usercache/biops/filecache/19452/meta > > > >> >> 2015-06-13 10:08:01,086 INFO [main] > > > >> >> org.apache.kylin.common.persistence.ResourceStore: Using metadata > > url > > > >> >> /yarn/nm/usercache/biops/filecache/19452/meta for resource store > > > >> >> 2015-06-13 10:08:01,327 INFO [main] > > > >> >>org.apache.kylin.cube.CubeDescManager: > > > >> >> Initializing CubeDescManager with config > > > >> >> /yarn/nm/usercache/biops/filecache/19452/meta > > > >> >> 2015-06-13 10:08:01,327 INFO [main] > > > >> >>org.apache.kylin.cube.CubeDescManager: > > > >> >> Reloading Cube Metadata from folder > > > >> >> /yarn/nm/usercache/biops/filecache/19452/meta/cube_desc > > > >> >> 2015-06-13 10:08:22,834 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled 100000 > > > >> >>records! > > > >> >> 2015-06-13 10:08:43,225 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled 200000 > > > >> >>records! > > > >> >> 2015-06-13 10:09:03,622 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled 300000 > > > >> >>records! > > > >> >> 2015-06-13 10:09:23,999 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled 400000 > > > >> >>records! > > > >> >> 2015-06-13 10:09:44,372 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled 500000 > > > >> >>records! > > > >> >> 2015-06-13 10:10:04,780 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled 600000 > > > >> >>records! > > > >> >> 2015-06-13 10:10:25,143 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled 700000 > > > >> >>records! > > > >> >> 2015-06-13 10:10:45,512 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled 800000 > > > >> >>records! > > > >> >> 2015-06-13 10:11:05,895 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled 900000 > > > >> >>records! > > > >> >> 2015-06-13 10:11:26,284 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 1000000 > > > >> >>records! > > > >> >> 2015-06-13 10:11:46,716 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 1100000 > > > >> >>records! > > > >> >> 2015-06-13 10:12:07,174 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 1200000 > > > >> >>records! > > > >> >> 2015-06-13 10:12:27,646 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 1300000 > > > >> >>records! > > > >> >> 2015-06-13 10:12:48,127 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 1400000 > > > >> >>records! > > > >> >> 2015-06-13 10:13:08,614 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 1500000 > > > >> >>records! > > > >> >> 2015-06-13 10:13:29,148 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 1600000 > > > >> >>records! > > > >> >> 2015-06-13 10:13:49,686 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 1700000 > > > >> >>records! > > > >> >> 2015-06-13 10:14:10,167 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 1800000 > > > >> >>records! > > > >> >> 2015-06-13 10:14:30,652 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 1900000 > > > >> >>records! > > > >> >> 2015-06-13 10:14:51,199 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 2000000 > > > >> >>records! > > > >> >> 2015-06-13 10:15:11,719 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 2100000 > > > >> >>records! > > > >> >> 2015-06-13 10:15:32,221 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 2200000 > > > >> >>records! > > > >> >> 2015-06-13 10:15:52,717 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 2300000 > > > >> >>records! > > > >> >> 2015-06-13 10:16:13,252 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 2400000 > > > >> >>records! > > > >> >> 2015-06-13 10:16:33,769 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 2500000 > > > >> >>records! > > > >> >> 2015-06-13 10:16:54,263 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 2600000 > > > >> >>records! > > > >> >> 2015-06-13 10:17:14,741 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 2700000 > > > >> >>records! > > > >> >> 2015-06-13 10:17:35,213 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 2800000 > > > >> >>records! > > > >> >> 2015-06-13 10:17:55,699 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 2900000 > > > >> >>records! > > > >> >> 2015-06-13 10:18:16,204 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 3000000 > > > >> >>records! > > > >> >> 2015-06-13 10:18:36,721 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 3100000 > > > >> >>records! > > > >> >> 2015-06-13 10:18:57,249 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 3200000 > > > >> >>records! > > > >> >> 2015-06-13 10:19:17,743 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 3300000 > > > >> >>records! > > > >> >> 2015-06-13 10:19:38,275 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 3400000 > > > >> >>records! > > > >> >> 2015-06-13 10:19:58,812 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 3500000 > > > >> >>records! > > > >> >> 2015-06-13 10:20:19,312 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 3600000 > > > >> >>records! > > > >> >> 2015-06-13 10:20:39,920 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 3700000 > > > >> >>records! > > > >> >> 2015-06-13 10:21:00,518 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 3800000 > > > >> >>records! > > > >> >> 2015-06-13 10:21:21,008 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 3900000 > > > >> >>records! > > > >> >> 2015-06-13 10:21:41,525 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 4000000 > > > >> >>records! > > > >> >> 2015-06-13 10:22:02,023 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 4100000 > > > >> >>records! > > > >> >> 2015-06-13 10:22:22,534 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 4200000 > > > >> >>records! > > > >> >> 2015-06-13 10:22:43,063 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 4300000 > > > >> >>records! > > > >> >> 2015-06-13 10:23:03,558 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 4400000 > > > >> >>records! > > > >> >> 2015-06-13 10:23:24,043 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 4500000 > > > >> >>records! > > > >> >> 2015-06-13 10:23:44,537 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 4600000 > > > >> >>records! > > > >> >> 2015-06-13 10:24:05,015 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 4700000 > > > >> >>records! > > > >> >> 2015-06-13 10:24:25,490 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 4800000 > > > >> >>records! > > > >> >> 2015-06-13 10:24:45,995 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 4900000 > > > >> >>records! > > > >> >> 2015-06-13 10:25:06,487 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 5000000 > > > >> >>records! > > > >> >> 2015-06-13 10:25:26,964 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 5100000 > > > >> >>records! > > > >> >> 2015-06-13 10:25:47,524 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 5200000 > > > >> >>records! > > > >> >> 2015-06-13 10:26:08,032 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 5300000 > > > >> >>records! > > > >> >> 2015-06-13 10:26:28,514 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 5400000 > > > >> >>records! > > > >> >> 2015-06-13 10:26:49,053 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 5500000 > > > >> >>records! > > > >> >> 2015-06-13 10:26:50,360 INFO [main] > > org.apache.hadoop.mapred.MapTask: > > > >> >> Spilling map output > > > >> >> 2015-06-13 10:26:50,360 INFO [main] > > org.apache.hadoop.mapred.MapTask: > > > >> >> bufstart = 0; bufend = 126646464; bufvoid = 268435456 > > > >> >> 2015-06-13 10:26:50,360 INFO [main] > > org.apache.hadoop.mapred.MapTask: > > > >> >> kvstart = 67108860(268435440); kvend = 45083392(180333568); > length > > = > > > >> >> 22025469/16777216 > > > >> >> 2015-06-13 10:26:50,360 INFO [main] > > org.apache.hadoop.mapred.MapTask: > > > >> >> (EQUATOR) 148671936 kvi 37167980(148671920) > > > >> >> 2015-06-13 10:26:54,802 INFO [SpillThread] > > > >> >> org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor > > > >> >>[.snappy] > > > >> >> 2015-06-13 10:26:54,883 INFO [SpillThread] > > > >> >> org.apache.kylin.job.hadoop.AbstractHadoopJob: The absolute path > > for > > > >> >>meta > > > >> >> dir is > > > >> >> > > > >> > > > > > > >>>>/yarn/nm/usercache/biops/appcache/application_1433833901375_0139/contai > > > >>>>ne > > > >> >>r_1433833901375_0139_01_000002/meta > > > >> >> 2015-06-13 10:27:09,614 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 5600000 > > > >> >>records! > > > >> >> 2015-06-13 10:27:30,135 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 5700000 > > > >> >>records! > > > >> >> 2015-06-13 10:27:50,657 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 5800000 > > > >> >>records! > > > >> >> 2015-06-13 10:27:59,691 INFO [SpillThread] > > > >> >> org.apache.kylin.job.hadoop.cube.CuboidReducer: Handled 100000 > > > >>records! > > > >> >> 2015-06-13 10:28:11,185 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 5900000 > > > >> >>records! > > > >> >> 2015-06-13 10:28:31,751 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 6000000 > > > >> >>records! > > > >> >> 2015-06-13 10:28:52,303 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 6100000 > > > >> >>records! > > > >> >> 2015-06-13 10:29:03,235 INFO [SpillThread] > > > >> >> org.apache.kylin.job.hadoop.cube.CuboidReducer: Handled 200000 > > > >>records! > > > >> >> 2015-06-13 10:29:12,873 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 6200000 > > > >> >>records! > > > >> >> 2015-06-13 10:29:33,424 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 6300000 > > > >> >>records! > > > >> >> 2015-06-13 10:29:54,168 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 6400000 > > > >> >>records! > > > >> >> 2015-06-13 10:30:14,771 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 6500000 > > > >> >>records! > > > >> >> 2015-06-13 10:30:18,479 INFO [SpillThread] > > > >> >> org.apache.kylin.job.hadoop.cube.CuboidReducer: Handled 300000 > > > >>records! > > > >> >> 2015-06-13 10:30:35,325 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 6600000 > > > >> >>records! > > > >> >> 2015-06-13 10:30:39,831 INFO [SpillThread] > > > >> >> org.apache.hadoop.mapred.MapTask: Finished spill 0 > > > >> >> 2015-06-13 10:30:39,831 INFO [main] > > org.apache.hadoop.mapred.MapTask: > > > >> >> (RESET) equator 148671936 kv 37167980(148671920) kvi > > > >>32705668(130822672) > > > >> >> 2015-06-13 10:30:55,807 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 6700000 > > > >> >>records! > > > >> >> 2015-06-13 10:31:16,301 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 6800000 > > > >> >>records! > > > >> >> 2015-06-13 10:31:36,804 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 6900000 > > > >> >>records! > > > >> >> 2015-06-13 10:31:57,302 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 7000000 > > > >> >>records! > > > >> >> 2015-06-13 10:32:17,790 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 7100000 > > > >> >>records! > > > >> >> 2015-06-13 10:32:38,294 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 7200000 > > > >> >>records! > > > >> >> 2015-06-13 10:32:58,830 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 7300000 > > > >> >>records! > > > >> >> 2015-06-13 10:33:19,354 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 7400000 > > > >> >>records! > > > >> >> 2015-06-13 10:33:39,866 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 7500000 > > > >> >>records! > > > >> >> 2015-06-13 10:34:00,371 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 7600000 > > > >> >>records! > > > >> >> 2015-06-13 10:34:20,872 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 7700000 > > > >> >>records! > > > >> >> 2015-06-13 10:34:41,387 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 7800000 > > > >> >>records! > > > >> >> 2015-06-13 10:35:01,884 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 7900000 > > > >> >>records! > > > >> >> 2015-06-13 10:35:22,368 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 8000000 > > > >> >>records! > > > >> >> 2015-06-13 10:35:42,868 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 8100000 > > > >> >>records! > > > >> >> 2015-06-13 10:36:03,361 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 8200000 > > > >> >>records! > > > >> >> 2015-06-13 10:36:23,981 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 8300000 > > > >> >>records! > > > >> >> 2015-06-13 10:36:44,529 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 8400000 > > > >> >>records! > > > >> >> 2015-06-13 10:37:05,034 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 8500000 > > > >> >>records! > > > >> >> 2015-06-13 10:37:25,609 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 8600000 > > > >> >>records! > > > >> >> 2015-06-13 10:37:46,152 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 8700000 > > > >> >>records! > > > >> >> 2015-06-13 10:38:06,725 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 8800000 > > > >> >>records! > > > >> >> 2015-06-13 10:38:27,296 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 8900000 > > > >> >>records! > > > >> >> 2015-06-13 10:38:47,905 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 9000000 > > > >> >>records! > > > >> >> 2015-06-13 10:39:08,430 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 9100000 > > > >> >>records! > > > >> >> 2015-06-13 10:39:28,952 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 9200000 > > > >> >>records! > > > >> >> 2015-06-13 10:39:49,460 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 9300000 > > > >> >>records! > > > >> >> 2015-06-13 10:40:09,987 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 9400000 > > > >> >>records! > > > >> >> 2015-06-13 10:40:30,551 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 9500000 > > > >> >>records! > > > >> >> 2015-06-13 10:40:51,080 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 9600000 > > > >> >>records! > > > >> >> 2015-06-13 10:41:11,579 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 9700000 > > > >> >>records! > > > >> >> 2015-06-13 10:41:32,122 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 9800000 > > > >> >>records! > > > >> >> 2015-06-13 10:41:52,623 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 9900000 > > > >> >>records! > > > >> >> 2015-06-13 10:42:13,128 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 10000000 > > > >> >>records! > > > >> >> 2015-06-13 10:42:33,628 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 10100000 > > > >> >>records! > > > >> >> 2015-06-13 10:42:54,128 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 10200000 > > > >> >>records! > > > >> >> 2015-06-13 10:43:14,629 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 10300000 > > > >> >>records! > > > >> >> 2015-06-13 10:43:35,149 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 10400000 > > > >> >>records! > > > >> >> 2015-06-13 10:43:55,764 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 10500000 > > > >> >>records! > > > >> >> 2015-06-13 10:44:16,303 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 10600000 > > > >> >>records! > > > >> >> 2015-06-13 10:44:36,857 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 10700000 > > > >> >>records! > > > >> >> 2015-06-13 10:44:57,349 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 10800000 > > > >> >>records! > > > >> >> 2015-06-13 10:45:17,840 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 10900000 > > > >> >>records! > > > >> >> 2015-06-13 10:45:38,441 INFO [main] > > > >> >> org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled > 11000000 > > > >> >>records! > > > >> >> 2015-06-13 10:45:41,057 INFO [main] > > org.apache.hadoop.mapred.MapTask: > > > >> >> Spilling map output > > > >> >> 2015-06-13 10:45:41,057 INFO [main] > > org.apache.hadoop.mapred.MapTask: > > > >> >> bufstart = 148671936; bufend = 6882957; bufvoid = 268435443 > > > >> >> 2015-06-13 10:45:41,057 INFO [main] > > org.apache.hadoop.mapred.MapTask: > > > >> >> kvstart = 37167980(148671920); kvend = 15142512(60570048); > length = > > > >> >> 22025469/16777216 > > > >> >> 2015-06-13 10:45:41,057 INFO [main] > > org.apache.hadoop.mapred.MapTask: > > > >> >> (EQUATOR) 28908413 kvi 7227096(28908384) > > > >> >> > > > >> >> Thanks, > > > >> >> > > > >> >> On Mon, Jun 15, 2015 at 3:11 PM, 周千昊 <[email protected]> wrote: > > > >> >> > > > >> >>> Hi Mishra > > > >> >>> It is as what you described, since the data size is 30MB > > only, > > > >> >>>hadoop > > > >> >>> will only manage to run MR in single mapper and reduce. > > > >> >>> As for your question about why it takes so long to run on a > > > >>small > > > >> >>> dataset, can you please dig into the web page of *map reduce > task > > > >> >>>status > > > >> >>> *to > > > >> >>> check how much time the MR job really take, so that we can make > > > >>sure if > > > >> >>> the > > > >> >>> time is consuming on the MR job or kylin job scheduling module > > > >> >>> > > > >> >>> Vineet Mishra <[email protected]>于2015年6月15日周一 下午4:51写道： > > > >> >>> > > > >> >>> > Shi, > > > >> >>> > > > > >> >>> > Hadoop is setup correctly on my cluster with the default block > > > >>size > > > >> >>>of > > > >> >>> > 128Mb and its indeed very much running multiple Mapper/Reducer > > > >>based > > > >> >>> jobs > > > >> >>> > for other cases. > > > >> >>> > > > > >> >>> > Its the only Kylin Cube building which is running through > single > > > >>M/R > > > >> >>> job. > > > >> >>> > > > > >> >>> > Moreover to my surprise, the 4th running job which is Build > Base > > > >> >>>Cuboid > > > >> >>> > Data shows Data size as 30Mb, is it the reason due to which > > single > > > >> >>> mapper > > > >> >>> > is getting invoked, if that being the case then even to > process > > > >>such > > > >> >>>a > > > >> >>> > small data set why is it taking around 50min. > > > >> >>> > > > > >> >>> > Thanks, > > > >> >>> > > > > >> >>> > On Mon, Jun 15, 2015 at 11:17 AM, Shi, Shaofeng < > > [email protected] > > > > > > > >> >>> wrote: > > > >> >>> > > > > >> >>> > > Yes you can adjust these parameters, for example give a > > smaller > > > >> >>>value > > > >> >>> for > > > >> >>> > > kylin.job.mapreduce.default.reduce.input.mb; but it only > > affects > > > >> >>>the > > > >> >>> > > reducer number; > > > >> >>> > > > > > >> >>> > > I suggest you investigate why there is only 1 mapper be > > started; > > > >> >>>Some > > > >> >>> > > factors like hadoop cluster size, HDFS file block size will > > > >>impact > > > >> >>> this; > > > >> >>> > > You can run a SQL (a query which need run MR, not simple > > select > > > >>*) > > > >> >>> with > > > >> >>> > > hive -e, and then use the MR job track URL to see how many > > > >>mappers > > > >> >>>be > > > >> >>> > > triggered; If it is still single, then the problem is in > your > > > >> >>>hadoop > > > >> >>> > > configuration; Otherwise it may exists in Kylin, check if > you > > > >>put > > > >> >>>some > > > >> >>> > > additional parameter in conf/kylin_job_conf.xml. > > > >> >>> > > > > > >> >>> > > > > > >> >>> > > On 6/15/15, 2:52 AM, "Vineet Mishra" < > [email protected]> > > > >> >>>wrote: > > > >> >>> > > > > > >> >>> > > >Can I have specification for these properties? > > > >> >>> > > > > > > >> >>> > > >KYLIN_JOB_MAPREDUCE_DEFAULT_REDUCE_COUNT_RATIO = > > > >> >>> > > >"kylin.job.mapreduce.default.reduce.count.ratio"; > > > >> >>> > > >KYLIN_JOB_MAPREDUCE_DEFAULT_REDUCE_INPUT_MB = > > > >> >>> > > >"kylin.job.mapreduce.default.reduce.input.mb"; > > > >> >>> > > >KYLIN_JOB_MAPREDUCE_MAX_REDUCER_NUMBER = > > > >> >>> > > >"kylin.job.mapreduce.max.reducer.number"; > > > >> >>> > > > > > > >> >>> > > >Thanks! > > > >> >>> > > > > > > >> >>> > > >On Sun, Jun 14, 2015 at 11:59 PM, Vineet Mishra < > > > >> >>> [email protected] > > > >> >>> > > > > > >> >>> > > >wrote: > > > >> >>> > > > > > > >> >>> > > >> Hi Shi, > > > >> >>> > > >> > > > >> >>> > > >> Its alright! > > > >> >>> > > >> So I was wondering my source hive Table is around 3 Gb, > > > >>despite > > > >> >>>of > > > >> >>> my > > > >> >>> > > >>hive > > > >> >>> > > >> table being partitioned and holding the data around 50-70 > > Mb > > > >>per > > > >> >>> > > >>partition > > > >> >>> > > >> the Mapper and Reducer getting spawned are single. The > > > >>amount of > > > >> >>> data > > > >> >>> > > >>that > > > >> >>> > > >> is being processed in the M/R is nothing as expected but > it > > > >> >>>takes > > > >> >>> hell > > > >> >>> > > >>lot > > > >> >>> > > >> of time. > > > >> >>> > > >> > > > >> >>> > > >> As mentioned in the trailing mail that the job is getting > > > >>very > > > >> >>> slow, > > > >> >>> > the > > > >> >>> > > >> process Build Base Cuboid Data itself takes around 50mins > > to > > > >>get > > > >> >>> > > >> completed. > > > >> >>> > > >> > > > >> >>> > > >> I can tweak the reducer parameter mentioned by you but > do u > > > >> >>>think > > > >> >>> that > > > >> >>> > > >> will make a difference since the mapper is where the most > > of > > > >>the > > > >> >>> time > > > >> >>> > is > > > >> >>> > > >> spent. > > > >> >>> > > >> > > > >> >>> > > >> Can you share your thoughts for performance tuning for > the > > > >>cube > > > >> >>> build! > > > >> >>> > > >> > > > >> >>> > > >> Thanks! > > > >> >>> > > >> > > > >> >>> > > >> On Sun, Jun 14, 2015 at 7:26 PM, Shi, Shaofeng > > > >> >>><[email protected]> > > > >> >>> > > wrote: > > > >> >>> > > >> > > > >> >>> > > >>> Hi, sorry, a busy weekend; > > > >> >>> > > >>> > > > >> >>> > > >>> Usually Kylin will request proper number of mapper and > > > >> >>>reducers; > > > >> >>> If > > > >> >>> > you > > > >> >>> > > >>> see single mapper/recudder, how much of your input and > > > >>output? > > > >> >>>If > > > >> >>> > your > > > >> >>> > > >>> cube is quite small, single mapper/reducer is possible; > > > >> >>> > > >>> > > > >> >>> > > >>> Number of mappers is decided by the FileInputFormat; But > > > >> >>>number of > > > >> >>> > > >>>reducer > > > >> >>> > > >>> was set by Kylin, see: > > > >> >>> > > >>> > > > >> >>> > > >>> > > > >> >>> > > >>> > > > >> >>> > > > > > >> >>> > > > >> >>> > > > >> > > > > https://github.com/apache/incubator-kylin/blob/master/job/src/main/java/ > > > >> >>> > > >>>org > > > >> >>> > > >>> /apache/kylin/job/hadoop/cube/CuboidJob.java#L141 > > > >> >>> > > >>> > > > >> >>> > > >>>< > > > >> >>> > > > > > >> >>> > > > >> > > https://github.com/apache/incubator-kylin/blob/master/job/src/main/java > > > >> >>> > > >>>/org/apache/kylin/job/hadoop/cube/CuboidJob.java#L141> > > > >> >>> > > >>> > > > >> >>> > > >>> > > > >> >>> > > >>> > > > >> >>> > > >>> > > > >> >>> > > >>> On 6/14/15, 5:25 PM, "Vineet Mishra" > > > >><[email protected]> > > > >> >>> wrote: > > > >> >>> > > >>> > > > >> >>> > > >>> >Urgent call, any follow up on this? > > > >> >>> > > >>> > > > > >> >>> > > >>> >On Fri, Jun 12, 2015 at 6:46 PM, Vineet Mishra > > > >> >>> > > >>><[email protected]> > > > >> >>> > > >>> >wrote: > > > >> >>> > > >>> > > > > >> >>> > > >>> >> > > > >> >>> > > >>> >> Why org.apache.kylin.job.hadoop.cube.CuboidReducer is > > > >> >>>running > > > >> >>> > Single > > > >> >>> > > >>> >> Mapper/Reducer for the job. Can I have the > > understanding > > > >> >>>behind > > > >> >>> > the > > > >> >>> > > >>> >>reason > > > >> >>> > > >>> >> of running it as single mapper/reducer. > > > >> >>> > > >>> >> > > > >> >>> > > >>> >> Thanks! > > > >> >>> > > >>> >> > > > >> >>> > > >>> >> On Fri, Jun 12, 2015 at 6:30 PM, Vineet Mishra > > > >> >>> > > >>><[email protected] > > > >> >>> > > >>> > > > > >> >>> > > >>> >> wrote: > > > >> >>> > > >>> >> > > > >> >>> > > >>> >>> Hi All, > > > >> >>> > > >>> >>> > > > >> >>> > > >>> >>> I am building a cube using Kylin and I could see > that > > > >>the > > > >> >>>job > > > >> >>> is > > > >> >>> > > >>> >>>running > > > >> >>> > > >>> >>> with Single Mapper and Reducer for some of the > > > >>intermediate > > > >> >>> > process > > > >> >>> > > >>> >>>such as > > > >> >>> > > >>> >>> > > > >> >>> > > >>> >>> Extract Fact Table Distinct Columns > > > >> >>> > > >>> >>> Build Dimension Dictionary > > > >> >>> > > >>> >>> Build N-Dimension Cuboid > > > >> >>> > > >>> >>> > > > >> >>> > > >>> >>> I am not sure what's the reason behind running the > job > > > >>with > > > >> >>> > single > > > >> >>> > > >>> M/R, > > > >> >>> > > >>> >>> is it really necessary or is it some default config. > > > >>which > > > >> >>> can be > > > >> >>> > > >>> >>>tweaked, > > > >> >>> > > >>> >>> its 70 Mins and the job status is 25% ! > > > >> >>> > > >>> >>> > > > >> >>> > > >>> >>> Urgent Call! > > > >> >>> > > >>> >>> > > > >> >>> > > >>> >>> Thanks! > > > >> >>> > > >>> >>> > > > >> >>> > > >>> >> > > > >> >>> > > >>> >> > > > >> >>> > > >>> > > > >> >>> > > >>> > > > >> >>> > > >> > > > >> >>> > > > > > >> >>> > > > > > >> >>> > > > > >> >>> > > > >> >> > > > >> >> > > > >> > > > >> > > > > > > > > >
