I am running kylin 2.5.1.
When I build one hour's cuboids, the step converting cuboid to HFILE took 8.33
minutes. Only 1 reduce task was created. Is there way to start more reduce
tasks?
The following is my kylin.properties file content:
#kylin.storage.hbase.region-cut-gb=5
kylin.storage.hbase.hfile-size-gb=1
Any suggestion is welcome.
Thanks.
Kang-sen
Log from kylin monitor step 13: (The data size is 1.26GB)
Counters: 50
File System Counters
FILE: Number of bytes read=966603663
FILE: Number of bytes written=1996309058
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=662733811
HDFS: Number of bytes written=1350608338
HDFS: Number of read operations=199
HDFS: Number of large read operations=0
HDFS: Number of write operations=5
Job Counters
Launched map tasks=48
Launched reduce tasks=1
Data-local map tasks=45
Rack-local map tasks=3
Total time spent by all maps in occupied slots (ms)=799000
Total time spent by all reduces in occupied slots (ms)=822064
Total time spent by all map tasks (ms)=799000
Total time spent by all reduce tasks (ms)=411032
Total vcore-milliseconds taken by all map tasks=799000
Total vcore-milliseconds taken by all reduce tasks=411032
Total megabyte-milliseconds taken by all map tasks=8999936000
Total megabyte-milliseconds taken by all reduce tasks=9259728896
Map-Reduce Framework
Map input records=28693452
Map output records=57386904
Map output bytes=6215636946
Map output materialized bytes=1020541171
Input split bytes=11748
Combine input records=0
Combine output records=0
Reduce input groups=57386904
Reduce shuffle bytes=1020541171
Reduce input records=57386904
Reduce output records=57386904
Spilled Records=114773808
Shuffled Maps =48
Failed Shuffles=0
Merged Map outputs=48
GC time elapsed (ms)=47218
CPU time spent (ms)=1278720
Physical memory (bytes) snapshot=130269708288
Virtual memory (bytes) snapshot=585895706624
Total committed heap usage (bytes)=149828927488
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=662722063
File Output Format Counters
Bytes Written=1350608338
-----------------------------------------------------------------------------------------------------------------------
Notice: This e-mail together with any attachments may contain information of
Ribbon Communications Inc. that
is confidential and/or proprietary for the sole use of the intended recipient.
Any review, disclosure, reliance or
distribution by others or forwarding without express permission is strictly
prohibited. If you are not the intended
recipient, please notify the sender immediately and then delete all copies,
including any attachments.
-----------------------------------------------------------------------------------------------------------------------
