Hello all,
In Build Base Cuboid Data step and Build N-Dimension steps,I find that
every step takes too time to handle records like the log. Should I configure
my hadoop's setting.
Thanks every much.
2016-01-20 16:22:06,092 INFO [main]
org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
hadoop-metrics2.properties
2016-01-20 16:22:06,143 INFO [main]
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period
at 10 second(s).
2016-01-20 16:22:06,143 INFO [main]
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system
started
2016-01-20 16:22:06,150 INFO [main] org.apache.hadoop.mapred.YarnChild:
Executing with tokens:
2016-01-20 16:22:06,150 INFO [main] org.apache.hadoop.mapred.YarnChild:
Kind: mapreduce.job, Service: job_1452787370536_0086, Ident:
(org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@dc42543)
2016-01-20 16:22:06,223 INFO [main] org.apache.hadoop.mapred.YarnChild:
Sleeping for 0ms before retrying again. Got null now.
2016-01-20 16:22:06,412 INFO [main] org.apache.hadoop.mapred.YarnChild:
mapreduce.cluster.local.dir for child:
nm-local-dir/usercache/kylin/appcache/application_1452787370536_0086
2016-01-20 16:22:06,716 INFO [main]
org.apache.hadoop.conf.Configuration.deprecation: session.id is deprecated.
Instead, use dfs.metrics.session-id
2016-01-20 16:22:07,045 INFO [main] org.apache.hadoop.mapred.Task: Using
ResourceCalculatorProcessTree : [ ]
2016-01-20 16:22:07,237 INFO [main] org.apache.hadoop.mapred.MapTask:
Processing split:
hdfs://**/kylin_metadata/kylin-8acccbc4-84b7-4774-b668-f64228c09672/kylin_intermediate_avro_global_device_20151219000000_20151221000000_8acccbc4_84b7_4774_b668_f64228c09672/000000_0:0+16387256
2016-01-20 16:22:07,283 INFO [main] org.apache.hadoop.mapred.MapTask:
(EQUATOR) 0 kvi 26214396(104857584)
2016-01-20 16:22:07,284 INFO [main] org.apache.hadoop.mapred.MapTask:
mapreduce.task.io.sort.mb: 100
2016-01-20 16:22:07,284 INFO [main] org.apache.hadoop.mapred.MapTask: soft
limit at 83886080
2016-01-20 16:22:07,284 INFO [main] org.apache.hadoop.mapred.MapTask:
bufstart = 0; bufvoid = 104857600
2016-01-20 16:22:07,284 INFO [main] org.apache.hadoop.mapred.MapTask:
kvstart = 26214396; length = 6553600
2016-01-20 16:22:07,289 INFO [main]
com.hadoop.compression.lzo.GPLNativeCodeLoader: Loaded native gpl library
from the embedded binaries
2016-01-20 16:22:07,291 INFO [main] com.hadoop.compression.lzo.LzoCodec:
Successfully loaded & initialized native-lzo library [hadoop-lzo rev
826e7d8d3e839964dd9ed2d5f83296254b2c71d3]
2016-01-20 16:22:07,295 INFO [main] org.apache.hadoop.mapred.MapTask: Map
output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2016-01-20 16:22:07,316 INFO [main]
org.apache.hadoop.conf.Configuration.deprecation: hadoop.native.lib is
deprecated. Instead, use io.native.lib.available
2016-01-20 16:22:07,317 INFO [main] org.apache.hadoop.io.compress.CodecPool:
Got brand-new decompressor [.lzo_deflate]
2016-01-20 16:22:07,322 INFO [main] org.apache.hadoop.io.compress.CodecPool:
Got brand-new decompressor [.lzo_deflate]
2016-01-20 16:22:07,322 INFO [main] org.apache.hadoop.io.compress.CodecPool:
Got brand-new decompressor [.lzo_deflate]
2016-01-20 16:22:07,322 INFO [main] org.apache.hadoop.io.compress.CodecPool:
Got brand-new decompressor [.lzo_deflate]
2016-01-20 16:22:07,328 INFO [main]
org.apache.kylin.job.hadoop.AbstractHadoopJob: The absolute path for meta
dir is
/**/tmp/nm-local-dir/usercache/kylin/appcache/application_1452787370536_0086/container_1452787370536_0086_01_000003/meta
2016-01-20 16:22:07,330 INFO [main] org.apache.kylin.common.KylinConfig: Use
KYLIN_CONF=/**/tmp/nm-local-dir/usercache/kylin/appcache/application_1452787370536_0086/container_1452787370536_0086_01_000003/meta
2016-01-20 16:22:07,344 INFO [main] org.apache.kylin.cube.CubeManager:
Initializing CubeManager with config
/**/tmp/nm-local-dir/usercache/kylin/appcache/application_1452787370536_0086/container_1452787370536_0086_01_000003/meta
2016-01-20 16:22:07,346 INFO [main]
org.apache.kylin.common.persistence.ResourceStore: Using metadata url
/**/tmp/nm-local-dir/usercache/kylin/appcache/application_1452787370536_0086/container_1452787370536_0086_01_000003/meta
for resource store
2016-01-20 16:22:07,558 INFO [main] org.apache.kylin.cube.CubeDescManager:
Initializing CubeDescManager with config
/**/tmp/nm-local-dir/usercache/kylin/appcache/application_1452787370536_0086/container_1452787370536_0086_01_000003/meta
2016-01-20 16:22:07,559 INFO [main] org.apache.kylin.cube.CubeDescManager:
Reloading Cube Metadata from folder
/**/tmp/nm-local-dir/usercache/kylin/appcache/application_1452787370536_0086/container_1452787370536_0086_01_000003/meta/cube_desc
2016-01-20 16:23:14,463 INFO [main]
org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled 100000 records!
2016-01-20 16:24:20,530 INFO [main]
org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled 200000 records!
2016-01-20 16:25:26,496 INFO [main]
org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled 300000 records!
2016-01-20 16:26:32,492 INFO [main]
org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled 400000 records!
2016-01-20 16:27:38,671 INFO [main]
org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled 500000 records!
2016-01-20 16:28:45,026 INFO [main]
org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled 600000 records!
2016-01-20 16:29:51,697 INFO [main]
org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled 700000 records!
2016-01-20 16:30:58,548 INFO [main]
org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled 800000 records!
2016-01-20 16:32:05,959 INFO [main]
org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled 900000 records!
2016-01-20 16:33:13,550 INFO [main]
org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled 1000000 records!
2016-01-20 16:34:20,893 INFO [main]
org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled 1100000 records!
2016-01-20 16:35:28,145 INFO [main]
org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled 1200000 records!
2016-01-20 16:36:11,918 INFO [main] org.apache.hadoop.mapred.MapTask:
Spilling map output
2016-01-20 16:36:11,918 INFO [main] org.apache.hadoop.mapred.MapTask:
bufstart = 0; bufend = 63651120; bufvoid = 104857600
2016-01-20 16:36:11,918 INFO [main] org.apache.hadoop.mapred.MapTask:
kvstart = 26214396(104857584); kvend = 21155656(84622624); length =
5058741/6553600
2016-01-20 16:36:11,918 INFO [main] org.apache.hadoop.mapred.MapTask:
(EQUATOR) 68735104 kvi 17183772(68735088)
2016-01-20 16:36:12,669 INFO [SpillThread]
org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor
[.lzo_deflate]
2016-01-20 16:36:12,706 INFO [SpillThread]
org.apache.kylin.job.hadoop.AbstractHadoopJob: The absolute path for meta
dir is
/**/tmp/nm-local-dir/usercache/kylin/appcache/application_1452787370536_0086/container_1452787370536_0086_01_000003/meta
2016-01-20 16:36:37,977 INFO [main]
org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled 1300000 records!
2016-01-20 16:37:37,524 INFO [SpillThread]
org.apache.kylin.job.hadoop.cube.CuboidReducer: Handled 100000 records!
2016-01-20 16:37:48,445 INFO [main]
org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled 1400000 records!
2016-01-20 16:39:00,123 INFO [SpillThread]
org.apache.kylin.job.hadoop.cube.CuboidReducer: Handled 200000 records!
2016-01-20 16:39:00,589 INFO [main]
org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled 1500000 records!
2016-01-20 16:39:46,678 INFO [main] org.apache.hadoop.mapred.MapTask:
Starting flush of map output
2016-01-20 16:40:21,932 INFO [SpillThread]
org.apache.kylin.job.hadoop.cube.CuboidReducer: Handled 300000 records!
2016-01-20 16:41:42,143 INFO [SpillThread]
org.apache.kylin.job.hadoop.cube.CuboidReducer: Handled 400000 records!
2016-01-20 16:43:02,409 INFO [SpillThread]
org.apache.kylin.job.hadoop.cube.CuboidReducer: Handled 500000 records!
2016-01-20 16:44:21,803 INFO [SpillThread]
org.apache.kylin.job.hadoop.cube.CuboidReducer: Handled 600000 records!
2016-01-20 16:45:41,249 INFO [SpillThread]
org.apache.kylin.job.hadoop.cube.CuboidReducer: Handled 700000 records!
2016-01-20 16:47:00,661 INFO [SpillThread]
org.apache.kylin.job.hadoop.cube.CuboidReducer: Handled 800000 records!
2016-01-20 16:48:20,193 INFO [SpillThread]
org.apache.kylin.job.hadoop.cube.CuboidReducer: Handled 900000 records!
2016-01-20 16:49:39,569 INFO [SpillThread]
org.apache.kylin.job.hadoop.cube.CuboidReducer: Handled 1000000 records!
2016-01-20 16:50:59,397 INFO [SpillThread]
org.apache.kylin.job.hadoop.cube.CuboidReducer: Handled 1100000 records!
2016-01-20 16:52:18,927 INFO [SpillThread]
org.apache.kylin.job.hadoop.cube.CuboidReducer: Handled 1200000 records!
2016-01-20 16:53:10,339 INFO [SpillThread] org.apache.hadoop.mapred.MapTask:
Finished spill 0
2016-01-20 16:53:10,339 INFO [main] org.apache.hadoop.mapred.MapTask:
(RESET) equator 68735104 kv 17183772(68735088) kvi 15983048(63932192)
2016-01-20 16:53:10,339 INFO [main] org.apache.hadoop.mapred.MapTask:
Spilling map output
2016-01-20 16:53:10,340 INFO [main] org.apache.hadoop.mapred.MapTask:
bufstart = 68735104; bufend = 83814293; bufvoid = 104857600
2016-01-20 16:53:10,340 INFO [main] org.apache.hadoop.mapred.MapTask:
kvstart = 17183772(68735088); kvend = 15983052(63932208); length =
1200721/6553600
2016-01-20 16:53:10,530 INFO [main]
org.apache.kylin.job.hadoop.AbstractHadoopJob: The absolute path for meta
dir is
/**/tmp/nm-local-dir/usercache/kylin/appcache/application_1452787370536_0086/container_1452787370536_0086_01_000003/meta
2016-01-20 16:54:30,303 INFO [main]
org.apache.kylin.job.hadoop.cube.CuboidReducer: Handled 100000 records!
2016-01-20 16:55:49,729 INFO [main]
org.apache.kylin.job.hadoop.cube.CuboidReducer: Handled 200000 records!
2016-01-20 16:57:09,126 INFO [main]
org.apache.kylin.job.hadoop.cube.CuboidReducer: Handled 300000 records!
2016-01-20 16:57:09,272 INFO [main] org.apache.hadoop.mapred.MapTask:
Finished spill 1
2016-01-20 16:57:09,277 INFO [main] org.apache.hadoop.mapred.Merger: Merging
2 sorted segments
2016-01-20 16:57:09,282 INFO [main] org.apache.hadoop.mapred.Merger: Down to
the last merge-pass, with 2 segments left of total size: 26184336 bytes
2016-01-20 16:57:10,555 INFO [main] org.apache.hadoop.mapred.Task:
Task:attempt_1452787370536_0086_m_000000_0 is done. And is in the process of
committing
2016-01-20 16:57:10,600 INFO [main] org.apache.hadoop.mapred.Task: Task
'attempt_1452787370536_0086_m_000000_0' done.
--
View this message in context:
http://apache-kylin.74782.x6.nabble.com/From-the-Build-Base-Cuboid-Data-step-to-Build-N-Dimension-steps-Too-much-time-is-taken-tp3351.html
Sent from the Apache Kylin mailing list archive at Nabble.com.
