Both mappers hangs in spilling map output. May some Hadoop configuration be
wrong with our cluster?
e: Handled 1200000 records!
2016-06-16 10:12:01,895 INFO [main]
org.apache.kylin.engine.mr.steps.BaseCuboidMapperBase: Handled 1300000 records!
2016-06-16 10:12:02,571 INFO [main]
org.apache.kylin.engine.mr.steps.BaseCuboidMapperBase: Handled 1400000 records!
2016-06-16 10:12:03,247 INFO [main]
org.apache.kylin.engine.mr.steps.BaseCuboidMapperBase: Handled 1500000 records!
2016-06-16 10:12:03,948 INFO [main]
org.apache.kylin.engine.mr.steps.BaseCuboidMapperBase: Handled 1600000 records!
2016-06-16 10:12:04,647 INFO [main]
org.apache.kylin.engine.mr.steps.BaseCuboidMapperBase: Handled 1700000 records!
2016-06-16 10:12:05,368 INFO [main]
org.apache.kylin.engine.mr.steps.BaseCuboidMapperBase: Handled 1800000 records!
2016-06-16 10:12:06,056 INFO [main]
org.apache.kylin.engine.mr.steps.BaseCuboidMapperBase: Handled 1900000 records!
2016-06-16 10:12:06,744 INFO [main]
org.apache.kylin.engine.mr.steps.BaseCuboidMapperBase: Handled 2000000 records!
2016-06-16 10:12:07,445 INFO [main]
org.apache.kylin.engine.mr.steps.BaseCuboidMapperBase: Handled 2100000 records!
2016-06-16 10:12:08,143 INFO [main]
org.apache.kylin.engine.mr.steps.BaseCuboidMapperBase: Handled 2200000 records!
2016-06-16 10:12:08,842 INFO [main]
org.apache.kylin.engine.mr.steps.BaseCuboidMapperBase: Handled 2300000 records!
2016-06-16 10:12:09,540 INFO [main]
org.apache.kylin.engine.mr.steps.BaseCuboidMapperBase: Handled 2400000 records!
2016-06-16 10:12:10,238 INFO [main]
org.apache.kylin.engine.mr.steps.BaseCuboidMapperBase: Handled 2500000 records!
2016-06-16 10:12:10,938 INFO [main]
org.apache.kylin.engine.mr.steps.BaseCuboidMapperBase: Handled 2600000 records!
2016-06-16 10:12:11,624 INFO [main]
org.apache.kylin.engine.mr.steps.BaseCuboidMapperBase: Handled 2700000 records!
2016-06-16 10:12:12,312 INFO [main]
org.apache.kylin.engine.mr.steps.BaseCuboidMapperBase: Handled 2800000 records!
2016-06-16 10:12:12,999 INFO [main]
org.apache.kylin.engine.mr.steps.BaseCuboidMapperBase: Handled 2900000 records!
2016-06-16 10:12:13,698 INFO [main]
org.apache.kylin.engine.mr.steps.BaseCuboidMapperBase: Handled 3000000 records!
2016-06-16 10:12:14,387 INFO [main]
org.apache.kylin.engine.mr.steps.BaseCuboidMapperBase: Handled 3100000 records!
2016-06-16 10:12:15,075 INFO [main]
org.apache.kylin.engine.mr.steps.BaseCuboidMapperBase: Handled 3200000 records!
2016-06-16 10:12:15,764 INFO [main]
org.apache.kylin.engine.mr.steps.BaseCuboidMapperBase: Handled 3300000 records!
2016-06-16 10:12:16,460 INFO [main]
org.apache.kylin.engine.mr.steps.BaseCuboidMapperBase: Handled 3400000 records!
2016-06-16 10:12:17,147 INFO [main]
org.apache.kylin.engine.mr.steps.BaseCuboidMapperBase: Handled 3500000 records!
2016-06-16 10:12:17,834 INFO [main]
org.apache.kylin.engine.mr.steps.BaseCuboidMapperBase: Handled 3600000 records!
2016-06-16 10:12:18,520 INFO [main]
org.apache.kylin.engine.mr.steps.BaseCuboidMapperBase: Handled 3700000 records!
2016-06-16 10:12:19,216 INFO [main]
org.apache.kylin.engine.mr.steps.BaseCuboidMapperBase: Handled 3800000 records!
2016-06-16 10:12:19,901 INFO [main]
org.apache.kylin.engine.mr.steps.BaseCuboidMapperBase: Handled 3900000 records!
2016-06-16 10:12:20,590 INFO [main]
org.apache.kylin.engine.mr.steps.BaseCuboidMapperBase: Handled 4000000 records!
2016-06-16 10:12:21,286 INFO [main]
org.apache.kylin.engine.mr.steps.BaseCuboidMapperBase: Handled 4100000 records!
2016-06-16 10:12:21,970 INFO [main]
org.apache.kylin.engine.mr.steps.BaseCuboidMapperBase: Handled 4200000 records!
2016-06-16 10:12:22,657 INFO [main]
org.apache.kylin.engine.mr.steps.BaseCuboidMapperBase: Handled 4300000 records!
2016-06-16 10:12:22,801 INFO [main] org.apache.hadoop.mapred.MapTask: Starting
flush of map output
2016-06-16 10:12:22,802 INFO [main] org.apache.hadoop.mapred.MapTask: Spilling
map output
2016-06-16 10:12:22,802 INFO [main] org.apache.hadoop.mapred.MapTask: bufstart
= 0; bufend = 190884986; bufvoid = 536870912
2016-06-16 10:12:22,802 INFO [main] org.apache.hadoop.mapred.MapTask: kvstart =
134217724(536870896); kvend = 116938212(467752848); length = 17279513/33554432
Cheers,
Jie
Am 16.06.2016 um 04:38 schrieb hongbin ma:
Since you intermediate table is merely 200M, each mapper deals with 100M
fact table. This is not very heavy for a mapper. Have you checked the log
for the mapper to know what is it doing in the 3600s?
On Thu, Jun 16, 2016 at 10:35 AM, hongbin ma <[email protected]> wrote:
3.7 GB of 8.4 GB virtual memory used
do you have any idea why the container is using lots of virtual memory?
is there any heavy contention on the memory resource? Not sure if it's the
root cause for slow mappers
--
Regards,
*Bin Mahone | 马洪宾*
Apache Kylin: http://kylin.io
Github: https://github.com/binmahone
