I am dealing with a table that has 71679920 records and 30 columns. It occupies about 13Gigs on HDFS. Two columns of this table contain latitude and longitude (both double) and another column contains geohash (12 char long strings). I am trying to index this table in Hive.
I run the following two queries in Hive shell: create index idx on table vzt_oct (slocnhash) as 'COMPACT' with deferred rebuild; alter index idx on vzt_oct rebuild; but I run up against Java Heap Space error. I have tried setting higher and higher values for mapreduce.map.memory.mb and mapreduce.reduce.memory.mb. The default value is 1024. I bump them up to 2000, 4000, 6000 upto 10000 after which I run in to Job execution error. Number of mappers are 50 and number of reducers are 54. I tried reducing the number of reducers as well but could not fix the problem. I have an EC2 hosted 3 node cluster composed of c3.2xlarge instances. Hadoop version 2.7.0 and Hive version 1.2.1. Each machine has 16Gigs of RAM. Questions: 1) Which parameters should I fiddle with? 2) Where are the error logs? $HADOOP_HOME/logs/userlogs? 3) Why is the tracking URL (line 14 in the attached log) not available after the job fails? 4) Why is the taskdetails.jsp page (line 122 in the attached log) never available?
Logging initialized using configuration in jar:file:/opt/hive/lib/hive-common-1.2.1.jar!/hive-log4j.properties set hive.cli.print.header=true alter index idx on vzt_oct rebuild Query ID = hadoopuser_20151130204742_a9167b79-99b7-476b-818f-127f92e1a461 Total jobs = 1 Launching Job 1 out of 1 Number of reduce tasks not specified. Estimated from input data size: 54 In order to change the average load for a reducer (in bytes): set hive.exec.reducers.bytes.per.reducer=<number> In order to limit the maximum number of reducers: set hive.exec.reducers.max=<number> In order to set a constant number of reducers: set mapreduce.job.reduces=<number> Starting Job = job_1448895898988_0002, Tracking URL = http://hadoop-e1e-m-01:8088/proxy/application_1448895898988_0002/ Kill Command = /home/hadoopuser/hadoop/bin/hadoop job -kill job_1448895898988_0002 Hadoop job information for Stage-1: number of mappers: 50; number of reducers: 54 2015-11-30 20:47:49,944 Stage-1 map = 0%, reduce = 0% 2015-11-30 20:48:04,544 Stage-1 map = 1%, reduce = 0%, Cumulative CPU 80.91 sec 2015-11-30 20:48:06,649 Stage-1 map = 6%, reduce = 0%, Cumulative CPU 136.25 sec 2015-11-30 20:48:08,761 Stage-1 map = 7%, reduce = 0%, Cumulative CPU 173.51 sec 2015-11-30 20:48:09,806 Stage-1 map = 10%, reduce = 0%, Cumulative CPU 202.74 sec 2015-11-30 20:48:10,857 Stage-1 map = 16%, reduce = 0%, Cumulative CPU 213.98 sec 2015-11-30 20:48:11,905 Stage-1 map = 19%, reduce = 0%, Cumulative CPU 237.63 sec 2015-11-30 20:48:12,950 Stage-1 map = 21%, reduce = 0%, Cumulative CPU 249.57 sec 2015-11-30 20:48:13,998 Stage-1 map = 22%, reduce = 0%, Cumulative CPU 258.29 sec 2015-11-30 20:48:15,051 Stage-1 map = 24%, reduce = 0%, Cumulative CPU 275.88 sec 2015-11-30 20:48:16,092 Stage-1 map = 27%, reduce = 0%, Cumulative CPU 296.2 sec 2015-11-30 20:48:17,127 Stage-1 map = 28%, reduce = 0%, Cumulative CPU 302.55 sec 2015-11-30 20:48:18,174 Stage-1 map = 36%, reduce = 0%, Cumulative CPU 333.17 sec 2015-11-30 20:48:19,236 Stage-1 map = 37%, reduce = 0%, Cumulative CPU 343.93 sec 2015-11-30 20:48:20,329 Stage-1 map = 39%, reduce = 0%, Cumulative CPU 347.63 sec 2015-11-30 20:48:22,481 Stage-1 map = 40%, reduce = 0%, Cumulative CPU 379.49 sec 2015-11-30 20:48:23,533 Stage-1 map = 41%, reduce = 0%, Cumulative CPU 391.39 sec 2015-11-30 20:48:24,564 Stage-1 map = 42%, reduce = 0%, Cumulative CPU 400.22 sec 2015-11-30 20:48:25,606 Stage-1 map = 47%, reduce = 1%, Cumulative CPU 411.02 sec 2015-11-30 20:48:27,697 Stage-1 map = 48%, reduce = 1%, Cumulative CPU 424.57 sec 2015-11-30 20:48:30,803 Stage-1 map = 50%, reduce = 2%, Cumulative CPU 454.59 sec 2015-11-30 20:48:31,850 Stage-1 map = 51%, reduce = 2%, Cumulative CPU 487.26 sec 2015-11-30 20:48:33,938 Stage-1 map = 52%, reduce = 2%, Cumulative CPU 491.14 sec 2015-11-30 20:48:34,970 Stage-1 map = 54%, reduce = 2%, Cumulative CPU 509.36 sec 2015-11-30 20:48:36,005 Stage-1 map = 56%, reduce = 3%, Cumulative CPU 546.5 sec 2015-11-30 20:48:37,048 Stage-1 map = 59%, reduce = 3%, Cumulative CPU 552.09 sec 2015-11-30 20:48:38,078 Stage-1 map = 63%, reduce = 3%, Cumulative CPU 577.68 sec 2015-11-30 20:48:39,108 Stage-1 map = 65%, reduce = 3%, Cumulative CPU 582.97 sec 2015-11-30 20:48:40,161 Stage-1 map = 69%, reduce = 4%, Cumulative CPU 588.19 sec 2015-11-30 20:48:41,247 Stage-1 map = 71%, reduce = 4%, Cumulative CPU 608.46 sec 2015-11-30 20:48:42,280 Stage-1 map = 72%, reduce = 4%, Cumulative CPU 612.66 sec 2015-11-30 20:48:43,307 Stage-1 map = 72%, reduce = 5%, Cumulative CPU 612.98 sec 2015-11-30 20:48:46,388 Stage-1 map = 73%, reduce = 5%, Cumulative CPU 624.57 sec 2015-11-30 20:48:49,469 Stage-1 map = 77%, reduce = 5%, Cumulative CPU 676.71 sec 2015-11-30 20:48:51,522 Stage-1 map = 78%, reduce = 5%, Cumulative CPU 683.47 sec 2015-11-30 20:48:52,553 Stage-1 map = 79%, reduce = 5%, Cumulative CPU 710.14 sec 2015-11-30 20:48:55,635 Stage-1 map = 82%, reduce = 5%, Cumulative CPU 751.07 sec 2015-11-30 20:48:56,666 Stage-1 map = 83%, reduce = 5%, Cumulative CPU 763.42 sec 2015-11-30 20:48:58,745 Stage-1 map = 85%, reduce = 5%, Cumulative CPU 783.17 sec 2015-11-30 20:48:59,771 Stage-1 map = 87%, reduce = 5%, Cumulative CPU 801.29 sec 2015-11-30 20:49:00,800 Stage-1 map = 88%, reduce = 6%, Cumulative CPU 818.9 sec 2015-11-30 20:49:01,828 Stage-1 map = 89%, reduce = 6%, Cumulative CPU 826.51 sec 2015-11-30 20:49:02,852 Stage-1 map = 93%, reduce = 6%, Cumulative CPU 844.7 sec 2015-11-30 20:49:03,877 Stage-1 map = 95%, reduce = 6%, Cumulative CPU 849.76 sec 2015-11-30 20:49:04,902 Stage-1 map = 97%, reduce = 6%, Cumulative CPU 864.92 sec 2015-11-30 20:49:05,934 Stage-1 map = 99%, reduce = 7%, Cumulative CPU 871.39 sec 2015-11-30 20:49:06,975 Stage-1 map = 100%, reduce = 7%, Cumulative CPU 873.76 sec 2015-11-30 20:49:08,044 Stage-1 map = 100%, reduce = 9%, Cumulative CPU 877.33 sec 2015-11-30 20:49:09,142 Stage-1 map = 100%, reduce = 12%, Cumulative CPU 891.59 sec 2015-11-30 20:49:10,196 Stage-1 map = 100%, reduce = 14%, Cumulative CPU 900.84 sec 2015-11-30 20:49:11,240 Stage-1 map = 100%, reduce = 17%, Cumulative CPU 915.64 sec 2015-11-30 20:49:12,320 Stage-1 map = 100%, reduce = 21%, Cumulative CPU 948.82 sec 2015-11-30 20:49:13,374 Stage-1 map = 100%, reduce = 25%, Cumulative CPU 974.19 sec 2015-11-30 20:49:14,454 Stage-1 map = 100%, reduce = 27%, Cumulative CPU 992.53 sec 2015-11-30 20:49:15,482 Stage-1 map = 100%, reduce = 31%, Cumulative CPU 1016.25 sec 2015-11-30 20:49:16,535 Stage-1 map = 100%, reduce = 32%, Cumulative CPU 1022.16 sec 2015-11-30 20:49:17,576 Stage-1 map = 100%, reduce = 34%, Cumulative CPU 1035.67 sec 2015-11-30 20:49:18,629 Stage-1 map = 100%, reduce = 36%, Cumulative CPU 1044.04 sec 2015-11-30 20:49:20,739 Stage-1 map = 100%, reduce = 37%, Cumulative CPU 1059.62 sec 2015-11-30 20:49:21,810 Stage-1 map = 100%, reduce = 38%, Cumulative CPU 1066.55 sec 2015-11-30 20:49:22,854 Stage-1 map = 100%, reduce = 40%, Cumulative CPU 1076.97 sec 2015-11-30 20:49:23,912 Stage-1 map = 100%, reduce = 42%, Cumulative CPU 1092.73 sec 2015-11-30 20:49:24,953 Stage-1 map = 100%, reduce = 41%, Cumulative CPU 1086.73 sec 2015-11-30 20:49:25,994 Stage-1 map = 100%, reduce = 43%, Cumulative CPU 1100.21 sec 2015-11-30 20:49:27,035 Stage-1 map = 100%, reduce = 45%, Cumulative CPU 1111.31 sec 2015-11-30 20:49:28,086 Stage-1 map = 100%, reduce = 52%, Cumulative CPU 1142.54 sec 2015-11-30 20:49:29,128 Stage-1 map = 100%, reduce = 54%, Cumulative CPU 1155.11 sec 2015-11-30 20:49:30,167 Stage-1 map = 100%, reduce = 57%, Cumulative CPU 1169.16 sec 2015-11-30 20:49:31,210 Stage-1 map = 100%, reduce = 62%, Cumulative CPU 1194.49 sec 2015-11-30 20:49:33,284 Stage-1 map = 100%, reduce = 64%, Cumulative CPU 1218.67 sec 2015-11-30 20:49:35,384 Stage-1 map = 100%, reduce = 66%, Cumulative CPU 1229.95 sec 2015-11-30 20:49:36,445 Stage-1 map = 100%, reduce = 68%, Cumulative CPU 1242.21 sec 2015-11-30 20:49:37,480 Stage-1 map = 100%, reduce = 70%, Cumulative CPU 1249.9 sec 2015-11-30 20:49:38,523 Stage-1 map = 100%, reduce = 74%, Cumulative CPU 1276.11 sec 2015-11-30 20:49:39,598 Stage-1 map = 100%, reduce = 76%, Cumulative CPU 1284.2 sec 2015-11-30 20:49:40,652 Stage-1 map = 100%, reduce = 77%, Cumulative CPU 1292.0 sec 2015-11-30 20:49:41,699 Stage-1 map = 100%, reduce = 82%, Cumulative CPU 1318.52 sec 2015-11-30 20:49:43,785 Stage-1 map = 100%, reduce = 85%, Cumulative CPU 1340.8 sec 2015-11-30 20:49:44,817 Stage-1 map = 100%, reduce = 86%, Cumulative CPU 1346.2 sec 2015-11-30 20:49:45,850 Stage-1 map = 100%, reduce = 93%, Cumulative CPU 1383.21 sec 2015-11-30 20:49:46,872 Stage-1 map = 100%, reduce = 94%, Cumulative CPU 1390.82 sec 2015-11-30 20:49:47,894 Stage-1 map = 100%, reduce = 97%, Cumulative CPU 1415.14 sec 2015-11-30 20:49:49,938 Stage-1 map = 100%, reduce = 94%, Cumulative CPU 1400.05 sec 2015-11-30 20:50:03,251 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1376.18 sec MapReduce Total cumulative CPU time: 22 minutes 56 seconds 180 msec Ended Job = job_1448895898988_0002 with errors Error during job, obtaining debugging information... Examining task ID: task_1448895898988_0002_m_000012 (and more) from job job_1448895898988_0002 Examining task ID: task_1448895898988_0002_m_000007 (and more) from job job_1448895898988_0002 Examining task ID: task_1448895898988_0002_m_000021 (and more) from job job_1448895898988_0002 Examining task ID: task_1448895898988_0002_m_000034 (and more) from job job_1448895898988_0002 Examining task ID: task_1448895898988_0002_m_000037 (and more) from job job_1448895898988_0002 Examining task ID: task_1448895898988_0002_r_000002 (and more) from job job_1448895898988_0002 Examining task ID: task_1448895898988_0002_r_000013 (and more) from job job_1448895898988_0002 Examining task ID: task_1448895898988_0002_r_000018 (and more) from job job_1448895898988_0002 Examining task ID: task_1448895898988_0002_r_000034 (and more) from job job_1448895898988_0002 Examining task ID: task_1448895898988_0002_r_000039 (and more) from job job_1448895898988_0002 Examining task ID: task_1448895898988_0002_r_000018 (and more) from job job_1448895898988_0002 Task with the most failures(4): ----- Task ID: task_1448895898988_0002_r_000018 URL: http://hadoop-e1e-m-01:8088/taskdetails.jsp?jobid=job_1448895898988_0002&tipid=task_1448895898988_0002_r_000018 ----- Diagnostic Messages for this Task: Error: Java heap space FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask MapReduce Jobs Launched: Stage-Stage-1: Map: 50 Reduce: 54 Cumulative CPU: 1376.18 sec HDFS Read: 13619965174 HDFS Write: 919877583 FAIL Total MapReduce CPU Time Spent: 22 minutes 56 seconds 180 msec
--------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
