hello everyone, when i use Hiveserver2 service , I am in my program inside the way through the JDBC to analyze SQL .
Sometimes there will be such a problem : ------------ MapredLocal task succeeded Launching Job 6 out of 6 Number of reduce tasks is set to 0 since there's no reduce operator Starting Job = job_1495161006766_0177, Tracking URL = http://master:8088/proxy/application_1495161006766_0177/ Kill Command = /home/hadoop/software/hadoop/bin/hadoop job -kill job_1495161006766_0177 Hadoop job information for Stage-6: number of mappers: 2; number of reducers: 0 2017-05-26 17:31:50,615 Stage-6 map = 0%, reduce = 0% 2017-05-26 17:32:04,106 Stage-6 map = 50%, reduce = 0%, Cumulative CPU 3.14 sec 2017-05-26 17:32:05,130 Stage-6 map = 100%, reduce = 0%, Cumulative CPU 6.51 sec MapReduce Total cumulative CPU time: 6 seconds 510 msec Ended Job = job_1495161006766_0177 Moving data to directory hdfs://master:9000/user/hive/warehouse/m9.db/testm91648 FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.StatsTask. PermGen space MapReduce Jobs Launched: Stage-Stage-5: Map: 1 Reduce: 1 Cumulative CPU: 1.85 sec HDFS Read: 7645 HDFS Write: 130 SUCCESS Stage-Stage-8: Map: 2 Cumulative CPU: 4.7 sec HDFS Read: 1463444 HDFS Write: 1909015 SUCCESS Stage-Stage-4: Map: 1 Reduce: 1 Cumulative CPU: 5.33 sec HDFS Read: 1915512 HDFS Write: 2175235 SUCCESS Stage-Stage-6: Map: 2 Cumulative CPU: 6.51 sec HDFS Read: 1309412 HDFS Write: 2678982 SUCCESS Total MapReduce CPU Time Spent: 18 seconds 390 msec Exception in thread "HiveServer2-Handler-Pool: Thread-39" Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "HiveServer2-Handler-Pool: Thread-39" ---- Once such a problem occurs, the hiveserver2 service hangs ,and I use the command (netstat -anp | grep 10000)can not monitor 10,000 ports . Software version : java 1.7 hadoop2.6.0-cdh5.4.0 apache-hive-2.1.0 SQL example: create table if not exists W_CONTENT_RECORD ( APP_CODE STRING, USER_ID STRING , N_COUNT BIGINT , H_COUNT_GR BIGINT , H_COUNT_JZ BIGINT ) PARTITIONED by (dt string ); create table if not exists TMP_LOG_COUNT_201705041553 as select APP_CODE, USER_ID, count(*) as LOG_COUNT from AS_OPERATION_LOG where (OPERATE_TIME >= '2017/03/28 09:00:00') and (OPERATE_TIME < '2017/04/13 10:00:00') group by APP_CODE,USER_ID; insert into W_CONTENT_RECORD partition (dt='2017/05/25 11:00') select a.APP_CODE as APP_CODE, a.USER_ID as USER_ID, a.LOG_COUNT as N_COUNT, b.count as H_COUNT_GR, b.count as H_COUNT_JZ from TMP_LOG_COUNT_201705041553 as a left outer join (select a1.APP_CODE,a1.USER_ID,(sum(a1.N_COUNT)/7) as count from W_CONTENT_RECORD as a1 where a1.dt in (select distinct(b1.dt) from W_CONTENT_RECORD as b1 where (b1.dt>='2017/04/10 00:00') and (b1.dt<='2017/04/20 23:00') and substr(b1.dt,12) = '10:00') group by a1.APP_CODE,a1.USER_ID) as b on a.APP_CODE=b.APP_CODE and a.USER_ID=b.USER_ID; create table if not exists TMP_SUM_COUNT_201705041553 as select APP_CODE,USER_ID,sum(N_COUNT) as N_COUNT,sum(H_COUNT_GR) as H_COUNT_GR,sum(H_COUNT_JZ) as H_COUNT_JZ from W_CONTENT_RECORD where dt in ('2017/04/18 09:00','2017/04/18 10:00','2017/04/18 11:00') group by APP_CODE,USER_ID; create table if not exists TMP_TOP_COUNT_201705041553 as select collect_set(a.APP_CODE)[0] as App_Code, a.USER_ID as User_ID, a.N_COUNT as n_count, collect_set(a.H_COUNT_GR)[0] as h_count_gr, collect_set(a.H_COUNT_JZ)[0] as h_count_jz from TMP_SUM_COUNT_201705041553 as a left outer join TMP_SUM_COUNT_201705041553 as b on a.USER_ID=b.USER_ID where a.N_COUNT<=b.N_COUNT and a.USER_ID<>'' and a.USER_ID is not null group by a.USER_ID,a.N_COUNT having count(b.N_COUNT)<=3; create table if not exists WARN_INFO( App_Code STRING , USER_ID STRING , WARN_SCORE STRING , RULE_ID STRING , TYPE_ID STRING )PARTITIONED by (dt string ); add jar /home/hadoop/m9server/udf/hiveUDF.jar; create temporary function ProportionOneDimensionUDF as 'com.trs.hive.hiveUDF.ProportionOneDimensionUDF'; insert into WARN_INFO partition (dt='2017/05/02 09:00') select App_Code,User_ID,ProportionOneDimensionUDF(0.3,n_count,h_count_gr) as SCORE_GR,1,1 from TMP_TOP_COUNT_201705041553 where n_count is not null and h_count_gr is not null and h_count_jz is not null; insert into WARN_INFO()_2016 partition (dt='2017/05/02 09:00') select App_Code,User_ID,ProportionOneDimensionUDF(0.3,n_count,h_count_jz) as SCORE_GR,2,1 from TMP_TOP_COUNT_201705041553 where n_count is not null and h_count_gr is not null and h_count_jz is not null; drop table if exists TMP_LOG_COUNT_201705041553; drop table if exists TMP_SUM_COUNT_201705041553; drop table if exists TMP_TOP_COUNT_201705041553; i try many methods ,but can not resovle it ,so i came here. thanks wei
