Hi, HDP2.3+Kylin 2.2
when run a bigdata set,hadoop it could't work,add set hive.auto.convert.join=false can workarounds for this bug: Where can I add this command in kylin? Error log: Create and distribute table, cmd: hive -e "USE default; DROP TABLE IF EXISTS kylin_intermediate_psn_pur_loan_20d88469_c478_41c6_a480_42c61aa0f897; CREATE EXTERNAL TABLE IF NOT EXISTS kylin_intermediate_psn_pur_loan_20d88469_c478_41c6_a480_42c61aa0f897 ( PSN_PUR_LOAN_DATE_CD date ,PSN_PUR_LOAN_ORG_CD string ,PSN_PUR_LOAN_CURR_CD string ,PSN_PUR_LOAN_QUAL_CD string ,PSN_PUR_LOAN_LON_BUSI_CD string ,PSN_PUR_LOAN_GBENT_SCALE_CD string ,PSN_PUR_LOAN_IND_INV_CD string ,PSN_PUR_LOAN_BASE_RATE_TIMES decimal(16,9) ,PSN_PUR_LOAN_LON_AMT decimal(25,5) ,PSN_PUR_LOAN_GRNT_AMT decimal(25,5) ,PSN_PUR_LOAN_BANK_GROUP_AMT decimal(25,5) ,PSN_PUR_LOAN_LON_RATE decimal(10,3) ,PSN_PUR_LOAN_OVDUE_RATE decimal(10,3) ,PSN_PUR_LOAN_RECA_PRCP decimal(20,5) ,PSN_PUR_LOAN_OVDUE_PRCP decimal(20,5) ,PSN_PUR_LOAN_RECA_INT decimal(20,5) ,PSN_PUR_LOAN_OVDUE_INT decimal(20,5) ,PSN_PUR_LOAN_STOT_DEINT decimal(20,5) ,PSN_PUR_LOAN_LON_BAL decimal(25,5) ) STORED AS SEQUENCEFILE LOCATION 'hdfs://rdc-bd12:8020/kylin/kylin_metadata/kylin-167704d8-7704-4877-9ff0-dfa3c4e82de8/kylin_intermediate_psn_pur_loan_20d88469_c478_41c6_a480_42c61aa0f897'; INSERT OVERWRITE TABLE kylin_intermediate_psn_pur_loan_20d88469_c478_41c6_a480_42c61aa0f897 SELECT PSN_PUR_LOAN.DATE_CD as PSN_PUR_LOAN_DATE_CD ,PSN_PUR_LOAN.ORG_CD as PSN_PUR_LOAN_ORG_CD ,PSN_PUR_LOAN.CURR_CD as PSN_PUR_LOAN_CURR_CD ,PSN_PUR_LOAN.QUAL_CD as PSN_PUR_LOAN_QUAL_CD ,PSN_PUR_LOAN.LON_BUSI_CD as PSN_PUR_LOAN_LON_BUSI_CD ,PSN_PUR_LOAN.GBENT_SCALE_CD as PSN_PUR_LOAN_GBENT_SCALE_CD ,PSN_PUR_LOAN.IND_INV_CD as PSN_PUR_LOAN_IND_INV_CD ,PSN_PUR_LOAN.BASE_RATE_TIMES as PSN_PUR_LOAN_BASE_RATE_TIMES ,PSN_PUR_LOAN.LON_AMT as PSN_PUR_LOAN_LON_AMT ,PSN_PUR_LOAN.GRNT_AMT as PSN_PUR_LOAN_GRNT_AMT ,PSN_PUR_LOAN.BANK_GROUP_AMT as PSN_PUR_LOAN_BANK_GROUP_AMT ,PSN_PUR_LOAN.LON_RATE as PSN_PUR_LOAN_LON_RATE ,PSN_PUR_LOAN.OVDUE_RATE as PSN_PUR_LOAN_OVDUE_RATE ,PSN_PUR_LOAN.RECA_PRCP as PSN_PUR_LOAN_RECA_PRCP ,PSN_PUR_LOAN.OVDUE_PRCP as PSN_PUR_LOAN_OVDUE_PRCP ,PSN_PUR_LOAN.RECA_INT as PSN_PUR_LOAN_RECA_INT ,PSN_PUR_LOAN.OVDUE_INT as PSN_PUR_LOAN_OVDUE_INT ,PSN_PUR_LOAN.STOT_DEINT as PSN_PUR_LOAN_STOT_DEINT ,PSN_PUR_LOAN.LON_BAL as PSN_PUR_LOAN_LON_BAL FROM TEST.PSN_PUR_LOAN as PSN_PUR_LOAN INNER JOIN TEST.DIM_LON_QUAL as DIM_LON_QUAL ON PSN_PUR_LOAN.QUAL_CD = DIM_LON_QUAL.QUAL_CD INNER JOIN TEST.DIM_LON_BUSI as DIM_LON_BUSI ON PSN_PUR_LOAN.LON_BUSI_CD = DIM_LON_BUSI.LON_BUSI_CD INNER JOIN TEST.DIM_GBENT_SCALE as DIM_GBENT_SCALE ON PSN_PUR_LOAN.GBENT_SCALE_CD = DIM_GBENT_SCALE.GBENT_SCALE_CD INNER JOIN TEST.DIM_CURR as DIM_CURR ON PSN_PUR_LOAN.CURR_CD = DIM_CURR.CURR_CD INNER JOIN TEST.DIM_DATE as DIM_DATE ON PSN_PUR_LOAN.DATE_CD = DIM_DATE.DATE_CD INNER JOIN TEST.DIM_LON_IND_INV as DIM_LON_IND_INV ON PSN_PUR_LOAN.IND_INV_CD = DIM_LON_IND_INV.IND_INV_CD INNER JOIN TEST.DIM_ORG as DIM_ORG ON PSN_PUR_LOAN.ORG_CD = DIM_ORG.ORG_CD WHERE (PSN_PUR_LOAN.DATA_DATE >= '20180131' AND PSN_PUR_LOAN.DATA_DATE < '20180201') ; " --hiveconf hive.merge.mapredfiles=false --hiveconf hive.auto.convert.join=true --hiveconf dfs.replication=2 --hiveconf hive.exec.compress.output=true --hiveconf hive.auto.convert.join.noconditionaltask=true --hiveconf mapreduce.job.split.metainfo.maxsize=-1 --hiveconf hive.merge.mapfiles=false --hiveconf hive.auto.convert.join.noconditionaltask.size=100000000 --hiveconf hive.stats.autogather=true SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/hdp/2.5.3.0-37/hive/lib/phoenix-4.8.2-HBase-1.1-hive.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/hdp/2.5.3.0-37/hive/auxlib/phoenix-4.8.2-HBase-1.1-hive.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/hdp/2.5.3.0-37/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] Logging initialized using configuration in file:/etc/hive/2.5.3.0-37/0/hive-log4j.properties OK Time taken: 2.163 seconds OK Time taken: 0.53 seconds OK Time taken: 0.398 seconds Query ID = hdfs_20180701104614_4150cdc6-b328-410c-ab53-0abe0e60e4c5 Total jobs = 5 SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/hdp/2.5.3.0-37/hive/lib/phoenix-4.8.2-HBase-1.1-hive.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/hdp/2.5.3.0-37/hive/auxlib/phoenix-4.8.2-HBase-1.1-hive.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/hdp/2.5.3.0-37/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] Execution log at: /tmp/hdfs/hdfs_20180701104614_4150cdc6-b328-410c-ab53-0abe0e60e4c5.log 2018-07-01 10:46:26 Starting to launch local task to process map join; maximum memory = 1046478848 2018-07-01 10:46:33 Processing rows: 200000 Hashtable size: 199999 Memory usage: 209462696 percentage: 0.20 2018-07-01 10:46:35 Processing rows: 300000 Hashtable size: 299999 Memory usage: 290444464 percentage: 0.278 2018-07-01 10:46:40 Processing rows: 400000 Hashtable size: 399999 Memory usage: 387538096 percentage: 0.37 2018-07-01 10:46:41 Processing rows: 500000 Hashtable size: 499999 Memory usage: 486199336 percentage: 0.465 2018-07-01 10:46:42 Processing rows: 600000 Hashtable size: 599999 Memory usage: 580640608 percentage: 0.555 2018-07-01 10:46:47 Processing rows: 700000 Hashtable size: 699999 Memory usage: 657217728 percentage: 0.628 2018-07-01 10:46:49 Processing rows: 800000 Hashtable size: 799999 Memory usage: 751664416 percentage: 0.718 2018-07-01 10:46:50 Processing rows: 900000 Hashtable size: 899999 Memory usage: 828289456 percentage: 0.792 2018-07-01 10:46:57 Processing rows: 1000000 Hashtable size: 999999 Memory usage: 908630072 percentage: 0.868 2018-07-01 10:47:11 Processing rows: 1100000 Hashtable size: 1099999 Memory usage: 994774944 percentage: 0.951 Execution failed with exit status: 3 Obtaining error information Task failed! Task ID: Stage-34 Logs: /tmp/hdfs/hive.log FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask apache_...@163.com
