Re: Re: it run error when run step 1. Execution failed with exit status: 3
it can work,Thanks a lot apache_...@163.com From: ShaoFeng Shi Date: 2018-07-01 11:16 To: dev Subject: Re: it run error when run step 1. Execution failed with exit status: 3 You can add this properties into conf/kylin_hive_conf.xml, restart Kylin, and then re-submit the job. 2018-07-01 11:07 GMT+08:00 Wangdp : > Hi, > > HDP2.3+Kylin 2.2 > > when run a bigdata set,hadoop it could't work,add set > hive.auto.convert.join=false can workarounds for this bug: > > Where can I add this command in kylin? > > Error log: > > Create and distribute table, cmd: > hive -e "USE default; > > DROP TABLE IF EXISTS kylin_intermediate_psn_pur_ > loan_20d88469_c478_41c6_a480_42c61aa0f897; > CREATE EXTERNAL TABLE IF NOT EXISTS kylin_intermediate_psn_pur_ > loan_20d88469_c478_41c6_a480_42c61aa0f897 > ( > PSN_PUR_LOAN_DATE_CD date > ,PSN_PUR_LOAN_ORG_CD string > ,PSN_PUR_LOAN_CURR_CD string > ,PSN_PUR_LOAN_QUAL_CD string > ,PSN_PUR_LOAN_LON_BUSI_CD string > ,PSN_PUR_LOAN_GBENT_SCALE_CD string > ,PSN_PUR_LOAN_IND_INV_CD string > ,PSN_PUR_LOAN_BASE_RATE_TIMES decimal(16,9) > ,PSN_PUR_LOAN_LON_AMT decimal(25,5) > ,PSN_PUR_LOAN_GRNT_AMT decimal(25,5) > ,PSN_PUR_LOAN_BANK_GROUP_AMT decimal(25,5) > ,PSN_PUR_LOAN_LON_RATE decimal(10,3) > ,PSN_PUR_LOAN_OVDUE_RATE decimal(10,3) > ,PSN_PUR_LOAN_RECA_PRCP decimal(20,5) > ,PSN_PUR_LOAN_OVDUE_PRCP decimal(20,5) > ,PSN_PUR_LOAN_RECA_INT decimal(20,5) > ,PSN_PUR_LOAN_OVDUE_INT decimal(20,5) > ,PSN_PUR_LOAN_STOT_DEINT decimal(20,5) > ,PSN_PUR_LOAN_LON_BAL decimal(25,5) > ) > STORED AS SEQUENCEFILE > LOCATION 'hdfs://rdc-bd12:8020/kylin/kylin_metadata/kylin-167704d8- > 7704-4877-9ff0-dfa3c4e82de8/kylin_intermediate_psn_pur_ > loan_20d88469_c478_41c6_a480_42c61aa0f897'; > INSERT OVERWRITE TABLE kylin_intermediate_psn_pur_ > loan_20d88469_c478_41c6_a480_42c61aa0f897 SELECT > PSN_PUR_LOAN.DATE_CD as PSN_PUR_LOAN_DATE_CD > ,PSN_PUR_LOAN.ORG_CD as PSN_PUR_LOAN_ORG_CD > ,PSN_PUR_LOAN.CURR_CD as PSN_PUR_LOAN_CURR_CD > ,PSN_PUR_LOAN.QUAL_CD as PSN_PUR_LOAN_QUAL_CD > ,PSN_PUR_LOAN.LON_BUSI_CD as PSN_PUR_LOAN_LON_BUSI_CD > ,PSN_PUR_LOAN.GBENT_SCALE_CD as PSN_PUR_LOAN_GBENT_SCALE_CD > ,PSN_PUR_LOAN.IND_INV_CD as PSN_PUR_LOAN_IND_INV_CD > ,PSN_PUR_LOAN.BASE_RATE_TIMES as PSN_PUR_LOAN_BASE_RATE_TIMES > ,PSN_PUR_LOAN.LON_AMT as PSN_PUR_LOAN_LON_AMT > ,PSN_PUR_LOAN.GRNT_AMT as PSN_PUR_LOAN_GRNT_AMT > ,PSN_PUR_LOAN.BANK_GROUP_AMT as PSN_PUR_LOAN_BANK_GROUP_AMT > ,PSN_PUR_LOAN.LON_RATE as PSN_PUR_LOAN_LON_RATE > ,PSN_PUR_LOAN.OVDUE_RATE as PSN_PUR_LOAN_OVDUE_RATE > ,PSN_PUR_LOAN.RECA_PRCP as PSN_PUR_LOAN_RECA_PRCP > ,PSN_PUR_LOAN.OVDUE_PRCP as PSN_PUR_LOAN_OVDUE_PRCP > ,PSN_PUR_LOAN.RECA_INT as PSN_PUR_LOAN_RECA_INT > ,PSN_PUR_LOAN.OVDUE_INT as PSN_PUR_LOAN_OVDUE_INT > ,PSN_PUR_LOAN.STOT_DEINT as PSN_PUR_LOAN_STOT_DEINT > ,PSN_PUR_LOAN.LON_BAL as PSN_PUR_LOAN_LON_BAL > FROM TEST.PSN_PUR_LOAN as PSN_PUR_LOAN > INNER JOIN TEST.DIM_LON_QUAL as DIM_LON_QUAL > ON PSN_PUR_LOAN.QUAL_CD = DIM_LON_QUAL.QUAL_CD > INNER JOIN TEST.DIM_LON_BUSI as DIM_LON_BUSI > ON PSN_PUR_LOAN.LON_BUSI_CD = DIM_LON_BUSI.LON_BUSI_CD > INNER JOIN TEST.DIM_GBENT_SCALE as DIM_GBENT_SCALE > ON PSN_PUR_LOAN.GBENT_SCALE_CD = DIM_GBENT_SCALE.GBENT_SCALE_CD > INNER JOIN TEST.DIM_CURR as DIM_CURR > ON PSN_PUR_LOAN.CURR_CD = DIM_CURR.CURR_CD > INNER JOIN TEST.DIM_DATE as DIM_DATE > ON PSN_PUR_LOAN.DATE_CD = DIM_DATE.DATE_CD > INNER JOIN TEST.DIM_LON_IND_INV as DIM_LON_IND_INV > ON PSN_PUR_LOAN.IND_INV_CD = DIM_LON_IND_INV.IND_INV_CD > INNER JOIN TEST.DIM_ORG as DIM_ORG > ON PSN_PUR_LOAN.ORG_CD = DIM_ORG.ORG_CD > WHERE (PSN_PUR_LOAN.DATA_DATE >= '20180131' AND PSN_PUR_LOAN.DATA_DATE < > '20180201') > ; > > " --hiveconf hive.merge.mapredfiles=false --hiveconf > hive.auto.convert.join=true --hiveconf dfs.replication=2 --hiveconf > hive.exec.compress.output=true --hiveconf > hive.auto.convert.join.noconditionaltask=true > --hiveconf mapreduce.job.split.metainfo.maxsize=-1 --hiveconf > hive.merge.mapfiles=false --hiveconf hive.auto.convert.join. > noconditionaltask.size=1 --hiveconf hive.stats.autogather=true > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in [jar:file:/usr/hdp/2.5.3.0-37/ > hive/lib/phoenix-4.8.2-HBase-1.1-hive.jar!/org/slf4j/impl/ > StaticLoggerBinder.class] > SLF4J: Found binding in [jar:file:/usr/hdp/2.5.3.0-37/ > hive/auxlib/phoenix-4.8.2-HBase-1.1-hive.jar!/org/slf4j/ > impl/StaticLoggerBinder.class] > SLF4J: Found binding in [jar:file:/usr/hdp/2.5.3.0-37/ > hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/ > StaticLoggerBinder.class] > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an > explanation. > SLF4J: Actual binding is of type [org.slf4j
Re: it run error when run step 1. Execution failed with exit status: 3
You can add this properties into conf/kylin_hive_conf.xml, restart Kylin, and then re-submit the job. 2018-07-01 11:07 GMT+08:00 Wangdp : > Hi, > > HDP2.3+Kylin 2.2 > > when run a bigdata set,hadoop it could't work,add set > hive.auto.convert.join=false can workarounds for this bug: > > Where can I add this command in kylin? > > Error log: > > Create and distribute table, cmd: > hive -e "USE default; > > DROP TABLE IF EXISTS kylin_intermediate_psn_pur_ > loan_20d88469_c478_41c6_a480_42c61aa0f897; > CREATE EXTERNAL TABLE IF NOT EXISTS kylin_intermediate_psn_pur_ > loan_20d88469_c478_41c6_a480_42c61aa0f897 > ( > PSN_PUR_LOAN_DATE_CD date > ,PSN_PUR_LOAN_ORG_CD string > ,PSN_PUR_LOAN_CURR_CD string > ,PSN_PUR_LOAN_QUAL_CD string > ,PSN_PUR_LOAN_LON_BUSI_CD string > ,PSN_PUR_LOAN_GBENT_SCALE_CD string > ,PSN_PUR_LOAN_IND_INV_CD string > ,PSN_PUR_LOAN_BASE_RATE_TIMES decimal(16,9) > ,PSN_PUR_LOAN_LON_AMT decimal(25,5) > ,PSN_PUR_LOAN_GRNT_AMT decimal(25,5) > ,PSN_PUR_LOAN_BANK_GROUP_AMT decimal(25,5) > ,PSN_PUR_LOAN_LON_RATE decimal(10,3) > ,PSN_PUR_LOAN_OVDUE_RATE decimal(10,3) > ,PSN_PUR_LOAN_RECA_PRCP decimal(20,5) > ,PSN_PUR_LOAN_OVDUE_PRCP decimal(20,5) > ,PSN_PUR_LOAN_RECA_INT decimal(20,5) > ,PSN_PUR_LOAN_OVDUE_INT decimal(20,5) > ,PSN_PUR_LOAN_STOT_DEINT decimal(20,5) > ,PSN_PUR_LOAN_LON_BAL decimal(25,5) > ) > STORED AS SEQUENCEFILE > LOCATION 'hdfs://rdc-bd12:8020/kylin/kylin_metadata/kylin-167704d8- > 7704-4877-9ff0-dfa3c4e82de8/kylin_intermediate_psn_pur_ > loan_20d88469_c478_41c6_a480_42c61aa0f897'; > INSERT OVERWRITE TABLE kylin_intermediate_psn_pur_ > loan_20d88469_c478_41c6_a480_42c61aa0f897 SELECT > PSN_PUR_LOAN.DATE_CD as PSN_PUR_LOAN_DATE_CD > ,PSN_PUR_LOAN.ORG_CD as PSN_PUR_LOAN_ORG_CD > ,PSN_PUR_LOAN.CURR_CD as PSN_PUR_LOAN_CURR_CD > ,PSN_PUR_LOAN.QUAL_CD as PSN_PUR_LOAN_QUAL_CD > ,PSN_PUR_LOAN.LON_BUSI_CD as PSN_PUR_LOAN_LON_BUSI_CD > ,PSN_PUR_LOAN.GBENT_SCALE_CD as PSN_PUR_LOAN_GBENT_SCALE_CD > ,PSN_PUR_LOAN.IND_INV_CD as PSN_PUR_LOAN_IND_INV_CD > ,PSN_PUR_LOAN.BASE_RATE_TIMES as PSN_PUR_LOAN_BASE_RATE_TIMES > ,PSN_PUR_LOAN.LON_AMT as PSN_PUR_LOAN_LON_AMT > ,PSN_PUR_LOAN.GRNT_AMT as PSN_PUR_LOAN_GRNT_AMT > ,PSN_PUR_LOAN.BANK_GROUP_AMT as PSN_PUR_LOAN_BANK_GROUP_AMT > ,PSN_PUR_LOAN.LON_RATE as PSN_PUR_LOAN_LON_RATE > ,PSN_PUR_LOAN.OVDUE_RATE as PSN_PUR_LOAN_OVDUE_RATE > ,PSN_PUR_LOAN.RECA_PRCP as PSN_PUR_LOAN_RECA_PRCP > ,PSN_PUR_LOAN.OVDUE_PRCP as PSN_PUR_LOAN_OVDUE_PRCP > ,PSN_PUR_LOAN.RECA_INT as PSN_PUR_LOAN_RECA_INT > ,PSN_PUR_LOAN.OVDUE_INT as PSN_PUR_LOAN_OVDUE_INT > ,PSN_PUR_LOAN.STOT_DEINT as PSN_PUR_LOAN_STOT_DEINT > ,PSN_PUR_LOAN.LON_BAL as PSN_PUR_LOAN_LON_BAL > FROM TEST.PSN_PUR_LOAN as PSN_PUR_LOAN > INNER JOIN TEST.DIM_LON_QUAL as DIM_LON_QUAL > ON PSN_PUR_LOAN.QUAL_CD = DIM_LON_QUAL.QUAL_CD > INNER JOIN TEST.DIM_LON_BUSI as DIM_LON_BUSI > ON PSN_PUR_LOAN.LON_BUSI_CD = DIM_LON_BUSI.LON_BUSI_CD > INNER JOIN TEST.DIM_GBENT_SCALE as DIM_GBENT_SCALE > ON PSN_PUR_LOAN.GBENT_SCALE_CD = DIM_GBENT_SCALE.GBENT_SCALE_CD > INNER JOIN TEST.DIM_CURR as DIM_CURR > ON PSN_PUR_LOAN.CURR_CD = DIM_CURR.CURR_CD > INNER JOIN TEST.DIM_DATE as DIM_DATE > ON PSN_PUR_LOAN.DATE_CD = DIM_DATE.DATE_CD > INNER JOIN TEST.DIM_LON_IND_INV as DIM_LON_IND_INV > ON PSN_PUR_LOAN.IND_INV_CD = DIM_LON_IND_INV.IND_INV_CD > INNER JOIN TEST.DIM_ORG as DIM_ORG > ON PSN_PUR_LOAN.ORG_CD = DIM_ORG.ORG_CD > WHERE (PSN_PUR_LOAN.DATA_DATE >= '20180131' AND PSN_PUR_LOAN.DATA_DATE < > '20180201') > ; > > " --hiveconf hive.merge.mapredfiles=false --hiveconf > hive.auto.convert.join=true --hiveconf dfs.replication=2 --hiveconf > hive.exec.compress.output=true --hiveconf > hive.auto.convert.join.noconditionaltask=true > --hiveconf mapreduce.job.split.metainfo.maxsize=-1 --hiveconf > hive.merge.mapfiles=false --hiveconf hive.auto.convert.join. > noconditionaltask.size=1 --hiveconf hive.stats.autogather=true > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in [jar:file:/usr/hdp/2.5.3.0-37/ > hive/lib/phoenix-4.8.2-HBase-1.1-hive.jar!/org/slf4j/impl/ > StaticLoggerBinder.class] > SLF4J: Found binding in [jar:file:/usr/hdp/2.5.3.0-37/ > hive/auxlib/phoenix-4.8.2-HBase-1.1-hive.jar!/org/slf4j/ > impl/StaticLoggerBinder.class] > SLF4J: Found binding in [jar:file:/usr/hdp/2.5.3.0-37/ > hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/ > StaticLoggerBinder.class] > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an > explanation. > SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] > > Logging initialized using configuration in file:/etc/hive/2.5.3.0-37/0/ > hive-log4j.properties > OK > Time taken: 2.163 seconds > OK > Time taken: 0.53 seconds > OK > Time taken: 0.398 seconds > Query ID = hdfs_20180701104614_4150cdc6-b328-410c-ab53-0abe0e60e4c5 > Total jobs = 5 > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in [jar:file:/usr/hdp/2.5.3.0-37/ > hive/lib/phoenix-4.8.2-HBase-1.1-hive.jar!/org/slf4j/impl/ >
it run error when run step 1. Execution failed with exit status: 3
Hi, HDP2.3+Kylin 2.2 when run a bigdata set,hadoop it could't work,add set hive.auto.convert.join=false can workarounds for this bug: Where can I add this command in kylin? Error log: Create and distribute table, cmd: hive -e "USE default; DROP TABLE IF EXISTS kylin_intermediate_psn_pur_loan_20d88469_c478_41c6_a480_42c61aa0f897; CREATE EXTERNAL TABLE IF NOT EXISTS kylin_intermediate_psn_pur_loan_20d88469_c478_41c6_a480_42c61aa0f897 ( PSN_PUR_LOAN_DATE_CD date ,PSN_PUR_LOAN_ORG_CD string ,PSN_PUR_LOAN_CURR_CD string ,PSN_PUR_LOAN_QUAL_CD string ,PSN_PUR_LOAN_LON_BUSI_CD string ,PSN_PUR_LOAN_GBENT_SCALE_CD string ,PSN_PUR_LOAN_IND_INV_CD string ,PSN_PUR_LOAN_BASE_RATE_TIMES decimal(16,9) ,PSN_PUR_LOAN_LON_AMT decimal(25,5) ,PSN_PUR_LOAN_GRNT_AMT decimal(25,5) ,PSN_PUR_LOAN_BANK_GROUP_AMT decimal(25,5) ,PSN_PUR_LOAN_LON_RATE decimal(10,3) ,PSN_PUR_LOAN_OVDUE_RATE decimal(10,3) ,PSN_PUR_LOAN_RECA_PRCP decimal(20,5) ,PSN_PUR_LOAN_OVDUE_PRCP decimal(20,5) ,PSN_PUR_LOAN_RECA_INT decimal(20,5) ,PSN_PUR_LOAN_OVDUE_INT decimal(20,5) ,PSN_PUR_LOAN_STOT_DEINT decimal(20,5) ,PSN_PUR_LOAN_LON_BAL decimal(25,5) ) STORED AS SEQUENCEFILE LOCATION 'hdfs://rdc-bd12:8020/kylin/kylin_metadata/kylin-167704d8-7704-4877-9ff0-dfa3c4e82de8/kylin_intermediate_psn_pur_loan_20d88469_c478_41c6_a480_42c61aa0f897'; INSERT OVERWRITE TABLE kylin_intermediate_psn_pur_loan_20d88469_c478_41c6_a480_42c61aa0f897 SELECT PSN_PUR_LOAN.DATE_CD as PSN_PUR_LOAN_DATE_CD ,PSN_PUR_LOAN.ORG_CD as PSN_PUR_LOAN_ORG_CD ,PSN_PUR_LOAN.CURR_CD as PSN_PUR_LOAN_CURR_CD ,PSN_PUR_LOAN.QUAL_CD as PSN_PUR_LOAN_QUAL_CD ,PSN_PUR_LOAN.LON_BUSI_CD as PSN_PUR_LOAN_LON_BUSI_CD ,PSN_PUR_LOAN.GBENT_SCALE_CD as PSN_PUR_LOAN_GBENT_SCALE_CD ,PSN_PUR_LOAN.IND_INV_CD as PSN_PUR_LOAN_IND_INV_CD ,PSN_PUR_LOAN.BASE_RATE_TIMES as PSN_PUR_LOAN_BASE_RATE_TIMES ,PSN_PUR_LOAN.LON_AMT as PSN_PUR_LOAN_LON_AMT ,PSN_PUR_LOAN.GRNT_AMT as PSN_PUR_LOAN_GRNT_AMT ,PSN_PUR_LOAN.BANK_GROUP_AMT as PSN_PUR_LOAN_BANK_GROUP_AMT ,PSN_PUR_LOAN.LON_RATE as PSN_PUR_LOAN_LON_RATE ,PSN_PUR_LOAN.OVDUE_RATE as PSN_PUR_LOAN_OVDUE_RATE ,PSN_PUR_LOAN.RECA_PRCP as PSN_PUR_LOAN_RECA_PRCP ,PSN_PUR_LOAN.OVDUE_PRCP as PSN_PUR_LOAN_OVDUE_PRCP ,PSN_PUR_LOAN.RECA_INT as PSN_PUR_LOAN_RECA_INT ,PSN_PUR_LOAN.OVDUE_INT as PSN_PUR_LOAN_OVDUE_INT ,PSN_PUR_LOAN.STOT_DEINT as PSN_PUR_LOAN_STOT_DEINT ,PSN_PUR_LOAN.LON_BAL as PSN_PUR_LOAN_LON_BAL FROM TEST.PSN_PUR_LOAN as PSN_PUR_LOAN INNER JOIN TEST.DIM_LON_QUAL as DIM_LON_QUAL ON PSN_PUR_LOAN.QUAL_CD = DIM_LON_QUAL.QUAL_CD INNER JOIN TEST.DIM_LON_BUSI as DIM_LON_BUSI ON PSN_PUR_LOAN.LON_BUSI_CD = DIM_LON_BUSI.LON_BUSI_CD INNER JOIN TEST.DIM_GBENT_SCALE as DIM_GBENT_SCALE ON PSN_PUR_LOAN.GBENT_SCALE_CD = DIM_GBENT_SCALE.GBENT_SCALE_CD INNER JOIN TEST.DIM_CURR as DIM_CURR ON PSN_PUR_LOAN.CURR_CD = DIM_CURR.CURR_CD INNER JOIN TEST.DIM_DATE as DIM_DATE ON PSN_PUR_LOAN.DATE_CD = DIM_DATE.DATE_CD INNER JOIN TEST.DIM_LON_IND_INV as DIM_LON_IND_INV ON PSN_PUR_LOAN.IND_INV_CD = DIM_LON_IND_INV.IND_INV_CD INNER JOIN TEST.DIM_ORG as DIM_ORG ON PSN_PUR_LOAN.ORG_CD = DIM_ORG.ORG_CD WHERE (PSN_PUR_LOAN.DATA_DATE >= '20180131' AND PSN_PUR_LOAN.DATA_DATE < '20180201') ; " --hiveconf hive.merge.mapredfiles=false --hiveconf hive.auto.convert.join=true --hiveconf dfs.replication=2 --hiveconf hive.exec.compress.output=true --hiveconf hive.auto.convert.join.noconditionaltask=true --hiveconf mapreduce.job.split.metainfo.maxsize=-1 --hiveconf hive.merge.mapfiles=false --hiveconf hive.auto.convert.join.noconditionaltask.size=1 --hiveconf hive.stats.autogather=true SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/hdp/2.5.3.0-37/hive/lib/phoenix-4.8.2-HBase-1.1-hive.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/hdp/2.5.3.0-37/hive/auxlib/phoenix-4.8.2-HBase-1.1-hive.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/hdp/2.5.3.0-37/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] Logging initialized using configuration in file:/etc/hive/2.5.3.0-37/0/hive-log4j.properties OK Time taken: 2.163 seconds OK Time taken: 0.53 seconds OK Time taken: 0.398 seconds Query ID = hdfs_20180701104614_4150cdc6-b328-410c-ab53-0abe0e60e4c5 Total jobs = 5 SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/hdp/2.5.3.0-37/hive/lib/phoenix-4.8.2-HBase-1.1-hive.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/hdp/2.5.3.0-37/hive/auxlib/phoenix-4.8.2-HBase-1.1-hive.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/hdp/2.5.3.0-37/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J:
