hei-wei commented on issue #2877:
URL:
https://github.com/apache/incubator-dolphinscheduler/issues/2877#issuecomment-637457310
hive表字段hive_columns_list:['INSTU_CDE', 'APPL_SEQ', 'APPL_CDE', 'APPLY_DT',
'BCH_CDE', 'FORM', 'CUST_ID', 'ID_TYP', 'ID_TYP_OTH', 'ID_NO', 'CUST_NAME',
'MSS_IND', 'REJ_IND', 'LOAN_TYP', 'TYP_SEQ', 'TYP_VER', 'LOAN_PROM',
'PRO_PUR_AMT', 'FST_PCT', 'FST_PAY', 'LOAN_CCY', 'PURPOSE', 'OTHER_PURPOSE',
'APPLY_AMT', 'APPRV_AMT', 'APPLY_TNR', 'APPRV_TNR', 'LOAN_FREQ', 'MTH_AMT',
'DSR_RATIO', 'COLL_RATIO', 'WF_APPR_STS', 'APP_ORIGIN', 'DOC_CHANNEL', 'SCORE',
'SCORE_GRADE', 'COMPL_RESULT', 'SIGN_DT', 'CONT_NO', 'COException in thread
"main" java.lang.RuntimeException: core-site.xml not found
[INFO] 2020-06-02 18:40:40.519 - [taskAppId=TASK-8-181-221]:[106] - ->
at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2577)
at
org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2503)
at
org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2409)
at
org.apache.hadoop.conf.Configuration.set(Configuration.java:1144)
at
org.apache.hadoop.conf.Configuration.set(Configuration.java:1116)
at
org.apache.hadoop.conf.Configuration.setBoolean(Configuration.java:1454)
at
org.apache.hadoop.util.GenericOptionsParser.processGeneralOptions(GenericOptionsParser.java:319)
at
org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:485)
at
org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:170)
at
org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:153)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:64)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hadoop.fs.FsShell.main(FsShell.java:372)
NT_AMT', 'MTD_CDE', 'RATE_TYP', 'MTD_MODE', 'REPC_OPT',
'BASIC_INT_RAT', 'PRICE_INT_RAT', 'SUPER_COOPR', 'COOPR_CDE', 'COOPR_NAME',
'COOPR_ZONE', 'COOPR_TEL', 'COOPR_SUB', 'SALER_NAME', 'SALER_MOBILE',
'OPERATOR_CDE', 'OPERATOR_NAME', 'OPERATOR_TEL', 'CRT_USR', 'CRT_DT',
'LAST_CHG_DT', 'LAST_CHG_USR', 'TYP_GRP', 'GUTR_OPT', 'CRT_BCH', 'LOAN_OPT',
'IN_STS', 'OUT_STS', 'VEH_CHASSIS', 'APPLY_TNR_TYP', 'APPRV_TNR_TYP',
'CRT_BCH_IND', 'RATE_RULE', 'DUE_DAY_OPT', 'DUE_DAY', 'APP_IN_ADVICE',
'APPLY_FST_PAY', 'APPLY_FST_PCT', 'AUTO_WF_IND', 'ACCT_OPT', 'ADVICE_CDE',
'IS_UPLOAD_CONT', 'IS_UPLOAD_LOAN', 'RISK_FLAG', 'MARKETING_CHANNEL',
'CODE_WORD', 'FLOAT_MODE', 'RISK_TIMES', 'RISK_ADVICE', 'NEW_CUST', 'RISK_AMT',
'GOODS_NUMB_POOL', 'GOODS_PROVINCE', 'GOODS_CITY', 'GOODS_AREA', 'GOODS_ADDR',
'SALER_REP_ID', 'SALER_REP_NAME', 'SALER_REP_MOBILE', 'REFEREE_NAME',
'REFEREE_NUIT', 'REFEREE_MOBILE', 'LEARN_ZY_MODE', 'OTHER_BANK_INSTU_DESC',
'OTHER_DESC', 'ADVICE_AMT', 'INCOME_AMT', 'SYS_FLAG', 'COOPR_CONT_NO',
'PAY_KIND', 'PAY_DT', 'QUICK_PAY_FLAG', 'PAY_TERM', 'SERNO', 'LMT_INIT_AMT',
'AUTO_MTD_CDE', 'AUTO_TNR_TYP', 'AUTO_TNR', 'EXCEP_IF', 'EXCEP_REASON',
'SIGN_MODEL', 'CANCLE_RESON', 'IS_OFTEN_CNO', 'OFTEN_CARD_NO',
'IS_UPLOAD_BILL', 'RISK_PRE_LEVEL', 'AUTO_RATE_MODE', 'AUTO_FIXED_OD_IND',
'AUTO_OD_INT_RATE', 'AUTO_INT_RAT', 'PLOAN_TNR_TYP', 'PLOAN_TNR',
'ADMIT_SERNO', 'REMARKS', 'APPLY_TYPE', 'CONSUME_SUM', 'APPLICANT_MOBILE',
'CONSUME_CURR_SUM', 'COOPPF_APPL_CDE', 'IS_REAL_APPLY', 'SPECIAL_CONT_FLAG',
'RISK_COEFFICIENT', 'CASH_AMT', 'SUP_CREDIT_CARD', 'IS_SIGLE_REPLAN',
'LOAN_EFFECT_POINT', 'IS_UPLOAD_DOC', 'CREDIT_CHECK_SIGN', 'CREDIT_CHECK_TIME',
'SERVER_NODE', 'IS_WHITE_LIST', 'WHITE_TYPE', 'IS_ADVICE_FREEZE',
'FREEZE_FLAG_TYP', 'FREEZ_REASON_CUST', 'CUST_TYPE', 'MARKET_CHANNEL',
'PACKAGE_CHANNEL', 'ANNUAL_INCOME_CAL', 'CUST_LEVEL', 'INDENT_TYPE',
'WL_COMPANY', 'CREDIT_RESULT', 'AC_INDENT_TYPE', 'ACCOUNT_MANAGER_LOGIN',
'PBOC_FLAG', 'THIRD_APPROVAL_FLAG', 'THIRD_APPROVAL_RESULT',
'THIRD_APPROVAL_DATE', 'GZ_PRICE_INT_RAT', 'REFEREE_NO',
'RISK_TERM_SPCL_PROC_FLAG', 'ADVICE_AMORT_TERM', 'LPR_RATE_TYPE',
'LPR_ISSUE_MAPRICE', 'LPR_FLOAT_RATE', 'LPR_ISSUE_DATE', 'PRODUCT_CHANNEL',
'SYS_CHANNEL', 'MANAGER_BRANCH'] len:179
源表LC_APPL与目标表SDATA.S001_LC_APPL字段个数和字段名一致。
/BIG_DATA/EDW/ZYXF_EDW/PYTHON_APP/log/20200602/source_to_sdata_S001_LC_APPL_20230825_0.log
[2020-06-02 18:40:35] [INFO] [source_to_sdata.py S001 LC_APPL 20230825
0 null null null]
源表LC_APPL与目标表SDATA.S001_LC_APPL字段个数和字段名一致。
执行之前检查并删除临时数据文件命令: hadoop fs -rm -r /user/taskctl/CMIS.LC_APPL
执行之前检查并删除临时数据文件
/BIG_DATA/EDW/ZYXF_EDW/PYTHON_APP/log/20200602/source_to_sdata_S001_LC_APPL_20230825_0.log
[2020-06-02 18:40:35] [INFO] [source_to_sdata.py S001 LC_APPL 20230825
0 null null null]
执行之前检查并删除临时数据文件命令: hadoop fs -rm -r /user/taskctl/CMIS.LC_APPL
sqoop import --connect "jdbc:oracle:thin:@10.94.30.50:1521:cmis"
--username="cmis" --password="cmis" --outdir
\/BIG_DATA/EDW/ZYXF_EDW/PYTHON_APP/tmp/.sqoop/java/ --table "CMIS.LC_APPL"
--columns
"INSTU_CDE,APPL_SEQ,APPL_CDE,APPLY_DT,BCH_CDE,FORM,CUST_ID,ID_TYP,ID_TYP_OTH,ID_NO,CUST_NAME,MSS_IND,REJ_IND,LOAN_TYP,TYP_SEQ,TYP_VER,LOAN_PROM,PRO_PUR_AMT,FST_PCT,FST_PAY,LOAN_CCY,PURPOSE,OTHER_PURPOSE,APPLY_AMT,APPRV_AMT,APPLY_TNR,APPRV_TNR,LOAN_FREQ,MTH_AMT,DSR_RATIO,COLL_RATIO,WF_APPR_STS,APP_ORIGIN,DOC_CHANNEL,SCORE,SCORE_GRADE,COMPL_RESULT,SIGN_DT,CONT_NO,CONT_AMT,MTD_CDE,RATE_TYP,MTD_MODE,REPC_OPT,BASIC_INT_RAT,PRICE_INT_RAT,SUPER_COOPR,COOPR_CDE,COOPR_NAME,COOPR_ZONE,COOPR_TEL,COOPR_SUB,SALER_NAME,SALER_MOBILE,OPERATOR_CDE,OPERATOR_NAME,OPERATOR_TEL,CRT_USR,CRT_DT,LAST_CHG_DT,LAST_CHG_USR,TYP_GRP,GUTR_OPT,CRT_BCH,LOAN_OPT,IN_STS,OUT_STS,VEH_CHASSIS,APPLY_TNR_TYP,APPRV_TNR_TYP,CRT_BCH_IND,RATE_RULE,DUE_DAY_OPT,DUE_DAY,APP_IN_ADVICE,APPLY_FST_PAY,APPLY_FST_PCT,AUTO_WF_IND,ACCT_OPT,ADVICE_CDE,IS_UPLOAD_CONT,IS_UPLOAD_LOAN,RISK_FLAG,MARKETING_CHANNEL,CODE_WORD,FLOAT_MODE,RISK_TIMES,RISK_ADVICE,NEW_CUST,RISK_AMT,GOODS_NUMB_POOL,GOODS_PROVINCE,GOODS_CITY,GOODS_AREA,GOODS_ADDR,SALER_REP_ID,SALER_REP_NAME,SALER_REP_MOBILE,REFEREE_NAME,REFEREE_NUIT,REFEREE_MOBILE,LEARN_ZY_MODE,OTHER_BANK_INSTU_DESC,OTHER_DESC,ADVICE_AMT,INCOME_AMT,SYS_FLAG,COOPR_CONT_NO,PAY_KIND,PAY_DT,QUICK_PAY_FLAG,PAY_TERM,SERNO,LMT_INIT_AMT,AUTO_MTD_CDE,AUTO_TNR_TYP,AUTO_TNR,EXCEP_IF,EXCEP_REASON,SIGN_MODEL,CANCLE_RESON,IS_OFTEN_CNO,OFTEN_CARD_NO,IS_UPLOAD_BILL,RISK_PRE_LEVEL,AUTO_RATE_MODE,AUTO_FIXED_OD_IND,AUTO_OD_INT_RATE,AUTO_INT_RAT,PLOAN_TNR_TYP,PLOAN_TNR,ADMIT_SERNO,REMARKS,APPLY_TYPE,CONSUME_SUM,APPLICANT_MOBILE,CONSUME_CURR_SUM,COOPPF_APPL_CDE,IS_REAL_APPLY,SPECIAL_CONT_FLAG,RISK_COEFFICIENT,CASH_AMT,SUP_CREDIT_CARD,IS_SIGLE_REPLAN,LOAN_EFFECT_POINT,IS_UPLOAD_DOC,CREDIT_CHECK_SIGN,CREDIT_CHECK_TIME,SERVER_NODE,IS_WHITE_LIST,WHITE_TYPE,IS_ADVICE_FREEZE,FREEZE_FLAG_TYP,FREEZ_REASON_CUST,CUST_TYPE,MARKET_CHANNEL,PACKAGE_CHANNEL,ANNUAL_INCOME_CAL,CUST_LEVEL,INDENT_TYPE,WL_COMPANY,CREDIT_RESULT,AC_INDENT_TYPE,ACCOUNT_MANAGER_LOGIN,PBOC_FLAG,THIRD_APPROVAL_FLAG,THIRD_APPROVAL_RESULT,THIRD_APPROVAL_DATE,GZ_PRICE_INT_RAT,REFEREE_NO,RISK_TERM_SPCL_PROC_FLAG,ADVICE_AMORT_TERM,LPR_RATE_TYPE,LPR_ISSUE_MAPRICE,LPR_FLOAT_RATE,LPR_ISSUE_DATE,PRODUCT_CHANNEL,SYS_CHANNEL,MANAGER_BRANCH"
-m 1 --hive-import --hive-overwrite --hive-table "SDATA.S001_LC_APPL"
--hive-partition-key "dt" --hive-partition-value "20230825"
--hive-drop-import-delims --fields-terminated-by "^" --lines-terminated-by
"\n" --null-string "\\\N" --null-non-string "\\\N"
/BIG_DATA/EDW/ZYXF_EDW/PYTHON_APP/log/20200602/source_to_sdata_S001_LC_APPL_20230825_0.log
[2020-06-02 18:40:35] [INFO] [source_to_sdata.py S001 LC_APPL 20230825
0 null null null]
执行全量抽取数据命令： sqoop import --connect
"jdbc:oracle:thin:@10.94.30.50:1521:cmis" --username="cmis" --password="cmis"
--outdir \/BIG_DATA/EDW/ZYXF_EDW/PYTHON_APP/tmp/.sqoop/java/ --table
"CMIS.LC_APPL" --columns
"INSTU_CDE,APPL_SEQ,APPL_CDE,APPLY_DT,BCH_CDE,FORM,CUST_ID,ID_TYP,ID_TYP_OTH,ID_NO,CUST_NAME,MSS_IND,REJ_IND,LOAN_TYP,TYP_SEQ,TYP_VER,LOAN_PROM,PRO_PUR_AMT,FST_PCT,FST_PAY,LOAN_CCY,PURPOSE,OTHER_PURPOSE,APPLY_AMT,APPRV_AMT,APPLY_TNR,APPRV_TNR,LOAN_FREQ,MTH_AMT,DSR_RATIO,COLL_RATIO,WF_APPR_STS,APP_ORIGIN,DOC_CHANNEL,SCORE,SCORE_GRADE,COMPL_RESULT,SIGN_DT,CONT_NO,CONT_AMT,MTD_CDE,RATE_TYP,MTD_MODE,REPC_OPT,BASIC_INT_RAT,PRICE_INT_RAT,SUPER_COOPR,COOPR_CDE,COOPR_NAME,COOPR_ZONE,COOPR_TEL,COOPR_SUB,SALER_NAME,SALER_MOBILE,OPERATOR_CDE,OPERATOR_NAME,OPERATOR_TEL,CRT_USR,CRT_DT,LAST_CHG_DT,LAST_CHG_USR,TYP_GRP,GUTR_OPT,CRT_BCH,LOAN_OPT,IN_STS,OUT_STS,VEH_CHASSIS,APPLY_TNR_TYP,APPRV_TNR_TYP,CRT_BCH_IND,RATE_RULE,DUE_DAY_OPT,DUE_DAY,APP_IN_ADVICE,APPLY_FST_PAY,APPLY_FST_PCT,AUTO_WF_IND,ACCT_OPT,ADVICE_CDE,IS_UPLOAD_CONT,IS_UPLOAD_LOAN,RISK_FLAG,MARKETING_CHANNEL,CODE_WORD,FLOAT_MODE,RISK_TIMES,RISK_ADVICE,NEW_CUST,RISK_AMT,GOODS_NUMB_POOL,GOODS_PROVINCE,GOODS_CITY,GOODS_AREA,GOODS_ADDR,SALER_REP_ID,SALER_REP_NAME,SALER_REP_MOBILE,REFEREE_NAME,REFEREE_NUIT,REFEREE_MOBILE,LEARN_ZY_MODE,OTHER_BANK_INSTU_DESC,OTHER_DESC,ADVICE_AMT,INCOME_AMT,SYS_FLAG,COOPR_CONT_NO,PAY_KIND,PAY_DT,QUICK_PAY_FLAG,PAY_TERM,SERNO,LMT_INIT_AMT,AUTO_MTD_CDE,AUTO_TNR_TYP,AUTO_TNR,EXCEP_IF,EXCEP_REASON,SIGN_MODEL,CANCLE_RESON,IS_OFTEN_CNO,OFTEN_CARD_NO,IS_UPLOAD_BILL,RISK_PRE_LEVEL,AUTO_RATE_MODE,AUTO_FIXED_OD_IND,AUTO_OD_INT_RATE,AUTO_INT_RAT,PLOAN_TNR_TYP,PLOAN_TNR,ADMIT_SERNO,REMARKS,APPLY_TYPE,CONSUME_SUM,APPLICANT_MOBILE,CONSUME_CURR_SUM,COOPPF_APPL_CDE,IS_REAL_APPLY,SPECIAL_CONT_FLAG,RISK_COEFFICIENT,CASH_AMT,SUP_CREDIT_CARD,IS_SIGLE_REPLAN,LOAN_EFFECT_POINT,IS_UPLOAD_DOC,CREDIT_CHECK_SIGN,CREDIT_CHECK_TIME,SERVER_NODE,IS_WHITE_LIST,WHITE_TYPE,IS_ADVICE_FREEZE,FREEZE_FLAG_TYP,FREEZ_REASON_CUST,CUST_TYPE,MARKET_CHANNEL,PACKAGE_CHANNEL,ANNUAL_INCOME_CAL,CUST_LEVEL,INDENT_TYPE,WL_COMPANY,CREDIT_RESULT,AC_INDENT_TYPE,ACCOUNT_MANAGER_LOGIN,PBOC_FLAG,THIRD_APPROVAL_FLAG,THIRD_APPROVAL_RESULT,THIRD_APPROVAL_DATE,GZ_PRICE_INT_RAT,REFEREE_NO,RISK_TERM_SPCL_PROC_FLAG,ADVICE_AMORT_TERM,LPR_RATE_TYPE,LPR_ISSUE_MAPRICE,LPR_FLOAT_RATE,LPR_ISWarning:
/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/bin/../lib/sqoop/../accumulo
does not exist! Accumulo imports will fail.
[INFO] 2020-06-02 18:40:41.924 - [taskAppId=TASK-8-181-221]:[106] - ->
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
20/06/02 18:40:41 INFO sqoop.Sqoop: Running Sqoop version:
1.4.6-cdh5.11.0
[INFO] 2020-06-02 18:40:47.819 - [taskAppId=TASK-8-181-221]:[106] - ->
20/06/02 18:40:41 WARN tool.BaseSqoopTool: Setting your password on the
command-line is insecure. Consider using -P instead.
20/06/02 18:40:42 INFO oracle.OraOopManagerFactory: Data Connector for
Oracle and Hadoop is disabled.
20/06/02 18:40:42 INFO manager.SqlManager: Using default fetchSize of
1000
20/06/02 18:40:42 INFO tool.CodeGenTool: Beginning code generation
20/06/02 18:40:47 INFO manager.OracleManager: Time zone has been set to
GMT
[INFO] 2020-06-02 18:40:50.180 - [taskAppId=TASK-8-181-221]:[106] - ->
20/06/02 18:40:47 INFO manager.SqlManager: Executing SQL statement: SELECT t.*
FROM CMIS.LC_APPL t WHERE 1=0
20/06/02 18:40:47 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is
/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34
Note:
/tmp/sqoop-taskctl/compile/9d598051260fa086176de9cf3c454c60/CMIS_LC_APPL.java
uses or overrides a deprecated API.
[INFO] 2020-06-02 18:40:52.307 - [taskAppId=TASK-8-181-221]:[106] - ->
Note: Recompile with -Xlint:deprecation for details.
20/06/02 18:40:50 INFO orm.CompilationManager: Writing jar file:
/tmp/sqoop-taskctl/compile/9d598051260fa086176de9cf3c454c60/CMIS.LC_APPL.jar
20/06/02 18:40:50 INFO manager.OracleManager: Time zone has been set to
GMT
20/06/02 18:40:50 INFO mapreduce.ImportJobBase: Beginning import of
CMIS.LC_APPL
20/06/02 18:40:50 INFO Configuration.deprecation: mapred.jar is
deprecated. Instead, use mapreduce.job.jar
20/06/02 18:40:50 INFO Configuration.deprecation: mapred.map.tasks is
deprecated. Instead, use mapreduce.job.maps
20/06/02 18:40:51 INFO client.RMProxy: Connecting to ResourceManager at
vm15211/10.94.152.11:8032
20/06/02 18:40:52 INFO db.DBInputFormat: Using read commited
transaction isolation
[INFO] 2020-06-02 18:40:57.730 - [taskAppId=TASK-8-181-221]:[106] - ->
20/06/02 18:40:52 INFO mapreduce.JobSubmitter: number of splits:1
20/06/02 18:40:52 INFO mapreduce.JobSubmitter: Submitting tokens for
job: job_1587388830783_1324
20/06/02 18:40:52 INFO impl.YarnClientImpl: Submitted application
application_1587388830783_1324
20/06/02 18:40:52 INFO mapreduce.Job: The url to track the job:
http://vm15211:8088/proxy/application_1587388830783_1324/
20/06/02 18:40:52 INFO mapreduce.Job: Running job:
job_1587388830783_1324
20/06/02 18:40:57 INFO mapreduce.Job: Job job_1587388830783_1324
running in uber mode : false
[INFO] 2020-06-02 18:41:10.802 - [taskAppId=TASK-8-181-221]:[106] - ->
20/06/02 18:40:57 INFO mapreduce.Job: map 0% reduce 0%
20/06/02 18:41:10 INFO mapreduce.Job: map 100% reduce 0%
[INFO] 2020-06-02 18:41:11.810 - [taskAppId=TASK-8-181-221]:[106] - ->
20/06/02 18:41:11 INFO mapreduce.Job: Job job_1587388830783_1324 completed
successfully
[INFO] 2020-06-02 18:41:11.927 - [taskAppId=TASK-8-181-221]:[106] - ->
20/06/02 18:41:11 INFO mapreduce.Job: Counters: 32
File System Counters
FILE: Number of bytes read=0
FILE: Number of bytes written=158024
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=87
HDFS: Number of bytes written=21403878
HDFS: Number of read operations=4
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Launched map tasks=1
Other local map tasks=1
Total time spent by all maps in occupied slots
(ms)=20142
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=10071
Total vcore-milliseconds taken by all map tasks=10071
Total megabyte-milliseconds taken by all map
tasks=15469056
Map-Reduce Framework
Map input records=25900
Map output records=25900
Input split bytes=87
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=135
CPU time spent (ms)=6170
Physical memory (bytes) snapshot=478605312
Virtual memory (bytes) snapshot=3037671424
Total committed heap usage (bytes)=487587840
Peak Map Physical memory (bytes)=478605312
Peak Map Virtual memory (bytes)=3037671424
File Input Format Counters
Bytes Read=0
File Output Format Counters
Bytes Written=21403878
20/06/02 18:41:11 INFO mapreduce.ImportJobBase: Transferred 20.4123 MB
in 20.9098 seconds (999.639 KB/sec)
20/06/02 18:41:11 INFO mapreduce.ImportJobBase: Retrieved 25900 records.
20/06/02 18:41:11 INFO manager.OracleManager: Time zone has been set to
GMT
20/06/02 18:41:11 INFO manager.SqlManager: Executing SQL statement:
SELECT t.* FROM CMIS.LC_APPL t WHERE 1=0
20/06/02 18:41:11 WARN hive.TableDefWriter: Column APPL_SEQ had to be
cast to a less precise type in Hive
20/06/02 18:41:11 WARN hive.TableDefWriter: Column TYP_SEQ had to be
cast to a less precise type in Hive
20/06/02 18:41:11 WARN hive.TableDefWriter: Column TYP_VER had to be
cast to a less precise type in Hive
20/06/02 18:41:11 WARN hive.TableDefWriter: Column PRO_PUR_AMT had to
be cast to a less precise type in Hive
20/06/02 18:41:11 WARN hive.TableDefWriter: Column FST_PCT had to be
cast to a less precise type in Hive
20/06/02 18:41:11 WARN hive.TableDefWriter: Column FST_PAY had to be
cast to a less precise type in Hive
20/06/02 18:41:11 WARN hive.TableDefWriter: Column APPLY_AMT had to be
cast to a less precise type in Hive
20/06/02 18:41:11 WARN hive.TableDefWriter: Column APPRV_AMT had to be
cast to a less precise type in Hive
20/06/02 18:41:11 WARN hive.TableDefWriter: Column MTH_AMT had to be
cast to a less precise type in Hive
20/06/02 18:41:11 WARN hive.TableDefWriter: Column DSR_RATIO had to be
cast to a less precise type in Hive
20/06/02 18:41:11 WARN hive.TableDefWriter: Column COLL_RATIO had to be
cast to a less precise type in Hive
20/06/02 18:41:11 WARN hive.TableDefWriter: Column SCORE had to be cast
to a less precise type in Hive
20/06/02 18:41:11 WARN hive.TableDefWriter: Column CONT_AMT had to be
cast to a less precise type in Hive
20/06/02 18:41:11 WARN hive.TableDefWriter: Column BASIC_INT_RAT had to
be cast to a less precise type in Hive
20/06/02 18:41:11 WARN hive.TableDefWriter: Column PRICE_INT_RAT had to
be cast to a less precise type in Hive
20/06/02 18:41:11 WARN hive.TableDefWriter: Column APPLY_FST_PAY had to
be cast to a less precise type in Hive
20/06/02 18:41:11 WARN hive.TableDefWriter: Column APPLY_FST_PCT had to
be cast to a less precise type in Hive
20/06/02 18:41:11 WARN hive.TableDefWriter: Column RISK_TIMES had to be
cast to a less precise type in Hive
20/06/02 18:41:11 WARN hive.TableDefWriter: Column RISK_AMT had to be
cast to a less precise type in Hive
20/06/02 18:41:11 WARN hive.TableDefWriter: Column GOODS_NUMB_POOL had
to be cast to a less precise type in Hive
20/06/02 18:41:11 WARN hive.TableDefWriter: Column ADVICE_AMT had to be
cast to a less precise type in Hive
20/06/02 18:41:11 WARN hive.TableDefWriter: Column INCOME_AMT had to be
cast to a less precise type in Hive
[INFO] 2020-06-02 18:41:15.004 - [taskAppId=TASK-8-181-221]:[106] - ->
20/06/02 18:41:11 WARN hive.TableDefWriter: Column LMT_INIT_AMT had to be cast
to a less precise type in Hive
20/06/02 18:41:11 WARN hive.TableDefWriter: Column AUTO_OD_INT_RATE had
to be cast to a less precise type in Hive
20/06/02 18:41:11 WARN hive.TableDefWriter: Column AUTO_INT_RAT had to
be cast to a less precise type in Hive
20/06/02 18:41:11 WARN hive.TableDefWriter: Column CONSUME_SUM had to
be cast to a less precise type in Hive
20/06/02 18:41:11 WARN hive.TableDefWriter: Column CONSUME_CURR_SUM had
to be cast to a less precise type in Hive
20/06/02 18:41:11 WARN hive.TableDefWriter: Column RISK_COEFFICIENT had
to be cast to a less precise type in Hive
20/06/02 18:41:11 WARN hive.TableDefWriter: Column CASH_AMT had to be
cast to a less precise type in Hive
20/06/02 18:41:11 WARN hive.TableDefWriter: Column ANNUAL_INCOME_CAL
had to be cast to a less precise type in Hive
20/06/02 18:41:11 INFO hive.HiveImport: Loading uploaded data into Hive
Logging initialized using configuration in
jar:file:/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/jars/hive-common-1.1.0-cdh5.11.0.jar!/hive-log4j.properties
OK
[INFO] 2020-06-02 18:41:21.063 - [taskAppId=TASK-8-181-221]:[106] - ->
Time taken: 2.599 seconds
Loading data to table sdata.s001_lc_appl partition (dt=20230825)
setfacl: Permission denied. user=taskctl is not the owner of
inode=dt=20230825
setfacl: Permission denied. user=taskctl is not the owner of
inode=dt=20230825
Partition sdata.s001_lc_appl{dt=20230825} stats: [numFiles=1,
numRows=0, totalSize=21403878, rawDataSize=0]
OK
Time taken: 0.844 seconds
**Exception in thread "main" java.lang.RuntimeException: core-site.xml
not found
[INFO] 2020-06-02 18:41:21.104 - [taskAppId=TASK-8-181-221]:[106] - ->
at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2577)**
at
org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2503)
at
org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2409)
at
org.apache.hadoop.conf.Configuration.set(Configuration.java:1144)
at
org.apache.hadoop.conf.Configuration.set(Configuration.java:1116)
at
org.apache.hadoop.conf.Configuration.setBoolean(Configuration.java:1454)
at
org.apache.hadoop.util.GenericOptionsParser.processGeneralOptions(GenericOptionsParser.java:319)
at
org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:485)
at
org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:170)
at
org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:153)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:64)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hadoop.fs.FsShell.main(FsShell.java:372)
SUE_DATE,PRODUCT_CHANNEL,SYS_CHANNEL,MANAGER_BRANCH" -m 1 --hive-import
--hive-overwrite --hive-table "SDATA.S001_LC_APPL" --hive-partition-key "dt"
--hive-partition-value "20230825" --hive-drop-import-delims
--fields-terminated-by "^" --lines-terminated-by "\n" --null-string "\\\N"
--null-non-string "\\\N"
刷新元数据完成：SDATA.S001_LC_APPL
/BIG_DATA/EDW/ZYXF_EDW/PYTHON_APP/log/20200602/source_to_sdata_S001_LC_APPL_20230825_0.log
[2020-06-02 18:40:35] [INFO] [source_to_sdata.py S001 LC_APPL 20230825
0 null null null]
刷新元数据完成：SDATA.S001_LC_APPL
alter table SDATA.S001_LC_APPL drop if exists partition(dt='20230825')
执行drop掉已存在的分区完成：SDATA.S001_LC_APPL
/BIG_DATA/EDW/ZYXF_EDW/PYTHON_APP/log/20200602/source_to_sdata_S001_LC_APPL_20230825_0.log
[2020-06-02 18:40:40] [INFO] [source_to_sdata.py S001 LC_APPL 20230825
0 null null null]
alter table SDATA.S001_LC_APPL drop if exists partition(dt='20230825')
执行drop掉已存在的分区完成：SDATA.S001_LC_APPL
timer start.command timeout = 62400
刷新元数据完成：SDATA.S001_LC_APPL
/BIG_DATA/EDW/ZYXF_EDW/PYTHON_APP/log/20200602/source_to_sdata_S001_LC_APPL_20230825_0.log
[2020-06-02 18:41:16] [INFO] [source_to_sdata.py S001 LC_APPL 20230825
0 null null null]
刷新元数据完成：SDATA.S001_LC_APPL
all end
sql : select count(1) from CMIS.LC_APPL
jdbc:oracle:thin:@10.94.30.50:1521:cmis
cmis/[email protected]:1521/cmis
EXEC SQL:select count(1) from CMIS.LC_APPL
*************
SQL ROWNUM:0 row(s)
/BIG_DATA/EDW/ZYXF_EDW/PYTHON_APP/log/20200602/source_to_sdata_S001_LC_APPL_20230825_0.log
[2020-06-02 18:41:16] [INFO] [source_to_sdata.py S001 LC_APPL 20230825
0 null null null]
select count(1) from CMIS.LC_APPL
检查源系统表LC_APPL行数:25900
sql : select count(1) from SDATA.S001_LC_APPL where dt='20230825'
/BIG_DATA/EDW/ZYXF_EDW/PYTHON_APP/log/20200602/source_to_sdata_S001_LC_APPL_20230825_0.log
[2020-06-02 18:41:20] [INFO] [source_to_sdata.py S001 LC_APPL 20230825
0 null null null]
select count(1) from SDATA.S001_LC_APPL where dt='20230825'
检查目标系统表SDATA.S001_LC_APPL行数:25900
记录数比较阈值: max = 0
/BIG_DATA/EDW/ZYXF_EDW/PYTHON_APP/log/20200602/source_to_sdata_S001_LC_APPL_20230825_0.log
[2020-06-02 18:41:20] [INFO] [source_to_sdata.py S001 LC_APPL 20230825
0 null null null]
比较源系统表和目标表行数
compareSourceAndTarget success.
LC_APPL:25900
SDATA.S001_LC_APPL:25900
compareSourceAndTarget success.
sTableNum: 25900
tTableNum: 25900
检查字段是否错位... checkColumnsCmd: hadoop fs -cat
/user/hive/warehouse/sdata.db/s001_lc_appl/dt=20230825/p* |awk -F '^' '{print
NF}' |grep -v 179 |wc -l
/BIG_DATA/EDW/ZYXF_EDW/PYTHON_APP/log/20200602/source_to_sdata_S001_LC_APPL_20230825_0.log
[2020-06-02 18:41:20] [INFO] [source_to_sdata.py S001 LC_APPL 20230825
0 null null null] 检查字段是否错位... checkColumnsCmd: hadoop fs -cat
/user/hive/warehouse/sdata.db/s001_lc_appl/dt=20230825/p* |awk -F '^' '{print
NF}' |grep -v 179 |wc -l
字段错位检查通过！ checkColumns success. res: 0
/BIG_DATA/EDW/ZYXF_EDW/PYTHON_APP/log/20200602/source_to_sdata_S001_LC_APPL_20230825_0.log
[2020-06-02 18:41:21] [INFO] [source_to_sdata.py S001 LC_APPL 20230825
0 null null null] 字段错位检查通过！ checkColumns success. res: 0
input ID: SDATA.S001_LC_APPL
[INFO] 2020-06-02 18:41:21.127 - [taskAppId=TASK-8-181-221]:[106] - ->
genDependence sql: INSERT INTO ETL_JOB_RUN_HIS VALUES(
'SDATA.S001_LC_APPL', 'SDATA', 'S001', 'LC_APPL',
'SDATA.S001_LC_APPL',
'/BIG_DATA/EDW/ZYXF_EDW/PYTHON_APP/SDATA/source_to_sdata.py',
'SDATA.S001_LC_APPL', '20230825', '0', '2020-06-02 18:41:21', '', ''
);
genDependence successful. 已生成作业依赖！
/BIG_DATA/EDW/ZYXF_EDW/PYTHON_APP/log/20200602/source_to_sdata_S001_LC_APPL_20230825_0.log
[2020-06-02 18:41:21] [INFO] [source_to_sdata.py S001 LC_APPL 20230825
0 null null null]
生成作业依赖：SDATA.S001_LC_APPL,SDATA,S001,LC_APPL,SDATA.S001_LC_APPL,/BIG_DATA/EDW/ZYXF_EDW/PYTHON_APP/SDATA/source_to_sdata.py,SDATA.S001_LC_APPL,20230825,0,2020-06-02
18:41:21,,
/BIG_DATA/EDW/ZYXF_EDW/PYTHON_APP/log/20200602/source_to_sdata_S001_LC_APPL_20230825_0.log
[2020-06-02 18:41:21] [INFO] [source_to_sdata.py S001 LC_APPL 20230825
0 null null null]
S001_LC_APPL抽取数执行成功！
##################################################
S001_LC_APPL抽取数执行成功！
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
