Toroidals commented on issue #12901: URL: https://github.com/apache/hudi/issues/12901#issuecomment-2696057193
> [@Toroidals](https://github.com/Toroidals) can you check the jobmanager.log and to see if there are any exception logs? or logs like "recreating and syncing" **This is the detailed error log， application_1740513854165_72700 There are no obvious errors in the Yarn logs，How can I provide you with the log files?** Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 2025-03-03T11:19:51,037 INFO [main] org.apache.hadoop.hive.conf.HiveConf - Found configuration file file:/usr/hdp/current/spark3-client/conf/hive-site.xml Hive Session ID = 6352b0ff-5630-4337-82c1-5c192d2eadbf 2025-03-03T11:19:51,227 INFO [main] SessionState - Hive Session ID = 6352b0ff-5630-4337-82c1-5c192d2eadbf 2025-03-03T11:19:51,433 INFO [main] org.apache.hadoop.hive.metastore.HiveMetaStoreClient - Trying to connect to metastore with URI thrift://crpprd6hd02:9083 2025-03-03T11:19:51,439 INFO [main] org.apache.hadoop.hive.metastore.HiveMetaStoreClient - HMSC::open(): Could not find delegation token. Creating KERBEROS-based thrift connection. 2025-03-03T11:19:51,493 INFO [main] org.apache.hadoop.hive.metastore.HiveMetaStoreClient - Opened a connection to metastore, current connections: 1 2025-03-03T11:19:51,496 INFO [main] org.apache.hadoop.hive.metastore.HiveMetaStoreClient - Connected to metastore. 2025-03-03T11:19:51,496 INFO [main] org.apache.hadoop.hive.metastore.RetryingMetaStoreClient - RetryingMetaStoreClient proxy=class org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient [email protected] (auth:KERBEROS) retries=1 delay=5 lifetime=0 Spark master: yarn, Application Id: application_1740513854165_72700 25/03/03 11:19:53 WARN DFSPropertiesConfiguration: Properties file file:/etc/hudi/conf/hudi-defaults.conf not found. Ignoring to load props file 25/03/03 11:19:53 WARN DFSPropertiesConfiguration: Cannot find HUDI_CONF_DIR, please set it as the dir of hudi-defaults.conf 25/03/03 11:20:00 WARN TableSchemaResolver: Could not find any data file written for commit, so could not get schema for table hdfs://prdhadoop/apps/hive/warehouse/hudi.db/hudi_xx_cmf_fin_business_sm_class_cdc 25/03/03 11:20:02 WARN TableSchemaResolver: Could not find any data file written for commit, so could not get schema for table hdfs://prdhadoop/apps/hive/warehouse/hudi.db/hudi_xx_cmf_fin_business_activity_cdc 25/03/03 11:20:04 WARN TableSchemaResolver: Could not find any data file written for commit, so could not get schema for table hdfs://prdhadoop/apps/hive/warehouse/hudi.db/hudi_xx_cmf_fnd_budget_account_def_cdc 25/03/03 11:20:06 WARN package: Truncated the string representation of a plan since it was too large. This behavior can be adjusted by setting 'spark.sql.debug.maxToStringFields'. Error in query: Table or view not found: ods.ods_xx_hr_org_unit_b_cdc; line 143 pos 19; 'Aggregate [unresolvedalias(count(1), None)] +- 'SubqueryAlias a +- 'Project ['ccch.header_claim_header_id, 'ccch.header_opr_company_id, 'ccch.header_company_segment, 'ccch.header_claim_type_id, 'ccch.header_claim_type_code, unresolvedalias('to_utc_timestamp('ccch.header_creation_date, Asia/Shanghai), None), unresolvedalias('to_utc_timestamp('ccch.header_gl_date, Asia/Shanghai), None), 't.account_line_id, 't.claim_header_id, 't.code_combination_id, 't.entered_dr, 't.entered_cr, 't.description, 't.object_version_number, unresolvedalias('to_utc_timestamp('t.creation_date, Asia/Shanghai), None), 't.created_by, 't.last_updated_by, unresolvedalias('to_utc_timestamp('t.last_update_date, Asia/Shanghai), None), 't.last_update_login, 't.program_application_id, 't.program_id, unresolvedalias('to_utc_timestamp('t.program_update_date, Asia/Shanghai), None), 't.request_id, 't.attribute_category, ... 90 more fields] +- 'Filter (('t._flink_cdc_ts_ms >= 2025-02-26 14:52:27) AND ('t._flink_cdc_ts_ms <= 2025-02-27 13:36:13)) +- 'Join LeftOuter, ('fbcm.project_id = 'cfbp1.project_id) :- 'Join LeftOuter, ('houb.unit_id = 'fbcm.unit_id) : :- Join LeftOuter, (company_id#625 = company_id#512) : : :- Join LeftOuter, ((budget_id#580 = account_id#515) AND NOT (_flink_cdc_op#615 = d)) : : : :- Join LeftOuter, ((comb_map_id#509 = budget_key#68) AND NOT (_flink_cdc_op#570 = d)) : : : : :- Join LeftOuter, ((business_activity_id#453 = business_activity_id#53) AND NOT (_flink_cdc_op#499 = d)) : : : : : :- Join LeftOuter, ((bz_small_class_id#410 = bz_small_class_id#52) AND NOT (_flink_cdc_op#443 = d)) : : : : : : :- Join LeftOuter, ((contract_id#81 = contract_header_id#292) AND NOT (_flink_cdc_op#400 = d)) : : : : : : : :- Join LeftOuter, (customer_id#73 = customer_id#233) : : : : : : : : :- Join LeftOuter, (budget_project_id#51 = project_id#178) : : : : : : : : : :- Join LeftOuter, (project_id#120 = project_id#87) : : : : : : : : : : :- Join Inner, (header_claim_header_id#107 = claim_header_id#20) : : : : : : : : : : : :- SubqueryAlias t : : : : : : : : : : : : +- SubqueryAlias spark_catalog.ods.ods_xx_cmf_clm_gl_account_lines_cdc : : : : : : : : : : : : +- Project [_hoodie_commit_time#14, _hoodie_commit_seqno#15, _hoodie_record_key#16, _hoodie_partition_path#17, _hoodie_file_name#18, account_line_id#19, claim_header_id#20, code_combination_id#21, entered_dr#22, entered_cr#23, description#24, object_version_number#25, creation_date#26, created_by#27, last_updated_by#28, last_update_date#29, last_update_login#30, program_application_id#31, program_id#32, program_update_date#33, request_id#34, attribute_category#35, attribute1#36, attribute2#37, ... 69 more fields] : : : : : : : : : : : : +- Relation ods.ods_xx_cmf_clm_gl_account_lines_cdc[_hoodie_commit_time#14,_hoodie_commit_seqno#15,_hoodie_record_key#16,_hoodie_partition_path#17,_hoodie_file_name#18,account_line_id#19,claim_header_id#20,code_combination_id#21,entered_dr#22,entered_cr#23,description#24,object_version_number#25,creation_date#26,created_by#27,last_updated_by#28,last_update_date#29,last_update_login#30,program_application_id#31,program_id#32,program_update_date#33,request_id#34,attribute_category#35,attribute1#36,attribute2#37,... 69 more fields] parquet : : : : : : : : : : : +- SubqueryAlias ccch : : : : : : : : : : : +- SubqueryAlias spark_catalog.doris_dwd.doris_dwd_rbs_cmf_clm_claim_headers_pub_upd : : : : : : : : : : : +- Relation doris_dwd.doris_dwd_rbs_cmf_clm_claim_headers_pub_upd[header_claim_header_id#107,header_opr_company_id#108,header_company_segment#109,header_claim_type_id#110,header_claim_type_code#111,header_creation_date#112,header_gl_date#113,province_code#114] orc : : : : : : : : : : +- SubqueryAlias cfpp : : : : : : : : : : +- SubqueryAlias spark_catalog.ods.ods_xx_cmf_fin_pa_projects_cdc : : : : : : : : : : +- Relation ods.ods_xx_cmf_fin_pa_projects_cdc[_hoodie_commit_time#115,_hoodie_commit_seqno#116,_hoodie_record_key#117,_hoodie_partition_path#118,_hoodie_file_name#119,project_id#120,org_id#121,org_name#122,template_flag#123,project_class#124,project_type#125,segment1#126,name#127,description#128,project_status_code#129,start_date#130,completion_date#131,pm_project_reference#132,pm_product_code#133,project_manager#134,source_system_code#135,source_project_id#136,source_creation_date#137,source_update_date#138,... 34 more fields] parquet : : : : : : : : : +- SubqueryAlias cfbp : : : : : : : : : +- SubqueryAlias spark_catalog.ods.ods_xx_cmf_fin_budget_projects_cdc : : : : : : : : : +- Relation ods.ods_xx_cmf_fin_budget_projects_cdc[_hoodie_commit_time#173,_hoodie_commit_seqno#174,_hoodie_record_key#175,_hoodie_partition_path#176,_hoodie_file_name#177,project_id#178,project_number#179,project_name#180,description#181,project_type#182,company_id#183,status#184,relevant_department_id#185,start_date#186,end_date#187,object_version_number#188,creation_date#189,created_by#190,last_updated_by#191,last_update_date#192,last_update_login#193,program_application_id#194,program_id#195,program_update_date#196,... 31 more fields] parquet : : : : : : : : +- SubqueryAlias cfac : : : : : : : : +- SubqueryAlias spark_catalog.ods.ods_xx_cmf_fin_ar_customers_cdc : : : : : : : : +- Relation ods.ods_xx_cmf_fin_ar_customers_cdc[_hoodie_commit_time#228,_hoodie_commit_seqno#229,_hoodie_record_key#230,_hoodie_partition_path#231,_hoodie_file_name#232,customer_id#233,customer_number#234,customer_name#235,customer_short_name#236,source_system_code#237,customer_type#238,orgcert_number#239,customer_class#240,inner_code#241,vat_registration_num#242,status#243,start_date#244,end_date#245,object_version_number#246,creation_date#247,created_by#248,last_updated_by#249,last_update_date#250,last_update_login#251,... 35 more fields] parquet : : : : : : : +- SubqueryAlias con : : : : : : : +- SubqueryAlias spark_catalog.ods.ods_xx_cmf_cm_contract_headers_cdc : : : : : : : +- Relation ods.ods_xx_cmf_cm_contract_headers_cdc[_hoodie_commit_time#287,_hoodie_commit_seqno#288,_hoodie_record_key#289,_hoodie_partition_path#290,_hoodie_file_name#291,contract_header_id#292,contract_self_number#293,contract_number#294,contract_name#295,province_code#296,create_company_id#297,create_dept_id#298,create_user_id#299,contact_tel#300,contract_subject#301,superior_sign_flag#302,retroactive_flag#303,sup_agreement_type#304,orig_contract_number#305,orig_contract_name#306,draft_date#307,approve_date#308,signed_date#309,archive_date#310,... 94 more fields] parquet : : : : : : +- SubqueryAlias cfbsc : : : : : : +- SubqueryAlias spark_catalog.ods.ods_xx_cmf_fin_business_sm_class_cdc : : : : : : +- Relation ods.ods_xx_cmf_fin_business_sm_class_cdc[_hoodie_commit_time#405,_hoodie_commit_seqno#406,_hoodie_record_key#407,_hoodie_partition_path#408,_hoodie_file_name#409,bz_small_class_id#410,seq_num#411,class_code#412,class_name#413,enabled_flag#414,object_version_number#415,creation_date#416,created_by#417,last_updated_by#418,last_update_date#419,last_update_login#420,program_application_id#421,program_id#422,program_update_date#423,request_id#424,attribute_category#425,attribute1#426,attribute2#427,attribute3#428,... 19 more fields] parquet : : : : : +- SubqueryAlias cfba : : : : : +- SubqueryAlias spark_catalog.ods.ods_xx_cmf_fin_business_activity_cdc : : : : : +- Relation ods.ods_xx_cmf_fin_business_activity_cdc[_hoodie_commit_time#448,_hoodie_commit_seqno#449,_hoodie_record_key#450,_hoodie_partition_path#451,_hoodie_file_name#452,business_activity_id#453,activity_code#454,activity_name#455,enabled_flag#456,object_version_number#457,creation_date#458,created_by#459,last_updated_by#460,last_update_date#461,last_update_login#462,program_application_id#463,program_id#464,program_update_date#465,request_id#466,attribute_category#467,attribute1#468,attribute2#469,attribute3#470,attribute4#471,... 32 more fields] parquet : : : : +- SubqueryAlias fbcm : : : : +- SubqueryAlias spark_catalog.ods.ods_xx_cmf_fin_budget_comb_map_cdc : : : : +- Relation ods.ods_xx_cmf_fin_budget_comb_map_cdc[_hoodie_commit_time#504,_hoodie_commit_seqno#505,_hoodie_record_key#506,_hoodie_partition_path#507,_hoodie_file_name#508,comb_map_id#509,province_code#510,pri_key#511,company_id#512,project_id#513,unit_id#514,account_id#515,activity_code#516,business_activity_id#517,brand_code#518,usergroup_code#519,product_code#520,channel_code#521,strategy_code#522,enabled_flag#523,start_date#524,end_date#525,orig_update_date#526,object_version_number#527,... 47 more fields] parquet : : : +- SubqueryAlias cfbad : : : +- SubqueryAlias spark_catalog.ods.ods_xx_cmf_fnd_budget_account_def_cdc : : : +- Relation ods.ods_xx_cmf_fnd_budget_account_def_cdc[_hoodie_commit_time#575,_hoodie_commit_seqno#576,_hoodie_record_key#577,_hoodie_partition_path#578,_hoodie_file_name#579,budget_id#580,budget_code#581,budget_name#582,province_code#583,enable_flag#584,object_version_number#585,creation_date#586,created_by#587,last_updated_by#588,last_update_date#589,last_update_login#590,program_application_id#591,program_id#592,program_update_date#593,request_id#594,attribute_category#595,attribute1#596,attribute2#597,attribute3#598,... 21 more fields] parquet : : +- SubqueryAlias fcb : : +- SubqueryAlias spark_catalog.ods.ods_xx_fnd_company_b_cdc : : +- Relation ods.ods_xx_fnd_company_b_cdc[_hoodie_commit_time#620,_hoodie_commit_seqno#621,_hoodie_record_key#622,_hoodie_partition_path#623,_hoodie_file_name#624,company_id#625,company_code#626,company_type#627,address#628,company_level_id#629,parent_company_id#630,chief_position_id#631,start_date_active#632,end_date_active#633,company_short_name#634,company_full_name#635,zipcode#636,fax#637,phone#638,contact_person#639,object_version_number#640,request_id#641,program_id#642,created_by#643,... 67 more fields] parquet : +- 'SubqueryAlias houb : +- 'UnresolvedRelation [ods, ods_xx_hr_org_unit_b_cdc], [], false +- SubqueryAlias cfbp1 +- SubqueryAlias spark_catalog.ods.ods_xx_cmf_fin_budget_projects_cdc +- Relation ods.ods_xx_cmf_fin_budget_projects_cdc[_hoodie_commit_time#712,_hoodie_commit_seqno#713,_hoodie_record_key#714,_hoodie_partition_path#715,_hoodie_file_name#716,project_id#717,project_number#718,project_name#719,description#720,project_type#721,company_id#722,status#723,relevant_department_id#724,start_date#725,end_date#726,object_version_number#727,creation_date#728,created_by#729,last_updated_by#730,last_update_date#731,last_update_login#732,program_application_id#733,program_id#734,program_update_date#735,... 31 more fields] parquet -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
