zcw5116 opened a new issue, #12257: URL: https://github.com/apache/dolphinscheduler/issues/12257
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar issues. ### What happened 1. DS Version: 3.1.0 2. Data Quality , datasource clickhouse check null 3. Spark Submit Missing main class the following logs: [LOG-PATH]: /opt/dolphinscheduler/worker-server/logs/20221007/7133023937376_2-25-29.log, [HOST]: Host{address='192.168.66.190:1234', ip='192.168.66.190', port=1234} [INFO] 2022-10-07 15:53:04.851 +0000 - Begin to pulling task [INFO] 2022-10-07 15:53:04.855 +0000 - Begin to initialize task [INFO] 2022-10-07 15:53:04.855 +0000 - Set task startTime: Fri Oct 07 15:53:04 UTC 2022 [INFO] 2022-10-07 15:53:04.856 +0000 - Set task envFile: /opt/dolphinscheduler/worker-server/conf/dolphinscheduler_env.sh [INFO] 2022-10-07 15:53:04.856 +0000 - Set task appId: 25_29 [INFO] 2022-10-07 15:53:04.856 +0000 - End initialize task [INFO] 2022-10-07 15:53:04.857 +0000 - Set task status to TaskExecutionStatus{code=1, desc='running'} [INFO] 2022-10-07 15:53:04.858 +0000 - TenantCode:xichen check success [INFO] 2022-10-07 15:53:04.859 +0000 - ProcessExecDir:/opt/ds_base_dir/exec/process/7120503728992/7133023937376_2/25/29 check success [INFO] 2022-10-07 15:53:04.859 +0000 - Resources:{} check success [INFO] 2022-10-07 15:53:04.860 +0000 - Task plugin: DATA_QUALITY create success [INFO] 2022-10-07 15:53:04.860 +0000 - data quality task params {"localParams":[],"resourceList":[],"ruleId":6,"ruleInputParameter":{"check_type":"0","comparison_type":1,"comparison_name":"10","failure_strategy":"0","operator":"4","src_connector_type":4,"src_datasource_id":1,"src_field":"c1_name","src_table":"dim_app_content","threshold":"10"},"sparkParameters":{"deployMode":"local","driverCores":1,"driverMemory":"512M","executorCores":2,"executorMemory":"2G","numExecutors":2,"others":"--conf spark.yarn.maxAppAttempts=1"}} [INFO] 2022-10-07 15:53:04.872 +0000 - Success initialized task plugin instance success [INFO] 2022-10-07 15:53:04.872 +0000 - Success set taskVarPool: null [INFO] 2022-10-07 15:53:04.873 +0000 - data quality task command: **${SPARK_HOME2}/bin/spark-submit --master local --driver-cores 1 --driver-memory 512M --num-executors 2 --executor-cores 2 --executor-memory 2G --conf spark.yarn.maxAppAttempts=1 /opt/dolphinscheduler/worker-server/libs/dolphinscheduler-data-quality-3.1.0.jar "{\"name\":\"$t(uniqueness_check)\",\"env\":{\"type\":\"batch\",\"config\":null},\"readers\":[{\"type\":\"JDBC\",\"config\**":{\"database\":\"tech\",\"password\":\"123456\",\"driver\":\"ru.yandex.clickhouse.ClickHouseDriver\",\"user\":\"default\",\"output_table\":\"tech_dim_app_content\",\"table\":\"dim_app_content\",\"url\":\"jdbc:clickhouse://ch001:8123/tech\"} }],\"transformers\":[{\"type\":\"sql\",\"config\":{\"index\":1,\"output_table\":\"duplicate_items\",\"sql\":\"SELECT c1_name FROM tech_dim_app_content group by c1_name having count(*) > 1\"} },{\"type\":\"sql\",\"config\":{\"index\":2,\"output_table\":\"duplicate_count\",\"sql\":\"SELECT COUNT(*) AS d uplicates FROM duplicate_items\"} }],\"writers\":[{\"type\":\"JDBC\",\"config\":{\"database\":\"ds\",\"password\":\"123456\",\"driver\":\"com.mysql.cj.jdbc.Driver\",\"user\":\"root\",\"table\":\"t_ds_dq_execute_result\",\"url\":\"jdbc:mysql://127.0.0.1:3306/ds?useUnicode=true&characterEncoding=UTF-8&useSSL=false&allowPublicKeyRetrieval=true&allowLoadLocalInfile=false&autoDeserialize=false&allowLocalInfile=false&allowUrlInLocalInfile=false\",\"sql\":\"select 0 as rule_type,'$t(uniqueness_check)' as rule_name,0 as process_definition_id,25 as process_instance_id,29 as task_instance_id,duplicate_count.duplicates AS statistics_value,10 AS comparison_value,1 AS comparison_type,0 as check_type,10 as threshold,4 as operator,0 as failure_strategy,'hdfs://mycluster:8020/user/xichen/data_quality_error_data/0_25_ch' as error_output_path,'2022-10-07 15:53:04' as create_time,'2022-10-07 15:53:04' as update_time from duplicate_count \"} },{\"type\":\"JDBC\",\"config\":{\"database\":\"ds\",\"passwo rd\":\"123456\",\"driver\":\"com.mysql.cj.jdbc.Driver\",\"user\":\"root\",\"table\":\"t_ds_dq_task_statistics_value\",\"url\":\"jdbc:mysql://127.0.0.1:3306/ds?useUnicode=true&characterEncoding=UTF-8&useSSL=false&allowPublicKeyRetrieval=true&allowLoadLocalInfile=false&autoDeserialize=false&allowLocalInfile=false&allowUrlInLocalInfile=false\",\"sql\":\"select 0 as process_definition_id,29 as task_instance_id,6 as rule_id,'1SYYJMCZZRKHUDNCCFM4ZRXJRQUM4DPPSQW5QMYGOLU=' as unique_code,'duplicate_count.duplicates'AS statistics_name,duplicate_count.duplicates AS statistics_value,'2022-10-07 15:53:04' as data_time,'2022-10-07 15:53:04' as create_time,'2022-10-07 15:53:04' as update_time from duplicate_count\"} },{\"type\":\"hdfs_file\",\"config\":{\"path\":\"hdfs://mycluster:8020/user/xichen/data_quality_error_data/0_25_ch\",\"input_table\":\"duplicate_items\"} }]}" [INFO] 2022-10-07 15:53:04.873 +0000 - tenantCode user:xichen, task dir:25_29 [INFO] 2022-10-07 15:53:04.873 +0000 - create command file:/opt/ds_base_dir/exec/process/7120503728992/7133023937376_2/25/29/25_29.command [INFO] 2022-10-07 15:53:04.874 +0000 - command : #!/bin/sh BASEDIR=$(cd `dirname $0`; pwd) cd $BASEDIR source /opt/dolphinscheduler/worker-server/conf/dolphinscheduler_env.sh ${SPARK_HOME2}/bin/spark-submit --master local --driver-cores 1 --driver-memory 512M --num-executors 2 --executor-cores 2 --executor-memory 2G --conf spark.yarn.maxAppAttempts=1 /opt/dolphinscheduler/worker-server/libs/dolphinscheduler-data-quality-3.1.0.jar "{\"name\":\"$t(uniqueness_check)\",\"env\":{\"type\":\"batch\",\"config\":null},\"readers\":[{\"type\":\"JDBC\",\"config\":{\"database\":\"tech\",\"password\":\"123456\",\"driver\":\"ru.yandex.clickhouse.ClickHouseDriver\",\"user\":\"default\",\"output_table\":\"tech_dim_app_content\",\"table\":\"dim_app_content\",\"url\":\"jdbc:clickhouse://ch001:8123/tech\"} }],\"transformers\":[{\"type\":\"sql\",\"config\":{\"index\":1,\"output_table\":\"duplicate_items\",\"sql\":\"SELECT c1_name FROM tech_dim_app_content group by c1_name having count(*) > 1\"} },{\"type\":\"sql\",\"config\":{\"index\":2,\"output_table\":\"duplicate_count\",\"sql\":\"SELECT COUNT(*) AS duplicates FROM duplicate_items\"} }],\"writers\":[{\"type\":\"JDBC\",\ "config\":{\"database\":\"ds\",\"password\":\"123456\",\"driver\":\"com.mysql.cj.jdbc.Driver\",\"user\":\"root\",\"table\":\"t_ds_dq_execute_result\",\"url\":\"jdbc:mysql://127.0.0.1:3306/ds?useUnicode=true&characterEncoding=UTF-8&useSSL=false&allowPublicKeyRetrieval=true&allowLoadLocalInfile=false&autoDeserialize=false&allowLocalInfile=false&allowUrlInLocalInfile=false\",\"sql\":\"select 0 as rule_type,'$t(uniqueness_check)' as rule_name,0 as process_definition_id,25 as process_instance_id,29 as task_instance_id,duplicate_count.duplicates AS statistics_value,10 AS comparison_value,1 AS comparison_type,0 as check_type,10 as threshold,4 as operator,0 as failure_strategy,'hdfs://mycluster:8020/user/xichen/data_quality_error_data/0_25_ch' as error_output_path,'2022-10-07 15:53:04' as create_time,'2022-10-07 15:53:04' as update_time from duplicate_count \"} },{\"type\":\"JDBC\",\"config\":{\"database\":\"ds\",\"password\":\"123456\",\"driver\":\"com.mysql.cj.jdbc.Driver\",\"user\":\"roo t\",\"table\":\"t_ds_dq_task_statistics_value\",\"url\":\"jdbc:mysql://127.0.0.1:3306/ds?useUnicode=true&characterEncoding=UTF-8&useSSL=false&allowPublicKeyRetrieval=true&allowLoadLocalInfile=false&autoDeserialize=false&allowLocalInfile=false&allowUrlInLocalInfile=false\",\"sql\":\"select 0 as process_definition_id,29 as task_instance_id,6 as rule_id,'1SYYJMCZZRKHUDNCCFM4ZRXJRQUM4DPPSQW5QMYGOLU=' as unique_code,'duplicate_count.duplicates'AS statistics_name,duplicate_count.duplicates AS statistics_value,'2022-10-07 15:53:04' as data_time,'2022-10-07 15:53:04' as create_time,'2022-10-07 15:53:04' as update_time from duplicate_count\"} },{\"type\":\"hdfs_file\",\"config\":{\"path\":\"hdfs://mycluster:8020/user/xichen/data_quality_error_data/0_25_ch\",\"input_table\":\"duplicate_items\"} }]}" [INFO] 2022-10-07 15:53:04.878 +0000 - task run command: sudo -u xichen sh /opt/ds_base_dir/exec/process/7120503728992/7133023937376_2/25/29/25_29.command [INFO] 2022-10-07 15:53:04.878 +0000 - process start, process id is: 26016 [INFO] 2022-10-07 15:53:07.238 +0000 - process has exited, execute path:/opt/ds_base_dir/exec/process/7120503728992/7133023937376_2/25/29, processId:26016 ,exitStatusCode:1 ,processWaitForStatus:true ,processExitValue:1 [INFO] 2022-10-07 15:53:07.241 +0000 - Send task execute result to master, the current task status: TaskExecutionStatus{code=6, desc='failure'} [INFO] 2022-10-07 15:53:07.241 +0000 - Remove the current task execute context from worker cache [INFO] 2022-10-07 15:53:07.241 +0000 - The current execute mode isn't develop mode, will clear the task execute file: /opt/ds_base_dir/exec/process/7120503728992/7133023937376_2/25/29 [INFO] 2022-10-07 15:53:07.242 +0000 - Success clear the task execute file: /opt/ds_base_dir/exec/process/7120503728992/7133023937376_2/25/29 [INFO] 2022-10-07 15:53:07.879 +0000 - -> 2022-10-07 23:53:06,923 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Exception in thread "main" org.apache.spark.SparkException: No main class set in JAR; please specify one with --class. at org.apache.spark.deploy.SparkSubmit.error(SparkSubmit.scala:972) at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:492) at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:898) at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1043) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1052) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) [INFO] 2022-10-07 15:53:07.881 +0000 - FINALIZE_SESSION ### What you expected to happen [Data Quality] No main class set in JAR ### How to reproduce [Data Quality] No main class set in JAR ### Anything else _No response_ ### Version 3.1.x ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
