qq774724635 opened a new issue, #12235: URL: https://github.com/apache/dolphinscheduler/issues/12235
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar issues. ### What happened ### Deployment model：standalone ### Environment： `export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.342.b07-1.el7_9.x86_64/jre # Database related configuration, set database type, username and password # export DATABASE=${DATABASE:-postgresql} # export SPRING_PROFILES_ACTIVE=${DATABASE} # export SPRING_DATASOURCE_URL # export SPRING_DATASOURCE_USERNAME # export SPRING_DATASOURCE_PASSWORD export DATABASE=mysql export SPRING_PROFILES_ACTIVE=mysql export SPRING_DATASOURCE_URL="jdbc:mysql://192.168.1.181:3306/dolphin?useUnicode=true&characterEncoding=UTF-8&useSSL=false&serverTimezone=Asia/Shanghai" export SPRING_DATASOURCE_USERNAME=bigData export SPRING_DATASOURCE_PASSWORD=P@ssw0rd654321 export SPRING_JACKSON_TIME_ZONE=${SPRING_JACKSON_TIME_ZONE:-Asia/Shanghai} # DolphinScheduler server related configuration export SPRING_CACHE_TYPE=${SPRING_CACHE_TYPE:-none} export SPRING_JACKSON_TIME_ZONE=${SPRING_JACKSON_TIME_ZONE:-UTC} export MASTER_FETCH_COMMAND_NUM=${MASTER_FETCH_COMMAND_NUM:-10} # Registry center configuration, determines the type and link of the registry center export REGISTRY_TYPE=${REGISTRY_TYPE:-zookeeper} export REGISTRY_ZOOKEEPER_CONNECT_STRING=${REGISTRY_ZOOKEEPER_CONNECT_STRING:-localhost:2181} # Tasks related configurations, need to change the configuration if you use the related tasks. export HADOOP_HOME=/opt/hadoop-2.7.3 export HADOOP_CONF_DIR=/opt/hadoop-2.7.3/etc/hadoop export SPARK_HOME1=/opt/spark-2.4.0 export SPARK_HOME2=/opt/spark-2.4.0 export PYTHON_HOME=/usr export HIVE_HOME=/opt/hive-2.3.7 #export FLINK_HOME=${FLINK_HOME:-/opt/soft/flink} export DATAX_HOME=/opt/datax export PATH=$HADOOP_HOME/bin:$SPARK_HOME1/bin:$SPARK_HOME2/bin:$PYTHON_HOME/bin:$JAVA_HOME/bin:$HIVE_HOME/bin:$FLINK_HOME/bin:$DATAX_HOME/bin:$PATH` common.properties： `# user data local directory path, please make sure the directory exists and have read write permissions data.basedir.path=/tmp/dolphinscheduler # resource storage type: HDFS, S3, NONE resource.storage.type=HDFS resource.upload.path=/dolphinscheduler # whether to startup kerberos hadoop.security.authentication.startup.state=false # java.security.krb5.conf path java.security.krb5.conf.path=/opt/krb5.conf # login user from keytab username [email protected] # login user from keytab path login.user.keytab.path=/opt/hdfs.headless.keytab # kerberos expire time, the unit is hour kerberos.expire.time=2 # resource view suffixs #resource.view.suffixs=txt,log,sh,bat,conf,cfg,py,java,sql,xml,hql,properties,json,yml,yaml,ini,js # if resource.storage.type=HDFS, the user must have the permission to create directories under the HDFS root path hdfs.root.user=hdfs fs.defaultFS=file:/// aws.access.key.id=minioadmin aws.secret.access.key=minioadmin aws.region=us-east-1 aws.endpoint=http://localhost:9000 # resourcemanager port, the default value is 8088 if not specified resource.manager.httpaddress.port=7777 # if resourcemanager HA is enabled, please set the HA IPs; if resourcemanager is single, keep this value empty #yarn.resourcemanager.ha.rm.ids=192.168.xx.xx,192.168.xx.xx # if resourcemanager HA is enabled or not use resourcemanager, please keep the default value; If resourcemanager is single, you only need to replace ds1 to actual resourcemanager hostname yarn.application.status.address=http://192.168.1.181:%s/ws/v1/cluster/apps/%s # job history status url when application number threshold is reached(default 10000, maybe it was set to 1000) yarn.job.history.status.address=http://192.168.1.181:19888/ws/v1/history/mapreduce/jobs/%s # datasource encryption enable datasource.encryption.enable=false # datasource encryption salt datasource.encryption.salt=!@#$%^&* # data quality option data-quality.jar.name=dolphinscheduler-data-quality-dev-SNAPSHOT.jar #data-quality.error.output.path=/tmp/data-quality-error-data # Network IP gets priority, default inner outer # Whether hive SQL is executed in the same session support.hive.oneSession=false # use sudo or not, if set true, executing user is tenant user and deploy user needs sudo permissions; if set false, executing user is the deploy user and doesn't need sudo permissions sudo.enable=true # network interface preferred like eth0, default: empty #dolphin.scheduler.network.interface.preferred= # network IP gets priority, default: inner outer #dolphin.scheduler.network.priority.strategy=default # system env path #dolphinscheduler.env.path=dolphinscheduler_env.sh # development state development.state=false # rpc port alert.rpc.port=50052 # Url endpoint for zeppelin RESTful API zeppelin.rest.url=http://localhost:8080` ### scene： After deploying spark, hadoop, hive, etc. and configuring spark to connect to hive metadata according to the information provided on the official website, the following error is always reported when performing data quality checks ### error： `[LOG-PATH]: /opt/apache-dolphinscheduler-3.0.0-bin/standalone-server/logs/20220930/7052280036032_2-376-859.log, [HOST]: Host{address='192.168.1.181:1234', ip='192.168.1.181', port=1234} [INFO] 2022-09-30 16:43:59.405 +0800 [taskAppId=TASK-20220930-7052280036032_2-376-859] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.dq.DataQualityTask:[83] - [WorkflowInstance-376][TaskInstance-859] - data quality task params {"localParams":[],"resourceList":[],"ruleId":6,"ruleInputParameter":{"check_type":"0","failure_strategy":"0","operator":"0","src_connector_type":0,"src_datasource_id":2,"src_field":"ID","src_table":"DIM_BE_MAINTAIN_FRT","threshold":"0"},"sparkParameters":{"deployMode":"client","driverCores":1,"driverMemory":"512M","executorCores":2,"executorMemory":"2G","numExecutors":2,"others":"--conf spark.yarn.maxAppAttempts=1 --driver-class-path /opt/apache-dolphinscheduler-3.0.0-bin/standalone-server/libs/standalone-server/mysql-connector-java-8.0.29.jar"}} [INFO] 2022-09-30 16:43:59.413 +0800 [taskAppId=TASK-20220930-7052280036032_2-376-859] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.dq.DataQualityTask:[181] - [WorkflowInstance-376][TaskInstance-859] - data quality task command: ${SPARK_HOME2}/bin/spark-submit --master yarn --deploy-mode client --driver-cores 1 --driver-memory 512M --num-executors 2 --executor-cores 2 --executor-memory 2G --queue default --conf spark.yarn.maxAppAttempts=1 --driver-class-path /opt/apache-dolphinscheduler-3.0.0-bin/standalone-server/libs/standalone-server/mysql-connector-java-8.0.29.jar /opt/apache-dolphinscheduler-3.0.0-bin/standalone-server/libs/dolphinscheduler-data-quality-dev-SNAPSHOT.jar "{\"name\":\"$t(uniqueness_check)\",\"env\":{\"type\":\"batch\",\"config\":null},\"readers\":[{\"type\":\"JDBC\",\"config\":{\"database\":\"warehouse_new\",\"password\":\"**************\",\"driver\":\"com.mysql.cj.jdbc.Driver\",\"user\":\"bigData\",\"output_table\":\"warehouse_new_DIM_BE_MAINTAI N_FRT\",\"table\":\"DIM_BE_MAINTAIN_FRT\",\"url\":\"jdbc:mysql://192.168.1.181:3306/warehouse_new?allowLoadLocalInfile=false&autoDeserialize=false&allowLocalInfile=false&allowUrlInLocalInfile=false\"} }],\"transformers\":[{\"type\":\"sql\",\"config\":{\"index\":1,\"output_table\":\"duplicate_items\",\"sql\":\"SELECT ID FROM warehouse_new_DIM_BE_MAINTAIN_FRT group by ID having count(*) > 1\"} },{\"type\":\"sql\",\"config\":{\"index\":2,\"output_table\":\"duplicate_count\",\"sql\":\"SELECT COUNT(*) AS duplicates FROM duplicate_items\"} }],\"writers\":[{\"type\":\"JDBC\",\"config\":{\"database\":\"dolphin\",\"password\":\"**************\",\"driver\":\"com.mysql.cj.jdbc.Driver\",\"user\":\"bigData\",\"table\":\"t_ds_dq_execute_result\",\"url\":\"jdbc:mysql://192.168.1.181:3306/dolphin?useUnicode=true&characterEncoding=UTF-8&useSSL=false&serverTimezone=Asia/Shanghai&allowLoadLocalInfile=false&autoDeserialize=false&allowLocalInfile=false&allowUrlInLocalInfile=false\",\"sql\":\"select 0 as rule_type,'$t(uniqueness_check)' as rule_name,0 as process_definition_id,376 as process_instance_id,859 as task_instance_id,duplicate_count.duplicates AS statistics_value,${comparison_name} AS comparison_value, AS comparison_type,0 as check_type,0 as threshold,0 as operator,0 as failure_strategy,'file:////user/admin/data_quality_error_data/0_376_test' as error_output_path,'2022-09-30 16:43:59' as create_time,'2022-09-30 16:43:59' as update_time from duplicate_count full join ${comparison_table}\"} },{\"type\":\"JDBC\",\"config\":{\"database\":\"dolphin\",\"password\":\"**************\",\"driver\":\"com.mysql.cj.jdbc.Driver\",\"user\":\"bigData\",\"table\":\"t_ds_dq_task_statistics_value\",\"url\":\"jdbc:mysql://192.168.1.181:3306/dolphin?useUnicode=true&characterEncoding=UTF-8&useSSL=false&serverTimezone=Asia/Shanghai&allowLoadLocalInfile=false&autoDeserialize=false&allowLocalInfile=false&allowUrlInLocalInfile=false\",\"sql\":\"select 0 as process_definition_id,859 as task_instance _id,6 as rule_id,'AOXATF9DIPSNL1TLP/YTDGK8TIRC3L+OUXFM8LSTGRK=' as unique_code,'duplicate_count.duplicates'AS statistics_name,duplicate_count.duplicates AS statistics_value,'2022-09-30 16:43:59' as data_time,'2022-09-30 16:43:59' as create_time,'2022-09-30 16:43:59' as update_time from duplicate_count\"} },{\"type\":\"hdfs_file\",\"config\":{\"path\":\"file:////user/admin/data_quality_error_data/0_376_test\",\"input_table\":\"duplicate_items\"} }]}" [INFO] 2022-09-30 16:43:59.414 +0800 [taskAppId=TASK-20220930-7052280036032_2-376-859] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.dq.DataQualityTask:[85] - [WorkflowInstance-376][TaskInstance-859] - tenantCode user:admin, task dir:376_859 [INFO] 2022-09-30 16:43:59.414 +0800 [taskAppId=TASK-20220930-7052280036032_2-376-859] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.dq.DataQualityTask:[90] - [WorkflowInstance-376][TaskInstance-859] - create command file:/tmp/dolphinscheduler/exec/process/6883044913472/7052280036032_2/376/859/376_859.command [INFO] 2022-09-30 16:43:59.414 +0800 [taskAppId=TASK-20220930-7052280036032_2-376-859] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.dq.DataQualityTask:[116] - [WorkflowInstance-376][TaskInstance-859] - command : #!/bin/sh BASEDIR=$(cd `dirname $0`; pwd) cd $BASEDIR source /opt/apache-dolphinscheduler-3.0.0-bin/standalone-server/conf/dolphinscheduler_env.sh ${SPARK_HOME2}/bin/spark-submit --master yarn --deploy-mode client --driver-cores 1 --driver-memory 512M --num-executors 2 --executor-cores 2 --executor-memory 2G --queue default --conf spark.yarn.maxAppAttempts=1 --driver-class-path /opt/apache-dolphinscheduler-3.0.0-bin/standalone-server/libs/standalone-server/mysql-connector-java-8.0.29.jar /opt/apache-dolphinscheduler-3.0.0-bin/standalone-server/libs/dolphinscheduler-data-quality-dev-SNAPSHOT.jar "{\"name\":\"$t(uniqueness_check)\",\"env\":{\"type\":\"batch\",\"config\":null},\"readers\":[{\"type\":\"JDBC\",\"config\":{\"database\":\"warehouse_new\",\"password\":\"**************\",\"driver\":\"com.mysql.cj.jdbc.Driver\",\"user\":\"bigData\",\"output_table\":\"warehouse_new_DIM_BE_MAINTAIN_FRT\",\"table\":\"DIM_BE_MAINTAIN_FRT\",\"url\":\"jdbc:mysql://192.168.1.181:3306/warehouse_new?allowLoadLocalInfile=false&autoDeserialize=false&allowLocalInfile=false&allowUrlInLocalInfile=false\"} }],\"transformers\":[{\"type\":\"sql\",\"co nfig\":{\"index\":1,\"output_table\":\"duplicate_items\",\"sql\":\"SELECT ID FROM warehouse_new_DIM_BE_MAINTAIN_FRT group by ID having count(*) > 1\"} },{\"type\":\"sql\",\"config\":{\"index\":2,\"output_table\":\"duplicate_count\",\"sql\":\"SELECT COUNT(*) AS duplicates FROM duplicate_items\"} }],\"writers\":[{\"type\":\"JDBC\",\"config\":{\"database\":\"dolphin\",\"password\":\"**************\",\"driver\":\"com.mysql.cj.jdbc.Driver\",\"user\":\"bigData\",\"table\":\"t_ds_dq_execute_result\",\"url\":\"jdbc:mysql://192.168.1.181:3306/dolphin?useUnicode=true&characterEncoding=UTF-8&useSSL=false&serverTimezone=Asia/Shanghai&allowLoadLocalInfile=false&autoDeserialize=false&allowLocalInfile=false&allowUrlInLocalInfile=false\",\"sql\":\"select 0 as rule_type,'$t(uniqueness_check)' as rule_name,0 as process_definition_id,376 as process_instance_id,859 as task_instance_id,duplicate_count.duplicates AS statistics_value,${comparison_name} AS comparison_value, AS comparison_type,0 as check_ty pe,0 as threshold,0 as operator,0 as failure_strategy,'file:////user/admin/data_quality_error_data/0_376_test' as error_output_path,'2022-09-30 16:43:59' as create_time,'2022-09-30 16:43:59' as update_time from duplicate_count full join ${comparison_table}\"} },{\"type\":\"JDBC\",\"config\":{\"database\":\"dolphin\",\"password\":\"**************\",\"driver\":\"com.mysql.cj.jdbc.Driver\",\"user\":\"bigData\",\"table\":\"t_ds_dq_task_statistics_value\",\"url\":\"jdbc:mysql://192.168.1.181:3306/dolphin?useUnicode=true&characterEncoding=UTF-8&useSSL=false&serverTimezone=Asia/Shanghai&allowLoadLocalInfile=false&autoDeserialize=false&allowLocalInfile=false&allowUrlInLocalInfile=false\",\"sql\":\"select 0 as process_definition_id,859 as task_instance_id,6 as rule_id,'AOXATF9DIPSNL1TLP/YTDGK8TIRC3L+OUXFM8LSTGRK=' as unique_code,'duplicate_count.duplicates'AS statistics_name,duplicate_count.duplicates AS statistics_value,'2022-09-30 16:43:59' as data_time,'2022-09-30 16:43:59' as create_time ,'2022-09-30 16:43:59' as update_time from duplicate_count\"} },{\"type\":\"hdfs_file\",\"config\":{\"path\":\"file:////user/admin/data_quality_error_data/0_376_test\",\"input_table\":\"duplicate_items\"} }]}" [INFO] 2022-09-30 16:43:59.421 +0800 [taskAppId=TASK-20220930-7052280036032_2-376-859] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.dq.DataQualityTask:[290] - [WorkflowInstance-376][TaskInstance-859] - task run command: sudo -u admin sh /tmp/dolphinscheduler/exec/process/6883044913472/7052280036032_2/376/859/376_859.command [INFO] 2022-09-30 16:43:59.422 +0800 [taskAppId=TASK-20220930-7052280036032_2-376-859] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.dq.DataQualityTask:[181] - [WorkflowInstance-376][TaskInstance-859] - process start, process id is: 1788 [INFO] 2022-09-30 16:44:01.422 +0800 [taskAppId=TASK-20220930-7052280036032_2-376-859] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.dq.DataQualityTask:[63] - [WorkflowInstance-0][TaskInstance-0] - -> 2022-09-30 16:44:00 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable [INFO] 2022-09-30 16:44:02.425 +0800 [taskAppId=TASK-20220930-7052280036032_2-376-859] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.dq.DataQualityTask:[63] - [WorkflowInstance-0][TaskInstance-0] - -> 2022-09-30 16:44:01 INFO SparkContext:54 - Running Spark version 2.4.0 2022-09-30 16:44:01 INFO SparkContext:54 - Submitted application: (uniqueness_check) 2022-09-30 16:44:01 INFO SecurityManager:54 - Changing view acls to: admin 2022-09-30 16:44:01 INFO SecurityManager:54 - Changing modify acls to: admin 2022-09-30 16:44:01 INFO SecurityManager:54 - Changing view acls groups to: 2022-09-30 16:44:01 INFO SecurityManager:54 - Changing modify acls groups to: 2022-09-30 16:44:01 INFO SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(admin); groups with view permissions: Set(); users with modify permissions: Set(admin); groups with modify permissions: Set() 2022-09-30 16:44:02 INFO Utils:54 - Successfully started service 'sparkDriver' on port 35638. 2022-09-30 16:44:02 INFO SparkEnv:54 - Registering MapOutputTracker 2022-09-30 16:44:02 INFO SparkEnv:54 - Registering BlockManagerMaster 2022-09-30 16:44:02 INFO BlockManagerMasterEndpoint:54 - Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information 2022-09-30 16:44:02 INFO BlockManagerMasterEndpoint:54 - BlockManagerMasterEndpoint up 2022-09-30 16:44:02 INFO DiskBlockManager:54 - Created local directory at /tmp/blockmgr-0905802e-97fd-4924-9877-24fae1ba490c 2022-09-30 16:44:02 INFO MemoryStore:54 - MemoryStore started with capacity 93.3 MB 2022-09-30 16:44:02 INFO SparkEnv:54 - Registering OutputCommitCoordinator 2022-09-30 16:44:02 INFO log:192 - Logging initialized @2662ms [INFO] 2022-09-30 16:44:03.426 +0800 [taskAppId=TASK-20220930-7052280036032_2-376-859] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.dq.DataQualityTask:[63] - [WorkflowInstance-0][TaskInstance-0] - -> 2022-09-30 16:44:02 INFO Server:351 - jetty-9.3.z-SNAPSHOT, build timestamp: unknown, git hash: unknown 2022-09-30 16:44:02 INFO Server:419 - Started @2777ms 2022-09-30 16:44:02 INFO AbstractConnector:278 - Started ServerConnector@52fc5eb1{HTTP/1.1,[http/1.1]}{0.0.0.0:4040} 2022-09-30 16:44:02 INFO Utils:54 - Successfully started service 'SparkUI' on port 4040. 2022-09-30 16:44:02 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@322803db{/jobs,null,AVAILABLE,@Spark} 2022-09-30 16:44:02 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1bbae752{/jobs/json,null,AVAILABLE,@Spark} 2022-09-30 16:44:02 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@460b6d54{/jobs/job,null,AVAILABLE,@Spark} 2022-09-30 16:44:02 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@76075d65{/jobs/job/json,null,AVAILABLE,@Spark} 2022-09-30 16:44:02 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3a4ba480{/stages,null,AVAILABLE,@Spark} 2022-09-30 16:44:02 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@27b71f50{/stages/json,null,AVAILABLE,@Spark} 2022-09-30 16:44:02 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@383790cf{/stages/stage,null,AVAILABLE,@Spark} 2022-09-30 16:44:02 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@ccd1bc3{/stages/stage/json,null,AVAILABLE,@Spark} 2022-09-30 16:44:02 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@878537d{/stages/pool,null,AVAILABLE,@Spark} 2022-09-30 16:44:02 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4455f57d{/stages/pool/json,null,AVAILABLE,@Spark} 2022-09-30 16:44:02 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@29fc1a2b{/storage,null,AVAILABLE,@Spark} 2022-09-30 16:44:02 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4d0b0fd4{/storage/json,null,AVAILABLE,@Spark} 2022-09-30 16:44:02 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7a24eb3{/storage/rdd,null,AVAILABLE,@Spark} 2022-09-30 16:44:02 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6c37bd27{/storage/rdd/json,null,AVAILABLE,@Spark} 2022-09-30 16:44:02 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@25d3cfc8{/environment,null,AVAILABLE,@Spark} 2022-09-30 16:44:02 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@30331109{/environment/json,null,AVAILABLE,@Spark} 2022-09-30 16:44:02 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2571066a{/executors,null,AVAILABLE,@Spark} 2022-09-30 16:44:02 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@74fe5966{/executors/json,null,AVAILABLE,@Spark} 2022-09-30 16:44:02 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4fe875be{/executors/threadDump,null,AVAILABLE,@Spark} 2022-09-30 16:44:02 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@677b8e13{/executors/threadDump/json,null,AVAILABLE,@Spark} 2022-09-30 16:44:02 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4a9486c0{/static,null,AVAILABLE,@Spark} 2022-09-30 16:44:02 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@562c877a{/,null,AVAILABLE,@Spark} 2022-09-30 16:44:02 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@67001148{/api,null,AVAILABLE,@Spark} 2022-09-30 16:44:02 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@64030b91{/jobs/job/kill,null,AVAILABLE,@Spark} 2022-09-30 16:44:02 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2032e725{/stages/stage/kill,null,AVAILABLE,@Spark} 2022-09-30 16:44:02 INFO SparkUI:54 - Bound SparkUI to 0.0.0.0, and started at http://enoch:4040 2022-09-30 16:44:02 INFO SparkContext:54 - Added JAR file:/opt/apache-dolphinscheduler-3.0.0-bin/standalone-server/libs/dolphinscheduler-data-quality-dev-SNAPSHOT.jar at spark://enoch:35638/jars/dolphinscheduler-data-quality-dev-SNAPSHOT.jar with timestamp 1664527442612 2022-09-30 16:44:03 INFO RMProxy:98 - Connecting to ResourceManager at /0.0.0.0:8032 [INFO] 2022-09-30 16:44:04.427 +0800 [taskAppId=TASK-20220930-7052280036032_2-376-859] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.dq.DataQualityTask:[63] - [WorkflowInstance-0][TaskInstance-0] - -> 2022-09-30 16:44:03 INFO Client:54 - Requesting a new application from cluster with 1 NodeManagers 2022-09-30 16:44:03 INFO Client:54 - Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container) 2022-09-30 16:44:03 INFO Client:54 - Will allocate AM container, with 896 MB memory including 384 MB overhead 2022-09-30 16:44:03 INFO Client:54 - Setting up container launch context for our AM 2022-09-30 16:44:03 INFO Client:54 - Setting up the launch environment for our AM container 2022-09-30 16:44:03 INFO Client:54 - Preparing resources for our AM container 2022-09-30 16:44:03 WARN Client:66 - Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME. [INFO] 2022-09-30 16:44:09.428 +0800 [taskAppId=TASK-20220930-7052280036032_2-376-859] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.dq.DataQualityTask:[63] - [WorkflowInstance-0][TaskInstance-0] - -> 2022-09-30 16:44:09 INFO Client:54 - Uploading resource file:/tmp/spark-ed1d373e-5e95-4d3b-b97b-0fa14e0266c9/__spark_libs__1561782813166038027.zip -> hdfs://192.168.1.181:54310/user/admin/.sparkStaging/application_1664519295996_0007/__spark_libs__1561782813166038027.zip [INFO] 2022-09-30 16:44:11.429 +0800 [taskAppId=TASK-20220930-7052280036032_2-376-859] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.dq.DataQualityTask:[63] - [WorkflowInstance-0][TaskInstance-0] - -> 2022-09-30 16:44:10 INFO Client:54 - Uploading resource file:/tmp/spark-ed1d373e-5e95-4d3b-b97b-0fa14e0266c9/__spark_conf__5327828120016222911.zip -> hdfs://192.168.1.181:54310/user/admin/.sparkStaging/application_1664519295996_0007/__spark_conf__.zip 2022-09-30 16:44:10 INFO SecurityManager:54 - Changing view acls to: admin 2022-09-30 16:44:10 INFO SecurityManager:54 - Changing modify acls to: admin 2022-09-30 16:44:10 INFO SecurityManager:54 - Changing view acls groups to: 2022-09-30 16:44:10 INFO SecurityManager:54 - Changing modify acls groups to: 2022-09-30 16:44:10 INFO SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(admin); groups with view permissions: Set(); users with modify permissions: Set(admin); groups with modify permissions: Set() [INFO] 2022-09-30 16:44:12.430 +0800 [taskAppId=TASK-20220930-7052280036032_2-376-859] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.dq.DataQualityTask:[63] - [WorkflowInstance-0][TaskInstance-0] - -> 2022-09-30 16:44:11 INFO Client:54 - Submitting application application_1664519295996_0007 to ResourceManager 2022-09-30 16:44:11 INFO YarnClientImpl:273 - Submitted application application_1664519295996_0007 2022-09-30 16:44:11 INFO SchedulerExtensionServices:54 - Starting Yarn extension services with app application_1664519295996_0007 and attemptId None [INFO] 2022-09-30 16:44:13.431 +0800 [taskAppId=TASK-20220930-7052280036032_2-376-859] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.dq.DataQualityTask:[63] - [WorkflowInstance-0][TaskInstance-0] - -> 2022-09-30 16:44:12 INFO Client:54 - Application report for application_1664519295996_0007 (state: ACCEPTED) 2022-09-30 16:44:12 INFO Client:54 - client token: N/A diagnostics: N/A ApplicationMaster host: N/A ApplicationMaster RPC port: -1 queue: default start time: 1664527451736 final status: UNDEFINED tracking URL: http://enoch:7777/proxy/application_1664519295996_0007/ user: admin [INFO] 2022-09-30 16:44:14.432 +0800 [taskAppId=TASK-20220930-7052280036032_2-376-859] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.dq.DataQualityTask:[63] - [WorkflowInstance-0][TaskInstance-0] - -> 2022-09-30 16:44:13 INFO Client:54 - Application report for application_1664519295996_0007 (state: ACCEPTED) [INFO] 2022-09-30 16:44:15.433 +0800 [taskAppId=TASK-20220930-7052280036032_2-376-859] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.dq.DataQualityTask:[63] - [WorkflowInstance-0][TaskInstance-0] - -> 2022-09-30 16:44:14 INFO Client:54 - Application report for application_1664519295996_0007 (state: ACCEPTED) [INFO] 2022-09-30 16:44:16.434 +0800 [taskAppId=TASK-20220930-7052280036032_2-376-859] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.dq.DataQualityTask:[63] - [WorkflowInstance-0][TaskInstance-0] - -> 2022-09-30 16:44:15 INFO Client:54 - Application report for application_1664519295996_0007 (state: ACCEPTED) [INFO] 2022-09-30 16:44:17.435 +0800 [taskAppId=TASK-20220930-7052280036032_2-376-859] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.dq.DataQualityTask:[63] - [WorkflowInstance-0][TaskInstance-0] - -> 2022-09-30 16:44:16 INFO Client:54 - Application report for application_1664519295996_0007 (state: ACCEPTED) 2022-09-30 16:44:17 INFO YarnClientSchedulerBackend:54 - Add WebUI Filter. org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter, Map(PROXY_HOSTS -> enoch, PROXY_URI_BASES -> http://enoch:7777/proxy/application_1664519295996_0007), /proxy/application_1664519295996_0007 2022-09-30 16:44:17 INFO JettyUtils:54 - Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /jobs, /jobs/json, /jobs/job, /jobs/job/json, /stages, /stages/json, /stages/stage, /stages/stage/json, /stages/pool, /stages/pool/json, /storage, /storage/json, /storage/rdd, /storage/rdd/json, /environment, /environment/json, /executors, /executors/json, /executors/threadDump, /executors/threadDump/json, /static, /, /api, /jobs/job/kill, /stages/stage/kill. [INFO] 2022-09-30 16:44:18.436 +0800 [taskAppId=TASK-20220930-7052280036032_2-376-859] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.dq.DataQualityTask:[63] - [WorkflowInstance-0][TaskInstance-0] - -> 2022-09-30 16:44:17 INFO YarnSchedulerBackend$YarnSchedulerEndpoint:54 - ApplicationMaster registered as NettyRpcEndpointRef(spark-client://YarnAM) 2022-09-30 16:44:17 INFO Client:54 - Application report for application_1664519295996_0007 (state: RUNNING) 2022-09-30 16:44:17 INFO Client:54 - client token: N/A diagnostics: N/A ApplicationMaster host: 192.168.1.181 ApplicationMaster RPC port: -1 queue: default start time: 1664527451736 final status: UNDEFINED tracking URL: http://enoch:7777/proxy/application_1664519295996_0007/ user: admin 2022-09-30 16:44:17 INFO YarnClientSchedulerBackend:54 - Application application_1664519295996_0007 has started running. 2022-09-30 16:44:17 INFO Utils:54 - Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 44485. 2022-09-30 16:44:17 INFO NettyBlockTransferService:54 - Server created on enoch:44485 2022-09-30 16:44:17 INFO BlockManager:54 - Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy 2022-09-30 16:44:17 INFO BlockManagerMaster:54 - Registering BlockManager BlockManagerId(driver, enoch, 44485, None) 2022-09-30 16:44:17 INFO BlockManagerMasterEndpoint:54 - Registering block manager enoch:44485 with 93.3 MB RAM, BlockManagerId(driver, enoch, 44485, None) 2022-09-30 16:44:17 INFO BlockManagerMaster:54 - Registered BlockManager BlockManagerId(driver, enoch, 44485, None) 2022-09-30 16:44:17 INFO BlockManager:54 - Initialized BlockManager: BlockManagerId(driver, enoch, 44485, None) 2022-09-30 16:44:17 INFO JettyUtils:54 - Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /metrics/json. 2022-09-30 16:44:17 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@27e44e9c{/metrics/json,null,AVAILABLE,@Spark} [INFO] 2022-09-30 16:44:22.437 +0800 [taskAppId=TASK-20220930-7052280036032_2-376-859] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.dq.DataQualityTask:[63] - [WorkflowInstance-0][TaskInstance-0] - -> 2022-09-30 16:44:21 INFO YarnSchedulerBackend$YarnDriverEndpoint:54 - Registered executor NettyRpcEndpointRef(spark-client://Executor) (192.168.1.181:58620) with ID 1 2022-09-30 16:44:21 INFO BlockManagerMasterEndpoint:54 - Registering block manager enoch:34122 with 912.3 MB RAM, BlockManagerId(1, enoch, 34122, None) [INFO] 2022-09-30 16:44:23.438 +0800 [taskAppId=TASK-20220930-7052280036032_2-376-859] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.dq.DataQualityTask:[63] - [WorkflowInstance-0][TaskInstance-0] - -> 2022-09-30 16:44:23 INFO YarnSchedulerBackend$YarnDriverEndpoint:54 - Registered executor NettyRpcEndpointRef(spark-client://Executor) (192.168.1.181:58628) with ID 2 2022-09-30 16:44:23 INFO YarnClientSchedulerBackend:54 - SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.8 [INFO] 2022-09-30 16:44:24.439 +0800 [taskAppId=TASK-20220930-7052280036032_2-376-859] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.dq.DataQualityTask:[63] - [WorkflowInstance-0][TaskInstance-0] - -> 2022-09-30 16:44:23 INFO SharedState:54 - loading hive config file: file:/opt/spark-2.4.0/conf/hive-site.xml 2022-09-30 16:44:23 INFO BlockManagerMasterEndpoint:54 - Registering block manager enoch:40145 with 912.3 MB RAM, BlockManagerId(2, enoch, 40145, None) 2022-09-30 16:44:23 INFO SharedState:54 - Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('file:/tmp/dolphinscheduler/exec/process/6883044913472/7052280036032_2/376/859/spark-warehouse'). 2022-09-30 16:44:23 INFO SharedState:54 - Warehouse path is 'file:/tmp/dolphinscheduler/exec/process/6883044913472/7052280036032_2/376/859/spark-warehouse'. 2022-09-30 16:44:23 INFO JettyUtils:54 - Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /SQL. 2022-09-30 16:44:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1068176{/SQL,null,AVAILABLE,@Spark} 2022-09-30 16:44:23 INFO JettyUtils:54 - Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /SQL/json. 2022-09-30 16:44:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@71789580{/SQL/json,null,AVAILABLE,@Spark} 2022-09-30 16:44:23 INFO JettyUtils:54 - Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /SQL/execution. 2022-09-30 16:44:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@9d3d54e{/SQL/execution,null,AVAILABLE,@Spark} 2022-09-30 16:44:23 INFO JettyUtils:54 - Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /SQL/execution/json. 2022-09-30 16:44:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@333398f{/SQL/execution/json,null,AVAILABLE,@Spark} 2022-09-30 16:44:23 INFO JettyUtils:54 - Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /static/sql. 2022-09-30 16:44:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@77896335{/static/sql,null,AVAILABLE,@Spark} 2022-09-30 16:44:24 INFO StateStoreCoordinatorRef:54 - Registered StateStoreCoordinator endpoint [INFO] 2022-09-30 16:44:26.440 +0800 [taskAppId=TASK-20220930-7052280036032_2-376-859] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.dq.DataQualityTask:[63] - [WorkflowInstance-0][TaskInstance-0] - -> 2022-09-30 16:44:25 INFO HiveUtils:54 - Initializing HiveMetastoreConnection version 1.2.1 using Spark classes. 2022-09-30 16:44:26 INFO metastore:376 - Trying to connect to metastore with URI thrift://192.168.1.181:9083 2022-09-30 16:44:26 INFO metastore:472 - Connected to metastore. 2022-09-30 16:44:26 INFO SessionState:641 - Created local directory: /tmp/admin 2022-09-30 16:44:26 INFO SessionState:641 - Created local directory: /tmp/50f5c351-bbbf-48fc-90fb-375a4ac8bbf2_resources [INFO] 2022-09-30 16:44:27.228 +0800 [taskAppId=TASK-20220930-7052280036032_2-376-859] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.dq.DataQualityTask:[375] - [WorkflowInstance-376][TaskInstance-859] - find app id: application_1664519295996_0007 [INFO] 2022-09-30 16:44:27.228 +0800 [taskAppId=TASK-20220930-7052280036032_2-376-859] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.dq.DataQualityTask:[205] - [WorkflowInstance-376][TaskInstance-859] - process has exited, execute path:/tmp/dolphinscheduler/exec/process/6883044913472/7052280036032_2/376/859, processId:1788 ,exitStatusCode:1 ,processWaitForStatus:true ,processExitValue:1 [INFO] 2022-09-30 16:44:27.440 +0800 [taskAppId=TASK-20220930-7052280036032_2-376-859] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.dq.DataQualityTask:[63] - [WorkflowInstance-0][TaskInstance-0] - -> 2022-09-30 16:44:26 INFO SessionState:641 - Created HDFS directory: /tmp/hive/admin/50f5c351-bbbf-48fc-90fb-375a4ac8bbf2 2022-09-30 16:44:26 INFO SessionState:641 - Created local directory: /tmp/admin/50f5c351-bbbf-48fc-90fb-375a4ac8bbf2 2022-09-30 16:44:26 INFO SessionState:641 - Created HDFS directory: /tmp/hive/admin/50f5c351-bbbf-48fc-90fb-375a4ac8bbf2/_tmp_space.db 2022-09-30 16:44:26 INFO HiveClientImpl:54 - Warehouse location for Hive client (version 1.2.2) is file:/tmp/dolphinscheduler/exec/process/6883044913472/7052280036032_2/376/859/spark-warehouse Exception in thread "main" org.apache.spark.sql.catalyst.parser.ParseException: mismatched input 'from' expecting <EOF>(line 1, pos 441) == SQL == select 0 as rule_type,'(uniqueness_check)' as rule_name,0 as process_definition_id,376 as process_instance_id,859 as task_instance_id,duplicate_count.duplicates AS statistics_value, AS comparison_value, AS comparison_type,0 as check_type,0 as threshold,0 as operator,0 as failure_strategy,'file:////user/admin/data_quality_error_data/0_376_test' as error_output_path,'2022-09-30 16:43:59' as create_time,'2022-09-30 16:43:59' as update_time from duplicate_count full join ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------^^^ at org.apache.spark.sql.catalyst.parser.ParseException.withCommand(ParseDriver.scala:241) at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:117) at org.apache.spark.sql.execution.SparkSqlParser.parse(SparkSqlParser.scala:48) at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parsePlan(ParseDriver.scala:69) at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:642) at org.apache.dolphinscheduler.data.quality.flow.batch.writer.JdbcWriter.write(JdbcWriter.java:74) at org.apache.dolphinscheduler.data.quality.execution.SparkBatchExecution.executeWriter(SparkBatchExecution.java:130) at org.apache.dolphinscheduler.data.quality.execution.SparkBatchExecution.execute(SparkBatchExecution.java:58) at org.apache.dolphinscheduler.data.quality.context.DataQualityContext.execute(DataQualityContext.java:62) at org.apache.dolphinscheduler.data.quality.DataQualityApplication.main(DataQualityApplication.java:70) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849) at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 2022-09-30 16:44:26 INFO SparkContext:54 - Invoking stop() from shutdown hook 2022-09-30 16:44:26 INFO AbstractConnector:318 - Stopped Spark@52fc5eb1{HTTP/1.1,[http/1.1]}{0.0.0.0:4040} 2022-09-30 16:44:26 INFO SparkUI:54 - Stopped Spark web UI at http://enoch:4040 2022-09-30 16:44:26 INFO YarnClientSchedulerBackend:54 - Interrupting monitor thread 2022-09-30 16:44:26 INFO YarnClientSchedulerBackend:54 - Shutting down all executors 2022-09-30 16:44:26 INFO YarnSchedulerBackend$YarnDriverEndpoint:54 - Asking each executor to shut down 2022-09-30 16:44:26 INFO SchedulerExtensionServices:54 - Stopping SchedulerExtensionServices (serviceOption=None, services=List(), started=false) 2022-09-30 16:44:26 INFO YarnClientSchedulerBackend:54 - Stopped 2022-09-30 16:44:26 INFO MapOutputTrackerMasterEndpoint:54 - MapOutputTrackerMasterEndpoint stopped! 2022-09-30 16:44:26 INFO MemoryStore:54 - MemoryStore cleared 2022-09-30 16:44:26 INFO BlockManager:54 - BlockManager stopped 2022-09-30 16:44:26 INFO BlockManagerMaster:54 - BlockManagerMaster stopped 2022-09-30 16:44:26 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:54 - OutputCommitCoordinator stopped! 2022-09-30 16:44:26 INFO SparkContext:54 - Successfully stopped SparkContext 2022-09-30 16:44:26 INFO ShutdownHookManager:54 - Shutdown hook called 2022-09-30 16:44:26 INFO ShutdownHookManager:54 - Deleting directory /tmp/spark-0d51c855-2d04-4883-a8c7-7b0d2afc31f6 2022-09-30 16:44:26 INFO ShutdownHookManager:54 - Deleting directory /tmp/spark-ed1d373e-5e95-4d3b-b97b-0fa14e0266c9 [INFO] 2022-09-30 16:44:27.442 +0800 [taskAppId=TASK-20220930-7052280036032_2-376-859] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.dq.DataQualityTask:[57] - [WorkflowInstance-0][TaskInstance-0] - FINALIZE_SESSION` ### What you expected to happen How to start a data quality check task properly ### How to reproduce Perform data quality checks in a stand-alone manner ### Anything else _No response_ ### Version 3.0.x ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
