xiaomaiyun opened a new issue, #11632: URL: https://github.com/apache/dolphinscheduler/issues/11632
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar issues. ### What happened I reference the global variable dt in sql(hive), but I can't get the parameters correctly. My configuration is as follows: task type: SQL Datasource types:Hive SQL Type: Non Query Global Variables:key=dt,value=$[yyyy-MM-dd-1] SQL Statement:load data inpath '/user/logstash/weblog/pageviews/${dt}' overwrite into table weblog.ods_click_pageviews partition(datestr='${dt}') log: [INFO] 2022-08-25 00:33:29.177 +0800 [taskAppId=TASK-20220825-6646621274688_8-1751-1992] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.sql.SqlTask:[385] - after replace sql , preparing : load data inpath '/user/logstash/weblog/pageviews/? overwrite into table weblog.ods_click_pageviews partition(datestr=?) The value of parameter dt is not obtained in sql 我在sql(hive)中引用了全局变量dt，但是我无法获取正确获取到参数。我的配置如下： task type：SQL Datasource types:Hive SQL Type:Non Query Global Variables:key=dt,value=$[yyyy-MM-dd-1] SQL Statement:load data inpath '/user/logstash/weblog/pageviews/${dt}' overwrite into table weblog.ods_click_pageviews partition(datestr='${dt}') 日志： [INFO] 2022-08-25 00:33:29.177 +0800 [taskAppId=TASK-20220825-6646621274688_8-1751-1992] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.sql.SqlTask:[385] - after replace sql , preparing : load data inpath '/user/logstash/weblog/pageviews/? overwrite into table weblog.ods_click_pageviews partition(datestr=?) sql中没有获取到参数dt的值 ### What you expected to happen Expected: load data inpath '/user/logstash/weblog/pageviews/2022-08-24' overwrite into table weblog.ods_click_pageviews partition(datestr='2022-08-24') Full log: [LOG-PATH]: /opt/dolphinscheduler/dolphinscheduler/worker-server/logs/20220825/6646621274688_8-1751-1992.log, [HOST]: Host{address='192.168.199.102:1234', ip='192.168.199.102', port=1234} [INFO] 2022-08-25 00:33:29.173 +0800 [taskAppId=TASK-20220825-6646621274688_8-1751-1992] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.sql.SqlTask:[120] - Full sql parameters: SqlParameters{type='HIVE', datasource=1, sql='load data inpath '/user/logstash/weblog/pageviews/${dt}' overwrite into table weblog.ods_click_pageviews partition(datestr='${dt}')', sqlType=1, sendEmail=null, displayRows=10, limit=0, segmentSeparator=, udfs='', showType='null', connParams='null', groupId='0', title='null', preStatements=[], postStatements=[]} [INFO] 2022-08-25 00:33:29.176 +0800 [taskAppId=TASK-20220825-6646621274688_8-1751-1992] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.sql.SqlTask:[121] - sql type : HIVE, datasource : 1, sql : load data inpath '/user/logstash/weblog/pageviews/${dt}' overwrite into table weblog.ods_click_pageviews partition(datestr='${dt}') , localParams : [],udfs : ,showType : null,connParams : null,varPool : [] ,query max result limit 0 [INFO] 2022-08-25 00:33:29.177 +0800 [taskAppId=TASK-20220825-6646621274688_8-1751-1992] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.sql.SqlTask:[91] - setSqlParamsMap: Property with paramName: dt put in sqlParamsMap of content load data inpath '/user/logstash/weblog/pageviews/${dt}' overwrite into table weblog.ods_click_pageviews partition(datestr='${dt}') successfully. [INFO] 2022-08-25 00:33:29.177 +0800 [taskAppId=TASK-20220825-6646621274688_8-1751-1992] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.sql.SqlTask:[91] - setSqlParamsMap: Property with paramName: dt put in sqlParamsMap of content load data inpath '/user/logstash/weblog/pageviews/${dt}' overwrite into table weblog.ods_click_pageviews partition(datestr='${dt}') successfully. [INFO] 2022-08-25 00:33:29.177 +0800 [taskAppId=TASK-20220825-6646621274688_8-1751-1992] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.sql.SqlTask:[385] - after replace sql , preparing : load data inpath '/user/logstash/weblog/pageviews/? overwrite into table weblog.ods_click_pageviews partition(datestr=?) [INFO] 2022-08-25 00:33:29.177 +0800 [taskAppId=TASK-20220825-6646621274688_8-1751-1992] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.sql.SqlTask:[394] - Sql Params are replaced sql , parameters:2022-08-24(VARCHAR)2022-08-24(VARCHAR) [INFO] 2022-08-25 00:33:29.177 +0800 [taskAppId=TASK-20220825-6646621274688_8-1751-1992] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.sql.SqlTask:[463] - can't find udf function resource [INFO] 2022-08-25 00:33:29.568 +0800 [taskAppId=TASK-20220825-6646621274688_8-1751-1992] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.sql.SqlTask:[367] - prepare statement replace sql : HikariProxyPreparedStatement@1481755034 wrapping org.apache.hive.jdbc.HivePreparedStatement@6f90d44e [ERROR] 2022-08-25 00:33:29.695 +0800 [taskAppId=TASK-20220825-6646621274688_8-1751-1992] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.sql.SqlTask:[211] - execute sql error: Error while compiling statement: FAILED: ParseException line 1:18 mismatched input '/' expecting StringLiteral near 'inpath' in load statement [ERROR] 2022-08-25 00:33:29.696 +0800 [taskAppId=TASK-20220825-6646621274688_8-1751-1992] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.sql.SqlTask:[164] - sql task error org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: ParseException line 1:18 mismatched input '/' expecting StringLiteral near 'inpath' in load statement at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:264) at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:250) at org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:309) at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:250) at org.apache.hive.jdbc.HiveStatement.executeUpdate(HiveStatement.java:448) at org.apache.hive.jdbc.HivePreparedStatement.executeUpdate(HivePreparedStatement.java:119) at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeUpdate(ProxyPreparedStatement.java:61) at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeUpdate(HikariProxyPreparedStatement.java) at org.apache.dolphinscheduler.plugin.task.sql.SqlTask.executeUpdate(SqlTask.java:305) at org.apache.dolphinscheduler.plugin.task.sql.SqlTask.executeFuncAndSql(SqlTask.java:202) at org.apache.dolphinscheduler.plugin.task.sql.SqlTask.handle(SqlTask.java:158) at org.apache.dolphinscheduler.server.worker.runner.TaskExecuteThread.run(TaskExecuteThread.java:208) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125) at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57) at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: ParseException line 1:18 mismatched input '/' expecting StringLiteral near 'inpath' in load statement at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:329) at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:207) at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:290) at org.apache.hive.service.cli.operation.Operation.run(Operation.java:260) at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:504) at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:490) at sun.reflect.GeneratedMethodAccessor92.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78) at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36) at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875) at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59) at com.sun.proxy.$Proxy35.executeStatementAsync(Unknown Source) at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:295) at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:507) at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1437) at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1422) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.parse.ParseException:line 1:18 mismatched input '/' expecting StringLiteral near 'inpath' in load statement at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:221) at org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:75) at org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:68) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:564) at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1425) at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1398) at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:205) ... 26 common frames omitted ### How to reproduce Reference right global parameter in sql ### Anything else [LOG-PATH]: /opt/dolphinscheduler/dolphinscheduler/worker-server/logs/20220825/6646621274688_8-1751-1992.log, [HOST]: Host{address='192.168.199.102:1234', ip='192.168.199.102', port=1234} [INFO] 2022-08-25 00:33:29.173 +0800 [taskAppId=TASK-20220825-6646621274688_8-1751-1992] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.sql.SqlTask:[120] - Full sql parameters: SqlParameters{type='HIVE', datasource=1, sql='load data inpath '/user/logstash/weblog/pageviews/${dt}' overwrite into table weblog.ods_click_pageviews partition(datestr='${dt}')', sqlType=1, sendEmail=null, displayRows=10, limit=0, segmentSeparator=, udfs='', showType='null', connParams='null', groupId='0', title='null', preStatements=[], postStatements=[]} [INFO] 2022-08-25 00:33:29.176 +0800 [taskAppId=TASK-20220825-6646621274688_8-1751-1992] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.sql.SqlTask:[121] - sql type : HIVE, datasource : 1, sql : load data inpath '/user/logstash/weblog/pageviews/${dt}' overwrite into table weblog.ods_click_pageviews partition(datestr='${dt}') , localParams : [],udfs : ,showType : null,connParams : null,varPool : [] ,query max result limit 0 [INFO] 2022-08-25 00:33:29.177 +0800 [taskAppId=TASK-20220825-6646621274688_8-1751-1992] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.sql.SqlTask:[91] - setSqlParamsMap: Property with paramName: dt put in sqlParamsMap of content load data inpath '/user/logstash/weblog/pageviews/${dt}' overwrite into table weblog.ods_click_pageviews partition(datestr='${dt}') successfully. [INFO] 2022-08-25 00:33:29.177 +0800 [taskAppId=TASK-20220825-6646621274688_8-1751-1992] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.sql.SqlTask:[91] - setSqlParamsMap: Property with paramName: dt put in sqlParamsMap of content load data inpath '/user/logstash/weblog/pageviews/${dt}' overwrite into table weblog.ods_click_pageviews partition(datestr='${dt}') successfully. [INFO] 2022-08-25 00:33:29.177 +0800 [taskAppId=TASK-20220825-6646621274688_8-1751-1992] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.sql.SqlTask:[385] - after replace sql , preparing : load data inpath '/user/logstash/weblog/pageviews/? overwrite into table weblog.ods_click_pageviews partition(datestr=?) [INFO] 2022-08-25 00:33:29.177 +0800 [taskAppId=TASK-20220825-6646621274688_8-1751-1992] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.sql.SqlTask:[394] - Sql Params are replaced sql , parameters:2022-08-24(VARCHAR)2022-08-24(VARCHAR) [INFO] 2022-08-25 00:33:29.177 +0800 [taskAppId=TASK-20220825-6646621274688_8-1751-1992] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.sql.SqlTask:[463] - can't find udf function resource [INFO] 2022-08-25 00:33:29.568 +0800 [taskAppId=TASK-20220825-6646621274688_8-1751-1992] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.sql.SqlTask:[367] - prepare statement replace sql : HikariProxyPreparedStatement@1481755034 wrapping org.apache.hive.jdbc.HivePreparedStatement@6f90d44e [ERROR] 2022-08-25 00:33:29.695 +0800 [taskAppId=TASK-20220825-6646621274688_8-1751-1992] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.sql.SqlTask:[211] - execute sql error: Error while compiling statement: FAILED: ParseException line 1:18 mismatched input '/' expecting StringLiteral near 'inpath' in load statement [ERROR] 2022-08-25 00:33:29.696 +0800 [taskAppId=TASK-20220825-6646621274688_8-1751-1992] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.sql.SqlTask:[164] - sql task error org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: ParseException line 1:18 mismatched input '/' expecting StringLiteral near 'inpath' in load statement at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:264) at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:250) at org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:309) at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:250) at org.apache.hive.jdbc.HiveStatement.executeUpdate(HiveStatement.java:448) at org.apache.hive.jdbc.HivePreparedStatement.executeUpdate(HivePreparedStatement.java:119) at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeUpdate(ProxyPreparedStatement.java:61) at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeUpdate(HikariProxyPreparedStatement.java) at org.apache.dolphinscheduler.plugin.task.sql.SqlTask.executeUpdate(SqlTask.java:305) at org.apache.dolphinscheduler.plugin.task.sql.SqlTask.executeFuncAndSql(SqlTask.java:202) at org.apache.dolphinscheduler.plugin.task.sql.SqlTask.handle(SqlTask.java:158) at org.apache.dolphinscheduler.server.worker.runner.TaskExecuteThread.run(TaskExecuteThread.java:208) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125) at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57) at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: ParseException line 1:18 mismatched input '/' expecting StringLiteral near 'inpath' in load statement at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:329) at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:207) at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:290) at org.apache.hive.service.cli.operation.Operation.run(Operation.java:260) at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:504) at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:490) at sun.reflect.GeneratedMethodAccessor92.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78) at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36) at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875) at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59) at com.sun.proxy.$Proxy35.executeStatementAsync(Unknown Source) at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:295) at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:507) at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1437) at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1422) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.parse.ParseException:line 1:18 mismatched input '/' expecting StringLiteral near 'inpath' in load statement at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:221) at org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:75) at org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:68) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:564) at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1425) at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1398) at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:205) ... 26 common frames omitted ### Version 3.0.0 ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
