[
https://issues.apache.org/jira/browse/AIRFLOW-7025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17061156#comment-17061156
]
ASF subversion and git services commented on AIRFLOW-7025:
----------------------------------------------------------
Commit 53d883943e7de034fa752ca7eceb8f93d1aaa6f1 in airflow's branch
refs/heads/v1-10-test from Kengo Seki
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=53d8839 ]
[AIRFLOW-7025] Fix SparkSqlHook.run_query to handle its parameter properly
(#7677)
cherry-picked from 2327aa5a2
> Fix SparkSqlHook.run_query to handle its parameter properly
> -----------------------------------------------------------
>
> Key: AIRFLOW-7025
> URL: https://issues.apache.org/jira/browse/AIRFLOW-7025
> Project: Apache Airflow
> Issue Type: Bug
> Components: hooks
> Affects Versions: 1.10.9
> Reporter: Kengo Seki
> Assignee: Kengo Seki
> Priority: Major
> Fix For: 1.10.10
>
>
> {{SparkSqlHook.run_query()}} has a parameter called {{cmd}}, which type is
> documented as string, but it doesn't work as expected.
> {code}
> In [1]: from airflow.providers.apache.spark.hooks.spark_sql import
> SparkSqlHook
>
> In [2]: SparkSqlHook(sql="SELECT 1", master="local[*]",
> conn_id="spark_default").run_query(cmd="--help")
>
> (snip)
> [2020-03-09 23:37:35,088] {spark_sql.py:149} INFO - b' childArgs
> [-e SELECT 1 - - h e l p]\n'
> {code}
> The passed argument "--help" is splitted into single characters. This is
> because the {{cmd}} parameter is concatenated to {{connection_cmd}} as a list
> of characters, as follows:
> {code:title=airflow/providers/apache/spark/hooks/spark_sql.py}
> 90 def _prepare_command(self, cmd):
> 91 """
> 92 Construct the spark-sql command to execute. Verbose output is
> enabled
> 93 as default.
> 94
> 95 :param cmd: command to append to the spark-sql command
> 96 :type cmd: str
> 97 :return: full command to be executed
> 98 """
> 99 connection_cmd = ["spark-sql"]
> (snip)
> 130 connection_cmd += cmd
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
