[
https://issues.apache.org/jira/browse/AIRFLOW-1326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ash Berlin-Taylor updated AIRFLOW-1326:
---------------------------------------
Description:
AIRFLOW-1184 introduced a bug making it impossible to include an argument with
an embedded space in the SparkSubmitOperator. (I believe the original bug
report should have been closed as wontfix/not-a-bug as it was user error in how
the SparkSubmitOperator is being called)
i.e. the correct way to achieve what was wanted in the bug is like this:
{code}
'application_args': [
'-f', ' foo',
'--bar', 'bar',
'--start', '{{ macros.ds_add(ds, -1)}}',
'--end', '{{ ds }}'
]
{code}
This was possibly compounded as the tests were showing something that would
never "work"
was:
Python Popen expect a list as command. Spark submit too, as:
* ['--option value']
is not the same as
* ['--option', 'value']
in regards of spark. eg spark logs : (yarn logs)
Error: Unknown option --end 2017-05-08
Error: Unknown option --begin 2017-05-07
Error: Unknown option --db_name mydb
Error: Missing option --begin
Error: Missing option --end
Error: Missing option --db_name
> CLONE - Contrib Spark Submit Hook _should not_ split argument array
> -------------------------------------------------------------------
>
> Key: AIRFLOW-1326
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1326
> Project: Apache Airflow
> Issue Type: Bug
> Components: contrib, hooks
> Affects Versions: Airflow 2.0, Airflow 1.8
> Reporter: Ash Berlin-Taylor
> Assignee: Vianney FOUCAULT
> Fix For: Airflow 2.0, Airflow 1.8
>
>
> AIRFLOW-1184 introduced a bug making it impossible to include an argument
> with an embedded space in the SparkSubmitOperator. (I believe the original
> bug report should have been closed as wontfix/not-a-bug as it was user error
> in how the SparkSubmitOperator is being called)
> i.e. the correct way to achieve what was wanted in the bug is like this:
> {code}
> 'application_args': [
> '-f', ' foo',
> '--bar', 'bar',
> '--start', '{{ macros.ds_add(ds, -1)}}',
> '--end', '{{ ds }}'
> ]
> {code}
> This was possibly compounded as the tests were showing something that would
> never "work"
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
