[
https://issues.apache.org/jira/browse/AIRFLOW-4717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ash Berlin-Taylor updated AIRFLOW-4717:
---------------------------------------
Summary: The spark_binary arg to SparkSubmitOperator has no effect when
connection exists. (was: The spark_binary option does not apply in
sparkSubmitOperator.)
> The spark_binary arg to SparkSubmitOperator has no effect when connection
> exists.
> ---------------------------------------------------------------------------------
>
> Key: AIRFLOW-4717
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4717
> Project: Apache Airflow
> Issue Type: Bug
> Components: hooks
> Affects Versions: 1.10.3
> Reporter: yunsu lee
> Assignee: yunsu lee
> Priority: Major
>
> Apache spark depending on the desto, the spark binary name may be different.
> (ex. spark2-submit)
> For this reason, the spark_binary option has been added to sparkSubmitOperator
> (Reference : [https://github.com/apache/airflow/pull/4360/files])
>
> However, this option does not work.
> This is because there is logic to hard-code and override the spark-binary
> option value in spark_submit_hook.py
> (Full path : airflow/contrib/hooks/spark_submit_hook.py)
>
> {code:java}
> ...
> conn_data['spark_binary'] = extra.get('spark-binary', "spark-submit")
> ...{code}
> It is necessary to delete the corresponding line.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
