yunsu lee created AIRFLOW-4717:
----------------------------------
Summary: The spark_binary option does not apply in
sparkSubmitOperator.
Key: AIRFLOW-4717
URL: https://issues.apache.org/jira/browse/AIRFLOW-4717
Project: Apache Airflow
Issue Type: Bug
Components: hooks
Affects Versions: 1.10.3
Reporter: yunsu lee
Apache spark depending on the desto, the spark binary name may be different.
(ex. spark2-submit)
For this reason, the spark_binary option has been added to sparkSubmitOperator
(Reference : [https://github.com/apache/airflow/pull/4360/files])
However, this option does not work.
This is because there is logic to hard-code and override the spark-binary
option value in spark_submit_hook.py
(Full path : airflow/contrib/hooks/spark_submit_hook.py)
{code:java}
...
conn_data['spark_binary'] = extra.get('spark-binary', "spark-submit")
...{code}
It is necessary to delete the corresponding line.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
