[
https://issues.apache.org/jira/browse/AIRFLOW-6086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Florian FERREIRA closed AIRFLOW-6086.
-------------------------------------
Resolution: Duplicate
> SparkSubmitOperator - Unable to override spark_binary
> ------------------------------------------------------
>
> Key: AIRFLOW-6086
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6086
> Project: Apache Airflow
> Issue Type: Bug
> Components: contrib, core
> Affects Versions: 1.10.6
> Reporter: Florian FERREIRA
> Priority: Major
>
> Hello,
> I have a connection "spark2_default" :
> || Conn Id || Conn Type || Host || Port || Is Encrypted || Is Extra
> Encrypted || Extra
> | 'spark2_default' | 'spark2' | 'yarn-cluster' | None | False | False |
> {"master":"yarn-cluster","deploy-mode":"cluster","spark-binary":"spark2-submit"}
> |
> Extra contains 'spark-binary' key that was use by airflow 1.10.2 to choose
> spark-submit operator. But in version 1.10.6 this config is ignore.
> I think that , in class SparkSubmitOperator in init function they has a
> default value "spark-submit" for spark_binary parameter.
> {code}
> spark_binary="spark-submit",
> {code}
> Therefore in class SparkSubmitHook when we control if spark_binary is empty
> it can't be.
> {code}
> conn_data['spark_binary'] = self._spark_binary or \
> extra.get('spark-binary', "spark-submit")
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
