[ https://issues.apache.org/jira/browse/AIRFLOW-6086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Florian FERREIRA closed AIRFLOW-6086. ------------------------------------- Resolution: Duplicate > SparkSubmitOperator - Unable to override spark_binary > ------------------------------------------------------ > > Key: AIRFLOW-6086 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6086 > Project: Apache Airflow > Issue Type: Bug > Components: contrib, core > Affects Versions: 1.10.6 > Reporter: Florian FERREIRA > Priority: Major > > Hello, > I have a connection "spark2_default" : > || Conn Id || Conn Type || Host || Port || Is Encrypted || Is Extra > Encrypted || Extra > | 'spark2_default' | 'spark2' | 'yarn-cluster' | None | False | False | > {"master":"yarn-cluster","deploy-mode":"cluster","spark-binary":"spark2-submit"} > | > Extra contains 'spark-binary' key that was use by airflow 1.10.2 to choose > spark-submit operator. But in version 1.10.6 this config is ignore. > I think that , in class SparkSubmitOperator in init function they has a > default value "spark-submit" for spark_binary parameter. > {code} > spark_binary="spark-submit", > {code} > Therefore in class SparkSubmitHook when we control if spark_binary is empty > it can't be. > {code} > conn_data['spark_binary'] = self._spark_binary or \ > extra.get('spark-binary', "spark-submit") > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)