[
https://issues.apache.org/jira/browse/AIRFLOW-6086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16983615#comment-16983615
]
ASF GitHub Bot commented on AIRFLOW-6086:
-----------------------------------------
flo076 commented on pull request #6680: [AIRFLOW-6086] Remove default value
spark_binary
URL: https://github.com/apache/airflow/pull/6680
Remove default value of spark_binary because it's better managed on spark
hook.
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following
[AIRFLOW-6086](https://issues.apache.org/jira/browse/AIRFLOW-6086/) issue
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> SparkSubmitOperator - Unable to override spark_binary
> ------------------------------------------------------
>
> Key: AIRFLOW-6086
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6086
> Project: Apache Airflow
> Issue Type: Bug
> Components: contrib, core
> Affects Versions: 1.10.6
> Reporter: Florian FERREIRA
> Priority: Major
>
> Hello,
> I have a connection "spark2_default" :
> || Conn Id || Conn Type || Host || Port || Is Encrypted || Is Extra
> Encrypted || Extra
> | 'spark2_default' | 'spark2' | 'yarn-cluster' | None | False | False |
> {"master":"yarn-cluster","deploy-mode":"cluster","spark-binary":"spark2-submit"}
> |
> Extra contains 'spark-binary' key that was use by airflow 1.10.2 to choose
> spark-submit operator. But in version 1.10.6 this config is ignore.
> I think that , in class SparkSubmitOperator in init function they has a
> default value "spark-submit" for spark_binary parameter.
> {code}
> spark_binary="spark-submit",
> {code}
> Therefore in class SparkSubmitHook when we control if spark_binary is empty
> it can't be.
> {code}
> conn_data['spark_binary'] = self._spark_binary or \
> extra.get('spark-binary', "spark-submit")
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
