[ 
https://issues.apache.org/jira/browse/AIRFLOW-5517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16935945#comment-16935945
 ] 

Elad commented on AIRFLOW-5517:
-------------------------------

Yep the default value of spark_binary in SparkSubmitOperator should be None 
(Currently it's "spark-submit")

*As a work aournd fix* you can just explicitly specify set spark_binary=None in 
your SparkSubmitOperator.

I guess that SparkSubmitOperator tests aren't great either if they didn't catch 
this when I fixed the hook.

I fix it once I finish with my other pending PRs (if someone else wants to pick 
it up before i do be my guest).

> SparkSubmitOperator: spark-binary parameter no longer taken from connection 
> extra
> ---------------------------------------------------------------------------------
>
>                 Key: AIRFLOW-5517
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-5517
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: contrib
>    Affects Versions: 1.10.4, 1.10.5
>            Reporter: Alexander Kazarin
>            Priority: Major
>             Fix For: 1.10.6
>
>
> We have an extra parameters in spark connection:
> {code:java}
> {"deploy-mode": "cluster", "spark-binary": "spark2-submit"}
> {code}
> After upgrade to 1.10.5 from 1.10.3 parameter 'spark-binary' in extra is no 
> longer take effect.
>  Broken after 
> [this|https://github.com/apache/airflow/commit/8be59fb4edf0f2a132b13d0ffd1df0b8908191ab]
>  commit, I think
> Workaround: call SparkSubmitOperator with spark_binary=None argument



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Reply via email to