[ https://issues.apache.org/jira/browse/AIRFLOW-5517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16935945#comment-16935945 ]
Elad commented on AIRFLOW-5517: ------------------------------- Yep the default value of spark_binary in SparkSubmitOperator should be None (Currently it's "spark-submit") *As a work aournd fix* you can just explicitly specify set spark_binary=None in your SparkSubmitOperator. I guess that SparkSubmitOperator tests aren't great either if they didn't catch this when I fixed the hook. I fix it once I finish with my other pending PRs (if someone else wants to pick it up before i do be my guest). > SparkSubmitOperator: spark-binary parameter no longer taken from connection > extra > --------------------------------------------------------------------------------- > > Key: AIRFLOW-5517 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5517 > Project: Apache Airflow > Issue Type: Bug > Components: contrib > Affects Versions: 1.10.4, 1.10.5 > Reporter: Alexander Kazarin > Priority: Major > Fix For: 1.10.6 > > > We have an extra parameters in spark connection: > {code:java} > {"deploy-mode": "cluster", "spark-binary": "spark2-submit"} > {code} > After upgrade to 1.10.5 from 1.10.3 parameter 'spark-binary' in extra is no > longer take effect. > Broken after > [this|https://github.com/apache/airflow/commit/8be59fb4edf0f2a132b13d0ffd1df0b8908191ab] > commit, I think > Workaround: call SparkSubmitOperator with spark_binary=None argument -- This message was sent by Atlassian Jira (v8.3.4#803005)