eladkal edited a comment on issue #5564: [AIRFLOW-4289] fix spark_binary argument being ignored in SparkSubmit… URL: https://github.com/apache/airflow/pull/5564#issuecomment-516081700 @ashb I need some help in resolving the failed test: ``` test_resolve_connection_yarn_default (tests.contrib.hooks.test_spark_submit_hook.TestSparkSubmitHook) ... INFO [airflow.contrib.hooks.spark_submit_hook.SparkSubmitHook] Could not load connection string , defaulting to yarn INFO [airflow.contrib.hooks.spark_submit_hook.SparkSubmitHook] Could not load connection string , defaulting to yarn INFO [airflow.contrib.hooks.spark_submit_hook.SparkSubmitHook] Spark-Submit cmd: ['spark-submit', '--master', 'yarn', '--name', 'default-name', 'test_application.py'] [31mFAILED[0m ``` Log: ``` [31m======================================================================[0m [31m22) FAIL: test_resolve_connection_yarn_default (tests.contrib.hooks.test_spark_submit_hook.TestSparkSubmitHook)[0m [31m----------------------------------------------------------------------[0m [0m Traceback (most recent call last):[0m [34mtests/contrib/hooks/test_spark_submit_hook.py[0m line [1m[36m246[0m[0m in [36mtest_resolve_connection_yarn_default[0m self.assertEqual(connection, expected_spark_connection) [31m [31m[1m[31mAssertionError[0m[0m[31m: [0m[31m{'master': 'yarn', 'queue': None, 'deploy_mode': None, 'spark[54 chars]ult'} != {'master': 'yarn', 'spark_binary': 'spark-submit', 'deploy_mo[64 chars]ult'}[0m [31m {'deploy_mode': None,[0m [31m 'master': 'yarn',[0m [31m 'namespace': 'default',[0m [31m 'queue': None,[0m [31m - 'spark_binary': None,[0m [31m + 'spark_binary': 'spark-submit',[0m [31m 'spark_home': None}[0m ``` The log shows that ` Could not load connection string , defaulting to yarn` Which means that `AirflowException` was raised due to ` conn = self.get_connection(self._conn_id)` as `self._conn_id` is empty string. In such case spark_binary won't be set. I'm open to suggestions how to address this.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
