hunnguye commented on issue #75332: URL: https://github.com/apache/airflow/issues/75332#issuecomment-6369693849
Tested [apache.spark: 6.3.4rc1](https://pypi.org/project/apache-airflow-providers-apache-spark/6.3.4rc1) (#71949) using the breeze enviroment and following versions - airflow version 3.3.2 - spark provider 6.3.4rc1 - python 3.12.14 Tested a spark submit against a dev spark cluster with following configuration: ``` { "queue": null, "deploy-mode": "cluster", "spark-binary": "spark-submit", "namespace": null, "principal": null, "keytab": null, "rest-scheme": "https", "rest-port": "6066", "yarn_resourcemanager_webapp_address": null } ``` With master being on https://master_url:7077 and api being on https://master_url:6066 1. kill job from spark => polling registers failing 2. kill job from airflow => spark ends the job with driver state = KILLED 3. job runs successfully in spark => airflow olls driver status with FINISHED -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
