sunank200 commented on issue #41324: URL: https://github.com/apache/airflow/issues/41324#issuecomment-2295417146
@EugeneMSOff Is the YARN binary available in the PATH of your Airflow instance? It seems that the YARN binary may not be available in your current environment, which could be the reason you're unable to kill Spark applications on the YARN cluster using the `on_kill()` method. Would you like to attempt installing the YARN binary in your Docker setup and try again? If you wish, you can create a PR to clarify that when YARN is used as the cluster manager (i.e., when --master yarn is specified), the YARN binary must be available in the PATH of the Airflow instance. This is necessary for operations such as killing YARN applications via the `on_kill()` method in `SparkSubmitHook`. Alternatively, we could consider adding a new feature, using the YARN ResourceManager REST API to manage the application state (e.g., killing), if the YARN binary is unavailable. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
