coffee34 opened a new issue, #37898: URL: https://github.com/apache/airflow/issues/37898
### Apache Airflow Provider(s) apache-livy ### Versions of Apache Airflow Providers 3.5.4 ### Apache Airflow version 2.7.2 ### Operating System Amazon MWAA ### Deployment Amazon (AWS) MWAA ### Deployment details I used LivyOperator (deferrable = True) to create spark job by livy, and execution_timeout is 2h. ### What happened When the spark job run for more than 2h, airflow will detect it's timeout and cancel the trigger. And then wake up the task to execute on_kill function. Then livy operator failed with `AttributeError: 'LivyOperator' object has no attribute '_batch_id'` when it execute on_kill function. ### What you think should happen instead The Livy operator should fail due to a timeout, and simultaneously, it should kill the Spark job. ### How to reproduce Utilize the LivyOperator in your workflow. Configure the operator with deferrable=True and execution_timeout=600 (for a 10-minute timeout). Execute a Spark job that is expected to run for longer than 10 minutes. ### Anything else Seems like when the livy operator is rewaked up after trigger is time out, the _batch_id [here](https://github.com/apache/airflow/blob/providers-apache-livy/3.5.4/airflow/providers/apache/livy/operators/livy.py#L123) is not initialized, which cause it failed to run on_kill function. ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
