potiuk commented on issue #60858: URL: https://github.com/apache/airflow/issues/60858#issuecomment-3779875663
There is not much to investigate - the on_failure_callback is not going to be called when task is killed - this is a given because the interpreter is killed without any chance for reaction. The only way to handle that is to make completely new flow where the supervisor should see if the task has been killed forecefully and **start a "run on_failure_callback" new task that will only do that. Supervisor cannot do it on it's own because supervisor does not parse the Dag that has on_failure_callback defined, so effectively it will need to run a new child task that will again parse the dag and only runs on_failure_callback. This has a number of edge cases to look at - for example handling the case where OOM happens during parsing the dag, This is quite clear what's going on here, it just needs quite complex implementation of handling this special flow. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
