potiuk commented on issue #60858: URL: https://github.com/apache/airflow/issues/60858#issuecomment-3780036032
> [@potiuk](https://github.com/potiuk), I've always wondered about this. What do you think about an `on_kill_callback`? I assume that would follow the framework of what you mentioned above. But curious your thoughts. Yes it could be posible, though I would rather extend the on_failure_callback and add type of failure there if possible. Actually .. I think it will not work in all cases anyway. If the task process is just killed with -9 or SEGV, this might work - but it is not likely to work with OOM - because generally OOM kills the whole Pod - which means that both task and supervisor wil get -9 (there might be some races) and neither of them will have a chance to react. What you **really** want to do in this case is to have an executor and deployment dependent behaviour - for examle K8S executor should monitor reason for POD failue - and then not even start another task, but even start another POD to run on_task_failure (or on_task_kill) - and in case of Celery Executor for example you would have to track which celery workers run which tasks, and do similar thing for **all**tasks that were runnning in the same worker .. Etc. etc. And it will look differently if airflow is run using different cluster technoloty - for example Nomad. There are **lots and lots** of flows that would need to be considered to make it robust and it's I think impossible to have a working solution without close coupling the solutions with deployment or limiting to single deployment configuration (say K8S executor). I would say that this one is "won't fix" kind of issues - and we need to simply accept there is no on_failure_callback generated in case of SIGKILL -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
