GitHub user potiuk added a comment to the discussion: KubernetesPodOperator finishes despite marked as failed
>From what I see in the code - KPO should use delete_namespaced_pod method >https://github.com/apache/airflow/blob/287b1d800d8748d23387936a88c943c2df241c57/providers/src/airflow/providers/cncf/kubernetes/operators/pod.py#L1043 depending on parameter you have there can be different grace_periods set - there is 0 for immediate, default depending on your POD default temaplate, you can also pas different parameter. Also in case POD cannot be deleted for whatever reason you should see a log. One scenario which I can see it happening is when you have long enough grace period and your running task also needs to respond to various SIGNALS (likely TERM/HUP) that K8S uses to kill your task. If you have long running, badly written C-code that does not respond to signals, or where you do not handle signal propagation in your container (for example use bash as entrypoint) https://www.kaggle.com/code/residentmario/best-practices-for-propagating-signals-on-docker the initial signals that K8S sends might be swallowed and the POD will continue running until grace period expires. That's theory. In theory either forcing grace period to 0 or properly handling signals and terminating your in-container process should do the job. But maybe there are other problems I am not aware of. GitHub link: https://github.com/apache/airflow/discussions/45830#discussioncomment-11902934 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
