nayanen opened a new issue #19698: URL: https://github.com/apache/airflow/issues/19698
### Description ### Description I have come across this issue where k8s customer using airflow application. The pod get terminated when there is brief API server interruption. API server interruption can happen due to lot of scenarios so ideally the application should handle it gracefully. Another key thing here is this happens only when you set the _"is_delete_pod_operator"_ set to _True_. Now if i read the [code](https://github.com/apache/airflow/blob/main/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py#L523-L533) i understand that the launcher.start_pod or launcher.monitor_pod have some logic which check heartbeat to the AI server and once its find some interruption it goes to the AirflowException part. So if we have some kind of nested exception or some retry logic to handle the brief interruption of API sever unavailability that should help to avoid the pod termination, because it just takes very less time for API server to re-establish the connection. ``` 'airflow_version': airflow_version.replace('+', '-'), 'kubernetes_pod_operator': 'True', } ) self.log.debug("Starting pod:\n%s", yaml.safe_dump(self.pod.to_dict())) final_state = None try: launcher.start_pod(self.pod, startup_timeout=self.startup_timeout_seconds) final_state, remote_pod, result = launcher.monitor_pod(pod=self.pod, get_logs=self.get_logs) except AirflowException: ``` The kubernetesPodOperator is provided as open-source Airflow and [more details](https://airflow.apache.org/docs/apache-airflow/1.10.12/_api/airflow/contrib/operators/kubernetes_pod_operator/index.html) about the behavior can be found below. ``` is_delete_operator_pod (bool) – What to do when the pod reaches its final state, or the execution is interrupted. If False (default): do nothing, If True: delete the pod ``` ### Use case/motivation _No response_ ### Related issues _No response_ ### Are you willing to submit a PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
