GitHub user karenbraganz added a comment to the discussion: Report of increased incidence of tasks getting stuck in queued in Airflow 2.10.5
@KarthikeyanDevendhiran the data from the log table suggests that these task instances are getting requeued as expected due to the changes introduced in [PR #43520](https://github.com/apache/airflow/pull/43520) but are still failing. >From the scheduler logs, I see that you are using the Kubernetes Executor. A >Kubernetes worker pod is created when the task is queued. However, this pod is >unable to run the task instance before the `task_queued_timeout` of 15 minutes >is reached, so the task instance keeps getting requeued and eventually fails. It is possible that some error occurs when the pod is being created, which prevents it from running the task. Could you please share the following information, which would help us understand the state of the worker pod: - KubernetesJobwatcher events for the worker pod by running the below command to filter for these events in the scheduler log. ``` kubectl logs <scheduler-pod-name> -n <namespace> | grep <worker-pod-name> ``` - Kubernetes events for the worker pod by running the below command. By default, these events are only stored for 60 minutes. ``` kubectl get events -n <namespace> --field-selector involvedObject.name=<worker-pod-name> ``` GitHub link: https://github.com/apache/airflow/discussions/51597#discussioncomment-13425697 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
