GitHub user KarthikeyanDevendhiran added a comment to the discussion: Report of
increased incidence of tasks getting stuck in queued in Airflow 2.10.5
Hi @karenbraganz, I’ve run the diagnostics as requested and observed consistent
logs across the worker pods that failed due to this issue.
**When executing the following command:**
``` kubectl logs airflow-scheduler-65cfcf89c9-67jlx -n airflow | grep
<worker-pod-name> ```
**I received logs indicating:**
```
Defaulted container "scheduler" out of: scheduler, git-sync,
scheduler-log-groomer, wait-for-airflow-migrations (init), git-sync-init (init)
```
**Additionally, when running:**
`kubectl get events -n airflow --field-selector
involvedObject.name=<worker-pod-name>`
**I found that:**
```
LAST SEEN TYPE REASON OBJECT
MESSAGE
12m Warning FailedMount pod/worker-pod-3j33j446
MountVolume.MountDevice failed for volume
"pvc-0de664bb-ff12-411c-aa2b-5ca54f2fd4ac" : rpc error: code = Internal desc =
Mount failed with error: rpc error: code = Unknown desc = exit status 1 ***
blobfuse2: A new version [2.5.0~preview.1] is available. Consider upgrading to
latest version for bug-fixes & new features. ***...
```
- Kubernetes events report a FailedMount error for volume
pvc-0de664bb-ff12-411c-aa2b-5ca54f2fd4ac, caused by a Blobfuse2 mount failure.
- The scheduler logs confirm that the pod is registered, but the task likely
fails before execution due to the volume issue.
I’m currently reviewing the PVC configuration to identify the root cause.
Please let me know if there are specific areas you'd like me to focus on or if
additional diagnostics would be helpful.
Thanks!
GitHub link:
https://github.com/apache/airflow/discussions/51597#discussioncomment-13453174
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
